U.S. Department of Housing and Urban Development, Office of Policy Development and
Research. A Guide to Evaluating Crime Control of Programs in Public Housing.
Washington, DC: Prepared for the U.S. Department of Housing and Urban Development
by KRA Corporation; 1997.  pp.3.1-3.15.

View entire document

Chapter 3
    Who Should Conduct Your Evaluation?

Evaluation is best thought of as a team effort. Although one person heads an evaluation team and has primary responsibility for the project, this individual will need assistance from others on your staff. An evaluation team will work together on the following tasks:

• Determining the focus and design of the evaluation.
• Developing the evaluation plan, performance indicators, and data collection instruments.
• Collecting, analyzing, and interpreting data.
• Preparing the report on evaluation findings.

Types of evaluation teams

You can assemble many types of evaluation teams. Three possible options include:

• Hiring an outside evaluator (option 1).
• Using an inhouse evaluation team supported by an outside consultant and program staff (option 2).
• Using an inhouse evaluation team supported by program staff (option 3).

Hiring an outside evaluator. Housing authorities typically do not have a research and evaluation staff and will probably need to hire an outside evaluator. This person would be supported by inhouse staff and would serve as a team leader. The evaluator could come from a research institute or a consulting firm. For more information on locating a candidate for this position, see the section later in this chapter on finding and hiring an outside evaluator.

Using an inhouse evaluation team supported by an outside consultant and program staff. If you feel that you have sufficient staff resources to implement the evaluation but need assistance with the technical aspects, you may want to hire an outside consultant. In this situation, an inhouse evaluator would serve as the team leader and be supported by both program staff end the outside consultant. If there are research resources within your Public Housing Agency (PHA), you may want to consider this option.

A consultant could support the evaluation by developing the evaluation design, conducting data analyses, and selecting or developing questionnaires. This person can also help you develop the evaluation plan and performance indicators.

Using an inhouse evaluation team supported by program staff. If resources are available within your PHA (that is, if you have research staff, evaluators, or program personnel who can assist with the evaluation), you could recruit these individuals to serve as evaluation team members.

The chart on the next page shows the possible advantages and disadvantages for each of the evaluation team options. Whatever team option you select, you will want to make sure that you or someone from the PHA become part of the team. Even if your role is limited to one of overall evaluation management, you will want to participate in all phases of the evaluation effort.

[[insert chart on page 3-3 here]]

Deciding what team is best for you. This decision will be influenced most by the resources and capabilities of your PHA. To determine what internal resources are available, you can examine your staff's skills and experience in planning an evaluation, designing data collection procedures and questionnaires, and collecting and analyzing information.

Below is a checklist to help you decide what type of team you may need. If you answer "no" to all of the resource questions, you may want to consider postponing your evaluation until you can obtain funds to hire an outside evaluator. You may also want to consider budgeting funds for an evaluation in your future program planning efforts.

If your answer to question 1 is "yes" but your answer is "no" to all other questions, you will need maximum assistance in conducting your evaluation and option 1 (an outside evaluator with inhouse support) is probably your best choice.

If you answer "no" to question 1 but "yes" to most of the other resource questions, then option 3 (inhouse staff only) may be an

[[chart material below - see page 3-4

Resources for Appropriate Team Selection  check one

Does your PHA have funds designated for evaluation purposes?

Have you successfully conducted previous evaluations of similar programs, components, or services?

Are there existing measures or indicators of performance currently in place?

Are existing program practices and information collection forms useful for evaluation purposes?

Can you collect evaluation information as part of your regular intake of residents?

Are there PHA staff who have training and experience in evaluation-related tasks?]]

appropriate choice for you. However, if you plan to use evaluation findings to seek program funding, you may want to consider using option 2 (inhouse team with outside consultant) and try to obtain evaluation funds from other areas of your agency's budget. If your answer to question 1 is "yes" and the remainder of your answers are mixed (some "yes" and some "no"), then either option 1 or option 2 could be effective.

Finding and hiring an outside evaluator

Careful selection of an outside evaluator can mean the difference between a positive and a negative experience. A good place to start is to consider using someone that you or another PHA has worked with successfully on another project. You may want to begin the search by interviewing that particular person. Other public agencies within your community may also be good sources for this type of referral.

PHAs located in or near a large city should have little trouble finding an evaluator. Many consulting firms have staff experienced in evaluation who have conducted housing-related evaluation studies. Most universities and colleges have faculty who can design and conduct evaluations. This manual includes a listing of national resources that may aid you in your search for experts.

Before hiring an evaluator, you will want to know the following:

• What level of experience does the evaluator have in the area of program evaluation?
• Does the evaluator have any experience conducting evaluations in the area of public housing or in a related area?
• Can the evaluator offer assistance in the full range of program evaluation activities including research design, data collection, data analysis, data interpretation, and dissemination of the results?
• Will the evaluator present information in a way that will be useful to you?
• How much will the project cost?
• How will the housing staff tee involved in the evaluation?
• Is the evaluator willing to work closely with you, the housing agency staff, and the tenant association?

A good outside evaluator will not dictate to you or your staff how the project will proceed but will instead work with you to conduct a successful evaluation. You should have input into determining the purpose of the evaluation, research questions, and performance indicators. Everyone on your evaluation team, including the outside evaluator, should be willing to collaborate with your PHA.

There are four basic steps for hiring an evaluator.

Step 1: Develop a statement of work. The first step in the hiring process is to develop a statement of work that details the general and specific requirements for the evaluator. General requirements should list the materials, services, and products to be provided by the evaluator.

In creating the statement of work, you will need to know the types of evaluation activities you want the evaluator to perform. Evaluator responsibilities can involve developing an evaluation plan, providing progress reports, developing data collection instruments and forms, collecting data, analyzing data, and writing reports.

The general requirements of the statement of work should list the frequency with which you expect to meet with the evaluator and the requirements for submitting written reports. Specific requirements should outline tasks. For example, if the tasks to be completed by the evaluator include a preliminary meeting; development of a research design, data collection plan, and sampling plan; data collection activities; data analysis; and a preliminary and final report, then the statement of work would specify each of these activities as a separate task. In addition, each task should have a timeframe for beginning and completion. Similarly, each task should have specific milestone dates for completion of work or submission of documents.

Step 2: Locate sources for evaluators. Potential sources you can use to find an evaluator include:

• Other public housing authorities that have used outside evaluators. PHAs or community-based organizations may have staff who have used outside evaluators and are able to recommend one, suggest methods of advertising for one, or provide other useful information. Contacting similar programs is one of the best ways to find an evaluator who understands your program, is sensitive to public housing residents, and can provide a useful evaluation that meets your needs.
• Evaluation divisions of State or local agencies. Most State or local government agencies have planning and evaluation departments. In addition, some local police departments employ research staff. You may be able to use individuals from these agencies to work with you on your evaluation, or they may be able to direct you toward other organizations with experience in conducting outside evaluations.
• Department of Housing and Urban Development (HUD). A number of offices HUD fund national evaluation studies conducted by contractors. These offices, such as the Office of Policy Development and Research and the Office of Public and Indian Housing, may be able to direct you to contractors who are experienced in conducting research and evaluation studies.
• Local colleges and universities. Departments of sociology, psychology, social work/social welfare, and public administration, as well as university-based research centers, are all possible sources within colleges and universities. Well-known researchers affiliated with these institutions may be readily identifiable. If they cannot personally assist you, they may be able to refer you to other individuals who are interested in performing local program evaluations.
• Technical assistance providers. Some Federal grant programs include a national technical assistance provider. If your agency is participating in this kind of grant program, then asking for assistance in identifying and selecting an evaluator is appropriate and reasonable.
• The public library. Reference librarians may be able to direct you to new sources. They can help identify local research firms and may be able to provide you with conference proceedings that list program evaluators.
• Research and consulting firms. Many experienced evaluators are part of research and consulting firms. They may be more difficult to identify initially but are sometimes listed in the yellow pages under "Research" or "Marketing Research." They can also be located by contacting your State human services departments to get a listing of the firms that have bid on recent contracts for evaluations of State programs.
• The United Way, American Public Welfare Association, Child Welfare League of America, Urban League, and local foundations. Your United Way or Urban League chapter, or area foundation staff and board members, may be able to provide you with names of local evaluators. They may also be able to provide insight on evaluations that were done well.
• The American Evaluation Association. Many evaluators belong to the American Evaluation Association. Based at the University of Virginia in Charlottesville, Virginia, this organization is able to provide a list of members in your area for a fee. Additional information on the American Evaluation Association is contained in the resources section of this manual.
• The American Criminal Justice Association (ACJA). ACJA has over 4,000 members who are concerned with the administration of criminal justice. ACJA publishes a semiannual journal, the Journal of the American Criminal Justice Association.
• The American Society of Criminology (ASC). ASC is a national organization concerned with criminology, research, and education involving the etiology, prevention, control, and treatment of crime and delinquency. Its membership includes over 2,500 professional and academic criminologists, practitioners, and academicians in many fields of criminal justice. ASC publishes a newsletter, The Criminologist, six times a year and a journal, Criminology, four times a year.
• The American Sociological Association. State and local chapters of the American Sociological Association may be able to direct you to members who conduct local program evaluations. Departments of sociology or criminology at local colleges and universities should be able to provide you with your State chapter's telephone number.

Step 3: Advertise and request applications. Once you have developed a statement of work and identified possible sources for evaluators, you are ready to advertise for applications. Advertising in the local paper, posting the position at a local college or university, and working with your local government's human resources department are possible ways of soliciting offers. Agency newsletters, local and national meetings, and professional journals are additional sources where you can post your request for applications or an advertisement. You will want to advertise as widely as possible, particularly if you are in a small community or are undertaking an evaluation for the first time. Using several advertising sources will help ensure that you receive more than one response. The following page lists some suggestions for preparing an effective advertisement.

If you have sufficient time, you may want to consider a two-step process for applications-advertising the position but also sending evaluators who respond to your advertisement more detailed information about your evaluation requirements. For example, you could send potential evaluators a brief description of the program and the evaluation questions you want to answer, along with a description of the size and location of the PHA where the evaluation will be conducted. This would give the candidate an opportunity to propose a plan that more closely corresponds to your program needs.

Step 4: Renew proposals and select an evaluator. The final step in hiring an evaluator is to review the proposals submitted and select an evaluator. In reviewing proposals, you will want to consider the candidate's writing style, type of evaluation plan proposed, experience working with your type of program and staff, familiarity with violence prevention initiatives and public housing residents, experience conducting similar evaluations, and proposed cost.

Once you have narrowed your selection to two or three candidates, you will want to contact them to schedule inperson interviews. The interview is an important part of the selection process. It will allow you to determine whether you and the evaluator are compatible. You can review the criteria for selecting an evaluator discussed earlier in this chapter and use them to guide your questions during the interview.

In most cases the ideal candidate will seem obvious after the interview is conducted, but sometimes it will be more difficult. As you do for other job applicants, you will need to check references.

It makes sense to follow whatever procedure your PHA has in place for contracting out for services. Remember to obtain the appropriate agency approval before notifying your chosen evaluator. If you currently operate a federally funded violence prevention program, check your grant requirements to determine whether you need Federal approval of your evaluator.

What to do when you have trouble hiring an evaluator. Despite your best intentions, you may encounter difficulties, including one or more of the following:

Few or no responses to your advertisement. PHAs located in isolated areas may only get a few responses to their advertisements. If this occurs you may want to contact 1 of the 80 local offices of the Department of Housing and Urban Development. They should be able to assist you by providing information about evaluators who have conducted similar evaluation studies in or near your community.

None of the applicants is compatible with program philosophy and staff. If applicants do not match program needs, you may find it helpful to network with other PHAs or directors of violence prevention programs in your city or State who have worked with evaluators. A compatible philosophy and approach are most important, and tradeoffs between them and the proximity of the evaluator may be needed to find the right one.

Money allocated for the evaluation is insufficient to hire a third party evaluator. In this instance, you will need to generate additional funds for the evaluation or negotiate with your evaluator to donate his or her services (inkind service). Many evaluators are committed to their profession and regularly discount their fees or donate a portion of their time to evaluation projects. Another option is to negotiate with a
college professor, using the evaluation dollars to pay for this person's time, while advanced degree students work under the direction of the professor to conduct some of the evaluation activities.

There may be researchers in your community who have an interest in conducting research in the area of violence prevention. They may be interested in using information about your violence prevention initiative and may be willing to provide evaluation services in exchange for access to participant and program information. For example, you can allow a university professor to have access to program and participant records in exchange for evaluation services such as instrument development or data analysis.

Managing an evaluation headed by an outside evaluator

Often when the decision is made to hire an outside evaluator, program managers and staff believe that the evaluation is out of their hands. This is not true. An outside evaluator cannot do the job effectively without the cooperation and assistance of PHA staff end residents. An evaluation is like any activity taking place within your agency; it needs to be managed.

Creating a contract. One mechanism for effectively managing the evaluation is to prepare a written contract for specifying the evaluator's roles and responsibilities. The contract should be prepared once you have received the appropriate approval to hire someone. Both the evaluator and the PHA staff authorized to hire outside services will need to sign the contract.

Your contract is a legally binding document which specifies the evaluation activities to be performed, the amount of time to complete the evaluation, and the cost. This document offers you protection by specifying who is expected to conduct the work and how the data that has been collected will be used. Every evaluation contract should include the following items:

• Who will perform evaluation tasks. Some evaluators delegate many of their responsibilities to less experienced staff end have little contact with the client once the contract is signed. To protect yourself from this scenario, the contract should specify what percentage of time the evaluator and his or her staff will devote to the evaluation.
• Who owns the evaluation data. In your contract, specify who has ownership of the data and to whom the information can be given. Release of information to outside parties should always be cleared with appropriate PHA staff. Any plans for publishing the evaluation results should be discussed and cleared before articles are written and submitted for publication.
• Your expectations about contacts between the evaluator and PHA. It is very important for an outside evaluator to keep the PHA, program staff, and residents informed about the status of the evaluation. Regular communication allows the PHA and other concerned parties to make important changes on an ongoing basis. The contract should specify expectations about the frequency of meetings and ongoing reporting requirements.

What to do if problems arise. Even with the best contracts, however, problems can arise during the course of the evaluation process. Examples of types of problems that may occur and possible solutions include the following:

Evaluation approaches differ-the program manager and evaluator do not see eye to eye. Try to reach a common ground where both programmatic and evaluation constraints and needs are met. If many reasonable attempts to resolve differences are made and severe conflicts still remain that could jeopardize the program or the evaluation, program staff should consider terminating the evaluation contract. This decision will need to be weighed carefully, as a new evaluator must be recruited and brought up to speed in midstream. In some situations, however, this may be the best option.

Evaluation of the program requires skills or analyses for which you did not originally plan. You may find that your evaluator is in agreement with your assessment and is willing to add another person to the evaluation team who has expertise and skills needed to undertake additional or different analyses. Many times additional expertise can be added to the evaluation team by using a few hours of a consultant's time. For example, programmers or statisticians may be necessary to augment the evaluation team.

The evaluator leaves, terminates the contract, or does not meet contractual requirements. If the evaluator leaves the area or terminates the contract, you will most likely be faced with recruiting a new one. In some instances programs have successfully maintained their ties to evaluators who have left the area, but this is often difficult.. When your evaluator does not meet contractual requirements and efforts to resolve the dispute have failed, you should turn the case over to your procurement office.

The evaluator does not have any experience working with low-income populations or community residents. It is not always possible to locate an evaluator who has the necessary experience in evaluation and experience working with public housing residents. It is a documented fact that very few evaluations have been conducted of violence prevention initiatives in public housing. You may have to educate the evaluator about the public housing development where your project is being offered and the characteristics of the surrounding community. The evaluator needs to understand how these factors may affect the evaluation and the questionnaires and procedures to be used. You may require that the evaluator work with members of the resident council and other key community leaders so that the evaluation is relevant to the participants' experiences and cultures. Remember, the outside evaluator works for you, and the implementation of a successful and valuable evaluation depends on you.