A Guide to Evaluating Crime
Control of Programs in Public Housing

Prepared for:
U.S. Department of Housing and Urban Development
Office of Policy Development and Research

Prepared by:
KRA Corporation

April 1997

FOREWORD

As part of our criminal justice research agenda, the Office of Policy Development and Research (PD&R) originally contracted with the KRA Corporation to review evaluation research on violence prevention initiatives. The objective of the review was to identify interpersonal violence prevention strategies that had been demonstrated to be effective for use in urban neighborhoods. We had hoped to produce a resource book from which public housing authorities and other social service agencies could glean helpful prevention information in their struggle to conquer the violent crime that plagues so many of our cities. However, KRA found few proven violence prevention strategies. There was no shortage of prevention programs, but rather a marked shortage of evaluations.

Therefore, in consultation with KRA, PD&R decided to produce a manual on how to conduct program evaluations of crime control programs with special emphasis on violence prevention efforts in public housing. Thus, this volume, A Guide to Evaluating Crime Control Programs in Public Housing, was written.

We hope that you find the Guide helpful.

Table of Contents

Introduction ........................................................................................................................................................................i

Chapter 1:   Why Should You Evaluate Your Program?..............................................................................................1-1

Chapter 2:   What Is Evaluation?..................................................................................................................................2-1

Chapter 3:   Who Should Conduct Your Evaluation?................................................................................................... 3-1

Chapter 4:   How Do You Prepare for an Evaluation?................................................................................................. 4-1

Chapter 5:   Developing an Evaluation Plan.................................................................................................................5-1

Chapter 6:   How Do You Get the Information You Need for Your Evaluation?........................................................6-1

Chapter 7:   Analyzing Evaluation Information............................................................................................................ 7-1

Chapter 8:   Reporting Your Findings........................................................................................................................... 8-1

Glossary

Resources

Introduction

The Office of Policy Development and Research in the Department of Housing and Urban Development (HUD) contracted with KRA Corporation (KRA) to review evaluation research on violence prevention initiatives and identify interpersonal violence prevention strategies that are effective for use in public housing. Successful prevention initiatives to be identified were to focus on ways to reduce accessibility of firearms, drug trafficking, conflicts between youth as individuals and as members of gangs, and abuse of alcohol and other drugs.

This study was designed to meet the pressing need for information to shape the efforts of those seeking to prevent or reduce the incidence of violence in public housing and similar environments. The original intent was to create an inventory of violence prevention initiatives that had been evaluated and shown to be effective in:

• Lowering the incidence of violent criminal behaviors in public housing and similar environments.
• Displacing criminal behavior known to be correlated with a high incidence of violence.

The inventory was to contribute to the creation of a resource document that could be used to plan local initiatives for reducing the incidence of violent criminal behavior in public housing and similar environments and reducing other criminal behavior associated with high rates of violence.

Very little evaluation research was identified upon completion of the inventory. HUD therefore changed the project to better meet the needs of Public Housing Agency (PHA) directors and others in the housing field. As an alternative to a document to assist in implementing violence prevention initiatives, HUD and KRA created this how-to manual on evaluating violence prevention efforts.

This manual is designed to do the following:

• Familiarize PHA directors and others in public housing with evaluation issues and procedures so they can participate in the process and effectively monitor the work of internal or external evaluators.
• Provide PHA directors and others in public housing with skills for thinking through the design of a violence prevention project, even if an evaluation is not planned.

Chapter 1
Why Should You Evaluate Your Program?

As a public housing administrator or key staff member in a public housing environment, you may have already asked yourself the question, "Why do I need to evaluate my program?" Most likely, your first responsibility has been to provide safe housing for residents. You may wonder why you need to evaluate, or measure the performance of, a program that seems to be working.

The first chapter of this manual presents reasons why you might want to evaluate an existing program or consider an evaluation before planning your next program. Have you wanted answers to the following questions about any of the programs in your public housing developments?

• Are our current programs meeting residents' needs?
• How can I measure the performance of our programs?
• How can I be sure a specific program is working?
• How can I find out if the violence prevention services we provide are leading to changes in the safety of residents?
• What works best to reduce violence and crime for our public housing residents?

This manual explains how program evaluation can answer each of these questions.

Benefits of evaluation

Evaluation measures performance. As a public housing administrator, you want to measure program performance. Evaluation provides tangible evidence that you are putting resources into programs that benefit residents. More importantly, it helps you direct those scarce resources to support programs that work. Evaluation is also just as useful to determine what doesn't work in a program and provides information you can use to improve your current efforts.

Evaluation demonstrates program benefits to funding sources and to the community. If you have a violence prevention program that works, you should share this success with funding sources, residents, and the community at large. The public or private agency or foundation that funded this program will want to know that it is supporting a successful project. Additionally, this information can be used to attract other potential funders. Agencies also often require programs to measure performance or provide information on program results, service quality, and customer (resident) satisfaction.

In addition, evaluation can provide useful information on the impact of the violence prevention program to a variety of audiences who are in the position to support your efforts, such as State and local officials, local law enforcement agencies, neighborhood associations, and community leaders. Evaluation will give you the evidence you need to obtain support for your program. You can use results to solicit funds from other funding sources, to support a request for additional funds to expand the program, or to justify offering the same program in another location.

Evaluation can help improve your program's effectiveness. Another benefit of conducting a program evaluation is that the findings will help you improve your program. You will be able to say with confidence that changes or improvements in the program are directly related to your program's evaluated intervention(s).

Evaluations create an opportunity to share information about what works with similar agencies. If you have a program that has been shown through evaluation to be effective, you can share this valuable information with other public housing agencies. You will have documented evaluation findings that show that the program design succeeded. Other agencies of similar size, in similar environments, will be able to replicate your program knowing that it works.
.

An evaluation can provide you with information on the process of implementing the program as well as the outcomes. Other public housing agencies will be able to benefit from lessons learned during your program implementation phase and the subsequent evaluation.

Common concerns about evaluation

Concern #1. Evaluation diverts resources from the program itself. Although an evaluation will cost something, it does not have to divert resources from other program activities. Consider developing a separate budget to support an evaluation. For example, when creating your initial program budget, you could divide the money into two distinct components-one for program activities and one for the evaluation. If your agency has funds set aside for research purposes, your evaluation could be funded from this pool of money. Or you could obtain financial support from a source other than the one funding your programmatic activities.

Concern #2. Evaluation increases the burden on program staff. Often program staff are responsible for collecting evaluation information because they are the most familiar with program participants and have the most contact with them. Despite this potential for increased burden, staff can benefit greatly from evaluation because it provides information that can help the staff improve their work with participants, learn more about program and participant needs, and validate their successes. Also, it's possible to decrease the burden on staff somewhat by incorporating evaluation into ongoing program activities. More information on how to do this appears in chapter 5.

Concern #3. Evaluation is too complicated. This is a common misconception. An evaluation is intended to answer basic questions about programs. Usually, evaluation questions include the following:

• How well did the program perform based on its stated performance indicators?
• Did the program meet its stated objectives? (For example, did the program prevent violence for the target population it served?)
• Did change occur, and what were the changes?

Concern #4. Evaluation may produce negative results and hurt the program. An evaluation may show that the program has not worked, but it is more likely to show that the program alone is not responsible for any changes that may have occurred. The goal of most evaluations is to determine whether changes can be attributed to a particular component of the program.

Concern #5. Evaluation is another form of monitoring. Evaluation is not the same as monitoring. However, often the information examined to monitor program operations is similar or can overlap with the information needed to conduct an evaluation. Monitoring looks at whether the program elements are being provided. For example, are staff doing what is called for by the program plan? Are they spending their time on appropriate program activities? Evaluation differs from monitoring in that it attempts to relate the desired outcomes to the program activities to see whether these activities produced the outcomes.

Tips for conducting a successful evaluation

Invest in planning. Before you begin, develop a plan that details what you are planning to evaluate, the timeframe for conducting the evaluation, who will do the evaluation, what resources are available, and what you plan to do with the findings. Ideally, the evaluation will be planned at the same time that the program gets underway. For more information on developing an evaluation plan, see chapter S.

Set aside adequate resources for the evaluation. Resources include not only funds to support the evaluation, but also staff time to complete evaluation activities.

Begin the evaluation during the initial stages of program implementation. It is always best to begin the evaluation during the initial stages of program implementation. Having information about the program from the very beginning enables you to make modifications if you determine that any aspect of the program is not working. Obviously, it is far better to make changes in a planned program early in the implementation phase than to carry out a program knowing that some aspect of it is not working well.

For example, say that as part of a violence prevention effort, your PHA has established a peer mediation program for youths ages 8 to 15 residing in two public housing developments. This 3-month program operates daily from 9 a.m. to noon in the local recreation center. During the first week of operation you determine that attendance is only 30 percent of what you expected. Most of the targeted youths have opted to go to the open swim at the recreation center or are working mornings with the summer youth employment program.

In this hypothetical situation, you would need to make a decision about how to modify the peer mediation program to ensure participation by its intended target population. Let's say you decide to change the hours of the peer mediation class to late afternoon and to offer an incentive to youths who complete the program. Flyers are distributed announcing the time change and weekly cookouts for all participants who attend daily.

This is an example of changing a program design in its early stages, when it becomes apparent that participation is likely to be much lower than expected. If this change in program design had not been made early in the implementation phase, the program may have experienced low levels of participation and may not have produced the desired outcomes.

Promote resident and staff participation in the evaluation. Participation of both the residents and housing authority staff in the evaluation is critical to its success. Residents and staff need to feel that both
the evaluation and the findings will benefit everyone involved. They should fully understand the importance of the evaluation process and how the evaluation findings can help them, their community, and the program. One way to encourage resident/staff ownership of and responsibility for the evaluation is to involve them in the evaluation. This can be accomplished during the period when you are collecting information about the program and participants (commonly referred to as the data collection phase of the program evaluation). Often staff are the best data collectors because they are the ones most actively involved in day-to-day program operations.

Be realistic about the burden of an evaluation. Evaluations require work. Even if you hire an outside evaluator to conduct one, you and your staff will have to spend time arranging for the evaluator to access data needed to carry out the study. Depending on the type of data collection planned for the evaluation, there may be the need to interview staff, review records, or distribute and collect survey questionnaires from residents and program participants. This is another reason that it is important to explain to staff end residents why you need an evaluation and how it will benefit them.

Ensure confidentiality of resident responses. Obtaining personal data is always sensitive; however, getting people to discuss their thoughts and/or involvement in illegal or violent activities is particularly touchy. When promising confidentiality of responses to the survey instruments, it is important to adhere to this commitment through all evaluation activities. Public housing residents who participate in any evaluation should be informed that they are taking part in one and that they have the right to refuse to participate without jeopardizing their participation in the program.

Have participants sign consent forms containing assurances that they will not be asked to leave their housing units based on information they supply for the evaluation and informing them that the evaluation is designed so that individual responses cannot be linked back to a specific participant. More information on informed consent and a sample consent form appear in chapter 6.

Consider cultural issues. You will want to ensure that the evaluation is relevant to and respectful of the cultural backgrounds and individuality of program participants. Chapter 6 also contains more discussion of cultural relevance.

Chapter 2
    What is Evaluation?

You and your staff have probably asked yourselves questions such as, Are our residents safe in their homes? Are crime rates decreasing? Are residents satisfied with new security efforts? Evaluation addresses these same questions using systematic methods to ensure that the answers are supported by evidence.

Program evaluation is simply a systematic method for collecting, analyzing, and using information to answer basic questions about your violence prevention program.

What can an evaluation tell you?

An evaluation can tell you whether you have been successful in meeting two primary program objectives:

Have you been successful in implementing your program? Are you implementing the activities that you initially planned? Are you reaching the intended target population? Are you reaching the intended number of residents? Are you developing planned collaborative relationships? These questions address your implementation objectives- what you plan to do, how you plan to do it, and who it is you want to reach.

Have you achieved the results/outcomes you expected? Are you seeing a reduction in crime in your complex? Do residents feel safer? Are residents exhibiting the expected changes in knowledge, attitudes, behaviors, awareness, or self-esteem? Are you seeing the expected changes in the community? These questions address your outcome objectives, which are your expectations about how your program will affect the incidence of acts of violence in your housing development and affect residents' satisfaction, knowledge, behavior, and attitudes.

An evaluation should answer both types of these objectives. You may have successfully implemented your program, but if you do not have information about program outcomes, you will not know if your program is resulting in decreased crime and violence. Similarly, the program may be successful in increasing security, but you will not be able to identify how or why these changes occurred if you do not have information about how the program was implemented.

To answer these questions, the evaluation plan must include performance measures. Performance measures may be based on quantitative or qualitative data. Quantitative data are easily expressed in numerical terms, such as the number of robberies per month or the number of residents participating in a specific violence prevention activity. On the other hand, qualitative data usually describe attitudes, such as residents' feelings of safety in their homes and their beliefs about how serious the problem of teenage drug use is. Both types of performance measures are useful in evaluating your violence prevention program.

What is involved in conducting an evaluation?

The term "systematic" in the definition of evaluation indicates that evaluation requires a structured and consistent method of collecting and analyzing information about your program. This means that an evaluation must be carefully planned and carried out. You can ensure that your evaluation is conducted in a systematic manner by following several basic steps.

Step 1: Assemble an evaluation team. Planning and implementing your evaluation should be a team effort. Even if you hire an outside consultant to help, you and your staff need to remain full partners in the evaluation effort. Chapter 3 discusses various evaluation teams and the roles that different team members may play in an evaluation.

Step 2: Prepare for the evaluation. This includes deciding what to evaluate, building a model of your program, and stating your objectives in measurable terms. The more attention you give to planning the evaluation, the more effective it will be. Chapter 4 will help you prepare for your evaluation.

Step 3: Develop an evaluation plan. An evaluation plan is a blueprint or a map of an evaluation. It details the design and methods that will be used to conduct the evaluation and analyze the findings. Information on what to include in a plan is provided in chapter S.

Step 4: Collect evaluation information. Once an evaluation plan is completed, you are ready to begin collecting information. This will require selecting and/or developing information collection procedures and instruments. This process is discussed in chapter 6.

Step 5: Analyze your evaluation information. After evaluation information is collected, it must be organized in a way that allows you to analyze it. Analysis should be conducted at various times during the course of the evaluation to allow you and your staff to obtain ongoing feedback about the program. This will either validate what you are doing or identify areas where changes may be needed. Chapter 7 discusses the analysis process.

Step 6: Prepare the evaluation report. The evaluation report should be a comprehensive document that describes the program in addition to providing the results of the implementation and outcome analysis. The report also should include an interpretation of the results for understanding program effectiveness. Chapter 8 is designed to assist you in preparing an evaluation report.

What will an evaluation cost?

Although a dollar amount cannot be specified here, it is possible to describe the kinds of information you can obtain from evaluations at different cost levels. Generally, the more data you collect (that is, collecting data at different points in time or from a variety of sources), the more costly the evaluation; however, the usefulness of the evaluation also increases.

Collecting data from one point in time and from a limited number of sources or using data that are already assembled for other purposes will minimize your evaluation costs. Examples of performance measures that this level of evaluation address include the following:

• The number and/or characteristics of residents participating in a violence prevention activity.
• Current resident satisfaction with security or a particular violence prevention activity.
• The number of violent acts occurring in the housing development at a specific point in time.

While the above-listed performance measures are useful, they do not provide the information needed to assess the change that occurred because of your program. To measure change, you need data from at least two points in time, usually immediately before the program activity was initiated and again when it is reasonable to assume that the program would have had an impact (perhaps 2 to 6 months after initiating the activity). Examples of performance measures at this level of evaluation include the following:

• The increase (or decrease) in the number of residents or types of residents served by a program activity.
• The increase (or decrease) in residents' satisfaction with security or a particular violence prevention activity.
• The decrease (or increase) in the number of violent acts occurring in the housing development.

Comparing data from two points in time allows you to assess changes that may have occurred because of your violence prevention program. Obviously, collecting and analyzing data from two or more points in time is more costly than a one-time data collection. However, if resources permit, this type of evaluation will provide significantly more information than the one-time data collection approach.

The most sophisticated level of evaluation-and the most costly one- allows you to determine that an observed change was caused by your program and not by some outside circumstances. This type of evaluation is the most costly one because it requires that the evaluation include data from sources not affected by your program, such as residents from the surrounding neighborhood.

For example, if your residents express increased satisfaction with security at their housing development, and residents of the surrounding neighborhood also express increased satisfaction with their security, then the increased satisfaction might be due to an increase in police activity or other community initiatives not directly related to your violence prevention program. Alternatively, if the neighborhood residents express either no change in satisfaction or a decrease in their satisfaction with security, then you have fairly strong evidence that your program is responsible for your residents' increased satisfaction.

As you can see, there is no single answer to the question, "What can an evaluation tell you?" posed earlier in this chapter. Evaluations can include a variety of performance measures and can provide different levels of information. The remaining chapters in this manual provide guidance in designing an evaluation that will meet both your information needs and budgetary constraints.

Chapter 3
    Who Should Conduct Your Evaluation?

Evaluation is best thought of as a team effort. Although one person heads an evaluation team and has primary responsibility for the project, this individual will need assistance from others on your staff. An evaluation team will work together on the following tasks:

• Determining the focus and design of the evaluation.
• Developing the evaluation plan, performance indicators, and data collection instruments.
• Collecting, analyzing, and interpreting data.
• Preparing the report on evaluation findings.

Types of evaluation teams

You can assemble many types of evaluation teams. Three possible options include:

• Hiring an outside evaluator (option 1).
• Using an inhouse evaluation team supported by an outside consultant and program staff (option 2).
• Using an inhouse evaluation team supported by program staff (option 3).

Hiring an outside evaluator. Housing authorities typically do not have a research and evaluation staff and will probably need to hire an outside evaluator. This person would be supported by inhouse staff and would serve as a team leader. The evaluator could come from a research institute or a consulting firm. For more information on locating a candidate for this position, see the section later in this chapter on finding and hiring an outside evaluator.

Using an inhouse evaluation team supported by an outside consultant and program staff. If you feel that you have sufficient staff resources to implement the evaluation but need assistance with the technical aspects, you may want to hire an outside consultant. In this situation, an inhouse evaluator would serve as the team leader and be supported by both program staff end the outside consultant. If there are research resources within your Public Housing Agency (PHA), you may want to consider this option.

A consultant could support the evaluation by developing the evaluation design, conducting data analyses, and selecting or developing questionnaires. This person can also help you develop the evaluation plan and performance indicators.

Using an inhouse evaluation team supported by program staff. If resources are available within your PHA (that is, if you have research staff, evaluators, or program personnel who can assist with the evaluation), you could recruit these individuals to serve as evaluation team members.

The chart on the next page shows the possible advantages and disadvantages for each of the evaluation team options. Whatever team option you select, you will want to make sure that you or someone from the PHA become part of the team. Even if your role is limited to one of overall evaluation management, you will want to participate in all phases of the evaluation effort.

Deciding what team is best for you. This decision will be influenced most by the resources and capabilities of your PHA. To determine what internal resources are available, you can examine your staff's skills and experience in planning an evaluation, designing data collection procedures and questionnaires, and collecting and analyzing information.

Below is a checklist to help you decide what type of team you may need. If you answer "no" to all of the resource questions, you may want to consider postponing your evaluation until you can obtain funds to hire an outside evaluator. You may also want to consider budgeting funds for an evaluation in your future program planning efforts.

If your answer to question 1 is "yes" but your answer is "no" to all other questions, you will need maximum assistance in conducting your evaluation and option 1 (an outside evaluator with inhouse support) is probably your best choice.

If you answer "no" to question 1 but "yes" to most of the other resource questions, then option 3 (inhouse staff only) may be an

appropriate choice for you. However, if you plan to use evaluation findings to seek program funding, you may want to consider using option 2 (inhouse team with outside consultant) and try to obtain evaluation funds from other areas of your agency's budget. If your answer to question 1 is "yes" and the remainder of your answers are mixed (some "yes" and some "no"), then either option 1 or option 2 could be effective.

Finding and hiring an outside evaluator

Careful selection of an outside evaluator can mean the difference between a positive and a negative experience. A good place to start is to consider using someone that you or another PHA has worked with successfully on another project. You may want to begin the search by interviewing that particular person. Other public agencies within your community may also be good sources for this type of referral.

PHAs located in or near a large city should have little trouble finding an evaluator. Many consulting firms have staff experienced in evaluation who have conducted housing-related evaluation studies. Most universities and colleges have faculty who can design and conduct evaluations. This manual includes a listing of national resources that may aid you in your search for experts.

Before hiring an evaluator, you will want to know the following:

• What level of experience does the evaluator have in the area of program evaluation?
• Does the evaluator have any experience conducting evaluations in the area of public housing or in a related area?
• Can the evaluator offer assistance in the full range of program evaluation activities including research design, data collection, data analysis, data interpretation, and dissemination of the results?
• Will the evaluator present information in a way that will be useful to you?
• How much will the project cost?
• How will the housing staff tee involved in the evaluation?
• Is the evaluator willing to work closely with you, the housing agency staff, and the tenant association?

A good outside evaluator will not dictate to you or your staff how the project will proceed but will instead work with you to conduct a successful evaluation. You should have input into determining the purpose of the evaluation, research questions, and performance indicators. Everyone on your evaluation team, including the outside evaluator, should be willing to collaborate with your PHA.

There are four basic steps for hiring an evaluator.

Step 1: Develop a statement of work. The first step in the hiring process is to develop a statement of work that details the general and specific requirements for the evaluator. General requirements should list the materials, services, and products to be provided by the evaluator.

In creating the statement of work, you will need to know the types of evaluation activities you want the evaluator to perform. Evaluator responsibilities can involve developing an evaluation plan, providing progress reports, developing data collection instruments and forms, collecting data, analyzing data, and writing reports.

The general requirements of the statement of work should list the frequency with which you expect to meet with the evaluator and the requirements for submitting written reports. Specific requirements should outline tasks. For example, if the tasks to be completed by the evaluator include a preliminary meeting; development of a research design, data collection plan, and sampling plan; data collection activities; data analysis; and a preliminary and final report, then the statement of work would specify each of these activities as a separate task. In addition, each task should have a timeframe for beginning and completion. Similarly, each task should have specific milestone dates for completion of work or submission of documents.

Step 2: Locate sources for evaluators. Potential sources you can use to find an evaluator include:

• Other public housing authorities that have used outside evaluators. PHAs or community-based organizations may have staff who have used outside evaluators and are able to recommend one, suggest methods of advertising for one, or provide other useful information. Contacting similar programs is one of the best ways to find an evaluator who understands your program, is sensitive to public housing residents, and can provide a useful evaluation that meets your needs.
• Evaluation divisions of State or local agencies. Most State or local government agencies have planning and evaluation departments. In addition, some local police departments employ research staff. You may be able to use individuals from these agencies to work with you on your evaluation, or they may be able to direct you toward other organizations with experience in conducting outside evaluations.
• Department of Housing and Urban Development (HUD). A number of offices HUD fund national evaluation studies conducted by contractors. These offices, such as the Office of Policy Development and Research and the Office of Public and Indian Housing, may be able to direct you to contractors who are experienced in conducting research and evaluation studies.
• Local colleges and universities. Departments of sociology, psychology, social work/social welfare, and public administration, as well as university-based research centers, are all possible sources within colleges and universities. Well-known researchers affiliated with these institutions may be readily identifiable. If they cannot personally assist you, they may be able to refer you to other individuals who are interested in performing local program evaluations.
• Technical assistance providers. Some Federal grant programs include a national technical assistance provider. If your agency is participating in this kind of grant program, then asking for assistance in identifying and selecting an evaluator is appropriate and reasonable.
• The public library. Reference librarians may be able to direct you to new sources. They can help identify local research firms and may be able to provide you with conference proceedings that list program evaluators.
• Research and consulting firms. Many experienced evaluators are part of research and consulting firms. They may be more difficult to identify initially but are sometimes listed in the yellow pages under "Research" or "Marketing Research." They can also be located by contacting your State human services departments to get a listing of the firms that have bid on recent contracts for evaluations of State programs.
• The United Way, American Public Welfare Association, Child Welfare League of America, Urban League, and local foundations. Your United Way or Urban League chapter, or area foundation staff and board members, may be able to provide you with names of local evaluators. They may also be able to provide insight on evaluations that were done well.
• The American Evaluation Association. Many evaluators belong to the American Evaluation Association. Based at the University of Virginia in Charlottesville, Virginia, this organization is able to provide a list of members in your area for a fee. Additional information on the American Evaluation Association is contained in the resources section of this manual.
• The American Criminal Justice Association (ACJA). ACJA has over 4,000 members who are concerned with the administration of criminal justice. ACJA publishes a semiannual journal, the Journal of the American Criminal Justice Association.
• The American Society of Criminology (ASC). ASC is a national organization concerned with criminology, research, and education involving the etiology, prevention, control, and treatment of crime and delinquency. Its membership includes over 2,500 professional and academic criminologists, practitioners, and academicians in many fields of criminal justice. ASC publishes a newsletter, The Criminologist, six times a year and a journal, Criminology, four times a year.
• The American Sociological Association. State and local chapters of the American Sociological Association may be able to direct you to members who conduct local program evaluations. Departments of sociology or criminology at local colleges and universities should be able to provide you with your State chapter's telephone number.

Step 3: Advertise and request applications. Once you have developed a statement of work and identified possible sources for evaluators, you are ready to advertise for applications. Advertising in the local paper, posting the position at a local college or university, and working with your local government's human resources department are possible ways of soliciting offers. Agency newsletters, local and national meetings, and professional journals are additional sources where you can post your request for applications or an advertisement. You will want to advertise as widely as possible, particularly if you are in a small community or are undertaking an evaluation for the first time. Using several advertising sources will help ensure that you receive more than one response. The following page lists some suggestions for preparing an effective advertisement.

If you have sufficient time, you may want to consider a two-step process for applications-advertising the position but also sending evaluators who respond to your advertisement more detailed information about your evaluation requirements. For example, you could send potential evaluators a brief description of the program and the evaluation questions you want to answer, along with a description of the size and location of the PHA where the evaluation will be conducted. This would give the candidate an opportunity to propose a plan that more closely corresponds to your program needs.

Step 4: Renew proposals and select an evaluator. The final step in hiring an evaluator is to review the proposals submitted and select an evaluator. In reviewing proposals, you will want to consider the candidate's writing style, type of evaluation plan proposed, experience working with your type of program and staff, familiarity with violence prevention initiatives and public housing residents, experience conducting similar evaluations, and proposed cost.

Once you have narrowed your selection to two or three candidates, you will want to contact them to schedule inperson interviews. The interview is an important part of the selection process. It will allow you to determine whether you and the evaluator are compatible. You can review the criteria for selecting an evaluator discussed earlier in this chapter and use them to guide your questions during the interview.

In most cases the ideal candidate will seem obvious after the interview is conducted, but sometimes it will be more difficult. As you do for other job applicants, you will need to check references.

It makes sense to follow whatever procedure your PHA has in place for contracting out for services. Remember to obtain the appropriate agency approval before notifying your chosen evaluator. If you currently operate a federally funded violence prevention program, check your grant requirements to determine whether you need Federal approval of your evaluator.

What to do when you have trouble hiring an evaluator. Despite your best intentions, you may encounter difficulties, including one or more of the following:

Few or no responses to your advertisement. PHAs located in isolated areas may only get a few responses to their advertisements. If this occurs you may want to contact 1 of the 80 local offices of the Department of Housing and Urban Development. They should be able to assist you by providing information about evaluators who have conducted similar evaluation studies in or near your community.

None of the applicants is compatible with program philosophy and staff. If applicants do not match program needs, you may find it helpful to network with other PHAs or directors of violence prevention programs in your city or State who have worked with evaluators. A compatible philosophy and approach are most important, and tradeoffs between them and the proximity of the evaluator may be needed to find the right one.

Money allocated for the evaluation is insufficient to hire a third party evaluator. In this instance, you will need to generate additional funds for the evaluation or negotiate with your evaluator to donate his or her services (inkind service). Many evaluators are committed to their profession and regularly discount their fees or donate a portion of their time to evaluation projects. Another option is to negotiate with a
college professor, using the evaluation dollars to pay for this person's time, while advanced degree students work under the direction of the professor to conduct some of the evaluation activities.

There may be researchers in your community who have an interest in conducting research in the area of violence prevention. They may be interested in using information about your violence prevention initiative and may be willing to provide evaluation services in exchange for access to participant and program information. For example, you can allow a university professor to have access to program and participant records in exchange for evaluation services such as instrument development or data analysis.

Managing an evaluation headed by an outside evaluator

Often when the decision is made to hire an outside evaluator, program managers and staff believe that the evaluation is out of their hands. This is not true. An outside evaluator cannot do the job effectively without the cooperation and assistance of PHA staff end residents. An evaluation is like any activity taking place within your agency; it needs to be managed.

Creating a contract. One mechanism for effectively managing the evaluation is to prepare a written contract for specifying the evaluator's roles and responsibilities. The contract should be prepared once you have received the appropriate approval to hire someone. Both the evaluator and the PHA staff authorized to hire outside services will need to sign the contract.

Your contract is a legally binding document which specifies the evaluation activities to be performed, the amount of time to complete the evaluation, and the cost. This document offers you protection by specifying who is expected to conduct the work and how the data that has been collected will be used. Every evaluation contract should include the following items:

• Who will perform evaluation tasks. Some evaluators delegate many of their responsibilities to less experienced staff end have little contact with the client once the contract is signed. To protect yourself from this scenario, the contract should specify what percentage of time the evaluator and his or her staff will devote to the evaluation.
• Who owns the evaluation data. In your contract, specify who has ownership of the data and to whom the information can be given. Release of information to outside parties should always be cleared with appropriate PHA staff. Any plans for publishing the evaluation results should be discussed and cleared before articles are written and submitted for publication.
• Your expectations about contacts between the evaluator and PHA. It is very important for an outside evaluator to keep the PHA, program staff, and residents informed about the status of the evaluation. Regular communication allows the PHA and other concerned parties to make important changes on an ongoing basis. The contract should specify expectations about the frequency of meetings and ongoing reporting requirements.

What to do if problems arise. Even with the best contracts, however, problems can arise during the course of the evaluation process. Examples of types of problems that may occur and possible solutions include the following:

Evaluation approaches differ-the program manager and evaluator do not see eye to eye. Try to reach a common ground where both programmatic and evaluation constraints and needs are met. If many reasonable attempts to resolve differences are made and severe conflicts still remain that could jeopardize the program or the evaluation, program staff should consider terminating the evaluation contract. This decision will need to be weighed carefully, as a new evaluator must be recruited and brought up to speed in midstream. In some situations, however, this may be the best option.

Evaluation of the program requires skills or analyses for which you did not originally plan. You may find that your evaluator is in agreement with your assessment and is willing to add another person to the evaluation team who has expertise and skills needed to undertake additional or different analyses. Many times additional expertise can be added to the evaluation team by using a few hours of a consultant's time. For example, programmers or statisticians may be necessary to augment the evaluation team.

The evaluator leaves, terminates the contract, or does not meet contractual requirements. If the evaluator leaves the area or terminates the contract, you will most likely be faced with recruiting a new one. In some instances programs have successfully maintained their ties to evaluators who have left the area, but this is often difficult.. When your evaluator does not meet contractual requirements and efforts to resolve the dispute have failed, you should turn the case over to your procurement office.

The evaluator does not have any experience working with low-income populations or community residents. It is not always possible to locate an evaluator who has the necessary experience in evaluation and experience working with public housing residents. It is a documented fact that very few evaluations have been conducted of violence prevention initiatives in public housing. You may have to educate the evaluator about the public housing development where your project is being offered and the characteristics of the surrounding community. The evaluator needs to understand how these factors may affect the evaluation and the questionnaires and procedures to be used. You may require that the evaluator work with members of the resident council and other key community leaders so that the evaluation is relevant to the participants' experiences and cultures. Remember, the outside evaluator works for you, and the implementation of a successful and valuable evaluation depends on you.

Chapter 4
    How Do You Prepare for an Evaluation?

There are three basic steps to building a strong foundation for 1 your evaluation.

• Step 1: Decide what to evaluate.
• Step 2: Specify your violence prevention program activities and the assumptions used in developing those activities.
• Step 3: State your program objectives in measurable terms.

Step 1: Decide what to evaluate

Your violence prevention program may be broad in scope, encompassing various activities, or it may focus on only one or two activities. You may choose to evaluate any or all parts of your program. Obviously, the more activities you include in your evaluation, the more resources will be required. If your resources are limited, you may want to narrow the scope of your evaluation. It is better to conduct an effective evaluation of a single program activity than to attempt a more comprehensive evaluation of many activities if you are lacking sufficient resources.

Step 2: Specify your violence prevention program activities and the assumptions used in developing those activities

Whether you decide to evaluate your entire program or a single program activity, you will need to develop a clear picture of each activity to be evaluated. Evaluation researchers usually refer to this picture as a "model." The model of your program should include the components listed below:

• The assumptions on which the activity was developed.
• A description of the program activities.
• Short-term program outcomes.
• Intermediate outcomes.
• The overall goal of the program activity.

The assumptions on which the activity was developed. Your decision to implement a particular violence prevention activity is based on a problem you identified and your belief that the activity will decrease or eliminate that problem. Listing these assumptions helps in understanding the objectives of the program activity. The table below lists some assumptions you might have that would lead to developing specific activities.

A description of the program activities. Listing the specific activities you plan to carry out (or have already implemented) identifies what it is you will be evaluating. This listing should include a description of the specific activity, who will perform the activity, who the activity is planned to reach, how many people the activity is designed to reach, and the timetable of the activity.

Worksheet 4-l(a) shows what might be included in a violence prevention activity designed to develop resident security patrols. A blank worksheet, Worksheet 4-l(b), is provided for your use in planning a similar violence prevention activity. Completing tables such as this helps in specifying your implementation objectives and outcome objectives.

• Column A: Enter each activity required to carry out the overall activity.
• Column B-1: Enter PHA staff or residents involved in implementing the activity.
• Column B-2: Enter any outside resources needed to implement the activity.
• Column C: Enter the people expected to be affected by the activity.
• Column D: Enter the time period in which the activity will be started and the duration of the activity.

Short-term program outcomes. These are the results you expect to see in the near future. I hey include areas such as keeping to the planned schedule and the number of people receiving the service as well as monitoring the effects of the program on participants.

Intermediate outcomes. These are the long-term results you expect to achieve after the program has been fully operational.

The overall goal of the program activity. This is the expected impact of the program.

Worksheet 4-1(a): Listing of Program Components

Worksheet 4-1(b): Listing of Program Components

Column A: Enter each step required to carry out the overall activity.                                                                                      Column B-1: Enter PHA staff or residents involved in implementing the activity.                                                                  Column B-2: Enter any outside resources needed to implement the activity.                                                                      Column C: Enter the people expected to be affected by the activity.                                                                                        Column D: Enter the time period in which the activity will be started and the duration of the activity.

Step 3: State your program objectives in measurable terms

Having clearly defined the steps needed to implement the violence prevention activity and identify the short-term and intermediate outcomes, you now need to develop specific criteria-or performance measures-that you will use to determine that the program objectives were achieved. Program objectives include program implementation measures, such as how you will implement your violence prevention activity, as well as the expected result of the activity.

Using our example of tenant security patrols, the outcome of "decreasing violent acts in the housing development" needs to be made more specific. For example, how much of a decrease are you expecting in assaults? In rapes? In murders? The specific goal you define should be realistic, but it should also be indicative of effectiveness. For example, it may be unrealistic to expect that no assaults will occur, but a decrease of only one percent is not an indication of program effectiveness. You can see that such performance measures are subjective. Knowledge of your housing development, the residents, and other community factors will influence how you define the performance measures.

Performance measures of short-term and intermediate goals may differ for the same goal. Again, using the example of tenant security patrols, your short-term goal of decreases in assaults may be 10 percent, but the intermediate goal may be 50 percent based on the assumption that as knowledge of the presence of tenant patrols becomes more widespread, the number of assaults will decrease further. Worksheet 2(a) provides examples of short-term and intermediate performance measures; Worksheet 4-2(b) is a blank worksheet provided for your use.

Defining the outcome objective in measurable terms identifies the information you will need to evaluate your program activity. Note, for example, that whenever you want to measure change, you will need information at two points in time. Chapter 6 discusses collecting information for the evaluation.

Worksheet 4-2(a):  Defining Performance Measures

Chapter 5
    Developing an Evaluation Plan

An evaluation plan is a written document that states the objectives of the evaluation, the questions that will be answered, the information that will be collected to answer these questions, and when collection of information will begin and end. You can think of the evaluation plan as the instructions for the evaluation. This plan can be used to guide you through each step of the evaluation process because it details the practices and procedures for successfully conducting your evaluation.

Once the evaluation plan has been completed, it is a good idea to have it reviewed by selected individuals for their comments and suggestions. Potential reviewers include:

• Public Housing Agency (PHA) administrators who can determine whether the evaluation plan is consistent with the agency's resources and evaluation objectives.
• Housing staff who can provide feedback on whether the evaluation will create an excessive burden for them and whether it is appropriate for residents.
• Professional evaluators.

This chapter describes the components for an evaluation plan and provides an outline for preparing a plan. Although you may never need to develop one without assistance, it is helpful for you to know what a plan is and how it is being used by the evaluator you select. The information contained in this chapter will help you:

• Work with an experienced evaluator (either an outside evaluator or someone within your PHA) to develop a plan.
• Obtain a basic understanding of what should be included in an evaluation plan to assist you as you review one.

A sample evaluation plan outline that may be used as a guide appears on the following pages. The major sections of the outline are:

• Section I: A description of the evaluation framework which specifies what you want to evaluate, what questions are to be addressed in the evaluation, and the timeframe for conducting the evaluation.
• Section II: A description of the program implementation objectives.
• Section III: A description of the program outcome objectives and performance measures.
• Section IV: Procedures for managing and monitoring the evaluation.

Section 1: The evaluation framework

This section of the evaluation plan presents the model for assessing your program activities (see chapter 4), program objectives, evaluation questions, and the timeframe for the evaluation (that is, when you will begin and end collection of evaluation information). Section I should also include a discussion of the context for the evaluation, particularly the aspects of the PHA, program staff, and residents that may affect the evaluation. If an outside evaluator is preparing the plan, the evaluator will need your help to prepare this section to ensure that the evaluation is tailored to the needs of your PHA and the residents.

Section 11: Evaluating implementation objectives

This section should provide detailed descriptions of what you plan to do, how you plan to do it, and who it is you want to reach. This information will be used to answer evaluation questions pertaining to your implementation objectives, such as: Are implementation objectives being attained? If not, why not? What barriers or challenges have been encountered? What has facilitated attainment of these objectives?

For each objective, the evaluation plan must describe the following:

• Types of information needed.
• Sources of information.
• Criteria for selecting information sources.
• Methods for collecting information, such as questionnaires and procedures.
• Timeframe for collecting information.
• Methods for analyzing information.

Types of information needed. Any information that is collected about your program activities or residents can be considered evaluation data. The types of information needed will be guided by the program objectives you seek to assess. For example, when your objective concerns what you plan to do, you will need to collect information on the types of services, activities, or initiatives that are developed and implemented; who received services, and their duration and intensity.

If the objective of your PHA is to provide increased security patrols at two sites, you will need to collect the following information:

• Number of resident volunteers.
• Number of hours in which security patrols operate.

When the objective concerns who will participate, you will need to collect information about residents' characteristics, the number of residents, how they were selected/recruited, barriers encountered in the selection/recruitment process, and factors that facilitated selection/ recruitment.

If the objective is to involve 50 residents in a 6-week crime and drug reduction program, for example, you will want to collect the following information:

• Age, sex, and race of participants.
• Number of participants previously involved in criminal or drug activity.
• Number of residents who are participating.
• Information on how the participants learned about the program.
• Amount of time residents participate in the program.
• Number of residents who successfully complete the program.

Sources of information. This refers to where, or from whom, you will obtain evaluation information. Again, the selection of sources will be guided by the objective you are assessing. For example:

• Information on services can come from program records or from interviews with program staff.
• Information on residents and recruitment strategies can come from program records and interviews with staff and residents.
• Information about barriers and facilitators to implementing the program or program activities can be obtained from interviews with relevant staff.

This section of your plan should also include a discussion of how you will maintain the confidentiality of information you obtain from your sources. In addition, it is wise to develop consent forms for those residents being asked to participate in the evaluation. The consent form should include a description of the evaluation objectives and how the information will be used. More information on maintaining confidentiality and a sample informed consent form appear in chapter 6.

Criteria for selecting information sources. If your initiative has a large number of staff members and/or residents, you can reduce the time and cost of the evaluation by including only a sample of them as sources for evaluation information. Sampling is a statistically reliable way of identifying a number of persons from the entire group of program participants who are representative of the group. An experienced evaluator will be able to advise you as to whether or not you should select a sample for your evaluation.

There are a variety of methods for sampling your sources.

• You can sample by identifying a specific timeframe for collecting evaluation-related information and including only those residents who participate during that timeframe.
• You can sample by randomly selecting the residents (or staff) to be used in the evaluation. For example, you might assign case numbers to residents and include only the even-numbered cases in your evaluation.
• You can sample based on specific criteria, such as length of time with the program (for staff) or characteristics of residents, such as age, gender, size of family, and length of time in complex.

Methods for collecting information. For each implementation objective you are assessing, your evaluation plan must specify what information will be collected (such as questionnaires and procedures) and who will collect it. To the extent possible, the collection of this information should be integrated into ongoing program operations. For example, in training programs, the registration forms for residents and the initial assessments of participating residents can be used to collect evaluation-related information as well as information relevant to conducting the training. There are a number of methods for collecting information including structured and open-ended interviews, paper and pencil inventories or questionnaires, observations, and systematic reviews of agency records or documents. The methods you select will depend upon the following:

• The evidence you need to establish that your objectives were attained. Performance measures make up this needed evidence. They are the indicators that the program activities reached their intended goals.
• The information sources, which can be questionnaires, interviews, case records, or observations.
• Your available resources. You will need to determine if you have the staff and funds available to collect the needed data.

Chapter 6 provides more information on these sources. The questionnaires of forms that you plan to use to collect evaluation information are usually included as part of your evaluation plan. You will not want to begin an evaluation until you have developed or selected all of the data collection instruments you plan to use. Developing or selecting questionnaires to use for the evaluation may require the assistance of an experienced evaluator.

Timeframe for collecting information. Although you will have already specified a general timeframe for the evaluation, you will need to specify one for collecting data relevant to each implementation objective. Times for data collection will again be guided by the objective being assessed.

Methods for analyzing information. This section of your evaluation plan describes the practices and procedures for use in analyzing the evaluation information. For assessing program implementation, the analyses will be primarily descriptive and may involve tabulating frequencies (of services and resident characteristics) and classifying narrative information into meaningful categories, such as types of barriers encountered, strategies for overcoming barriers, and types of facilitating factors. An experienced evaluator can help your evaluation team design an analysis plan. More information on analyzing program implementation information is provided in chapter 7.

Section III: Evaluating outcome objectives and performance measures

The practices and procedures for evaluating whether the outcome objectives of your program have been met are similar to those for evaluating implementation objectives. To evaluate outcome objectives you will probably use both qualitative and quantitative performance measures. The performance measures will enable you to answer the following questions:

• Did residents and/or the community demonstrate changes in knowledge, attitudes, behaviors, or awareness? Performance indicators could include self-reported increased knowledge, a change in attitude, a reduction in criminal activity or violent behavior, and increased awareness of their own or others' violent behavior.
• Are the changes the result of the program's activities? Are the reported changes in knowledge, attitudes, behavior, or awareness a direct result of your program? Did the changes occur after involvement in the program? Are there other factors that may have influenced the changes?

Two commonly used evaluation designs that can help you to answer these questions are:

• Comparison of conditions before and after a program is established.
• Comparison of conditions before and after a program is established, using a comparison group.

A comparison of conditions before and after the violence prevention program is implemented requires that you collect information at least twice-once before the program is implemented and then again either sometime after the program has been in effect (when you could expect the program to have had a measurable impact) or after the program has ended. You can collect outcome information as often as you like after the program has been implemented, but you must collect it on residents and/or the community before implementing the program. This information is called baseline information and is essential for demonstrating that a change occurred.

If you are implementing an education or training program, this type of design can be effective for evaluating immediate changes in participants' knowledge and attitudes. In these types of programs, you can assess residents' knowledge and attitudes prior to the training and immediately after training with some degree of certainty that any changes observed resulted from your interventions.

A comparison of conditions before and after the violence prevention program is implemented using a comparison group also requires that you collect information at a minimum of two points in time and that you collect information from individuals (or about a housing development or neighborhood) not affected by your violence prevention program. The purpose of a comparison group is to determine if changes you find in your residents or housing development conditions are attributable to your program and not to some other reason. Comparison data might be obtained from the following:

• Housing development residents not participating in the violence prevention program but who are similar to program participants in most other ways; for example, male teenagers not participating in a midnight basketball program.
• Crime statistics from a nearby housing development that has characteristics similar to your housing development's characteristics, such as type of development building (that is, highrise or garden apartments), number of teenagers, and level of criminal activity before your violence prevention program was implemented.

There are obvious cost considerations when including a comparison group in your evaluation design. You must be able to identify a group of individuals or a housing development or neighborhood that is similar to your residents, development, or neighborhood. You must be able to obtain data from such individuals or about the development or neighborhood. Both of these tasks will require some research and additional data collection activity. Although there are additional costs, the information from a comparison group will provide significantly more evidence concerning the effectiveness of your program if your program participants have more positive scores on performance measures than the comparison group. You can state with more certainty that your program was effective in bringing about the observed change and that this is not due to some other reason.

Pretesting information collection instruments. Your evaluation plan will need to include a discussion of your plans for testing out your questionnaires before using them for evaluation. This process is commonly referred to as pretesting. Chapter 6 provides information on pretesting instruments.

Analyzing participant outcome information. Your plan for evaluating outcomes should include a description of how you intend to analyze the data that has been collected. The analyses are intended to answer the questions about whether change occurred and whether changes that occurred can be attributed to your program.

Section IV: Procedures for managing and monitoring the evaluation

This section of the evaluation plan can be used to describe the procedures you intend to use to manage the evaluation. If PHA staff are to be responsible for data collection, you will need to describe how they will be trained and monitored. You may want to develop a data collection manual that describes the processes and procedures for staff to use. This will ensure consistency in information collection and will be useful for staff who are hired after the evaluation begins. Various types of monitoring activities are discussed in chapter 6.

The final section of your evaluation plan should include a discussion of how you will handle any changes in program operations that may occur during the time the evaluation is being conducted. For example, if a particular component is discontinued or added to the program or program activities, you will need to have procedures for documenting when this change occurred, the reasons for the change, and whether particular residents were involved in the program prior to or after the change. This will help you determine whether the change had any impact on attainment of expected results and/or outcomes.

After your evaluation plan is complete and the questionnaires have been pretested, you are ready to begin collecting evaluation information.The following chapter discusses information collection.

Chapter 6
   How Do You Get the Information You Need for Your Evaluation?

After you have completed your evaluation design, you are ready to begin collecting information, the process commonly referred to by evaluators as the data collection phase. This chapter will provide you with steps and suggestions to help you in this process. The information you collect will provide some immediate feedback on whether the program has been effective in reaching its objective(s).

Information collection generally consists of six steps. These steps were introduced earlier in this manual as part of the process of developing your evaluation plan.

Step 1: Determine the kinds of information you need for your evaluation

Your outcome objectives, if stated in measurable terms, will guide the decision about the kinds of information needed. As discussed in chapter 2, both quantitative and qualitative performance measures will be needed to describe program outcomes. You will need to collect information that can be used to demonstrate that your program or program activities have been effective.

For example if one of your general participant outcome objectives is to integrate children from the housing complex into ongoing leagues or other competitive activities in the surrounding community, that objective, stated in measurable terms, will determine the types of information needed. That objective stated in measurable terms may be:

To increase the number of extracurricular activities that youths participate in and to reduce their incidence of behavioral problems as reported by crime statistics, their parents, and self-reports by the youngsters themselves.

Note that any given objective may have multiple measures. In this example, the types of information you will need to assess attainment of this objective are:

• Program participation rates.
• Number and type of interactions with the juvenile justice system.
• Parental observation of improvements in their children's behaviors.
• Student self-reports of their lack of involvement with the criminal justice system.

Specifying the information needed will ensure that you do not collect more than you need. It also keeps the cost and time required for the evaluation to a minimum. Given that most Public Housing Agencies (PHAs) have limited resources, you will want to collect only information that is actually needed for the evaluation.

Step 2: Identify the best sources for the information you need

Every data element usually has a range of potential information sources, including:

• Program records, such as case records and program pretest and posttest scores on any tests given.
• Program management information systems.
• Program reports and documents.
• Program staff.
• Program participants.
• Family members of participants.
• Members of a control or comparison group.
• Staff of collaborating agencies.
• Records from other agencies, such as health agencies, schools, criminal justice agencies, mental health agencies, child welfare agencies, and social service agencies.
• Community leaders.
• Outside experts.
• The general public.
• National databases.

To decide the best sources for information, ask yourself three questions:

1. What sources are likely to provide the most accurate information?

2. What sources are the least costly or time consuming?

3. Does the information collection pose an undue burden on the sources?

Having accurate data sources for the evaluation is the most important factor. For example, it may be less costly or time consuming to obtain information about services from interviews with program staff, but staff may not provide as accurate information about services as case records could. When you interview staff, you are relying on their memories, but when you review case records, you should be able to obtain information about what actually did happen.

If you choose to use case records to obtain evaluation information, however, you will need to make sure that staff are consistent in recording evaluation information in the records. Sometimes case record reviews can be difficult to use for evaluation purposes because they are incomplete or do not report either participant or service-related information in a consistent manner.

Step 3: Select or develop data collection instruments

There is a variety of types of data collection instruments, including:

• uestionnaires or surveys (mail, inperson, or telephone).
• Case record extraction forms.
• Observation forms.

Choosing a particular survey methodology will depend on your PHA's budget, the number and type of questions you need to ask, number of residents being surveyed, availability of staff and/or residents to conduct the surveys, and other factors. Types of surveys include mail, telephone, and inperson. Most likely, you will need to develop your own questionnaire to collect information that specifically addresses your particular program objectives. This is not a complicated process. The discussion on the following pages outlines the steps in developing surveys. Each type of data collection instrument has both advantages and disadvantages, which are listed in the chart on the following page.

Developing a mail survey. The success of a mail survey depends on obtaining the cooperation of the resident, who must be motivated to complete and return the survey within the time specified. Therefore, the design of any mail survey must convey to the resident the following:

• Purpose and importance of the survey (this can be done through a cover letter).
• Clear instructions.
• Confidentiality.
• Established timeframe for completing and returning the survey.

The cover letter. of the simplest ways to convey the importance of the survey is through a cover letter. If possible, the letter should be personalized with the resident's name rather than using a generic "Dear Resident." The cover letter should spell out the purpose and objectives of the survey, who is being asked to respond to the survey and why, and how the resident was selected to receive the questionnaire. The letter should be signed by an official such as the PHA project manager or some other person to show that the study has been given a high profile.

The letter must include a contact person and telephone number in case the resident has questions. In addition, the letter should include a pre-addressed, stamped envelope for returning the survey. An example of a cover letter appears on the following page.

Clear instructions for completing the survey. The instructions for completing a self-administered mail survey are extremely important to ensure that the resident reports accurate data. In self-administered questionnaires the question wording and sequencing can have an effect on data quality. The resident who is completing the survey must first comprehend the questions. To ensure that the resident understands each question and each of the possible answers, you will need to be certain that:

• Instructions are clear and concise. Residents or other respondents should be able to clearly understand how to answer the questions.
• Simple skip instructions guide the respondents through the questions. Skip instructions enable the respondents to maneuver their way through the questionnaire and should be prominently placed so that the respondents do not overlook them.
• Boxed instructions are included, as needed, to bring attention to certain instructions.
• Instructions are consistent throughout the survey and are visually emphasized. A questionnaire must be laid out consistently to ensure that the respondents understand the instructional patterns of the survey. (The use of bold typeface or italics is one way to bring attention to an item.)
• The format of questions is consistent. Consistency of the survey instrument is critical if the respondents are to follow instructional patterns.
• Critical questions do not come at the end of the survey.

Using existing data collection instruments. Many existing instruments can be used to assess participant outcomes. The Center for the Study & Prevention of Violence (CSPV), University of Colorado at Boulder maintains VIOEVAL, a database that stores reference information about survey instruments. Examples of survey references include the self-reporting delinquency scale, the self-efficacy scale, the violence scale, and the self-esteem scale. Contact information for CSPV is included in the resources section at the end of this manual.

Before you select a standardized assessment instrument for your evaluation, be sure to ask an outside professional for advice, and ask administrators of similar programs about their experiences using the instrument. In addition, you should review each item on the instrument to ensure that the information it asks for is consistent with your expectations about how program participants will change.

If you are unable to find an appropriate existing instrument to assess participant outcome objectives, you will need to develop your own. Developing your own outcome assessment instrument is a complex process and may require the assistance of an expert to ensure its usefulness for your evaluation.

Criteria for selecting or developing an appropriate instrument. Whether you decide to use an existing instrument or to develop your own, the instrument you use should meet the following criteria.

• It should include questions that can be used to measure the concepts addressed or affected by your program. For example, if you are providing alcohol and drug abuse prevention training, you would want an instrument to measure changes in knowledge of alcohol and drug abuse.
• It should be appropriate for your participants in terms of age or developmental level, language, and ease of use. Questions should be written in simple and easy-to-understand language.
• It should respect and reflect the participants' cultural backgrounds. The definitions, concepts, and items in the instrument should be relevant to the participants' community and experience. [See the sidebar on the following page.]
• It should be able to be completed in a reasonable timeframe that is not a burden.

Step 4: Establish procedures for collecting information

Once you decide what type of instrument you will use to collect evaluation information, you must establish a set of procedures to ensure that this information will be collected in a consistent and systematic manner. Everyone involved in collecting evaluation information must be trained in these procedures:

• When you will collect the information. The timeframe during which the data is to be collected must be clearly specified. There is some information that may need to be collected before the program starts and other information that needs to be collected at the end of the program. Having the timeframe spelled out will ensure that the information is collected as scheduled.
• Where you will collect the information. You will need to determine the sources from which the information will be collected. In some instances you may be using program records, while in other instances you may be relying on participants coming to a specific location to complete the survey instrument or to participate in a group discussion about their experiences. You will need to determine where you will collect the information and convey this to program participants.
• Who will collect the information. This responsibility must be clearly specified or you will risk having some information collection activities fall through the cracks. In some situations you will need to be sure that information collectors meet certain criteria. For example, they may need to be familiar with the culture or the language of the individuals they are interviewing or observing. If the survey is being administered by interviewers (for example, residents hired and trained to conduct interviews), those persons must be properly trained to administer the survey. Training will ensure that the interviewers are familiar with the survey instrument.
  

• How you will collect the information. You have already decided what instrument to use to collect the information. However, you also need to establish the procedures for administering the instrument. Will it be administered to a group or to individuals? If you are collecting information from children, will other family members be present? If you are collecting information from individuals with a low level of literacy, will the data collectors read the items to them? The methods you use will depend in large part on the type of program and the characteristics of the participants. Violence prevention training and education programs, for example, may have participants complete the instrument in a group setting.

It is very useful to develop a manual that describes exactly what is expected in the information collection process. This helps maintain the quality of the evaluation effort, especially when new staff are hired.

Case record extraction form. If you are using program records as a source of information, you will need to develop a case record extraction form to use. This form provides a place for recording all of the information from participant records needed for the evaluation. Similarly, if you plan to interview program staff, you must develop interviews that focus specifically on the evaluation's information needs. Sometimes, in developing these instruments, you or your evaluator may decide that certain types of information would be "interesting" to collect. However, if the information does not relate directly to your program or outcome objectives, you should resist this urge.

Observation forms or checklists. Another popular evaluation technique is the observation form or checklist. These tools are useful to record information about the environment where the program is located, the number of residents participating when the program is visited, and the behaviors of participating residents and service providers or staff.

Step 5: Pretest the information collection instruments and procedures

Before you begin collecting evaluation information, you will need to pretest your instruments and procedures. The pretest will determine whether the instruments and procedures obtain the information that you want, are not excessively burdensome, and are appropriate for your participant population. Use the pretest information to make any necessary revisions before you begin your evaluation. The kinds of information that can be obtained from a pretest include:

• How long it takes to complete interviews, obtain information from participant records, or fill out questionnaires.
• Whether self-administered questionnaires can be completed by participants without assistance.
• Whether the records you need are readily available.
• Whether you can collect the necessary information in the established timeframe.
• Whether letters to inform participants of the evaluation or any required consent forms can be easily delivered.

You may pretest your instruments with a small number of individuals or program records. You should instruct individuals involved in the pretest to take notes and make comments on the process of using the instruments. These notes and comments may be reviewed to determine whether changes are needed in the instruments or procedures.

You must also review the completed instruments to assess the number of incomplete answers, unlikely answers, comments included in the margins, or other indicators that revisions are necessary. Generally, you will probably need to improve the wording of some questions and instructions for the respondent, as well as delete or add items.

Step 6: Conduct and monitor data collection

After you have completed steps 1 through 5, you are ready to begin collecting evaluation information. This process should be carefully monitored. Monitoring will ensure consistency in the data collection process and that everyone adheres to the time intervals established for collecting information from individual participants.

As part of the monitoring process, you may want to establish a schedule for submitting completed data collection instruments to the evaluation team. This will ensure that instruments are not lost and that confidentiality is maintained. Completed data collection instruments should be treated confidentially; it is a good idea to have completed forms submitted immediately to a member of the evaluation team.

To ensure quality control, information collection staff, particularly if they also are program staff, must be fully educated about the importance of carefully administering and consistently completing evaluation instruments. The first priority of program staff is usually providing services or training to participants, and little effort is given to collecting evaluation information. Encourage your staff to focus on evaluation collection as an important aspect of providing services or training to program participants.

Once evaluation information is collected, you can begin to analyze it. This process may take place on an ongoing basis or after all data have been collected. The procedures for analyzing and interpreting the evaluation information are discussed in the following chapter.