National Institute of Justice. Evaluating Drug Control and System Improvement Projects, Guidelines for Projects Supported by the Bureau of Justice Assistance. Washington, DC: Prepared for the U.S. Department of Justice, National Institute of Justice by Abt Associates, Inc.; 1989 (Reprint 1992). pp. 1-4.

View entire document

Chapter 1: Developing a Strategy for Evaluation

Evaluation involves the systematic assessment of whether and to what extent projects or programs are implemented as intended and whether they achieve their intended objectives. It entails asking questions about projects or programs (or even of a larger constellation of programs that comprise a state's strategy), acquiring information, and analyzing that information. Evaluations vary, therefore, according to the types of questions posed, the methods used for acquiring information, and the types of analyses conducted.

This document provides a summary description of various approaches to evaluation, their essential attributes, and their relative strengths and weaknesses. In addition, the choices posed by the various demands of designing and managing an evaluation are identified, with some discussion of several issues relevant to administrative decisions about when and what to evaluate, and by which means.

The discussion here is focused on the evaluation of specific projects, and not higher-level entities (programs or strategies that include two or more projects). Evaluating such entities - especially their collective impacts - poses special analytic problems not addressed here.

A Continuum of Evaluative Activities

There is no single method of evaluation that is best suited to all purposes and all projects. Instead, the most appropriate method for answering questions depends upon many factors, including:

1) the type of question posed;

2) the nature of the project and its possible effects, and the constraints these features impose on the ability to answer the questions asked;

3) the availability of data needed for an evaluation;

4) how certain the decision maker (or the "client" for the evaluation) needs to be about the data and the conclusions that are produced by the study;

5) what level of resources the decision maker/client is willing to devote to getting the answer; and

6) the time that is available for the study.

Typically, choosing an evaluation strategy requires accepting trade-offs between time, cost, and confidence that one can place in the study's findings. The optimal balance is one in which the evaluation provides the most valuable analysis of project implementation or the most plausible estimates of the project's effects, is most likely to be conducted successfully, and provides the most useful results for administrative, planning, and policy purposes.

Determining Whether to Evaluate at All

Before trying to determine which kind of evaluation approach best suits both the needs of important stakeholders and the nature of the project, a threshold decision first has to be made: whether to evaluate the project at all. States that fund a number of different projects in several program areas will find it difficult to evaluate every project. Rather than attempting to do so, administrators should focus their evaluation resources so that they provide the most useful information possible.

In deciding which projects to evaluate, the following questions should be considered:

How central is the project to the state's strategy?

How costly is it, relative to others?

Are the project's objectives such that progress towards meeting them is difficult to estimate accurately with existing monitoring procedures?

How much knowledge exists about the effectiveness of the type of project being supported? Other things being equal, where more uncertainty exists about a project's effects, the need for evaluation is greater.

Are evaluations underway elsewhere that are assessing similarly designed projects? If so, the administrator may choose to wait until the results of those evaluations are in, and to devote evaluation resources instead to projects about which less is known.

Focusing the Evaluation: Which Questions to Ask?

Because evaluations are undertaken to answer questions, framing the question - or questions - clearly is the most critical element in designing and planning evaluations. Evaluations that are not focused precisely are likely to be wasteful of scarce resources.

Projects as Input-Output Processes

To identify the types of questions that can be asked of a project, and thereby to distinguish among different types of evaluations, it is helpful to conceive of projects as input-output processes.

Inputs refer to all the resources that are devoted to the project, including (among others): money, staff and volunteers, knowledge, equipment and space, as well as such things as the networks of possible supporters and allies that staff bring to the project, and the good will of others that can be called upon. In addition, one of the key inputs are the plans and assumptions upon which the project is built.

Activities are the things people do that, taken together, constitute the project. Examples include patrolling the borders to intercept drug smugglers with experimental technology, investigating crimes and suspects, providing close surveillance of persons sentenced to intensive probation supervision, collecting and testing urine samples from persons under supervision, or responding to calls alleging spouse abuse in certain prescribed and innovative ways.

Results are the goods or services that are produced by these activities. Examples include arrests of drug dealers, convictions, investigations forwarded, seizures of illegal drugs, or provision of intensive probation supervision. As conceived here, results are the more immediate outcomes of project activities, directly flowing from those activities, as distinguished from longer-term or less immediate results that we define as "outcomes."

Outcomes refer to the ultimate end products of a project's activities. These differ from results in that they are the more long-term consequences of activities, or changes that result from those activities. Some of these outcomes may be intended and foreseen, and their occurrence can be taken to indicate that the program's goals were achieved. (Goals are distinct from outcomes, because they define the desired states, conditions, or events against which actual outcomes are assessed.) Other outcomes may be unexpected, unwanted, and unintended. Examples of intended end products of a concentrated drug intervention effort would include not only the seizure of illegal drug shipments (the results of the activities) but also the diminution of the supply of drugs on the street, reduced drug use, and lower incidence of street crimes by addicts. Unintended and undesirable outcomes might include an increase in the incidence of street crimes by drug abusers if a restriction in supply produced a rise in the price of the drug without a parallel drop in demand. In these circumstances, abusers might be compelled to steal more to purchase the same amount of drugs.

Questions About Implementation, Results, and Outcomes

Conceiving of projects as input-output processes permits one to catalogue possible types of evaluative questions. One type of question addresses aspects of a project's implementation, such as features or problems associated with the way project activities are organized or carried out. These include: How well or effectively do the project's administrators and staff combine the various resources (or inputs) they are given? Why are admissions into a drug treatment program so much lower than anticipated? Were assumptions about the target population erroneous? Were recruitment or assignment procedures ineffectively designed, or staff poorly managed? These are just a few examples of a number of problems in program administration or implementation that evaluative questions can address.

The data the evaluator examines to explain these include information about program activities and the way they are organized, in addition to the nature of the inputs (resources, planning assumptions, etc.) and their suitability to the problem the program is designed to address.

Evaluative questions might also focus on the pattern of observed project results. For example, why is the dropout rate from a drug treatment program so high? What accounts for the apparent variation among police precincts in arrest rates in an experimental patrol project? Why are the conviction rates of persons arrested following investigations by multi-jurisdictional task forces so variable (or how do they compare with conviction rates of persons investigated by non-task force officials)? To what extent was the shortened time to criminal case disposition a result of a delay-reduction program? Did the experimental alternative sentencing program actually result in offenders being sentenced to it in lieu of imprisonment? To explain these results, or to account for variations that are observed in them, the evaluator would examine elements that were expected to produce results - both the project's inputs and activities. In addition, the evaluator may want to explore whether other forces or events operating outside the project contribute to the observed results.

Finally, evaluative questions may also focus on whether and to what extent the project has achieved its ultimate goals. Project goals can include such things as reducing the incidence of crime, decreasing the prevalence of drug use and abuse, reducing the supply of illegal drugs, or rehabilitating prisoners. Outcome or impact evaluations are designed to assess whether and to what extent a project has accomplished these goals. This requires analyzing information about those elements of the project that were expected to produce these outcomes, including the inputs and the activities of the project, as well as the more immediate results of these activities. Such evaluations are among the most demanding, because these end products, or outcomes, often pose the most difficult measurement problems. Moreover, the evaluator is faced with the difficult methodological problem of attributing the outcomes to the project itself and ruling out (or at least accounting for) the effects of other forces operating independently of the project.

Matching Evaluations to Types of Projects

Not all types of evaluations are suited to all projects. The appropriateness of an evaluation approach depends in part upon the project's stage of development and "maturity." Moreover, it depends upon whether the requirements of certain types of evaluations can be met.

In the early stages of project development, the most pressing problems confronting managers generally involve project implementation rather than assessments of outcomes. Indeed, a prerequisite for achieving results, and longer-term or less immediate outcomes, is a project that operates with at least some modicum of stability and effectiveness. Translating a project plan into operational reality requires solving a number of problems that emerge in the early months of a project's existence. Projects often have yet a different set of organizational problems after the initial development activity has passed, when they confront a transition to becoming "institutionalized," transformed from a distinct and perhaps innovative "project" into a more routine agency operation. State administrators and program managers may decide in these instances that the most productive use of an evaluator is to assist them in devising effective remedies to specific difficulties in project or program implementation.

Once projects have developed some stability in their activities, evaluating outcomes or results may be feasible. Whether to focus on the less immediate outcomes or the more direct results of a project should be determined by the relative importance of each to the broader state strategy. For certain types of project, their immediate results have the most significance. For example, a project to develop "model drug control legislation" could reasonably be assessed by examining whether such bills were indeed drafted and passed, ignoring whether the new law actually led to reduced drug trafficking or abuse. Other examples include spending funds to enhance capabilities of forensic laboratories, develop automated fingerprinting identification systems, provide assistance to victims or witnesses, establish financial information sharing systems for investigating money laundering, or provide training to staff in financial investigation techniques. Evaluation questions that are likely to be posed of these programs include how many such services, or how much equipment was provided.

In other types of programs, administrators and managers may decide also that it is sufficient for their purposes to assess whether and to what extent programs achieve their immediate results, without demonstrating further that these results produce longer-range or broader crime control benefits or improvements in justice system operations. For example, a program designed to enhance the investigation of drug trafficking cases so that they are more likely to lead to convictions could reasonably be assessed according to whether conviction rates actually increased. This result is assumed to be of value in and of itself, without demonstrating that improved conviction rates of drug traffickers have a positive effect on the supply or use of drugs. Consequently, the evaluation would focus on documenting and explaining these results, without attempting to measure more downstream effects.

If evaluating the results or outcomes of a project has high priority, certain conditions must exist to support conclusions. First, as mentioned above, the operation of the program must have achieved some measure of stability and successful implementation. Some evaluations have documented that projects have not produced the effects desired, without first establishing that the project in fact operated as planned. In such a circumstance, it is difficult to determine if the successfully implemented project failed to achieve its objectives or if the project was so inadequately implemented that any real test of its underlying concept was unfeasible. Or, if program activities change rapidly - as they often do in newly established projects - the project's effects may vary widely, making it difficult to attribute particular results to particular features of the project.

Second, outcomes or results must be susceptible to measurement. For example, evaluating whether an enforcement program succeeds at deterring would-be drug traffickers from entering the illegal market imposes difficult problems of measurement.

Third, the critical features of the program that are thought to produce the desired outcomes or results must be able to be distinguished from other possible causes. Evaluating whether or not a drug treatment program for probationers reduces drug abuse and criminal recidivism is difficult if all program participants are also placed under intensive probation supervision and are subjected to random urine testing.

Faced with less-than-optimal conditions, evaluators may develop substitute measures or rely on complicated statistical methodologies to attempt to distinguish among various simultaneous causes. This may be accomplished after the fact - which is to say, after a program has been in operation for some time - but at some cost in the ability to develop supportable conclusions. To increase the odds of the project's being able to support an evaluation of its outcomes, evaluators involved in the program planning stage will be able to propose modifications in organizational operation that will better prepare the project for subsequent evaluation.