National Institute of Justice. Evaluating Drug Control and System Improvement Projects, Guidelines for Projects Supported by the Bureau of Justice Assistance. Washington, DC: Prepared for the U.S. Department of Justice, National Institute of Justice by Abt Associates, Inc.; 1989 (Reprint 1992). pp.15-16.
Chapter 5: Using Evaluation Findings for Decision Making
Consider two truisms: "Evaluation activities should meet the information needs of decision makers who fund them," and "The purpose of evaluations is to provide feedback to decision makers about program operations and their effectiveness so that their decisions can be as fully informed as possible." Most experienced administrators and evaluators however, know that this often does not happen. Evaluations may be undertaken because they are required, and the reports are subsequently shelved with little comment or effect. This may occur for several reasons, including:
failing to address directly the policy makers' or program administrators' principal questions;
not communicating the results and nature of the study in a way that can be readily understood by a client;
framing the presentation of the study and its findings without a clear understanding of who the primary and secondary audiences are;
not meshing the conclusion of the study with the schedule upon which policy or programmatic decisions are made;
developing findings that are perceived as challenging by the stakeholder and are therefore resisted.
Several strategies can be employed to overcome these obstacles.
Increasing Relevance for Decision Makers
Evaluations may not fit well with decision makers' interests, and therefore will be seen as irrelevant for at least two reasons. First, the questions that the evaluator chooses to focus the study upon do not correspond closely enough with the decision makers' principal concerns. Secondly, the evaluation's findings may fail to suggest clear and explicit recommendations for action.
The first problem can be overcome by communication throughout the study between the decision maker/client and the evaluator. In the design phase of the study, the concerns expressed by decision makers can be translated with the evaluator's assistance into questions that are capable of being addressed by the evaluation. The decision maker may also become more aware of the implications of various choices of study design for the conclusions that may ultimately be drawn from the evaluation. Many aspects of the evaluation design reflect choices and trade-offs where there is no single correct answer. Through discussions with the evaluator, decision makers will understand better these choices and be able to assess the advantages and disadvantages of various options, as well as the consequences of choices for the study's results.
The frequent absence of clear recommendations for action in the evaluation springs from deeper tensions (Moore, 1983). Scientific studies -- as well as evaluation that use scientific methods -- aim to develop accurate descriptions of the observable world and powerful explanations of why phenomena occur. Translating these descriptions and explanations into prescriptions for action is not a scientific activity. Nor do most descriptions and explanations directly employ specific policy decisions. At best, they can estimate with varying degrees of certainty the likely consequences of alternative courses of action. Weighing the desirability of such options, however, involves making value judgments and calculating the optimal trade-offs between costs and benefits, advantages and disadvantages. Most evaluations, even extremely rigorous and thorough ones, do not test the validity of all elements considered in these optimizing calculations. Nor are values derived from evidence or inferences about evidence. Values come from outside the realm of science.
Whether the evaluation is to include recommendations for action (beyond undertaking further study) should be discussed among evaluators and decision makers. Decision makers may feel that they are better able to make the translation from findings to action themselves (Lipton, 1984,155). If the evaluator is to stop at presenting findings, without making recommendations, the findings should be present in ways that facilitate the decision makers' developing conclusions on their own. If, on the other hand evaluators present recommendations for action, they should make explicit the values that they bring to their work and upon which they base their recommendations. If the evaluator's values are separate clearly from the descriptions, explanations, and other findings of the evaluation, the decision maker will be able to see how the recommendations were developed. Decision makers will also be able to bring different values or concerns to the findings and develop their own recommendations, which may differ from those of the evaluator. Maintaining a distinct boundary between the more strictly structured enterprise of evaluating criminal justice projects and the task of making recommendations about them will protect the integrity of both tasks, thereby maximizing the chances that decision makers will be able to use evaluation findings for different types of political and administrative decisions.
Explicating the Advantages and Disadvantages of Different Options
The practical concerns of policy makers involve choosing between alternatives. Learning that a project did or did not produce the desired effect is helpful, but clarifying both the alternatives and their relative costs and benefits, or advantages and disadvantages, is generally more useful.
Communicating the Findings in Understandable Terms
Policy makers are generally most interested in study results and recommendations. Discussions of study design, limitations of data, and methodological problems encountered are of less importance. To the extent that these matters affect the ability to draw conclusions from the data, they should not be ignored, but the evaluator should avoid long or technical discussions in the main text of the report. The readers should be alerted that the findings are qualified, but referred to an appendix for a fuller and, perhaps, more technical discussion of them (Majchrzak, 1984, 93; Lipton & Appel, 1984, 158-159). The tendency to write for other researchers alone should be discouraged.
Writing for the Intended Audience
Evaluations are often written for many different audiences, which may weaken the relevance of the study's findings for the primary audience. State administrators often have different interests in a project than other high-level policy makers, or project managers. For example, an evaluation that says that a project has not had its intended effects will be of little use to project managers, but of more interest to policy makers. Because it is difficult to be all things to all people in an evaluation, the evaluator should establish who is the primary audience and write to that audience's interests. This will direct the type of questions addressed, the way findings are discussed, as well as the depth with which they are examined (Weiss, 1972, 119).
Meshing the Evaluation's Schedule With Time Frames for Decision Making
The demands for policy or programmatic decisions often impose nearly impossible schedules upon evaluations, especially upon the more demanding evaluations that aim to estimate a project's effects. The frequent result is that the evaluation's findings come too late, after the point where the effective decision was made. Several strategies exist for ameliorating this mis-match of schedules. One is for the evaluator to report frequently to the principal client throughout the course of the evaluation. Unfortunately, what the decision maker often wants most are the final results of the evaluation. This poses a dilemma. The evaluator may resist the pressure to present tentative findings, but the opportunity to have input into the decision process may pass irretrievably. The alternative is to present the tentative findings, which may be little more than crude or rough estimates, knowing that these will be refined and perhaps even revised afterwards (Lipton, 1984, 158).
Preparing Clients For Challenging Findings
To avoid the appearance of a "surprise attack" by the evaluator on key stakeholders, it may be useful to involve these stakeholders in the study at various stages. Evaluators might even conduct mock analyses with them before the data are collected to prepare them for possible findings. At this early stage, standards of project performance can be established in coordination with project staff so that they know how their work is going to be measured. Also, surprises can be avoided by reporting findings on an interim basis. Quarterly reports or ongoing discussions of preliminary results may be used to keep key administrators apprized of the status of the evaluation. In discussing findings, alternative interpretations can be explored, which may contribute to a more balanced final report. Similarly, early trade-offs that are made regarding design, measurement and sampling should be discused when findings are reported so that the audience can make appropriate inferences from the results or, at least, can evaluate the inferences that the evaluator makes.
The outcome of such a strategy, however, can be to shift the project's operations during the course of the evaluation. Such midstream changes may affect the measurement of impact and the ability to identify cause and effect relationships. This is less a problem with evaluations of project implementation, but if the evaluation design requires constancy in program operations (as many impact evaluations do), the analysis may become confounded. Evaluators who work closely with program administrators during the course of the evaluation should anticipate analytic problems and devise a method for dealing with them. Positive changes which improve the program are a successful outcome of the evaluation process even if they complicate the analysis.