Office of Juvenile Justice and Delinquency Prevention. Evaluating Juvenile Justice Programs: A
Design Monograph for State Planners. Washington, DC: Prepared for the U.S. Department of
Justice, Office of Juvenile Justice and Delinquency Prevention by Community Research Associates, Inc.; 1989.  pp.47-49.

View entire document

Using Evaluations: Audiences and Products.

Evaluation projects, and their products are important elements in the overall policy analysis and feedback process. New juvenile justice programs, or continuations of those already established, represent policy action and choices, as does the decision not to create new programs. If evaluations of the kind described in this monograph are not undertaken, policy analysis and feedback will still take place. Program funders, backers, critics, and beneficiaries will receive information about the program in various forms, mostly informal, anecdotal, and subjective, and will make decisions about the program that will affect its future. With evaluations a certain, though sometimes small, amount of objectivity is injected into the policy analysis process, and program decisions begin to made in light of this more scientific evidence. This is the primary function of evaluation

Sometimes, even though the value of evaluation for policy analysis is recognized its potential is not fully realized. This occurs, for example, when the multiple users or consumers of evaluation information are not considered and the evaluation products do not reach them. Often, the evaluation product is one of more reports read by the funders of the evaluation effort and/or the funders of the program being evaluated, and the policy analysis stops there. But there may be other parties with a need and desire to know evaluation results, and it may be in your interest to inform them.

Others who have interest in evaluation information include:

• Program clients or potential clients. Reading the results, if presented from a lay person's perspective, may improve their self image or their image of the program which would be a positive development.
• Other criminal justice professionals. More often than not the program you evaluate exists in some form somewhere else, or a very similar one exists, and the people associated with that program would surely read your findings with great interest. It is likely that similar methods are being tried with adults in other places, indicating an even greater audience for your findings.
• Potential supporters of the program. Thoughtful portrayal of evaluation findings can help build community support and consensus that is needed for program survival. The public education function of evaluation information should not be overlooked.

With a little thought at the front end of an evaluation effort, audiences such as these can be identified for your evaluation results, doing so helps identify the type of evaluation to be conducted, the specific types in information to collect, and the appropriate products that should be developed.

It is the extreme case in which evaluation data stand on their own; they are almost always presented (though not tabulated) with their recipients in mind.

For example, assume that an 18-month evaluation of a jail removal project has been completed and the authors are preparing to present their findings to the funders. More often than not t least two reports of the findings will be prepared. One is an "Executive Summary" or some other non-technical review of the findings that is brief, to the point, and geared towards the political and administrative issues at hand. The other is a full, or technical report that explains in detail the study design, methods, limitations, and findings. Both reports are valid, reflecting the awareness that there are different audiences, or consumers, of the evaluation information.

In another example, a new program may be viewed with skepticism by school or law enforcement officials, especially if it utilizes an unfamiliar method such as play therapy or psychodrama. One product should be a detailed description of the program itself, with the law enforcement and school official in mind. Once these officials understand the program they will be less skeptical. In the meantime, a good monitoring or process evaluation effort can be accomplished that meets other goals as well. The program description might be excerpted from the final report and published in a tasteful manner.

In a final example, a comparative evaluation effort, might produce some important products for potential clients, above and beyond the final report. A video outlining the pros and cons of the program, or that compares responses and opinions from juveniles in experimental and control programs, might be produced. This would be a great value for potential funders and clients.

When the three type of outcomes are considered-knowledge production, consensus building, and instrumental-a variety of possible audiences and products are suggested, depending on the nature of the evaluation. For example:

• Program descriptions. The evaluation may produce the first and only detailed description of program goals, objectives, and operations. This is a valuable product in itself, and will usually have a receptive audience.
• Process feedback. Outside of the published evaluation findings, there may be other information uncovered during the course of the study that is not appropriate for a final product, but that has value for program staff and funders. Most evaluators will provide an informal debriefing to cover such matters. To those people this information may be as important, if not more so, than the published findings.
• Good news. In addition to presenting favorable findings to funders, evaluation products can be geared toward interested outside audiences whose concern is not with the specific program, but with perhaps replicating it or portions of it, or selling their constituencies on the idea of implementing similar efforts. If handled appropriately, evaluation information can provide useful public or marketing information.
• Bad news. Most evaluation will uncover potential "bad news," that is, information that suggests there are problems with the program under investigation, or that findings will be contrary to the expectations of program funders or managers. Evaluators should be comfortable in presenting this evaluation information too. Generally, three rules should guide the presentation of potential bad news:

(1)  Present the information as soon as you can, but only when you are confident that your information and sources are correct; don't present hunches or preliminary indications as confirmation of program problems. You may find after further inquiry that your initial supposition was incorrect.

(2)  Present the information to an audience limited to program managers or administrators (and perhaps to project funders, depending on how closely they are involved in your work). At the start of an evaluation project, the appropriate individuals who will receive preliminary information and briefings should be identified, and they should be the only persons to receive potential bad news. Then they can make the decisions regarding how to use the information you provide.

(3)  Present the potential bad news in as positive a manner as possible. Program managers should be glad to receive information that helps them solve problems, or that directs their decisions regarding program modification. If your potential bad news comes early enough, the y may be in a position to make corrections, lessen the impact , and improve the program. You will often find that your information will be well received, and perhaps that the news is not perceived by the recipients to be as bad as you thought it might be.

As evaluation audiences and products are important to consider in using evaluations, so too is product dissemination. It is not just what you produce and who you produce it for, but how you get it to the intended audience that can be the difference in making your evaluation useful.

If the intended target does not normally read long evaluation reports, and is one that you have not traditionally targeted for you product, then dissemination strategy is critical. In some instances this amounts to a marketing and product presentation strategy.

If the intended audience would benefit most from a brief and direct presentation-of program operations, evaluation findings, preliminary findings-then a road show might be warranted. This could include slides, videos, presentations by former clients, posters and/or product demonstrations, and might prove more effective in delivering findings than a written report. Such a product requires careful consideration of resources and priorities.

In another instance, the final evaluation report might be delivered to a limited audience, say 10 to 20 funders, board members, legislative committee members, but a brief monograph or pamphlet describing the program and the most pertinent findings might go to a broader audience of criminal justice professionals or community members. Again, this may do more good for the program than the detailed report.

Program funders, managers, and evaluators can be creative and broad-minded in their consideration of these issues, and they can influence evaluation products to meet many needs. The most effective way to do this is to specify the products and audiences in program and evaluation requests for proposals (RFPs). These are discussed in Chapter Five.