Office of Juvenile Justice and Delinquency Prevention. Evaluating Juvenile Justice
Programs: A
Design Monograph for State Planners. Washington, DC: Prepared for the U.S. Department of
Justice, Office of Juvenile Justice and Delinquency Prevention by Community Research
Associates, Inc.; 1989. pp.28-29.
Basic Outcome Evaluation
With basic outcome evaluation, the logical sequence from program activities, to program processes, to program outcomes is made for a single program. Such analysis cannot be attempted without the antecedent monitoring and process evaluation that outcome evaluation suggests. In the case of the LRE program, two outcomes measures were chosen, and they were taken before and after the implementation of the program. The measures were:
(1) Student scores on scales measuring correlates of law abiding behavior.
(2) A self-report survey on various criminal and delinquent behaviors.
For this evaluation within a single school, program success is defined as whether, or to what extent, student attitudes and behavior changed in the expected directions, as determined by comparisons of time series data. Do the scores and other
Outcome evaluation involves developing and analyzing data to assess program impact and effectiveness.
indicators change in expected directions following (and perhaps during) the LRE program? Are these short-term outcomes, measured soon after classes ended, predictive of longer-term behavior, which might be measured by follow up studies?
Other approaches to basic outcome evaluation might include comparisons within a program. Such activities might involve comparing LRE curriculum to other curricula, or comparing different LRE curricula, predicting "high risk" students at the outset of the program and focusing follow-up efforts on them, studying early and intermediate indicators of successful outcome, or measuring outcome at various points during the program.
The value of a basic outcome evaluation such as this lies in providing the best information possible about program performance. In a single school, with this evaluation design, a finding that LRE students scored about the same or worse on post-program measures in comparison to pre-program measures would have hurt the overall program. If nothing else, the findings would have stimulated reconsideration of the program's goals and procedures. There would have been no information showing that it made a difference. In this case, though, differences were observed in the expected directions. Had the outcome findings been inconclusive the evaluators would turn to the process and monitoring data to explore the reasons, and probably would have found some helpful clues. The LRE program did just that and found other benefits such as favorable feedback from parents, and improvements in police officer handling of juveniles.
A basic outcome evaluation that uses comparisons within a program, as the LRE project did, allows the researcher and program administrator to address the question "Did the program make a difference?" While it doesn't explain what would happen if the program was not implemented, it provides information regarding program impact and program effects.
Comparative Outcome Evaluation
In a comparative outcome evaluation, long-term outcome measures are collected for more than one program, usually for the program under inquiry and a control group of programs, but they may be collected for multiple programs and control groups. This was the research design for the LRE program evaluation. The outcome measures described above were collected for LRE students and control student groups, before and after the program was implemented in five different schools. With the exception of collecting even longer-term measures, e.g., follow-up examination of attitudes and criminal behaviors after one or more years, replicating the research design for one program over multiple programs provides valuable evaluation information. This is especially true when comprehensive monitoring and process evaluation data have been collected in the course of program implementation.
The benefits of comparative outcome evaluation often include all of the benefits of lower levels of evaluation since those kinds of information are necessary to support it. The benefits of comparative outcome evaluation also include:
Comparative outcome evaluations, then, deserve the highest degree of confidence, especially if a pre/post comparison and a comparison with other controls is employed. Often it will not be possible to design and implement a thorough comparative outcome evaluation. Pre/post only designs, or comparison with controls only, will provide evaluators with good information regarding program performance.