A Typology of Evaluation Levels

Office of Juvenile Justice and Delinquency Prevention. Evaluating Juvenile Justice Programs: A
Design Monograph for State Planners. Washington, DC: Prepared for the U.S. Department of
Justice, Office of Juvenile Justice and Delinquency Prevention by Community Research Associates, Inc.; 1989. pp. 23-30.

View entire document

A Typology of Evaluation Levels

This chapter has presented many concepts and ideas for the specialist/evaluator planning evaluation. To reinforce these issues this chapter offers a typology of evaluation levels using an actual juvenile justice program example-a Law Related Education program implemented and evaluated in Colorado. The multitude of issues and questions raised is not intended to confuse the reader, but to convey the notion that there are many options for evaluating, and many good reasons for making the decision to evaluate. Once the reader accepts the following:

That evaluation data either do exist, or can be developed.
That an evaluation effort need not be exhaustive to be useful, and
That any evaluation information, if logically and carefully developed, will provide decisionmakers with more information than they would have had otherwise, even if it dies not provide every single piece of useful information.

Then the decision to evaluate will be readily made. This section demonstrate even more clearly the various types of possible, and useful, evaluations.

Dimensions of the Typology

The typology of evaluation levels relies on the distinction drawn between monitoring, process, and outcome evaluation, and on the level of comparison you intend to achieve, or are able to achieve, given data and resource limitations. Generally, evaluations will be program, or program component, specific, or they will make use of comparisons with other programs or client groups. Combining three levels of evaluation with two general comparative perspectives reveals six evaluation types, as Exhibit C illustrates. As this typology unfolds below, it will become clear that each of these evaluation types has a useful purpose. Choosing one or the other is not a right versus wrong issue. In some instances the resources available and the demand for information will dictate that no more than a basic evaluation should be attempted. In other instances, a basic or comparative process evaluation will satisfy decisionmakers. It is the rare evaluation effort that either can support or, if the financial resources are available can achieve a true comparative outcome evaluation. Comparative outcome evaluations should be used, however, for the most critical, long-term problems faced by juvenile justice, and for the most promising strategies for addressing those problems.

It is important that the range of possibilities be given serious consideration as you make evaluation plans.

The Law Related Education Program

This program will be used to explain the evaluation typology presented above. It is important to understand that this example-a Law Related Education program-was actually implemented. The evaluation type employed was a comparative outcome evaluation. In this section, five hypothetical evaluations are described to define the other components in the typology, and the comparative outcome evaluation is described from the information produced by the program.

With any evaluation effort it is important to understand a program's goals, objectives, and operations before selecting an evaluation approach and methods. These are reviewed here for the Law Related Education (LRE) program.

LRE Goal:

Provide instruction to students to build a conceptual and practical understanding of law, enforcement, and judicial processes, leading to improved citizenship skills, a desire to work within the legal system to settle grievances and deal with criminal problems, an understanding of the basis for rules and favorable attitudes towards enforcement and justice.

LRE Objective:

Provide 30 to 40 semester hours of LRE to school age children (middle to junior high school age) in seven schools.

LRE Procedures:

Each of the seven schools selected a teaching team that was trained in the LRE curriculum.
Throughout a semester the team taught law related topics including mock judicial procedures, and utilized legal and law enforcement professionals in class exercises, visits to courts, rides in patrol cars, and home security audits.

LRE Rationale:

The educational activities offered as the LRE program are expected to increase understanding of law enforcement, and judicial processes because the standard school curricula do not cover such topics.

Exhibit C

AN EVALUATION TYPOLOGY COMBINING LEVELS OF EVALUATION
AND COMPARATIVE PERSPECTIVES

Comparative Perspective

LEVEL OF EVALUATION	Program Only	Program and Comparison
Monitoring	Basic Monitoring	Comparative Monitoring
Process Evaluation	Basic Process Evaluation	Comparative Process Evaluation
Outcome Evaluation	Basic Outcome Evaluation	Comparative Outcome Evaluation

The new educational material will challenge perceptions of these phenomena among school children the are based on television portrayal and popular perceptions among peers. If it is carefully and thoughtfully presented, the LRE curriculum will change these conceptions among the students and foster respect for law abiding behavior.

Basic Monitoring

A basic monitoring evaluation is concerned with answering simple questions about program activities and rationale. A good way to approach the problem is to ask

Monitoring is the process of developing and analyzing data to count and/or identify specific program activities and operations.

the following question: "Who is doing what, when, where, and how often and with what resources?" For the LRE program the following answers are typical:

Teachers are providing instruction to students, the targets of their activities, with the aid of law enforcement and judicial professionals.
The two basic activities are teaching and learning.
These activities take place at certain times over a school semester, typically one or two classes a week.
These activities take place in the school setting, and at various other outside locations, such as courtrooms and police stations.

Answering the "How often" question might entail collecting data in the following areas:

The number of students receiving the instruction:

-- total number of students taught, and subtotals for various student type-age, sex, ethnicity, family characteristics,

-- number of students per class, or average number if there is variation, or absences per class.

Teacher data, including the total number and types of teachers involved.
Teacher/student ratio.
Hours of instruction offered, by different teachers, and by course component, e.g., lecturing versus field trips, or school teacher versus other professional.
Adherence to, or deviation from, the established teaching schedule.
Program cost information, including salaries, travel, supplies, special materials, and the like.
Other descriptive program information, such as course plans and outlines, learning objectives, and teacher training.

Basic monitoring information for a program has the following utilities:

It provides operating and descriptive information for funders and other overseers of the program, for interested outsiders (students, parents, other schools), and most importantly for program staff and administrators. It answers such questions as "What does the LRE program do?"
It provides the basic information necessary for comparing achievements to stated or desired objectives. If the goal to provide 40 hours of instruction in a school semester was missed by 5 hours, the answer may be found in the scheduling or other descriptive types of information. Such an inquiry may lead to a refinement of the goal, or to scheduling adjustments to avoid the problem in the future. The 5-hour shortfall, a piece of basic monitoring information, may highlight a significant operational problem and lead to its solution.
It provides educational information for all involved in the program. Evaluations often will be undertaken for new programs, and the program participants or administrators will learn more about the process and impact of the program if monitoring data are available than they would in the absence of such data. They are usually glad to have the opportunity to review them, even if few surprises result.
Basic monitoring data serve a preventive maintenance function, by tracking indicators of critical elements which, if they deviate too much from the expected norm, signal a program problem. For example, if absentee or student turnover rates reach a high level some immediate action may be required to stabilize the program.

Most program leaders or administrators collect this basic information, or at least some subset of it. Collecting the data, if it is done carefully and reviewed periodically, is evaluation of a basic sort. Comparisons are made between expectations and observed results, or at least data are relied on the set expectations. This is evaluation activity.

It is, of course, not sufficient to simply collect and analyze program data to complete a successful monitoring program. The data and findings should be integrated into the decisionmaking process at the program and/or higher levels.

Comparative Monitoring

In a comparative evaluation effort, basic monitoring data are collected for other similar programs, or for subjects in control groups which do not receive the intervention but are monitored for comparison purposes. The LRE program used both types of comparison in its evaluation. LRE programs were implemented and monitored in seven different schools, and in five of the seven control groups, consisting of students randomly assigned to traditional civics or social science classes, were monitored. In this manner, comparisons were made for schools that implemented an LRE program under slightly different circumstances, reflecting slight variations across schools in student ages and racial mixtures, and also within schools that did and did not receive the LRE program.

The value of comparative monitoring information lies in the comparative perspective it provides. With basic monitoring information, the only comparison possible is internal to the program, i.e., comparison with program goals and objectives. Comparative monitoring allows such comparisons, but also allows comparing performance with other programs. For example, if one program attains 90% of its planned instruction hours, while another programs achieve more (95%-100%) or less (70/5-85%), more had been learned than a simple internal check against program objectives.

More important, however, is the confidence in interpreting findings that the comparative approach provides. When describing and evaluating programs, especially when short or long term outcomes are discussed, it is valuable to consider alternative possible explanations of your findings. It is important to know whether an increase in clients, a change in client behavior, or a response to a program initiative is really the product of the program under study or some other factor, such as increased arrests, client education level, or outside influences. The comparative perspective brings more information to the analyst, and allows control or analysis of factors outside the program. In this manner it increases the ability to distinguish program effects from other influences, and thus gives the evaluator more confidence. Chapter Four addresses this issue in more detail under the section "Threats to Validity."

Basic Process Evaluation

If the planned outcome for the LRE program consists of certain attitudes and behaviors among the students, or changes in attitudes and behaviors, then there must be some process by which the program activities, as measured by monitoring, produce the expected outcome. By considering the program activities in combination or in some sequence, and by considering the mechanism by which the activities produce the result, you enter the realm of process evaluation.

For example, qualitative and quantitative measures of student-teacher relations or interactions might provide a process measure that is more predictive or program success than simply counting the number of hours spent in class. The effect of time on the program, as indicated by turnover or absenteeism, is another valuable process measure. Qualitative assessments of the program's link with other school activities,

Process evaluation involves developing and analyzing data to assess program processes and procedures, esp., determining the connections between various program activities.

may also prove valuable. For example, did the field trips interfere with other classes or extra-curricular activities, or of the program's acceptance by the school administration. Test and quiz grades may also be good interim measures of program success.

It is these kinds of information that help explain how the various program activities operate together. They produce short-term outcome measures which, if positive, can be expected to produce positive results in the long run as well. Basic process evaluation is valuable for other reasons, including:

It enables a better understanding of monitoring issues. In examining process, you achieve a better understanding of the formal and informal processes that operate within a program.
Process information provides important feedback that may allow interim adjustments before a program goes too far in the wrong direction.
For programs that do not undergo long-term outcome evaluations, process evaluations provide assessments of short-term performance.

Comparative process Evaluation

Comparative process evaluation employs the same measurements and assessments used in basic process evaluation, but for comparable or control programs. When comparisons are used for process evaluation, the benefit for evaluators is even greater. Principally, the comparative perspective at the process level provides much more confidence in the findings because the number of cases (programs, or students within programs) increases, and because information from different programs introduces different perspectives and controls into the evaluation.

Consider the LRE program. Process evaluations were conducted in six of the seven schools (one school received minimal administrative support). The LRE programs varied in their student types-the grade ranged from junior high to middle school, and one was a multi-level grade school; one was more racially mixed than the other. The schools also varied in their implementations of the LRE program. This produced variations in quantitative and qualitative information that broadened the entire project's understanding of LRE and its potential, more than a single case study would have done.

Additionally, comparative process measures provide relative measures; that is short-term performance measures that can be compared with measures from other programs. Relative measures permit comparisons of marginal performance differences across programs, and also allow evaluators to address a variety of policy-related questions-What if we taught more hours? What if there was a diversity of students in the class? What if fewer field trips were taken? These questions can be answered with more confidence when enough cases are present to produce variation in the variables of interest.

Basic Outcome Evaluation

With basic outcome evaluation, the logical sequence from program activities, to program processes, to program outcomes is made for a single program. Such analysis cannot be attempted without the antecedent monitoring and process evaluation that outcome evaluation suggests. In the case of the LRE program, two outcomes measures were chosen, and they were taken before and after the implementation of the program. The measures were:

(1) Student scores on scales measuring correlates of law abiding behavior.

(2) A self-report survey on various criminal and delinquent behaviors.

For this evaluation within a single school, program success is defined as whether, or to what extent, student attitudes and behavior changed in the expected directions, as determined by comparisons of time series data. Do the scores and other

Outcome evaluation involves developing and analyzing data to assess program impact and effectiveness.

indicators change in expected directions following (and perhaps during) the LRE program? Are these short-term outcomes, measured soon after classes ended, predictive of longer-term behavior, which might be measured by follow up studies?

Other approaches to basic outcome evaluation might include comparisons within a program. Such activities might involve comparing LRE curriculum to other curricula, or comparing different LRE curricula, predicting "high risk" students at the outset of the program and focusing follow-up efforts on them, studying early and intermediate indicators of successful outcome, or measuring outcome at various points during the program.

The value of a basic outcome evaluation such as this lies in providing the best information possible about program performance. In a single school, with this evaluation design, a finding that LRE students scored about the same or worse on post-program measures in comparison to pre-program measures would have hurt the overall program. If nothing else, the findings would have stimulated reconsideration of the program's goals and procedures. There would have been no information showing that it made a difference. In this case, though, differences were observed in the expected directions. Had the outcome findings been inconclusive the evaluators would turn to the process and monitoring data to explore the reasons, and probably would have found some helpful clues. The LRE program did just that and found other benefits such as favorable feedback from parents, and improvements in police officer handling of juveniles.

A basic outcome evaluation that uses comparisons within a program, as the LRE project did, allows the researcher and program administrator to address the question "Did the program make a difference?" While it doesn't explain what would happen if the program was not implemented, it provides information regarding program impact and program effects.

Comparative Outcome Evaluation

In a comparative outcome evaluation, long-term outcome measures are collected for more than one program, usually for the program under inquiry and a control group of programs, but they may be collected for multiple programs and control groups. This was the research design for the LRE program evaluation. The outcome measures described above were collected for LRE students and control student groups, before and after the program was implemented in five different schools. With the exception of collecting even longer-term measures, e.g., follow-up examination of attitudes and criminal behaviors after one or more years, replicating the research design for one program over multiple programs provides valuable evaluation information. This is especially true when comprehensive monitoring and process evaluation data have been collected in the course of program implementation.

The benefits of comparative outcome evaluation often include all of the benefits of lower levels of evaluation since those kinds of information are necessary to support it. The benefits of comparative outcome evaluation also include:

Increased confidence in the evaluation findings regarding any single program, due to the benefit of controlled program comparisons.
A better understanding of the possible range and varieties of outcomes that an LRE program might produce. This is especially helpful for those interested in replicating the program.

Comparative outcome evaluations, then, deserve the highest degree of confidence, especially if a pre/post comparison and a comparison with other controls is employed. Often it will not be possible to design and implement a thorough comparative outcome evaluation. Pre/post only designs, or comparison with controls only, will provide evaluators with good information regarding program performance.

Summary

This chapter has reviewed various approaches to evaluation. Before actual research planning takes place, a number of issues must be considered to help focus the evaluation and to prepare the research design. Good program candidates for evaluation should be identified, since all programs can be evaluated but available resources will not allow it. The program issues of accessibility, length of operation, history, expense, nature of controversy surrounding it, and external pressures to evaluate should be considered in making the selection.

If you give careful consideration to these and other issues you will usually find that (1) evaluation is not as difficult or esoteric as it seems, (2) you have been doing it in some fashion already and may as well take credit for the good work. (3) that there may be a broader audience for the information being produced, especially if the manner and format of presentation are adjusted a bit, and (4) providing objective data about juvenile justice programs will be appreciated by many in the field.

Now, having decided that evaluation can be accomplished, you face decisions about how to conduct them. The next chapter will review the basics of evaluation design and other relevant research issues.