Office of Juvenile Justice and Delinquency Prevention. Evaluating Juvenile Justice Programs: A
Design Monograph for State Planners. Washington, DC: Prepared for the U.S. Department of
Justice, Office of Juvenile Justice and Delinquency Prevention by Community Research Associates, Inc.; 1989.  pp. 27-28.

View entire document

Basic Monitoring

A basic monitoring evaluation is concerned with answering simple questions about program activities and rationale. A good way to approach the problem is to ask

Monitoring is the process of developing and analyzing data to count and/or identify specific program activities and operations.

the following question: "Who is doing what, when, where, and how often and with what resources?" For the LRE program the following answers are typical:

• Teachers are providing instruction to students, the targets of their activities, with the aid of law enforcement and judicial professionals.
• The two basic activities are teaching and learning.
• These activities take place at certain times over a school semester, typically one or two classes a week.
• These activities take place in the school setting, and at various other outside locations, such as courtrooms and police stations.

Answering the "How often" question might entail collecting data in the following areas:

• The number of students receiving the instruction:

             --    total number of students taught, and subtotals for various student type-age, sex, ethnicity, family characteristics,

             --    number of students per class, or average number if there is variation, or absences per class.

• Teacher data, including the total number and types of teachers involved.
• Teacher/student ratio.
• Hours of instruction offered, by different teachers, and by course component, e.g., lecturing versus field trips, or school teacher versus other professional.
• Adherence to, or deviation from, the established teaching schedule.
• Program cost information, including salaries, travel, supplies, special materials, and the like.
• Other descriptive program information, such as course plans and outlines, learning objectives, and teacher training.

Basic monitoring information for a program has the following utilities:

• It provides operating and descriptive information for funders and other overseers of the program, for interested outsiders (students, parents, other schools), and most importantly for program staff and administrators. It answers such questions as "What does the LRE program do?"
• It provides the basic information necessary for comparing achievements to stated or desired objectives. If the goal to provide 40 hours of instruction in a school semester was missed by 5 hours, the answer may be found in the scheduling or other descriptive types of information. Such an inquiry may lead to a refinement of the goal, or to scheduling adjustments to avoid the problem in the future. The 5-hour shortfall, a piece of basic monitoring information, may highlight a significant operational problem and lead to its solution.
• It provides educational information for all involved in the program. Evaluations often will be undertaken for new programs, and the program participants or administrators will learn more about the process and impact of the program if monitoring data are available than they would in the absence of such data. They are usually glad to have the opportunity to review them, even if few surprises result.
• Basic monitoring data serve a preventive maintenance function, by tracking indicators of critical elements which, if they deviate too much from the expected norm, signal a program problem. For example, if absentee or student turnover rates reach a high level some immediate action may be required to stabilize the program.

Most program leaders or administrators collect this basic information, or at least some subset of it. Collecting the data, if it is done carefully and reviewed periodically, is evaluation of a basic sort. Comparisons are made between expectations and observed results, or at least data are relied on the set expectations. This is evaluation activity.

It is, of course, not sufficient to simply collect and analyze program data to complete a successful monitoring program. The data and findings should be integrated into the decisionmaking process at the program and/or higher levels.

Comparative Monitoring

In a comparative evaluation effort, basic monitoring data are collected for other similar programs, or for subjects in control groups which do not receive the intervention but are monitored for comparison purposes. The LRE program used both types of comparison in its evaluation. LRE programs were implemented and monitored in seven different schools, and in five of the seven control groups, consisting of students randomly assigned to traditional civics or social science classes, were monitored. In this manner, comparisons were made for schools that implemented an LRE program under slightly different circumstances, reflecting slight variations across schools in student ages and racial mixtures, and also within schools that did and did not receive the LRE program.

The value of comparative monitoring information lies in the comparative perspective it provides. With basic monitoring information, the only comparison possible is internal to the program, i.e., comparison with program goals and objectives. Comparative monitoring allows such comparisons, but also allows comparing performance with other programs. For example, if one program attains 90% of its planned instruction hours, while another programs achieve more (95%-100%) or less (70/5-85%), more had been learned than a simple internal check against program objectives.

More important, however, is the confidence in interpreting findings that the comparative approach provides. When describing and evaluating programs, especially when short or long term outcomes are discussed, it is valuable to consider alternative possible explanations of your findings. It is important to know whether an increase in clients, a change in client behavior, or a response to a program initiative is really the product of the program under study or some other factor, such as increased arrests, client education level, or outside influences. The comparative perspective brings more information to the analyst, and allows control or analysis of factors outside the program. In this manner it increases the ability to distinguish program effects from other influences, and thus gives the evaluator more confidence. Chapter Four addresses this issue in more detail under the section "Threats to Validity."