National Institute of Justice. Learning from Demonstration Programs. Washington, DC:
Paper
prepared for the U.S. Department of Justice, National Institute of Justice by Abt
Associates Inc. pp.10-11.
IV. Install a Monitoring System to Provide Continuous Feedback.
Every new project goes through a "shake-down" period, while people adjust to new roles and duties, procedures are modified and refined, and expectations are tested against reality. Some problems at this point are the norm. The key is to identify and address them promptly and for that, a good monitoring system is invaluable.
A monitoring system is a system for collecting and organizing management information.
Installing a monitoring system entails designing formats for collecting data on the
internal operations of a program, and developing procedures for periodically tabulating
and analyzing those data. As a mechanism for generating statistics that describe what a
program is doing, the monitoring system is a key information source for impact evaluation.
Equally important, long before an impact evaluation has been completed, the monitoring
system provides on-going feedback on the attainment of service delivery or activity
objectives. As such, it serves as a form of early warning system, alerting program
managers to operational weaknesses that require corrective action. A variety of questions
regarding the status and productivity of program efforts can be addressed by a monitoring
system. Are treatment programs reaching the intended numbers and types of recipients? How
many of what kinds of arrests are enforcement actions producing? Have those treatment
services or enforcement actions been implemented at planned levels of effort? What is
happening to the arrests made or participants treated? As these questions imply, a
monitoring system will typically include three classes of case-level data:
(1) Information on the background characteristics and intake status of program targets
(socioeconomic and criminal characteristics of all cases treated or processed);
(2) Data characterizing the activities of the program (information tracking cases through a treatment regime or criminal process);
(3) A battery of outcome scores (follow-up data describing the relevant characteristics of all subjects or target areas at selected intervals after program intervention).
Notably, outcome scores should not be confused with impact measures. The fact that
subjects exposed to a particular treatment or surveillance strategy have produced clean
urine or remained arrest free is not necessarily an indication of long term behavioral
change. Outcome scores only describe the status of subjects at a particular moment in
time--without reference to what might have happened in the absence of program
intervention. Only an impact evaluation can correlate the status of subjects or target
areas with the activities of a demonstration program.
Organizing the case-level data collected by a monitoring system can be accomplished in any
number of ways depending on the data processing capabilities of the implementing agency.
In small programs, a simple paper filing system and periodic manual tabulation procedures
might be adequate. In a larger program, data might be converted for use on a personal
computer and all tabulations and cross-tabulations handled with standard statistical
software packages (such as SAS or SPSS). Alternatively, forms might be processed by state
or municipal computing facilities or shipped to outside processing firms for data entry
and analysis.
In addition to case-level data, a monitoring system should also maintain statistics
describing the human and financial resources allocated to a program. Staff
characteristics, deployment patterns, line item costs, and the amount of staff time
allocated to various demonstration activities are useful descriptors of a program's
resource needs. Indeed, a monitoring system that provides both case-level data and
information on staff allocation and related costs provides program managers with a
powerful diagnostic tool for identifying strong and weak links in the activities and
organization of a demonstration program