Implement a process evaluation to document what is done, when, by whom, to whom

National Institute of Justice. Learning from Demonstration Programs. Washington, DC: Paper
prepared for the U.S. Department of Justice, National Institute of Justice by Abt Associates Inc. p.12.

View entire document

V. Implement a process evaluation to document what is done, when, by whom, to whom.

Process evaluation adds a qualitative dimension to the descriptive statistics furnished by a monitoring system. From a monitoring system, one might learn, for instance, how many counseling sessions of what duration were received by participants in a drug treatment program; or how many new cases were opened for investigation by a law enforcement agency. A process evaluation would expand these indicators by providing information on particular modalities of treatment or tactics of investigation. In addition to furnishing richer data on some of the more quantitatively elusive aspects of program operations, a process assessment can also capture important information about a program's social, legal, and political context. Since local cultures play a key role in shaping the direction and results of new public policies, understanding the local environment is crucial to understanding how a program works.

Field observation, reviews of media reports and legislative hearings, and interviews of program personnel, host agency staff, key personnel in related agencies, and community and political leaders, are important sources of descriptive and historical data on the program and the environment in which it operates. These data serve a number of important purposes:

First and foremost, understanding exactly how a program operates (and how it may be influenced by various environmental factors) can be a great aid in developing analytic models appropriate to the task of investigating program impact.
Qualitative data are also essential for interpreting the results of a quantitative inquiry. It makes no sense, for instance, to declare that a program had no discernable effect on its target population or area, if observation has revealed that the activities described by the available statistics either failed to take place or were implemented in a qualitatively different form.
Finally, in cases where program impacts cannot be empirically measured, a process evaluation or case study may serve as a surrogate source of information about the nature of a particular reform. Although the information will necessarily focus more on "what happened" than whether particular goals were achieved, case material may nonetheless shed important light on some of the apparent strengths and limitations of the program process.