National Institute of Justice. Evaluating Drug Control and System Improvement Projects, Guidelines for Projects Supported by the Bureau of Justice Assistance. Washington, DC: Prepared for the U.S. Department of Justice, National Institute of Justice by Abt Associates, Inc.; 1989 (Reprint 1992). Pages 11-13
Chapter 3: Methods of Analyzing Data
The repertoire of analytic techniques that have been used in evaluation studies is large. It includes, among others, case studies, simple descriptive statistics, before/after comparisons, cohort studies, time-trend comparisons or other longitudinal methods (such as "panel analysis"), cross-sectional comparisons, cost-benefit analysis, controlled experiments and quasi-experimental statistical analyses, and methods drawn from the fields of operations research and systems analysis. What follows is a very brief (and incomplete) catalogue of some of these analytic approaches. Cursory descriptions of their key attributes, relative advantages and disadvantages are provided, along with some suggestions about what the evaluator and administrator might consider when choosing among them.
In general, these methods vary (1) in the emphasis they place on descriptive as opposed to explanatory analysis, (2) the extent to which they support strong conclusions about the project's operations and/or impacts, and (3) their reliance on qualitative compared to quantitative data.
Case Studies
A case study is an inquiry that investigates a contemporary phenomenon within its real-life context, when the boundaries between phenomenon and context are not clearly evident, and in which multiple sources of evidence are used (Yin, 1981). Case studies rely heavily on description but when undertaken for evaluative purposes, they may involve interpretation and analysis. Case studies typically make extensive use of qualitative data drawn from interviews and observation, although they may include quantitative data as well. They have traditionally been used in the sciences as an exploratory research strategy, undertaken for the purposes of learning how a particular phenomenon (such as a project or program) operates, as well as for developing hypotheses about it. Because the evaluator becomes steeped in richly detailed information in the course of undertaking a case study, a comprehensive understanding of a project and its complexities can be developed. Another advantage of case studies is that they be done quite quickly. Information can be collected on site through observation, interviews, examination of administrative records, or any other sources of information. A report may be written to organize information around key questions that are framed by the evaluator and the "client," either the state administrator or the program manager, or both.
The principal shortcomings of case studies as an evaluation method are that they demand competence and experience on the part of the evaluator. The analyst must evaluate information and those who provide it, distinguish significant features from insignificant ones, analyze data (impressionistic as well as more structured) quickly to discern patterns, devise ingenious ways of testing hypotheses against data, and find new data for these tests. The quality and usefulness of a case study therefore relies very heavily upon the good judgment and experience of the evaluator. The best case studies are sometimes done by those who are capable of working with more complex research designs.
Another problematic feature of the case study method is its limited ability to yield strong conclusions about whether, and to what extent, a project produces the effects that planners and managers intend for it to have. In some instances, success or failure is readily apparent. Case study methods may be sufficient to develop explanations of either these successes or failures. In instances where success of failure is more difficult to discern, however, and where the possible explanations are many, more controlled measurement and statistical analysis may be required to estimate the nature and size of a relationship between various observed effects and their causes. Some have argued however, that the case study method can be formalized so that it can be an effective tool for explanatory analysis. Evaluators choosing to undertake case studies are advised to consider various strategies for maximizing the reliability and validity of data collected and the procedures they use to interpret them (Yin, 1984, 1988).
Quantitative Descriptions
Evaluations may require documenting or measuring, in quantitative terms, the activities and the results of a project. While this can be done in a more quantitative case study, the analyst may chose to use relatively little qualitative data and focus instead on building a description from counts of activities, clients, staff hours, or other measures of what goes on in a project.
If a more complex description is required, evaluators may choose to develop models that describe the key components of the project, how they are related to one another, and measures that characterize these relations in quantitative terms. These may include the models such as those used in operations research (Tien, 1983). Their advantage is that they permit very precise specifications of how various elements of a project affect the operations of other elements in the project. If designed well, with accurate data, these models can be used to simulate alternative ways of organizing the project and to estimate the impact that these alternative may have.
Before/After Studies
When the evaluator is asked to assess the impact of a project, or to assess its effectiveness in accomplishing its goals, methods for testing cause-and-effect hypotheses are called for. One such method compares the target population or conditions before and after the project begins its operations. This "bargain basement" approach seeks to establish that participation in, or implementation of, the project is at least associated with the desired change. This design, termed a before/after comparison, requires obtaining data about the conditions that prevailed before the project intervention was initiated. (The analyst may find it possible to rely on data that another agency collected before the project's intervention occurred.) If the desired changes are shown to occur after the intervention, support is given to the assertion that the project caused the change to happen.
Confidence in the findings of such before/after comparisons depend, however, upon whether factors other than the project's interventions changed as well. Was the observed change really due to another force that operated independently of the project? Many of the conditions targeted by criminal justice projects are influenced by demographic, social, legal, and economic forces that operate independently of a project's intervention. Any increase or decrease in the observed outcomes may be affected by these outside factors and may therefore be unrelated to the project. To rule out these other possible explanations, the analyst must devise strategies for testing them. One such method is to collect data on these other possible causes and to impose statistical controls to isolate their effects from the project's operations. (See "quasi-experimental techniques," below.)
Yet another drawback of before/after studies is that estimates of the project's effect might be obscured by taking too few measurements. If the phenomena being measured are subject to random fluctuations, comparing only tow snapshots may not provide true pictures of the conditions before, during and after program intervention.
Time-Series Analyses
One method of compensating for fluctuating rates in the targeted conditions is to take several measures before and after the implementation of the project (or, if subjects are being studied, before and after they are exposed to the project). This research design, known as "time series analysis," first observes trends in the conditions existing before the project's intervention and then analyzes the trend data statistically. This trend can then be extrapolated into the future, to the point after which the project was implemented. By comparing what was projected to occur as a result of the pre-existing trends with what actually occurred, the analyst obtains some indication of what the project's impact may be. If outcome measures are fairly stable, minor random fluctuations or outside influences such as shifting demographics in the target populations may be accounted for in the trend projections. Time-series designs do not rule out all other possible explanations of the observed changes, however.
Experimental Designs
The investigative technique that provides the analyst maximum control, so that the relationship between a particular element of a project and the desired outcome can be isolated from other causal forces and measured accurately, is the laboratory experiment. If all factors are held constant (or "controlled"), and an effect is observed after one factor changes, one is in the strongest position to say that the manipulated factor caused the observed effect. Branches of science that have been able to impose laboratory conditions upon the matters they investigate have been able to develop powerful explanations of complex phenomenon.
Outside the laboratory, one can approximate laboratory conditions in field experiments (Campbell and Stanley, 1963). If one were in the laboratory studying the effects of a treatment regime, one might be able to control not only environment but also individual differences among subjects. (Laboratory mice, for example, are bred from a common genetic stock expressly for the purpose of experimentation.) In the field, however, such control over subjects and their environments cannot be gained. The strategy for approximating this control is to assign at random equally eligible subjects (cases/arrestees/addicts, or whatever) to two groups. The subjects in one group (the "experimental group") are then exposed to or given the "treatment" - be it a drug treatment project, enhanced prosecution, or whatever else the project is designed to do - while the other group (the "Control group") is not. Random assignment provides optimal assurance that any differences in the outcomes observed in the two groups can be attributed to the experimental treatment, and not to pre-existing differences or to chance.
While it may seem difficult to undertake experiments in criminal justice settings, several studies with experimental designs have been carried out with much success, yielding powerful findings (Lempert and Visher, 1987). Unfortunately, such studies are complicated, often vulnerable to a number of threats that may spoil the ability to draw strong conclusions, and generally very costly and time-consuming.
Quasi-Experimental Studies
Where random assignment of participants to treatment or control groups is not feasible for practical, ethical, or legal reasons, the evaluator may choose quasi-experimental evaluation designs to approximate the advantages of random selection. One such design is to identify a comparison group that is similar to the treatment group in those characteristics thought to be capable of influencing the outcome under examination (Campbell and Stanley, 1963). The strength of this design rests on the extent to which all the influential characteristics are accounted for in selecting the control group. The analyst can then account statistically for differences between groups that might influence the observed outcomes. The only requirements are that no differentiating characteristic belongs uniquely to one group, and that such competing factors be measured in both groups.
Because the use of a non-random comparison group does not eliminate all alternative explanations for the relationship between treatment and outcome, this type of design requires much more complicated analysis and yields less certain results that true experiments. Nonetheless, quasi-experimental designs can produce findings that are much stronger than other types of evaluation methods that impose fewer controls (e.g., case studies, before/after comparisons, descriptive models).
Some projects are not well suited to either experimental or quasi-experimental evaluation designs because their operations fluctuate too much due to their newness. Both evaluation designs require that treatment be constant and uniform throughout the time that the data are collected. If programs have not reached a state of relative stability in operations, the expense and time required of an experiment is likely to be wasted. In such instances, a focus on program implementation is likely to be more fruitful.