Bureau of Justice Assistance - Center for Program Evaluation and Performance Measurement

Bibliography

Descriptions of Types of Impact Research Designs

ONE-SHOT CASE STUDY
The one-shot case study involves the measurement of an identified "outcome" after a treatment or program has been implemented. However, there are no measures taken or available for comparison (i.e., status before the program, or outcome of a comparison or control group). Without a comparison measure, there is no means for inferring that the "outcome" was actually influenced by the treatment or program.

ELEMENTARY QUASI-EXPERIMENTAL DESIGNS
BEFORE-AFTER DESIGN
The elementary quasi-experimental design known as the before-after design involves the measurement of "outcome" indicators (e.g., arrest rates, attitudes) prior to implementation of the treatment, and subsequent re-measurement after implementation. Any change in the measure is attributed to the treatment. This design provides a significant improvement over the one-shot study because it measures change in the factor(s) to be impacted. However, this design suffers from threats of history - the possibility that some alternate factor (besides the treatment) has actually caused the change.

COMPARATIVE POST-TEST DESIGN
The elementary quasi-experimental design known as the comparative post-test design involves the measurement of outcomes for both the treatment group as well as a comparison group. However, unlike more sophisticated designs, selection of participants into the treatment and comparison groups is not done randomly. While such a design to some extent overcomes the issues of a one-shot study by allowing comparisons of success, this design is typically plagued by threats due to selection bias. That is, an alternate explanation for differences between group outcomes is that some alternate factor, which was related to the selection process, has actually caused the differences in outcomes.

QUASI-EXPERIMENTAL DESIGNS
COMPARATIVE CHANGE
The quasi-experimental design known as the comparative change design allows for the measurement of change in relevant outcome factors (using a pre- and post-test) and provides for comparison of this change between a treatment group and a non-random comparison group. Because comparison and treatment groups are not randomly selected, alternate explanations due to prior differences between groups continue to be a threat.

INTERRUPTED TIME SERIES
The interrupted time series design involves repeated measurement of an indicator (e.g., reported crime) over time, encompassing periods both prior to and after implementation of a program. The goal of such an analysis is to assess whether the treatment (or program) has "interrupted" or changed a pattern established prior to the program=s implementation. However, the impact of alternate historical events may threaten the interpretation of the findings.

COMPARATIVE TIME SERIES
The quasi-experimental design known as the comparative time series tracks some outcome of interest for periods before and after program implementation for both the treatment group as well as a non-randomly selected comparison group. Because comparison and treatment groups are not randomly selected, alternate explanations due to prior differences between groups continue to be a threat.

EXPERIMENTAL DESIGNS
RANDOMIZED-COMPARATIVE POST-TEST
In the experimental design known as the randomized comparative post-test design a treatment and control group are randomly selected for study. The treatment group is given the treatment, while the control group is not. Both groups are tested or measured after the treatment. The test results of the two groups are compared.

RANDOMIZED-COMPARATIVE CHANGE
In the experimental design known as the randomized comparative change design a treatment and control group are randomly selected for study. Both groups are administered a pre-test. The treatment group is given the treatment, while the control group is not. Both groups are tested or measured after the treatment. The test results of the two groups are compared. The pretest allows a check on the randomization process, and allows for control of any differences found.

Back to Impact Evaluations by Type of Research Design

| | |
| | |