Institute for Law and Justice. Urban Street Gang Enforcement. Washington, DC: Prepared for the U.S. Department of Justice, Bureau of Justice Assistance; 1997. pp. 103-106.
Impact Evaluation
An impact evaluation measures the program's effects and the extent to which its goals were attained. Although evaluation designs may produce useful information about a program's effectiveness, some may produce more useful information than others. For example, designs that track effects over extended time periods (time series designs) are generally superior to those that simply compare periods before and after intervention (pre-post designs); comparison group designs are superior to those that lack any basis for comparison; and designs that use true control groups (experimental designs) have the greatest potential for producing authoritative results. At a minimum, gang suppression programs should use pre-post designs. Even better is the use of longer time series analyses and comparison or control group designs.
Dependent Variables
The major limiting factor on evaluations is the availability of baseline data on the
dependent variables of interest. Examples of dependent variables are gang-related crime,
citizens' fear of crime, and weapons offenses. Although data on arrests, reported crime,
and calls for service for prior time periods are available in most jurisdictions, it may
be impossible to determine from these data which events are gang related. Some
jurisdictions find it informative to examine pre- and post-data surrounding events that
frequently involve gang members (for example, all weapons offenses, including shots fired;
homicides; drug trafficking; and driveby shootings).
Less commonly found are data on other variables, such as fear of crime, levels of
disorder, or satisfaction with the police.
If no prior data exist for an important dependent variable, the department must gather the data prior to the start of a new program. This effort may involve producing special computer runs on selected types of incidents or geographic areas, conducting a special community survey, taking photographs that show gang influence in targeted neighborhoods, or hand tabulating data from existing records.
Comparison or Control Groups
Whenever possible, it is desirable to identify a comparison or control group. This may be a group of persons or an area in the community that does not receive the intervention but has characteristics similar to the group that does. Another possible comparison group is a similar, nearby community. The same pre- and post-intervention measurements should be made of the comparison group as of the group receiving the intervention.
Basic Evaluation Procedures
Regardless of the evaluation design, certain fundamental procedures should be followed, the most important of which are outlined in the eight steps that follow.
Carefully State the Hypothesized Effects. In the project planning phase, the department should carefully state its expectations of a gang enforcement strategy that is implemented correctly. There may be one anticipated effect or many, examples of which are:
In some programs, the effects may be expected to occur in stages: for example, the program will reduce street-level drug sales by gang members, which will reduce fear of crime. In this case, one stated effect is anticipated to result in another.
Identify Possible Unintended Effects. Most programs have potential associated risks that also should be identified early in the planning phase to allow the department an opportunity to address them better. The evaluation can assess whether the unintended effects occurred. Examples of unintended but anticipated effects are:
Define Measurement Criteria. The indicators that will be measured (for example, gang-related crime and community support) must be clearly defined to obtain consistent, reliable measurements. What is meant by gang related crime? Should a gang motive be present before crimes are so classified? Are measures of some crimes more important than others? How are satisfaction with the police, fear of crime, or community support defined? Terms should be clarified so the department and evaluators understand exactly what is to be measured and how.
Determine Appropriate Time Periods. Data collection can be costly, and information about results is important to people with investments in the program. For practical reasons, evaluators often must compromise ideal time periods. This step responds to issues such as how far back in time baseline data should be collected and how long the program should operate to give it a fair opportunity to show results.
Generally, the BJA Urban Street Gang Program demonstration sites obtained data on key dependent variables (for example, drug arrests, driveby shootings, and gang-related homicides) for a 5-year period before project commencement. This method helped identify trends and aberrations. For example, an unusual problem or a unique special operation could have resulted in an unusually high arrest rate in a given year.
The demonstration sites were expected to devote the first 3 months to the needs assessment and planning processes, with program activities occurring for the next 12 to 15 months. The impact evaluation should be based on data accumulated for at least 12 months of program activities. The demonstration sites were required to submit their evaluation reports in their 18th month.
Although not a steadfast rule, most jurisdictions should be able to make some assessments about program impact after 12 months of activities. However, the appropriate period may vary considerably, depending on the nature of the problem, the type and complexity of the response, and other factors.
Monitor Program Implementation. Systematic program monitoring is
required to aid the site in correcting and overcoming problems, to effect management
improvements that may reduce future implementation failures, and to interpret program
results correctly.
Collect Data Systematically. The data collected on program
implementation, hypothesized effects, and unintended effects must be as accurate as
possible. If more than one person collects data, each must follow the same rules and use
the same definitions. If data are collected over a long period, the same rules and
definitions must be used throughout.
Analyze Data. Data analysis should produce a description of the program as it was implemented. If the evaluation design is strong enough, analysis can go beyond describing what happened and provide convincing explanations of why it happened. The analysis should present the evidence that helps determine whether the program had its hypothesized effects and whether it resulted in any unintended effects.
In some situations, it may be useful to determine whether differences between target and control groups are statistically significant. Various statistical techniques may be used to determine whether observed differences are likely due to the program design or whether they occurred by chance.
For most jurisdictions, statistical significance will be less important than other considerations, such as if the program effect was large enough to make a substantial difference and if enough benefits were derived to justify program costs. These judgments must be made by the department, city or county administrators, and residents, and the evaluation results should assist in making informed decisions.
Replicate the Program. Replication of a program ensures that the
documented program effects were not a chance occurrence, the result of unobserved
intervening factors, or limited to one place or one time period. Once a program component
is found to have consistent effects in several applications, the strategy may be used
confidently with predictable results. This feature underscores the importance of
evaluations.