Bureau of Justice Assistance. Improving the Nation's Criminal Justice System:
Findings and Results From State and Local Program Evaluations. Effective Programs Monograph No. 1. December
1997. (NCJ 166821)
View entire document
Appendix A
Identifying Effective Criminal Justice Programs: Guidelines and Criteria for the
Nomination of Effective Programs
The Bureau of Justice Assistance (BJA) has created the Intensive Program Evaluation (IPE)
Initiative to respond to the Attorney General's charge to "find out what works and
spread the word." This new initiative establishes a mechanism to validate the
effectiveness of criminal justice programs based on published criteria, including
evaluation results, and to disseminate information about approved effective programs
through Department of Justice networks directly to practitioners. The approval of
effective programs for dissemination relies on the evaluations to be conducted by the
Program Effectiveness Review Panel. This Panel will provide independence and rigor in its
recommendations to the Director. In the development of the guidelines, BJA relied heavily
upon the Panel, as well as upon past activities of the National Institute of Justice, to
identify exemplary programs.
The program objectives are to:
• Enhance the ability of State and local agencies to generate and use evaluation results
for strategy development, program improvement, and effective program identification.
• Identify and document useful approaches to designing and conducting evaluations at State
and local levels.
BJA needs to identify effective State and local criminal justice programs, practices, and
products as part of broader efforts at the national level to improve the criminal justice
system by disseminating useful program information to policymakers and practitioners. It
is an approach that might be called "leading by example." Through this approach,
information on successful programs will be disseminated to the field in a credible and
timely fashion. The products from this initiative are the results from the 56 laboratories
(50 States and 6 territories) put in place under the Byrne Formula Grant Program.
Program effectiveness guidelines and criteria are presented here for use in submitting
potential programs, as well as for the Program Effectiveness Review Panel in its review of
nominated programs. The Panel reviews the programs and submits its recommendations to the
Director. Once the effective programs have been approved, BJA bulletins will be published.
I. Abstract
The abstract should be a 1-page description (200-300 words) of the program that provides a
concise statement of concrete, observable outcomes. The abstract should briefly describe
the following aspects of the program: (1) goals, (2) purposes and needs addressed, (3)
method of operation, (4) audience, and (5) expected result(s).
II. Basic Information
Basic information should be approximately one page.
A. Project Title
Location
Contact Person
Give the title of the project (including any acronym or abbreviation), the name of the
applicant agency, and the address and a daytime telephone number of a contact person
within the applicant agency.
B. Original Developer
Applicant Agency
Provide the name(s) and title(s) of those who originally developed the program. Describe
the mission of the applicant agency.
C. Years of Project
Provide (1) date(s) developed, (2) date(s) operated, and (3) date(s) evaluated.
D. Source(s) and Level(s) of Development and Dissemination Funding
List sources of funding for the project and amounts by year. Categories of sources include
Federal, State, local, and other.
III. Description of Program
Describe the program in approximately five to six pages.
A. Background, Foundation, and Theoretical Framework
Discuss briefly the history of how and why the program was developed. Present the
theoretical or empirical framework upon which the program is based.
B. Purposes and Needs Addressed/Problem Statement
Describe the specific needs the program was designed to address. Needs should be linked to
the target audience and special features of the program.
C. Goals
Provide a clear and concise statement of the program goals. Include only those goals that
relate directly to claims of effectiveness. In the case of evaluation models designed to
meet intermediate objectives, make the link to the ultimate purpose of the program.
D. Objectives
Objectives are the intermediate effects or results to be achieved by the program in
pursuing its ultimate goal. Objectives measure the extent to which program goals are being
accomplished. Identify appropriate objectives that logically flow from program goals.
Objectives should be stated in terms of outcomes (expected effects or results). A
distinction should be made between outputs (quantities produced) and effects/results.
E. Intended Audience
Identify the relevant demographic characteristics of the population for which the
objectives are designed.
F. Features: How the Program Operates
Provide a complete description of how the program actually operates, identifying all
features critical to its implementation. Include the following topics as they apply to the
project: (1) scope (Does the project supplement or replace an existing program, or is it a
component of a larger program?), (2) staff activities and staffing patterns, (3) staff
development activities, (4) management activities, and (5) monitoring and evaluation
procedures.
G. Significance of Program Design Compared with Design of Similar Programs
Describe the features of the program that distinguish it from similar programs. Discuss
ways in which the program addresses special problems. Note innovative or unique features.
IV. Potential for Replication
Describe the potential for replicating the program in two to three pages.
A. Settings and Participants (Development and Evaluation Sites)
Briefly describe the community(ies) where the intervention was developed or field tested.
Socioeconomic, ethnic, and geographic descriptions are appropriate.
B. Replicable Components and Documentation
Indicate which aspects of the program are appropriate for dissemination to other sites. If
the program has developed support materials for dissemination, indicate the type of
documentation available.
C. User Requirements
Describe the minimum requirements necessary for implementing the project at another site
(e.g., special staff, facilities, staff training time).
D. Costs (for Implementation and Operation)
Present a brief explanation of the recurring and nonrecurring costs associated with
adopting the project. Costs such as personnel costs, special equipment, and materials and
supplies that are necessary for installing and/or maintaining the program at an adopting
site should be discussed. Costs associated with the development of the original program
should be excluded from this discussion.
V. Evidence
Provide evidence in approximately six to eight pages.
A. Impact Statement(s)
The specific evaluation methods used to determine the impact of the program should be
explained.
Succinctly state the major accomplishments of the program and its activities. Include a
brief description of the type of evidence used to support the fact that an impact was
determined and a description of the nature of the change that was demonstrated.
Generally, an impact statement includes (1) the target group for which results are
available, (2) the nature of the change, (3) the process used for measuring the impact of
the program, and (4) the standards by which to judge whether the gains achieved are
significant.
A clear impact statement is critical because the panel judges the adequacy of evidence
based on the claim. Further, the statement identifies the project objectives/outcomes that
will be approved for dissemination (i.e., only those objectives/outcomes that are
reflected in the impact statement(s) and supported by convincing evidence will be
approved).
B. Description of Methodology
1. Design
An evaluation design usually addresses three factors: (1) the timing of data collection
(e.g., pretests and posttests or different points in a time series), (2) the groups
involved (e.g., a group receiving the program and a comparison group receiving an
alternative program), and (3) the way in which a standard of comparison will be determined
(e.g., a treatment group's gain or change will be compared with national or State
benchmarks).
Describe the type of design used for each claim and the reason for the choice. Address any
assumptions or problems inherent in the research design that was used.
2. Sample
The discussion of sampling procedures should answer four questions: (1) Who participated
in the study? (2) How was the sample selected? (3) How many participants were included in
the final sample? (4) How representative is the sample of the target population and
program participants as a whole?
3. Instruments and Procedures
This section should describe the instruments and/or procedures and how each assessment
technique relates to the outcomes. Provide sufficient information so that a judgment can
be made about the technical strength and appropriate use of the measure (e.g., validity,
reliability, levels, subscales).
It is especially important to describe validity and reliability procedures for
project-developed instruments; in such cases, the procedures for instrument development
and field testing also should be explained.
4. Data Collection
Describe the procedures used to select and train testers and the actual strategies used to
assure quality control during data collection. Indicate the periods of data collection,
the persons responsible for supervising the data collection, and the scoring and data
summary procedures. It is especially important to describe in detail the data collection
and quality control procedures for qualitative evaluations.
5. Data Analysis
If data are quantitative in nature, indicate the statistical technique(s) and levels of
significance used in the analysis.
If data are qualitative in nature, describe the procedures used to code and categorize or
reduce information for summary purposes. Describe ways in which linkages were made across
data elements to draw and verify conclusions.
C. Description of Results
Present detailed results of analyses in table or chart form, if appropriate. Sufficient
detail should be provided for the reader to check conclusions independently. Also,
summarize the results for the claims in narrative form, relating the specific outcomes to
the accomplishment of goals.
D. Summary of Supplementary Evidence
Provide additional evidence that supports the results, including anecdotal information,
perceptions of quality, and levels of satisfaction. Supplementary evidence can also be
evidence of generalizability.
E. Interpretation and Discussion of Results
1. Relationship Between Effect and Treatment
Summarize the results of all data related to the claim that the treatment was effective.
Link the results to specific features of the program design.
2. Control of Rival Hypotheses
Provide evidence of program attribution, i.e., evidence that suggests that the effects can
be attributed to the program and not to some other equally plausible factor. As
appropriate to the design, show how the following alternative explanations can be
eliminated from consideration: maturation, other treatments, historical factors,
statistical regression, attrition, differential selection of groups, and testing. (Note:
Sound evaluation design can control most rival hypotheses; however, other data may be used
to show attribution of effects.)
F. Significance of Results
1. Relationship of Results to Needs
Demonstrate how the results obtained are important: How do the results meet the needs for
which the project was designed? Establish the importance of the needs, and demonstrate
that the results are broad enough and powerful enough to be viewed as important.
2. Comparison of Results With Results From Other Programs
Compare the results with results of similar projects or national or statewide initiatives,
if appropriate.