Office of Juvenile Justice and Delinquency Prevention. Evaluating Juvenile Justice
Programs: A
Design Monograph for State Planners. Washington, DC: Prepared for the U.S. Department of
Justice, Office of Juvenile Justice and Delinquency Prevention by Community Research
Associates, Inc.; 1989. pp. 40-41.
Special Topics in Program Evaluation
Sources of Data
Official Sources of Data
In many situations the most convenient sources of data are official records maintained by juvenile or criminal justice agencies. Arrests, juvenile court referrals, and adjudications are example of frequently used official data in the evaluation of juvenile justice programs. When recidivism is employed as an outcome criterion it is most often measured from official sources. Although these data are useful and often readily available, you must exercise caution in their use. Remember that the principal reason for which the data were initially collected is as an accounting of the activities of criminal justice agencies. Arrests are more a measure of police activity than of criminal behavior. However, arrest data are a preferred method of measuring recidivism and constitute the best data available from official sources, since they have been least affected by the filtering process of the juvenile justice system.
...official justice records should always be viewed as an underestimate of the actual amount of criminal or delinquent behavior.
In addition to measuring subsequent delinquent behavior, official data sources include the behavior of agency members themselves. If an arrest is made of a juvenile offender it will be represented in the office police records. However, if a youth engages in subsequent delinquent behavior it may not come to the attention of the police or, if it does, the officer may choose not to make an arrest In either case, it would not be reflected in official police records. For this reason, official justice records should always be viewed as an underestimate of the actual amount of criminal or delinquent behavior. Also, changes in organization activities or policy can have an effect on official data which should not be mistaken for changes in crime and delinquency. As long as the evaluator is aware of the potential pitfalls of these data and represents them in the report, official records are a valuable source of evaluation data.
Self Reported Data
Instead of relying on criminal or juvenile justice agencies to tell us about the behavior of youth many researchers advocate asking the youths themselves about their delinquent activities. While this process may seem incredible to some, self reported procedures have repeatedly been found to be valuable and reliable in delinquency research (see Hindelang, et. Al, 1981). In this procedure, generally the youth is asked to complete a questionnaire indicating the frequency of his/her involvement in specific types of delinquent activities. While there is general agreement between self reported data and official statistics in identifying the most serious and persistent offenders, self reported instruments offer more accurate and precise measurement of the numbers and variety of delinquent activities.
Interviews and Questionnaires
One of the most valuable sources of data is directly asking program participants, staff, or other individuals questions pertinent to the evaluation. While these approaches may involve somewhat different methods, they are similar in that the evaluator is attempting to elicit information directly from those knowledgeable about, or involved with program activities. Measurement of self reported delinquency is this type of activity which can be administered either using an interview or questionnaire format.
The choice of format depends upon the specific context of the study, the study population, and the type of information to elicited. For example, if one is studying an incarcerated population in which the participants may have a low reading ability, conducting interviews may be a more valid procedure than administering a questionnaire. Similarly the evaluator may be attempting to elicit sensitive information from program administrators or asking questions that do not lend themselves to categorized responses. In these situations an interview approach may also be more appropriate. While interviews often have the advantage of being able to probe the meaning of responses they can be more costly and more time consuming than an questionnaire. Furthermore, an interview strategy requires trained personnel to convey the meaning of questions, establish rapport with the subject, and probe responses. Questionnaires have the advantage of being easily administered to a large group of subjects, and the fact that each person is answering identical questions increases the reliability of responses.
Questionnaires have the advantage of being easily administered to a large group of subjects.
Regardless of which of these methods is employed, the evaluator must exercise great caution in the selection of items to be included. It is imperative that questions be chosen that are clear and simple, elicit only one response, do not have a double meaning, and measure what they are intended to measure. Insuring that these conditions are met increases the validity of the study.
Surveys
Public opinion polls and surveys seem to dominate our modern life. The widespread use of surveys may give the impression that designing and conducting surveys is an east task. True the administration of a survey is straightforward, which is one of the main advantages of this method. But, producing meaningful information through surveys is a much more complex and difficult task. Surveys that contain ill chosen questions and poor response options, produce information this is likely to be invalid and not reliable, not to mention misleading and potentially damaging. While volumes have been written on survey design and analysis, the following sections will provide some basic guidelines for developing survey items that may be most useful in juvenile justice program evaluation.