National Institute of Justice. Evaluating Drug Control and System Improvement Projects, Guidelines for Projects Supported by the Bureau of Justice Assistance. Washington, DC: Prepared for the U.S. Department of Justice, National Institute of Justice by Abt Associates, Inc.; 1989 (Reprint 1992). pp. 5-7.
Chapter 2: Obtaining Information for Evaluations
Questions frame not only the focus of the evaluation but also direct the evaluator to the kind of information that must be collected. They do not necessarily determine precisely the kind of information required, nor the method by which it is to be obtained, however. Administrators and evaluators face choices that involve questions of suitability, expense, timing, and how much confidence the administrator needs to have in the data and the conclusions drawn from them. Generally, these choices require making trade-offs between time, money, and confidence. The precise nature of these trade-offs in each case depend upon the features of the program being evaluated as well as the questions that are asked about it. In considering how to strike the balance between resources, feasibility, and confidence, administrators should ask evaluators to specify the trade-offs that particular evaluation plans entail.
Use Existing Data or Collect New Information?
Evaluations may be able to employ information that others have collected, usually for other purposes, rather than going to the expense and energy of collecting new information. In determining whether to use existing information rather than collect new data, administrators should ask evaluators to consider, at least, the following matters:
How available are the existing data, and can they be obtained for research?
How valid are they? Do they actually measure what they intend to measure? This is often more complicated than meets the eye. For example, studies of sentencing practices have been done by collecting and then analyzing data on all persons sitting in a jail or prison system on a single day. Such a data collection strategy biases the sentenced population in favor of those with longer terms, because these persons fill a disproportionately larger number of beds. More valid data would be obtained by analyzing those entering the jail or prison over a specified period of time. (Both of these data collection methods fail to encompass all sentenced persons, obviously, because they ignore those not given incarceration sentences.)
The validity of data can be assessed also by determining whether they refer to a subset of the population or practices of interest, or whether data reflect the entire universe of that which is being examined. If they are a subset, are there any reasons to suspect that they might be unrepresentative of the larger universe? Even if the data are not drawn from subsets or samples, how complete are the data? If information is missing, is there any reason to suspect that information is excluded systematically as the result of some practice? For example, studies that rely on drawing random samples of case records in the files of prosecutors may unintentionally exclude the more difficult or serious one if these files are kept at the assigned prosecutors' desks. Or is there reason to expect that those charged with collecting or recording the data have an interest in under-or-over-reporting them?
How reliable are the available data? (Reliability refers to the dependability of the measure used to obtain the data. Data are said to be reliable if their measurement is consistent, so that upon repeated measurement they remain the same regardless of transient personal, environmental, or instrumental factors.) Were the data collected in interviews that were sometimes carried out on site, while others were not? If so, would this plausibly have a biasing effect on some responses? Were those charged with recording the information likely to be reliable, and motivated or supervised adequately so that their performance was consistent? Was the instrument used to collect the information problematic, such as a poorly-worded questionnaire that may have yielded ambiguous (and unreliable) responses? If the data are collected by means of self-reporting, how dependable are they likely to be, given the means by which they were collected?
The principal advantage of using existing data is that they are often inexpensive or even provided free to the evaluator, and they may be abundant, representing an expenditure of hundreds of thousands of dollars. The disadvantage, however, is that there may not be a perfect fit between what the evaluator is trying to measure and the purposes for which the data were collected. Moreover, it is often difficult to assess how reliable the data are, and how reliably they were entered, without conducting random checks of the information against the records used by the persons who originally recorded the information.
Collecting New Data
Where the needed data do not exist, the evaluator will have no choice but to obtain those data directly. Although it may be more costly than using already-existing data, collecting new data has distinct advantages. Greater control can be achieved over the measures that are used, as well as over the procedures and staff employed to collect the data. The reliability and validity of these data may thereby be increased.
To choose the most appropriate data collection strategy, the evaluator should assess the relative advantages and disadvantages of alternative approaches and clarify the nature of the trade-offs that are faced in the particular evaluation. The administrator's input into the decision should include some guidance regarding the level of resources that may be devoted to collecting data, the constraints on time available to collect them (for example, must administrative or policy decisions be made on a schedule that is too tight for an extended data collection effort?), and the amount of confidence that the administrator needs to have in the results of the evaluation for decision purposes. How much of a difference will it make for the administrator to have information that is more rather than less reliable and valid? Because the answer to this question bears heavily on the degree of precision demanded of the data -- and the requisite costs of getting it -- it is helpful for the administrator to set at least the upper and lower boundaries for the evaluator.
The means of collecting new data include direct observation, interviews (either in person or by telephone), surveys, drawing information from administrative records, or piggy-backing data collection instruments onto existing recording procedures.
Direct Observation
Obtaining data by on-site observation has the advantage of providing an opportunity to learn in detail how the project works, the context within which it exists, and what its various consequences are. On-site observation is especially useful for exploratory evaluations, where the primary purpose is describing the program, identifying its significant features, exploring the possible consequences of its activities, and developing hypotheses about how and why it works. It is an especially useful strategy for identifying "side effects," or consequences that are not intended.
A major disadvantage of direct observation is that data collection may lack focus, unless discipline is exercised. The reliability of data collected by observation may also be questionable, depending upon the competence and perceptiveness of the observer, and his or her relationship to those being evaluated. Generally, this data collection strategy is better suited to exploratory and descriptive evaluations, or evaluations of project implementation, than to tests of hypotheses about a project and its consequences.
If precision of measurement and high reliability is desired, evaluators may design forms for recording their observations. Such forms may specify uniform procedures and categories for such things as counting events, the time that it takes for them to happen, the numbers of persons processed, the numbers and types of persons who interact during the course of a specific transaction, or other categories.
Interviews
Interviews are often coupled with direct observation during site-visits, but may be conducted separately, either in person or by telephone. For exploratory purposes, such interviews may be relatively unstructured and open-ended. Where the evaluation calls for more focused data collection, a questionnaire can be developed. This increases the reliability of the data because respondents are answering uniformly-phrased questions. The virtues of structured interviews over other types of data collection methods include more control over obtaining the data, (unlike mail surveys, it is more difficult for the person interviewed not to respond at all), and an ability to clarify answers by probing matters that emerge in the course of the interview. The chief disadvantage is cost and time, which can be substantial if the program is a large one. In many circumstances, interviewing may not be feasible at all.
Collecting Data from Administrative Records
Programs operating within the criminal justice system often keep a variety of different records for administrative purposes. These include case files on the defendants/offenders, management information systems organized to coordinate processing of cases or individuals, institutional performance records of persons detained or sentenced or put under some other form of surveillance, records of prior arrests or convictions, as well as financial accounting information about agencies. Evaluators will almost always draw at least some of the required data from these records. The advantages of such information are obvious: it is often plentiful, the cost of recording it has been born by others, and some of it may even be in computer-readable format. Whether or not the evaluation can rely entirely on it depends upon whether the data describe exactly what the evaluator is examining, how reliable the data are, and whether the recording of them has been biased. Many of the same issues arise for the use of administrative records as for using other existing data, described above.
Adapting Administrative Recording-Keeping Procedures for Evaluations
One of the most effective and least costly methods of collecting data is to change the project's existing procedures for recording information so that the needed information is captured. For example, existing intake forms might be augmented to include additional questions asked of all project participants; or records of services rendered to those participants might be supplemented with requests for additional information needed by the evaluator. The advantages of this method are substantial: the evaluator may define precisely what it is that is to be collected, rather than having to rely on already-existing indicators that may not provide direct measures. The evaluator will also not be required to hire and supervise a staff of data collectors. The costs of data collection will be carried by the project, however, and recording these additional data may be time consuming. Because data collected in this way often permit powerful analyses, evaluators and administrators are encouraged to plan ahead for evaluations, to design data collection instruments that can be used by project staff for a period of time before the analytic work begins, and to monitor how staff comply with the procedures for recording the data. Because project staff may resist doing the additional work, agreements have to be negotiated among staff, project management, and those involved in the evaluation.
Surveys
When administrative records do not contain the data required of the evaluation, it may be collected by pencil and paper surveys. Like questionnaires administered during interviews, surveys permit control over the phrasing of questions. In circumstances where anonymity of respondents matters, such surveys may reduce the likelihood of biased reporting and thereby raise the validity of the information. Mail surveys are also relatively easy to administer; responses can be coded quickly, readily putting a format for analysis (including computer-readable formats), and all often at a low cost. They may not be suited to data collection on small projects, however, where the key actors can be surveyed in person by interview. Moreover, a principal liability of using surveys distributed by mail or other means is that respondents often have few incentives to complete and return them. Return rates are often very low.
Confidentiality of Data
Evaluators may have to collect information about people that is sensitive in nature. For example, the evaluation may seek to learn about past or current criminal activity. Measures of this can be obtained by asking persons to reveal this information about themselves, on the condition that this information shall not be divulged to others or described in ways that reveal the identity of the respondent. If such confidentiality is to be assured, evaluators, administrators, and relevant parties should establish a formal agreement to guarantee that information obtained for evaluative purposes will not be revealed to project administrators, staff, or to other agencies.
Protection of Human Subjects
In addition, research projects that involve putting subjects at risk, or that offer a particular benefit to some (e.g., a treatment) while denying it to others, must conform to guidelines established by the U.S. Department of Health and Human Services, as well as any other guidelines required by other relevant agencies. (For example, drug treatment programs in hospital settings will have to conform to human protection guidelines established by the hospitals). Questions regarding this matter should be addressed to the Bureau of Justice Assistance.