Office of Juvenile Justice and Delinquency Prevention. Evaluating Juvenile Justice
Programs: A Design Monograph for State Planners. Washington, DC: Prepared for the
U.S. Department of Justice, Office of Juvenile Justice and Delinquency Prevention by
Community Research Associates, Inc.; 1989. pp.56-57.
Choosing an Evaluator
Regardless of whether you follow an RFP process for your evaluation, you must choose an evaluator. Sometimes this choice will be made simply by the restrictions you face-not enough money to hire an outside evaluator, you have competent evaluators in your office, or self-evaluation by the funded program is deemed appropriate. Other times you will be faced with a dilemma-for example, hiring an outside evaluator when you have the expertise in your office; or imposing your office evaluator on a program with staff capable of conducting evaluations. Selecting an evaluator is a critical decision, one that often hinges on political considerations, or that raises political issues once the decision is made
In this section, we discuss two aspects of choosing an evaluator: (1) the pros and cons of utilizing outside evaluators, and (2) recognizing a good evaluator when you see one.
Inside Versus Outside Evaluators
There are practical and logistical aspects to his issue. First two types of inside evaluators need to be distinguished. An inside evaluator may include you or one of your staff, if you are evaluating a program funded by your office, or it may mean a staff person(s) from the program itself.
Generally, the more interest exhibited in your program by people and professionals outside of your state or local environment, the more you will want to consider an outside evaluator.
Depending on the situation, or on one's perspective, your staff may be viewed as outsiders. If your office provided the program funding, however, you are not likely to be viewed as an outsider in terms of evaluation.
On the practical side, it is almost always less expensive to use an inside evaluator than to contract for an outside evaluator, but there are exceptions. Using program personnel as evaluators is probably the least expensive but produces the lease objective results. Contracting with a university or college may prove to be a cost effective means of obtaining outside evaluators. Sometimes, based on cost considerations alone, using inside evaluators is the only option for conducting a program evaluation. In many instances, it is a perfectly logical and appropriate thing to do.
You may wish to use an outside evaluator because someone with broad experience in program evaluation is needed, or because the need for an objective perspective is great, which is often the case for expensive and controversial programs. You may, perhaps, intend to replicate a juvenile justice program evaluation conducted elsewhere and need the same evaluator, or at least an outside evaluator to be consistent. Wherever it is essential to be as objective as possible about the evaluation, you should consider an outside evaluator. Generally, the more interest exhibited in your program by people and professionals outside of your state or local environment, the more you will want to consider an outside evaluator. You cannot escape the fact that evaluations performed by insiders run the risk of being criticized as lacking objectivity.
To repeat, it will not always be fiscally possible to hire an outside evaluator. But, well designed and documented evaluation research effort will almost always provide sufficient and reliable evaluation information.
Recognizing a Good Evaluator
Evaluators are made (often self-made) and not born, so it is possible to identify what they are made of. There are three critical qualities of a good evaluator-experience, skill, and brains. Each will be reviewed here.
Program evaluators learn by doing, and the key to conducting good evaluations is knowing the ins and outs of the political and logistical aspects of program evaluation. Because, there are few academic programs in the country that have a program evaluation curriculum, a good evaluator is usually one who has done a number of them, and whose references will vouch for the work done.
Evaluation skills refers to knowledge about research design, methodology and statistics. These are taught in academic programs, and they can also be easily identified in a review of written materials provided by a prospective evaluator. Like experience in evaluation, research design and methods skills are refined in practice. They will most often be found in who has utilized those skills in prior program evaluations.
Brains refers to the thought process in program evaluation research. The best evaluations are often found in unique or creative applications of research skills to the particularities of the program being evaluated. This may in a special sampling strategy, utilization of a measure from another discipline, creative use of archival records, or an effective explanation of statistical methods to non-technical readers that indicates competence and confidence in the subject matter.
You have three sources of information that will help you determine the experience,
skills, and brains of an evaluator. They are (1) the written examples of past work
performed by the evaluator, (2) the resumes and references provided, and (3) the actual
evaluation plan submitted for the program at hand. Careful consideration of each of these
with further discussions with the potential evaluator when you feel it is necessary, will
held you in the selection process.