U.S. Department of Housing and Urban Development. Program Evaluation and Analysis: A
Technical Guide for State and Local Governments. Washington, DC: Prepared for the U.S.
Department of Housing and Urban Development by Public Technology, Inc.; 1978. pp.
25-26.
Step 4-Verifying the Accuracy of the Data
One of the most frequently overlooked aspects of program evaluation is verifying
the accuracy of the data. While treated here as a separate step for emphasis, the
discussion of the previous step correctly suggests that data accuracy should be verified
during data collection. In this way, the analyst can take actions to correct or improve
the data immediately, rather than initiate a second collection effort later. There are
three major types of data inaccuracies-clerical errors, subjective errors, and
methodological errors.
1. Clerical Errors. Clerical errors are one of the most common sources of
inaccuracy. Such errors (transposed digits recording the wrong figure, etc.) frequently
occur when data are transferred from original source documents to summary reports or data
collection worksheets. Clerical errors can be detected by checking a sampling of the data
collection worksheets against the original source documents. If more than 10 percent of
the sample entries are incorrect, the analyst can take one of several remedial actions.
If more than one person has been recording the data in question, the analyst should try to
determine whether the high error rate is uniform among all collectors or is found only in
the work done by one or more individuals. The employee completing each worksheet can be
identified by a code on the sheet itself. If the high error rate is restricted to one or
more individuals, the analyst can either review collection procedures with those
individuals and stress the importance of accuracy to the employees and their supervisor,
or request that a more accurate employee be assigned to recollect the same data. Should
the high error rate prove to be uniform among all collectors, the analyst should review
the collection procedures with all employees and appropriate supervisors to determine
whether the worksheets are poorly designed or the data collection procedures incomplete or
confusing.
If data collection accuracy does not improve, analysts may want to consider collecting the
data themselves or finding another way to measure the criterion in question. Another
remedial course is to postpone the evaluation while improved data collection procedures
are developed. This will usually mean postponing the evaluation for one program period
(one month to one year). Naturally, the earlier in the evaluation process this
determination can be made, the fewer dollar and personnel resources will be wasted on an
incomplete effort.
2. Subjective Judgment Errors. Data involving subjective judgments will
require more involved accuracy checks than outlined above. When dealing with subjective
ratings such as those provided by inspectors or social services counselors, the analyst
must make an effort to determine the accuracy of the rating system. This is accomplished
by examining the rating scale to determine how clear and comprehensive the descriptions
are of the various rating categories. In addition, the analyst should attempt to determine
how much training the field personnel have had in the use of the scale and how often the
training is reviewed. The review question can be significant, since experience has shown
that extended use of a subjective scale often result in "compressed" ratings;
i.e.. Fewer ratings toward the extremes of the scale. Periodic reviews of the scale with
supervisors can help alleviate this tendency.
The analyst may also find it useful to examine the turnover rate among field personnel,
since high turnover often results in inconsistent ratings over the evaluation of the
ratings by getting several people independently to apply the rating scale to the same
situation or site at the same time.
3. Methodological Errors. Of the data collection techniques menti8oned, surveys
are most prone to methodological error. The analyst should review the survey instrument
(questionnaire ) for possible bias, the sample selection method, the size of the sample,
the degree of training given to surveyors, and the methods used to analyze responses. The
references found in Appendix B should provide the information needed to make most of these
determinations. No survey can be 100 percent accurate. What the analyst should watch for
are instances in which opinions or results are not clear-cut on a specific question and
there is some evidence of significant inaccuracy in the survey. Management should be
caution that does not have a high degree of reliability. Data from flawed surveys can
still be used, but with due caution.
Another type of methodological error can sometimes be avoided by double-checking of the
analyst's thought processes. It is very easy to get so involved in what you are doing that
relatively simple errors go unnoticed. For example, an evaluation director reported that
one of his associates was deeply involved in establishing criteria and collecting data on
the effectiveness of fire suppression services. The analyst hit on the idea of using the
percentage of a building that was consumed by fire as a criterion for effectiveness of the
fire department. The evaluation director hastened to point out that since the fire
department had no control over how long a building had been burning before an alarm was
turned in, and that a building might well be fully engulfed before the department was
notified, the proposed criterion was neither fair nor valid. There is a much better chance
of avoiding such errors if the work of an analyst is checked by at least one other
analyst.
It is generally inadvisable to continue the evaluation with data errors greater than 10
percent. If an evaluation is continued under such circumstances, the analyst should be
sure to identify clearly resulting distortions in the evaluation report. Managers must
understand that they cannot place the same degree of confidence in evaluations with
questionable data as in evaluations with highly reliable data.
In summary, five major options can be pursued if key data are discovered to be inaccurate:
(1) The evaluation team can seek other, perhaps less direct, ways of getting acceptable
data. (2) Improved procedures can be adopted for collecting the data and the evaluation
postponed until new, reliable data can be gathered. (3) the evaluation team can seek to
improve the quality of the data by such methods as clear supervision of the collection
effort, or the use of better collection forms. (4) The evaluation can be continued with
the clear warning that management should be cautious in using the data in question for
decision making. (5) The evaluation can be canceled as infeasible. While the most suitable
option will depend on the specifies of the situation, analysts will probably feel the most
confident with the second option, where practical. The important point is to recognize
that inaccurate data can badly undermine the credibility of an evaluation, and the analyst
should guard against this problem.