Program
Evaluation
and Analysis

A TECHNICAL GUIDE
for State and Local
Governments

Prepared for

U.S. DEPARTMENT OF HOUSING AND URBAN DEVELOPMENT
Office of Policy Development and Research
Washington. D. C. 20410

PUBLIC TECHNOLOGY, INC.

Washington, D.C. 20036

under Contract #2237

CHAPTER III: PROGRAM EVALUATION ....
    Task 5-Selecting an Evaluation Design .......
        Design #1: Planned vs. Actual Performance .
        Design #2: Time Trend .................
        Design #3 Before Program vs. After Program
    Task 6-Data Collection ...................
        Step 1-Identifying the Data ...............
        Step 2-Determining Data Availability ......
            1. Existing Records and Statistics ........
            2. Client Perception Surveys ............
            3. Special Data Collection Techniques ....
        Step 3-Physically Collecting the Data .....
            1. Existing Records ant Statistics ........
            2. Client Perception Surveys ............
            3. Special Data Collection Techniques ....
        Step 4-Verifying the Accuracy of the Data ..
            1. Clerical Errors .....................
            2. Subjective Judgment Errors ...........
            3. Methodological Errors ...............
    Task 7-Synthesizing the Data ..............
        Step 1-Organizing the Data ..............
        Step 2-Making Comparisons .............
            Design _#1: Planned vs. Actual ..........
            Design #2 Time Trend ...............
            Design #3: Before Program vs. After Program .............
        Step 3-Checking the Validity of Evaluation Results ...............
        Step 4-Drawing Conclusions and Making Recommendations ........
    Postscript ................................

CHAPTER IV: PROGRAM ANALYSIS ........
    Task 8-Developing Program Alternatives .....
        Step 1-Generating Program Alternatives ....
        Step 2-Screening Alternatives ............
    Task 9-Data Collection ...................
        Step 1-Estimating Service Demand ........
        Step 2-Collecting Cost Data .............
        Step 3-Collecting Effectiveness Data ......
        Step 4-Collecting Feasibility Information ...
    Task 10-Synthesizing the Data .............
        Step 1-Organizing the Data ..............
        Step 2-Comparing the Data ..............
        Step 3-Drawing Conclusions .............

CHAPTER V: COMMUNICATION AND FOLLOW-UP..................
    Task 11 Communicating the Results ........
        Step 1-Preparing a Draft Report .........
        Step 2- Reviewing the Draft Report.........
        Step 3-Transmitting the Final Report ......
    Task 12-Follow-up Obligation .............
        Step 1-Forming an Implementation Team ..
        Step 2-Preparing an Implementation Workplan ......................
        Step 3-Monitoring Implementation Progress...
Summary ................................

APPENDIXES .............................

APPENDIX A: PRINCIPLES OF EVALUATION

APPENDIX B: SAMPLE SURVEYS ...........
    Potential Benefits .........................
    Selected Documentation ....................
    User Experience.............................
    Adaptation Considerations ..................

APPENDIX C: PROGRAM EVALUATION AND ANALYSIS STUDIES ....

FIGURES
    1 The Program Evaluation and Analysis Process
    2 Project Selection Paper ....................
    3 Orientation Questions .....................
    4 Workplan Outline ........................
    5 Effective Objectives .......................
    6 Clientele Group Classification ...............
    7 Suggested Interview Questions ..............
    8 Evaluation Criteria .......................
    9 Traffic Fatalities Time Trend Comparison .....
    10 Data Availability Worksheet ................
    11 Data Location ...........................
    12 Table Comparison of Evaluation Criteria .....
    13 Bar Chart Comparison of Evaluation Criteria .
    14 Graph Comparison of Evaluation Criteria .....
    15 Sources for Program Alternatives ............
    16 Brainstorming ...........................
    17 Alternative Summary .....................
    18 Sources for Demand Estimates .............
    19 Sample Cost Estimation Worksheet ..........
    20 Cost Estimation Principles .................
    21 Feasibility Guidelines .....................
    22 Cost Comparisons ........................
    23 Report Outline ...........................
    24 Suggested Review Questions ................
    25 San Diego County Report Review System .....
    26 San Diego County Office of Program Evaluation(OPE)
            Implementation Workplan.........
    27 City of San Diego Action Plan ..............
    28 Invalidity Factors........................
    29 Contacts: Surveys ........................

City, and State governments are finding it increasingly difficult to balance their budgets as rising costs and service demands continue to outstrip revenue. This situation increases the importance and difficulty of making resource allocation decisions for public programs. Such decisions can be considerably improved if local officials have better information on the effectiveness of existing and proposed programs.

To help provide this type of assistance to local and State governments. the Department of Housing and Urban Development. Office of Policy Development and Research has provided financial and technical support to Public Technology, Inc. (PTI). to develop this procedural guide on the we of program evaluation and program analysis. The guide is based on research funded by the HUD Office of Policy Development and Research. plus other work done by city. county, and State governments and by universities.

PTI's Technology Exchange Program. which is partially sponsored by the Office of Policy Development and Research's Division of Product Dissemination and Transfer. prepared this guide. The Technology Exchange engages in five basic activities that were employed in developing this guide:

Problem Description. The Program actively encourages the participation of city, county, and State government representatives in defining common, high, priority problems.
Information Collection. The Program works systematically to acquire information about innovative techniques applicable to the problems defined by State and local government practitioners.
Product Engineering. The Program works to translate, synthesize, and assemble promising ideas. concepts. methods. and procedures into war-oriented products.
Package Production. The Program then produces the communication tools needed to support transfer activities including an Executive Summary. a Management Report. Technical Reports. and a Training Package.
Information Dissemination and Technical Assistance. The Program works to actively encourage awareness. interest. evaluation. and adoption of the innovative techniques by city. county. and State governments.

The Technology Exchange Program does not engage in research and development. but rather works to pull together available information about existing techniques and to place this information in the hands of the State or local practitioner in an easy-to-use form. Thus. this guide does not purport to present new research: it is a synthesis of existing techniques. a guide to what is now known about program evaluation end program analysis.

This Technical Guide is part of a package of tools designed to assist State and local governments the complete communication package consists of four parts:

Program Evaluation and Analysis: An Executive Summary-A brief overview written for elected officials and chief               executives that describes the rationale for using program evaluation and analysis to support the decision-making process and covers the package organization, content, and scope.
Program Evaluation and Analysis: A Management Report for State and Local Governments-A somewhat longer document. written for chief executives and senior administrators, that explains how to plan. organize. staff. implement. and monitor a program evaluation and review effort and outlines the existing techniques described in the Technical Guide.
Program Evaluation and Analysis: A Technical Guide For State and Local Governments--A procedural guide to the tools and techniques of program evaluation and analysis. written for administrators. analysts. and other staff who will directly conduct program evaluations and analyses. The Guide describes each task and step in detail. presents selected documentation, and provides options for adapting many steps to local conditions.
Program Evaluation and Analysis Training Program--Regional workshops for State and local government practitioners interested in implementing the concepts or techniques documented in the Management Report and Technical Guide. conducted by PTI staff personnel. plus on-site technical assistance to a limited number of jurisdictions.

State and local government officials may address inquiries concerning any of the above to:

Program Evaluation
Public Technology, Inc.
1140 Connecticut Avenue. N.W.
Washington. D. C. 20036
(202) 452-7700

This Guide was with financial assistance from the Division of Product Dissemination and Transfer of the Office of Policy Development and Research. The Division of Community Development and Management Research provided technical assistance.

The specifications for this package and the necessary policy. technical. and editorial guidance for its preparation were provided by a User Requirements Committee comprising the following members:

Leroy Whiting
( Chairman )Assistant Executive
Director Model Cities Chicago
Committee on Urban Opportunity
Chicago,Illinois

Richard Heiwig
Assistant City Manager
Dayton. Ohio

Thomas C Kelly
County Manager
Volusia County. Florida

Frank Kirk
Director of Local Government Affairs State of Illinois
Springfield, Illinois

Howard McMahan
Justin Industries. Inc.
Fort Worth Texas
Formerly City Manager.
Oklahoma City, Oklahoma

James E. Phelps
Budget and Research Director
Long Beach, California

Honorable Paul Soglin
Mayor
Madison, Wisconsin

William 0. Talley
City Manager
Anaheim, California

Dean Vanderbilt
Management Services Director
Dallas, Texas

Significant contributions to this Guide were also made by: Gary Brown. Director of Evaluation. Winston Salem. North Carolina: Camille Cates. Director of Intergovernmental Relations. Sunnyvale. California: Donald Fisk. Director. and Harold Kosakoff. Deputy Director. Office of Program Evaluation. San Diego County. California: Rack Fukuhara. International City Management Association; Charles E. Hill. Budget and Research Director. and Patrick Manion. Management Assistant. Phoenix. Arizona: Charles Kirchner. Special Assistant to the Director. Department of Local Government Affair Stan: of Illinois: Dave Knapp. Assistant Director of Financial Management. City of San Diego. California: Emmett Moten. Director, Policy Planning and Analysis. blew Orleans. Louisiana: John Niles. District of Columbia Office of Budget and Management Systems: John Tow. Management Analysis Supervisor. Long Beach, California: and Victor Swyden. Councilman. Kansas City. Missouri. A special word of acknowledgment is also due for Heather Aveilhe. Han Siegel and Harley Fitts of the Department of Housing and Urban Development for their active and positive participation in this project.

The material presented in this Guide is a staff synthesis of perspectives of User Requirements Committee members: materials taken from published research and other printed sources: and visits to leading-edge jurisdiction such as Kansas City. Missouri: Multnomah County, Oregon: New Orleans. Louisiana: Phoenix. Arizona: San Diego County. the City of San Diego. and Sunnyvale. California and Winston Salem. North Carolina

Some of the material in this Guide was developed from two publications of the Urban Institute:

Practical Program Evaluation for State ant Local Government Officials. by Harry Hatry.  Richard E. Winnie and Donald M. Fisk. 1973.

Program Analysis for State and Local Governments by Harry Hatry. Louis Blair. Donald Fisk and Wayne Kimmel. 1976.

In some instances, excerpts from these documents have been used verbatim. or with some modification. We gratefully acknowledge the cooperation and guidance of Harry Hatry and Jane Woodward of the Urban Institute.

The PTI Technology Exchange Program is managed by C. Nelson Hoy. The staff members responsible for this document are: Ken Steil, Project Director:  David Pearl and Walter Webb, Editors: and Marcia House, Project Secretary.

CHAPTER ONE Introduction

This Technical Guide is written or county manager. administrative assistants or interns and management or budget analysts responsible for conducting program evaluations or analyses. The guise assumes that such users have certain basic analytical skills but no specific experience with program evaluation and analysis. Within this context the Guide documents a process for addressing whether a particular government program is producing the desired result and for determining the most effective ant efficient way to allocate resources for improved future performance.

PROBLEM OVERVIEW
State and local governments are finding it increasingly difficult to balance their budgets because costs are increasing at a faster rate than revenues. This situation results in large pan from inflated costs for labor materials and equipment: public demands for expanded. improved. or additional services; employee demands for higher pay. shorter hours, or additional fringe benefit: lower tax revenues due to a depressed local economy: delays in real estate reassessments: and public resistance to higher taxes coupled with inflation.

Public administrator have several alternatives available to them in dealing with this squeeze: ( I ) improve effectiveness (2) improve efficiency. (3) decrease expenditures. (4) increase tax revenues or (5) combine two or more of these alternatives. Program evaluation and analysis are management-oriented tools designed to help public administrator with the first two alternatives -( 1) improve program effectiveness and (2) improve program efficiency.

TECHNOLOGY OVERVIEW
Program evaluation and analysis are two closely related technologies that can be used together or separately. Program evaluation measures program effectiveness. It involve reviewing objectives reviewing evaluation criteria. collecting data. synthesizing data, drafting reports and implementing result. Program analysis concern the design of new programs or redesign of old ones. It involves selecting programs. defining problems. defining or redefining objectives establishing evaluation criteria. generating program alternatives estimating costs ant effectiveness, studying feasibility drafting reports, ant implementing results.

The two technologies presented here are complementary. Evaluation provides information on the impact of existing government efforts and highlights areas that need improvement. Analysis can then be employed to help determine the most effective form for these improvements to take. A~. . program improvements have been implemented, evaluation is once again needed to assess their success and the cycle begins again. The cycle can also begin with program analysis used to determine the best way to institute a totally new program, followed by evaluation of the program activities. While evaluation ant analysis can be used separately the payoff is greatly increased by using both processes together.

For the purposes of this Guide. the two technologies can be understood as comprising a single process. Figure I presents a schematic diagram showing the major tasks in the program evaluation and analysis process. For simplicity and clarity. the diagram does no' attempt to show all of the possible decision points and feedback loops in the evaluation/analysis process. For example. difficulties in cats collection (Task 6 or 9) could easily force a revision of the project workplan (Task 2).

Program evaluation and analysis can be viewed as:

A structured process for staff personnel to follow in situations where the public administrator needs  better information for decision making. A structured process for determining whether a partially program is producing desired results or effects. rational method for designing a new program or redesigning an old one to efficiently produce desired results or effects, and
A method for helping program manatee diagnose and make improvements in their operations.

Program evaluation ant analysis attempt to answer questions about the effectiveness and efficiency of government programs by identifying quantifiable indicators of program performance. Naturally, some programs (such as street maintenance, refuse collection and vehicle maintenance) lent themselves quite readily to this approach while others (such as recreation. library services. and welfare programs) are much more Difficult to quantify. Social action programs often have vague or ill-defined goals ant objectives that span many years. While evaluation ant analysis can provide meaningful information on such programs. these types of programs will usually require greater experience and technical capabilities than many jurisdictions will have when they first undertake program evaluation ant analysis. Successful evaluation ant analysis depends on:

The existence of or the ability to formulate meaningful goals objectives. ant evaluation criteria for public programs; The ability to measure program effectiveness through the collection and interpretation of data: The willingness of public officials to support the process by basing resource allocation decisions on the results of evaluation and analysis; and The commitment on the part of local officials to implement the recommendations of evaluation and analysis projects

POTENTIAL BENEFITS
Program evaluation and analysis offers several potentials benefits to city and country governments:

Elected officials, chief executives, ant public administrators benefit by having better information to aid their decision making, thus giving them greater confidence in those decisions:

Program agency personnel have the opportunity to ex- amine and influence the future direction of their program well as benefit from an outside view of the program that will help them gain a fresh perspective:

The public benefits by receiving more effective and efficient government services for their tax dollar: and all parties benefit from the valuable insights into government programs gained from examination of a program and its basic premises.

Figure 1. THE PROGRAM EVALUATION AND ANALYSIS PROCESS. The diagram below summarizes the 12 major tasks in the program evaluation and analysis process. Note that both processes have common beginning end ending tasks and that the process is cyclical. To maintain clarity. the diagram does not attempt to show all of the possible decision points and feedback loops in the process.

INSERT FIGURE 1 HERE

ORGANIZATIONAL REQUIREMENTS
A program evaluation or analysis project. as outlined in this Technical Guide can be undertaken by a single analyst working either full. or part-time. Jurisdictions new to evaluation ant analysis are not likely to assign more than one staff to an initial effort.

The analyst should be a generalist with analytical ability-analytical ability being defined as the ability to ask the right questions. A college degree in business administration. industrial engineering. public administration or any number of fields is very helpful but not essential. as is some experience in government operations background in statistics is valuable. The analyst should also be inquisitive, resourceful and open-minded enough to ask the type of questions that assume nothing is given. Above-average verbal and written communication, skills are very important. The analyst bean the responsibility of doing most of the actual work of data collection synthesis and report writing. A Food candidate might be a budget or management analyst, a planner. or an administrative intern.

Even when a project is formally assigned to a single individual. project success requires broader involvement and cooperation. Top management supervision ensures that the evaluation or analysis effort effectively addresses management needs. Good liaison with the program agency secures the assistance of agency staff. The expertise of specialists in various departments supplements the generalized skills of the analyst.

Recognizing these needs this Technical Guide presents program evaluation and analysis in a project management framework. That is to say. once a program has been selected for evaluation or analysis. the conduct of the evaluation or analysis is viewed as a work project in and of itself. As such. the evaluation or analysis project has its own organization. objectives. and staffing requirements.

While the analyst will probably possess most of the skills required. most jurisdictions will find that it is more efficient to use a team approach. A team or multi-disciplinary approach will usually be beneficial because: ( I ) it allows the best talent available in each functional area to be assigned as needed, (2) it helps to train additional personnel in program evaluation and analysis principles and techniques and (3) it helps to foster increased cooperation among involved government personnel. Neither evaluation nor analysis can be successfully conducted without close cooperation from program agency personnel, so management should lay the necessary groundwork.

A project team consists of group of people assembled for the explicit purpose of evaluating or analyzing a government program. The group disbands once its mission is accomplished. While the size and composition of this group or "team" will vary with the complexity of the project and will probably even vary somewhat over the duration of the project, two key people will be required in addition to the analyst: The team leader and the agency liaison.

The team leader should be someone with a good track record, have a good general undemanding of governmental operations, and have the ability to work with and motivate a variety of personalities. If team leaders are not part of a central evaluation staff, then they should represent the management function of the jurisdiction and have authority that extends across departmental lines.

The team leader is responsible for the management of the project in accordance with the approved workplan and for interfacing with lop management and elected officials. The team leader parcels out specific assignments within the team and runs external political interference for the team. A good candidate might an assistant city manager. budget director. or assistant to the mayor.

The agency liaison should have a good grasp of all agency operations. access to the department head. and the ability to work with others. He or she will be responsible for helping the team leader and analyst hammer out the details of the workplan. providing access to agency personnel and information, providing substantive guidance on program purposes and background, and keeping the department head informed of the study progress. A good candidate would be an assistant department head or administrative assistant.

Smaller jurisdictions may combine the functions of team leader and analyst in one person. In larger jurisdictions, the team leader may actually supervise the work of several teams conducting several evaluation and analysis projects. The Guide assumes that the team leader and analyst will pet form the work required for program selection, and that the agency liaison will be selected during project workplan preparation. After the workplan has been approved, additional personnel will be added to the team as required. Depending on the scope and time frame of the project. the analyst is the only person who may be full-time on the project.

TECHNICAL GUIDE APPROACH
The Guide presents the program evaluation and analysis process as a series of tasks, with specific steps laid out for each task. Not all evaluations or analyses will require every step specified in this Guide in exactly the level of detail provided. However, each task and step represents a concern that must be dealt with in some way.

For example, the task concerned with generating program alternative during program analysis involves consulting many information sources and thoroughly screening alternatives. Analysis of a small program may not warrant consulting every information source and using every screening technique described in the Guide. but the basic purpose of seeking innovative ways to accomplish program objectives must be recognized and accomplished. Toward this end, the Guide describes alternative levels of effort for as many of the steps as possible.

While discussion of tasks and steps is employed, this form of presentation does not necessarily mean that the Guide is a how-to-do-it manual in the strictest sense. Thus, the major intent of the Guide is to describe an overall program evaluation and analysis approach. The tasks and steps are guides to the techniques that seem the most appropriate for jurisdictions with little or no formal evaluation and analysis experience. As a jurisdiction gains experience and expertise, more sophisticated techniques, such as mathematical modeling, can be used to evaluate and analyze more complex and less well defined programs. Consequently, this document should be viewed as a detailed primer and reference work on program evaluation and analysis, with the understanding that a jurisdiction will probably outgrow some of the techniques presented here as experience is gained.

CHAPTER ORGANIZATION
This Technical Guide is organized into four major chapter and three appendices. Chapter 11. "Preparatory Tasks," discusses how to select the appropriate programs for evaluation and analysis, how to define the project scope and prepare a project workplan, how to select the project team members and draw upon outside resources, how to define or redefine program goals and objectives, and how to establish criteria for measuring program performance (Tasks 14 in Figure 1)

Chapter III. "Program Evaluation," discusses how to elect the proper evaluation design, how to collect the necessary information and data, and how to synthesize this information and data and draw conclusions about program results (Tasks 5-7 in Figure 1).

Chapter IV. "Program Analysis," discusses how to develop alternative program approaches, how to estimate costs for each alternative, how to estimate effectiveness for each alternative. how to assess feasibility for each alternative. and how to analyze the options available Tasks 8-10 in Figure 1).

Chapter V. "Follow-Up Procedures," discusses how to prepare a draft report on program evaluation or analysis, how to review the draft report with all interested parties, how to communicate the findings to top management, and how to organize and monitor an implementation effort (Tasks 11 and 12 in Figure 1).

Appendix A. "Principles of Evaluation," introduces the theory of evaluation, including a presentation of the three major types of evaluation designs, and a discussion of the sources of invalidity in program evaluation.

Appendix B. "Sample Surveys," briefly describes sample surveys, discusses their use in the evaluation and analysis process, provides references to selected documentation that will provide assistance in actually conducting a survey, and provides contacts with jurisdictions that have used surveys.

Appendix C. "Program Evaluation and Analysis Studies," provides a functionally organized list of evaluation and analysis projects that have been conducted by five local governments and cites persons to contact in each jurisdiction for additional information about those studies.

CHAPTER TWO Preparatory Tasks

INSERT DIAGRAM HERE

Program evaluation and analysis are closely related technologies that can be used separately or together. Both begin with preparatory tasks including: (1) selecting critical programs, (2) preparing a workplan. (3) completing a project team. and (4) defining goals and objectives and establishing performance criteria. Since these procedures are common to both techniques, they will be discussed only once.

Task 1 involves several possible ways to select evaluation and analysis projects, including some specific tips for selecting a jurisdiction's first project. Task 2 covers planning the work of a specific evaluation or analysis project, such as defining the problem, establishing project scope, and estimating time and manpower requirements. Task 3 includes identifying and organizing the people needed to perform the various project jobs. Task 4 stresses the importance of and provides a methodology for defining or identifying goals, objectives, and performance criteria in program evaluation and analysis.

The analyst is deeply involved in all four tasks described in this chapter, and the analyst is complemented by a team leader, serving in an administrative capacity. An agency liaison is chosen early in Task 2-Preparing a Workplan. Subsequently. skills needed for the project can be identified. and other members of the project team chosen.

The tasks discussed in this chapter will lay the groundwork for the specific steps of evaluation or analysis. In most cases several possible levels of effort are described so that the level of detail and precision can be tailored to meet local requirements. Chapter III provides the same type of guidance for the tasks of evaluation, while Chapter IV discusses analysis. Chapter V again deals with concerns that are common to both techniques, the communication of project results and the implementation of recommendations.

TASK 1-SELECTING CRITICAL PROGRAMS
    Obviously, no jurisdiction has the resources to evaluate or analyze all of its programs. Therefore, it is important to determine what programs get priority treatment. There are several ways of making this selection, depending upon local conditions, but the first two steps, preparing an initial list of candidate programs and screening the list, are common to all selection methods. Beyond these common steps, a more structured and detailed approach to project selection is presented in Step 3-Preparing Additional Information on Remaining Candidate Programs. The amount of structure in the selection process will normally depend on such factors as the size of the jurisdiction, the resources committed to program evaluation and analysis, and whether elected officials are to be involved in program selection. As a general rule, the more people involved in selection, the greater the need for a formalized structure.

Step 1-Preparing the Initial List
    Selecting programs begins with the chief executive or his staff preparing a list of candidate program. Most jurisdictions have an unwritten list of problem programs that immediately come to mind during' any discussion of possible areas for improved performance. The best way to formalize such a list is to examine the most common sources for program suggestions:

Operating Agencies - Frequently, department heads and other agency employees will be acutely aware of programs that need study and improvement. Such programs are often mentioned in agency budget requests as initiatives for new programs or          expansion of existing programs.

Staff Agencies - Budget and research, planning, and the chief executive's staff personnel frequently have a depth and breadth  of knowledge of government programs.

Elected Officials - Councilmen and other elected officials are often aware of critical programs and make them either campaign issues or subjects for legislative initiatives.

Community Groups - Service clubs, improvement associations and special interest groups may be particularly concerned with and draw public attention to certain programs.

Citizen Surveys - Many jurisdictions are beginning to rely on sample surveys of citizen perceptions to identify areas of concern, either in specific programs or across the full spectrum of government activities. Once the candidate list has been written, the        screening step can begin.

Step 2-Screening the Program List
       The list of candidate programs compiled in Step 1 should be examined carefully to select the most important and appropriate ones. This is best accomplished by applying the following selection criteria to the list:

Timing - Is the program approaching a decision point, such as the budget or expiration of key legislation? Is there time for the analysis to be done before decisions must be made?

Scope - Is the program significant enough in impact or amount of government resources involved to merit the effort?

Performance - Does there appear to he substantial room for improving program performance? Past evaluation results are          particularly helpful here.

Capabilities - Are the dollar and personnel resources available to Reform the analysis? Would the analyst require the use of    outside expertise?

Data Availability - Do sufficient data exist to undertake the analysis, and can needed data be gathered within the time               available

Political Feasibility - Does the program have such strong support from special interest groups (labor unions, citizens' associations. etc.) that a chance in operations is unlikely, regardless of analysis results?

Applicability - Does the program lend itself to measurement? Can reasonable estimates be made of the effectiveness of current operations or future alternatives?

A jurisdiction embarking on an initial formalized evaluation or analysis project should naturally be concerned about the success of this first project. The experience of several jurisdictions indicates that several additional criteria should be applied to increase the probability of selecting a successful first project. These criteria are:

Completion Time Span-The first project should have a relatively short time span, probably 3 - 4 months maximum. Local decision makers may lose interest in program evaluation and analysis if the initial project takes too long to complete.

Payoff-A special effort should be made to select a project that will produce easily visible benefits.

Program Perceptions-The first program selected should be one that has a positive or neutral image to decision makers. Selection of a program that is viewed unfavorably will only strengthen the misconception that evaluation and analysis are negative processes intended to criticize and embarrass operating agencies. It is helpful if the program agency head is favorably disposed to the conduct of the study.

A positive response to all of the above criteria means that the project is relatively safe in terms of the probability of producing positive, visible results. As a jurisdiction gains experience and confidence in evaluation and analysis, local officials will probably wish to attempt more challenging projects for which success is less sure but the potential payoff greater.

For example, a good first project might be a street maintenance program that is readily quantifiable, relatively narrow in scope and objectives and appears to have some room for improvement. As the local staff gains experience from such projects, local officials may make the conscious decision to evaluate or analyze more challenging programs, such as local recreation and social action programs.

While all of the selection criteria listed above are important, practitioners unanimously agree that the issue of timing is by far the most important. Since the basic purpose of evaluation and analysis is to provide information for decision making it is vital that the results be available when a decision needed. From a practical standpoint, this often means that the time when the results are needed coupled with knowledge of the available personnel resources, will frequently determine how rigorous an evaluation or analysis can be. Phrased another way there is almost always enough time to perform some level of evaluation or analysis to aid decision maker

This should not be interpreted as encouragement for less rigorous and therefore potentially inaccurate, studies: it merely recognizes the necessity for evaluation and analysis to conform to the real needs of decision makers. A more detailed discussion of time frame and scope tradeoffs will be presented in Task 2-Preparing a Workplan.

A further aid to selecting evaluation and analysis projects is the provision of time and cost estimates early in the selection process. Naturally the actual time, manpower, and cost can vary considerably according to the final scope of the project but most managers and elected officials find it useful to have tough estimates to aid their deliberations. These initial estimates may be expressed simply as ranges for calendar time, manpower and cost. For example, the analyst may estimate that an evaluation of a street maintenance program will take 3-4 months, involve 1-2 man-months of analytical time, and cost $4,000 to $7,000 to obtain meaningful results. As mentioned above, tradeoffs within these categories can be made but the estimates give decision makers some idea of the relative magnitude of the respective candidate studies.

In addition to the factors discussed above, decision makers may wish to select a program for evaluation to confirm their impression that it is performing well. If the evaluation verifies their opinions, elected officials or top management may be able to silence critics of an effective although unpopular program.

The availability of program evaluation results is an additional criterion in determining the subject of program analysis. Evaluation examines the past performance of a program to determine program effectiveness and efficiency. Evaluation results are not a deciding factor in the fate of a program: however, they are a clear indication of program performance and should focus attention on areas that need improvement.

For several reasons, the availability of evaluation results should carry considerable weight in the selection of programs for analysis. First, the fast selection of a program for evaluation indicates a high level of concern on the part of local decision makers, if the selection was based on local priorities rather than Federal requirements. Second, evaluation results point to major problem areas and analysis can suggest improvements. Third, the evaluation effort lays the groundwork for analysis by identifying objectives and criteria and familiarizing analysts with agency operations and data sources. Also, communication and cooperation links with the program agency developed during evaluation can be maintained and expanded. When program operations are favorably evaluated, it is often a waste of resources to conduct a subsequent analysis: however, in such cases analysis may indicate how the program might address the needs of different client groups, or incorporate different activities.

The above two steps may provide enough information for many jurisdictions to select evaluation and analysis projects. However, if more precision or the involvement of more people is desired in program selection, then the procedure outlined in the following step is in order.

Step 3-Preparing Additional Information on Remaining Programs
Local decision makers will find it very helpful to have additional information in a uniform format and level of detail about each of the programs that survived the screening. A good vehicle for accomplishing this purpose is a project selection paper. Such a paper is a written presentation that attempts to identify and describe the main features of a program. The paper provides an extremely useful starting point for evaluation and analysis, as well as serves as a project selection tool, since it requires a careful definition of problems addressed by the program.

A suggested outline for such a paper is presented in Figure 2. A paper following such an outline would require several hours to several days to prepare, depending on the size and complexity of the program and the amount of background knowledge possessed by the writer. The paper should be no longer than two typewritten pages to facilitate use by top management and elected officials.

The project selection papers should be submitted to the mayor, council, or other policy makers for deliberation and final issue selection. Depending on the level of effort committed by decision makers, several evaluations or analyses might be conducted simultaneously. The project selection papers should be transmitted to the study team leader or analyst for preparation of a project workplan (Task 2).

Alternative Selection procedures
The structured process outlined above can be followed for the first several rounds of projects or until the decision makers and department heads become sold on the value of evaluation and analysis. If a full-time evaluation and analysis unit has been established, the jurisdiction may wish to allow departments to bring projects directly to the attention of the evaluation staff. The evaluation director then selects the projects deemed to be the most fruitful, turning down or postponing only those less fruitful projects that cannot be covered with existing resources. Such a less-formal procedure will foster a greater feeling of cooperation and confidence in the operating departments, since they request assistance themselves rather than have evaluation and analysis imposed on them from outside.

As will be seen later, the proper working relationship between operating departments and the evaluation staff is very important in maximizing the benefits of evaluation and analysis. This less-formal approach would of course still allow the council, mayor, or manager as appropriate to mandate evaluation and analysis of politically important programs.

The experience of at least one jurisdiction that initially followed such an informal approach indicates that more structure may be needed to select programs as requests for evaluative and analytical help increase. Winston-Salem, North Carolina, has recently instituted a Management Information System Committee that screens evaluation requests as one of its functions. The committee, composed of the city manager and key department heads, meets monthly to decide which evaluation projects should be added to the work load of the evaluation staff and what priority each project should receive. This allows considerable flexibility in the screening and scheduling of the evaluation work load.

Figure 2. PROJECT SELECTION PAPER. Below is a suggested content outline for a document that can provide additional information in selecting evaluation and analysis projects. A separate paper should be prepared for each candidate program. The completed paper should be no longer than 2 typewritten pages.

TASK 2-PREPARING A WORKPLAN
The preparation of a workplan is a very important Ask in both evaluation and analysis. In addition to providing guidance for the project team, the workplan serves as a vehicle for assuring that all concerned understand the precise scope of the project before work begins. Six steps are involved in workplan preparation: (1) defining the problem that the program addresses, (2) orienting the analyst, (3) establishing project scope, (4) identifying work steps, (5) estimating time and manpower requirements, and (6) approving the workplan.

It is not absolutely necessary to prepare a workplan as detailed as the one described in this task, but it is vital that consensus on the scope of the project be obtained before work begins. It is also important that the analyst and local decision makers recognize and make full use of the workplan preparation and approval process to discuss tradeoffs between completion date, scope, personnel and dollar resources, and technical precision of the project.

Some of the initial for the evaluation or analysis is actually begun during workplan preparation. The analyst must become somewhat familiar with the organization, mission, and background of the program agency, as well as conduct a preliminary survey of the data routinely kept by the agency in order to prepare an accurate workplan. The time spent on such activities during workplan preparation is not wasted; in fact, the time required to conduct the project is usually reduced by at least as much time as was devoted to those activities.

Step 1-Defining the Problems Addressed by the program
The first step in preparing the workplan is to clearly define the problems addressed by the program. This step is often overlooked since the problems often seem obvious. Experience has shown, however, that original problem statements are often vague, incomplete, or misleading. For example. a city began an analytical study to determine how to substantially increase productivity in records microfilming program because the capacity of existing storage facilities was being taxed. The problem was initially perceived as being one of selecting, purchasing, and installing the most appropriate additional microfilm hardware.

Careful examination of the problem soon revealed that the actual problem was much larger and more complex than originally stated. The city did not know what information it was storing, what information needed to be stored, or what frequency or mode of access to the stored information was required. Analysts eventually determined that a lengthy, detailed records management program study was needed to solve the problem that was initially thought to be a microfilm hardware problem. Obviously, such a discovery can have a significant impact on workplan preparation. The best way to clarify problems is to discuss them with elected officials, management, the program agency head, and several program staff members.

Step 2-Orienting the Analyst
Once the basic problems addressed by the program have been defined, the analyst's second step is to become familiar with the program. As a matter of courtesy, the analyst and/or team leader should always begin by contacting the head of the program agency, stating the purpose and asking what procedure the department head prefers the analyst to use in contacting operation personnel. It is appropriate to suggest that the department head name someone from the agency to serve as agency liaison on the project team.

The orientation process may take from one day to several months, depending on the size of the agency, the size and background of the study team, and the complexity of the study. Depending upon the issues and the situation, the analyst might be looking for preliminary answers to general questions such as those presented in Figure 3.

Figure 3. ORIENTATION QUESTIONS. The analyst may wish to seek the answers to questions such as those below to become oriented to the program agency being evaluated or analyzed.

Step 3- Establishing Project Scope
After the analyst becomes familiar with the program, the third step is to establish a scope for the proposed program evaluation or analysis. One approach to this task is to formulate several very specific questions to be answered by the study. Another approach is to establish specific objectives for the evaluation or analysis project. Obviously, these questions or objectives relate directly to what top management needs to know to make its decisions. The project scope should be stated in writing.

It is important to establish the scope before work begins so that all concerned will understand precisely what issues will be addressed and to what level of detail. Establishing this information will forestall after-the-fact misunderstandings about what the study was supposed to accomplish.

Initial guidance on defining the scope should come from the program selection process. If the project was selected by elected officials or top management, the analyst must try to determine the decisions they hope to make about the program and then tailor the scope to provide the kind of information needed to support such decisions This may not be easy ant may involve some educated guesses on the part of the analyst, even after interviewing elected officials and top management. The accuracy of these guesses will be verified by presenting the completed workplan to appropriate officials for approval before actual work begins (Step 6).

Depending on local circumstances, an effective project scope can range from broad to detailed. For example, a jurisdiction that does not have goals and objectives established for its programs might undertake a broad study to accomplish the following :

1. Identify program goals and objectives.
2. Develop evaluation criteria.
3. Define subprograms and activities currently performed.
4. Define existing service levels for all activities.
5. Establish target values for evaluation criteria.

Once an evaluation of this scope has been accomplished, subsequent evaluations and analyses of program activities can be more rigorous. In all cases the statement of project scope should clearly indicate which activities are to be examined and in what level of detail.

An example of a more detailed scope can be seen in the following excerpt from the San Diego County evaluation of the County general relief welfare program:
1. Effectiveness Questions and Concerns

1. What are the demographic differences between persons applying for general relief and persons receiving general relief?
2. Is the current general relief grant sufficient to meet the basic needs of the general relief clients?
3. How effective are the general relief program employment and job training activities in assisting client to prepare for and find employment?
4. How does San Diego County's general relief grant level and eligibility criteria compare with those of other jurisdictions?

2. Efficiency Questions and Concerns

1. Is the program effective in screening out ineligible applicants??
2. Is the program efficient in servicing eligible applicants?
3. Are operating procedures as efficient as possible (i.e., are there areas of work duplication, do bottlenecks occur, are procedures standardized?)?
4. Are current program staff-to-client ratios optimal?
5. Is the current program's organization structured to operate at optimum efficiency?

Step 4 - Identify Elements of the Workplan
The fourth step in this task is for the analyst to determine the work that will be necessary to complete the study. While there may be slight variation, the following major elements will always be included in the workplan:

1. Orient the agency to the evaluation or analysis effort.
2. Identify program objectives and evaluation criteria
3. Prepare a flow chart or schematic diagram of all activities of an ongoing program.
4. In program evaluation, select the evaluation design. In program analysis, generate program alternatives
5. Determine data requirements
6. Determine data availability.
7. Gather data.
8. Analyze data.
9. Prepare a draft report
10. Review the draft report with the agency and affected community groups.
11. Prepare the final report.
12. Present results of the study.

Step 5 - Estimating time and Manpower Requirements
The fifth step is to estimate how much time and manpower will be required to perform each of the workplan elements identified in Step 4.

Very little concrete guidance can be given in estimating the time or effort for the above tasks because the time will vary considerably according to program size and complexity and the manpower available to the project team. Each task can take from one day to severe, months, depending upon the specific circumstances. Analysts should however, be able to develop a realistic workplan by reading through the entire process presented sensed in this Technical Guide so that they have a clear understanding of what is involved in each of the elements outlined above and can make careful estimates of the specific situation. The value of preparing a workplan should be increasingly obvious, especially for first time evaluations and analyses. Analysts will undoubtedly begin to get a feel for the time and effort involved as they gain experience, but a workplan should still be prepared to ensure that nothing is overlooked.

One extremely important precaution about making firm time commitments for the study. If existing data are inaccurate, the analyst may have to formulate a plan for developing data from scratch. This problem will be addressed in greater detail in Task 6, but the analyst should be aware now than this problem could occur since it obviously can seriously affect the work schedule.

Step 6-Approving the workplan
The information developed in the preceding five steps should be incorporated into a single written document. A suggested outline for an evaluation or analysis workplan is presented in Figure 4. The completed workplan should be submitted to elected officials or top management as well as to the program agency, so that all parties understand and agree on the scope of the study before it begins. It is during this step of the process that tradeoffs with respect to scope, time span, manpower, and technical rigor are normally made.

Management or elected officials may be willing to sacrifice the answer to one or more effectiveness or efficiency questions in order to have the project completed at an earlier date: or they may wish to apply additional resources to obtain more information. Whatever changes are made, it is important that the agreed upon scope be committed to writing to avoid after the fact misunderstandings about what the project was supposed to accomplish.

Due to unforeseen circumstances, it may be necessary to alter the scope of the project after work commences. For example, the project team may discover that the data needed to complete a part of the project are not available, or that some available data are inaccurate. When circumstances dictate a change in project scope, it is important that the team leader discuss the problem with management and/or elected officials to arrive at a new understanding of what is to be accomplished. Local officials may be satisfied with the reduced scope, or they may mandate that the project be postponed until the difficulties can be resolved, or they may wish the project terminated.

Whatever the case, the project team should not make unilateral decisions that change the scope of the project. The project workplan plan must represent a contract with management and/or elected officials if evaluation and analysis are to pin or retain credibility with local decision makers.

Figure 4. WORKPLAN OUTLINE. This outline covers the topics suggested for either a program evaluation or analysis workplan.

TASK 3-COMPLETING THE PROJECT TEAM
Preparation of the workplan should give the analyst a fairly clear idea of the skills required to conduct the study. It is not necessary to have a central staff with a wide range of specialized skills, since personnel with needed expertise can be borrowed on a temporary, part-time basis from other departments or can be recruited from outside government on a volunteer basis.

This task comprises three steps: (1) Identifying skills, (2) obtaining the appropriate personnel, and (3) briefing the project team.

Step 1- Identifying the Required Skills
The analyst should study the methodology section of the project workplan and make a list of the skills required to conduct the various steps. While the specifics will vary somewhat from one project to another, several basic skills are required for all studies. These skills will usually be divided among a team leader, an agency liaison, and the analyst, as specified in Chapter I.

In addition, many studies will require some form of technical expertise that neither the team leader nor the analyst possesses. Budget analysts or accountants may be needed to analyze the cost of performing certain tasks so that the costs can be related to results. Statistical expertise may be required to analyze evaluation criteria values. Specific program-related expertise is often required.

A good way for the analyst to approach this step is to prepare brief (two or three sentences) written descriptions of each of the perspectives required. These descriptions can then be used to determine which specific individuals should be added to the project team. It is possible that one person can provide two or more of the perspectives contained in the descriptions. Here, a note of caution to the analyst-try to avoid predetermining the solution by your selection of technical experts.

For example, the use of computer specialists on a program analysis project team will virtually guarantee that the alternatives proposed and selected will involve the use of a computer. While this may indeed turn out to be the best alternative, the analyst must still be careful not to inadvertently narrow the range of options by the selection of too narrow a set of perspectives.

Step 2. Obtaining Appropriate Personnel
Once the necessary skills have been identified the next step is to find individuals with those skills. Most of the personnel will be available within the government, either in the program agency or in staff agencies such as finance or planning. Most of the specific types of program-related expertise will have to come from within the program agency itself. If, for example, you are studying a housing rehabilitation program, then code enforcement and housing rehabilitation within the program agency should provide most of the expert rise. However, it may still be possible to get outside assistance from a civil engineer in the public works department or from housing specialists in State or Federal agencies.

In addition to these personnel, it may be necessary in some cases to augment government personnel with outside resource people such as a consultant to help familiarize the team with a specialized subject matter or to perform a specific task as a complex statistical analysis. Although it is possible to contract with a consulting firm to perform an entire evaluation or analysis, this approach provides a jurisdiction with little internal capacity for evaluation and analysis. Furthermore, a jurisdiction that has not performed several studies itself will probably experience difficulty in communicating with and providing guidance to a consultant.

An additional possible source of expertise is a local college or university. Academic personnel have often been used to help design a questionnaire or to conduct a survey to measure client perceptions of program performance. Statistical and industrial engineering skills, as well as business skills, can often be found in local colleges. However, it is best to use academic personnel in narrowly defined roles rather than to allow them to conduct the entire study since some academicians have a tendency to concentrate on aspects that are of interest to themselves rather than on the practical needs of local decision makers.

Some communities have made good use of local talent on study groups. and there is no reason why this source of expertise cannot be tapped for evaluation and analysis studies. Specific technical expertise is sometimes provided by local firms as a community service. There are even some instances in which a local firm has made a standing agreement to provide this expertise on an as-needed basis.

Such arrangements work well as long as the company keeps a firm commitment to provide the services when they are needed by the government rather than when persons can be spared by the company. In fact, the question of time availability is the biggest drawback to using personnel resources other than government employees or contractor. Community volunteer function well if given adequate guidance by the government and if they provide their services in the appropriate time frame. Such volunteers usually have full-time jobs and they can devote only evening and weekend time to the project. Ideally. the government may be able to find retired people with the necessary skills who are willing to volunteer their time to fit the study schedule. Several jurisdictions have found it useful to maintain community volunteer talent banks listing available expertise and experience.

In conducting program analysis it is important to create a project team that will generate new innovative, and practical approaches to program alternatives. A special effort should be made to include people who bring a fresh perspective to the problem-people who are known to be open-minded and creative. New employees who come from private industry or another jurisdiction
might provide such a perspective. If all the team members are intimately familiar with the program area, they may be too close to the problems and traditional approaches to offer useful alternatives.

Step 3 - Briefing the Project Team
After specific personnel have been located the team leader should call a meeting of the project team. Each member should be provided with a copy of the project workplan. The team leader should discuss the study objectives and methodology and explain what pan each person will have in the study. Questions about timing and possible conflicts with other duties should be ironed out at this point. Often, members of the project team can make suggestions to improve the workplan. Such suggestions should be incorporated as long as they do not change the scope of the project or adversely impact the completion date. If either of these conditions occur, the team leader must seek approval of the changes from appropriate local decision makers.

Specific assessment and discussion should also take place regarding the impact of project work on the regular tasks and responsibilities of the team members. One way to ensure willing cooperation of team members is to assure them that provisions will be made to get their regular work done if a conflict should arise. Obviously, such assurances will not always be possible, but caution should be used to make sure that evaluation and analysis project work does not become a burdensome extra responsibility for team member.

TASK 4- ESTABLISHlNG GOALS OBJECTIVES AND EVALUATION CRITERIA
With the project team selected, the next major step is to investigate the program's goals, objectives, and evaluation criteria. A program goal is a broad statement of intended accomplishments or a description of a general condition deemed desirable. Goal setting should be primarily the responsibility of elected officials and public administrators. A program objective is a specific, well defined and measurable condition that must be attained in order to accomplish a stated pal. Objective setting should be primarily the responsibility of public administrator and their staffs. or operational personnel. Evaluation criteria are the actual instruments used to measure progress toward objectives. Evaluation criteria are normally formulated by project personnel.

Ideally, all government programs should have goals and objectives explicitly stated as part of the program planning process. In reality, very few government programs have explicit, meaningful goal and objective statements. This poses a problem in program evaluation, since there is no clearly stated direction with which actual performance can be compared. Often Often and objectives are scattered throughout program documentation. In such cases, the analyst must identify goal and objective statements and phrase them clearly.

Many times, the program documentation contains much of the information to compose goals and objectives. In such instances, the evaluator is best advised to assist program personnel, management, and elected officials to establish meaningful goals and objectives to be used in future evaluations and to guide program activities. In the latter case, evaluative effort should not be as rigorous and critical as in cases where goals and objectives clearly exist and are recognized by the program staff.

Since program analysis concerns future program activities, it is always appropriate to establish goals and objectives for a new program and redefine them for existing programs. It will usually be necessary to establish evaluation criteria for all programs, even those that already have effective goals and objectives.

The above discussion should make it obvious that the analyst's role in goal and objective setting will not be the same in every project. The analyst may need to do nothing more than make sure that the set of clearly stated goals and objectives is the most current available or he may need to participate in a full-blown goal- and objective-setting process. In most cases, he will assemble and restate goals and objectives drawn from program documentation and other sources. The steps outlined in this task are designed to provide some guidance to the analyst for each of the cases discusses above. The four major steps are (1) Review program material, (2) define program goals, (3) define objectives, and (4) establish performance criteria.

Step 1-Reviewing Program Material
The first task for the analyst is to review source material relating to the program and to get a general idea of the overall purpose behind the program. Some suggested sources for leads to program goals ant objectives are:

Budget document-The program agency's annual budget request and justification will often have statements of program goals and objectives. Such statements may appear under other names, such as program purpose or program scope.

Program personnel-Perhaps the most important source is the program agency personnel themselves. Their knowledge of program operations and history as well as access to records containing policy memoranda etc., make them the prime source. This activity should be the first concern of the analyst during the agency orientation task of either evaluation or analysis.

Enabling legislation-Many jurisdictions include an indication in their budgets as to the legal basis for tine program. Examination of the charter or applicable statutes will frequently give insights as to the intent and scope of the program.

Policy messages of elected officials-Such documents as"state of-the-city" messages frequently provide insight into what elected officials perceive to be the functions of various key programs.

Expressions made by legislators, citizen groups or individual citizens at hearings before a local council or in the press-Testimony before committees considering a bill to create, expand, abolish, or evaluate a program may contain useful discussions of both explicit and implied objectives.

Minutes of boards and commissions-Many local government programs have some policy-wide oversight body to give overall direction. The minutes or annual reports of such bodies will frequently provide insights into the goals of the program

Study of these sources may revert clear goal and objective statements that fit the characteristics listed in Steps 2 and 3 in which case the analyst should proceed to develop evaluation criteria as outlined in Step 4.

Step 2- Defining Program Goals
A goal statement should describe in general terms something to be accomplished. A goal statement should be written with several factors in mind:

A goal covers long time spans relative to objectives:
A goal can be either intangible or tangible:
A goal should be people- or community oriented:
A goal should not predetermine the details of program activities;
A goal should reflect the direction daired by the general public, elected officials, and public administrators- not staff personnel: and
A goal should be expressed as a desired outcome or condition to be achieved rather than as an action or process.

Examples of program goal statements include:
Traffic Engineering-Safe, efficient, and convenient movement of people and goods.
Fire Department-The highest level of public physical safety with the resources available.
Economic Development-Economic opportunities for persons who have not enjoyed economic equality.

Many programs have several related activities, each of which may have one or more subgoals. For example, a fire department will usually have separate organizational activities for fire suppression, training, fire prevention, ambulance services, and administrative support. Subgoals for these activities might be:

Fire Suppression-Rapid suppression of fire.
Fire Prevention-Reduction in incidence of fires.
Training-More effective and efficient fire department personnel.

As the examples show the subgoals support the general program goal and address a segment of the program mission.

If the analyst cannot find or derive goal statements such as those listed above, it will be necessary to establish goals from scratch. Ideally, goal setting should be directed by the chief administrative officer of a jurisdiction with the direct input of elected officials. As a matter of practicality the analyst may find it more efficient to draft goal Statements in conjunction with the agency head and present these draft goals to the chief administrator and elected officials to stimulate discussion.

The chief administrator and elected officials can be expected to take a greater interest in the goal-setting process as they begin to grasp the importance of goals and objectives in the management of government programs. This means that the goal-setting discussions may be relatively brief for the first several programs studied but may increase in length and intensity for subsequent programs. The analyst should keep this factor in mind, as it can affect the length of time necessary to conduct a study ant therefore should influence workplan preparation.

Step 3-Defining Program Objectives
Once consensus has been reached on the more general goal statement, the analyst's next job is to review, redefine, or define specific and measurable objectives. As a matter of practicality, much of the groundwork for the formulation of objectives will have been done during the development of goal statements. The analyst may even wish to develop the goals and objectives at the same time and to present both to the agency head, top management, and elected officials through the procedure presented above. This consolidated effort will work best when there appears to the analyst to be little question or disagreement on the goal statements as drafted. However, when the program goals seem to be controversial, the analyst should make sure that the goals are agreed upon before attempting to develop objectives.

An objective should describe something to be accomplished in specific, well-defined and measurable terms. Objectives are derived from goals by, first, formulating a strategy for reaching the goal and, second, establishing one or more objectives necessary to make this strategy work. In the case of the fire department example, the subgoals represent an expression of the chosen strategy. That is, in order to achieve the overall program goal of maintaining public physical safety, the strategic elements of fire prevention, suppression, training, and medical assistance are necessary. Specific objectives are then developed for each subgoal.

An objective should be written with these factors in mind:

An objective is something that must be accomplished in order to achieve a goal:
An objective is not a program or project function, activity, task, or step:
An objective should not predetermine in any fashion the solution to a problem or way to do something:
An objective should relate to the needs of groups of citizens or the community as a whole:
An objective should explicitly consider unintended or negative effects:
An objective should be achievable within a specific time frame: and
An objective should be expressed as a desired outcome or condition to be achieved rather than as an action or process.

Figure 5 presents Several examples of effective objectives that follow the above guidelines. If effective objectives cannot be found in or derived from program documentation the analyst will have to develop them from scratch.

In developing objectives, the analyst should take into consideration the effects the objectives have on various population or clientele groups. Different groups may be affected by a program in different degrees. It is important to identify such groups and to collect data reflecting program impacts on them. An "average" crime rate or "average" family income for a jurisdiction will not adequately reflect possible major differences that may exist among segments of the population. The following points should be considered:

Each program will have some groups that are intended beneficiaries; i.e., clients of the service.
Each program is likely to significantly affect certain other groups that are not intended beneficiaries. These effects may be detrimental or beneficial.
The citizens of the community or state considered as a whole often make up a category that should be explicitly identified.
In some cases, future citizens may be an important group to consider explicitly because their interests are closely related to the program.

Figure 5. EFFECTIVE OBJECTIVES. Below are examples of program objectives determined according to me guidelines presented in Step 3.

The analyst will find that the preparation of a clientele group profile will help to develop objectives that are people-oriented by creating a picture of the group that is the target for the program. Figure 6 contains a suggested list of characteristics for inclusion in such a profile. Most of this information can be obtained from census data. Each program is likely to have at least some unique clientele groupings.

It is important that the program objectives be developed in close cooperation with program personnel, especially for programs of long standing, because the analyst is developing the standards against which programs will be measured and it is only fair that everyone agree on the essentials at the outset. Also, should an analyst attempt to develop objectives from the other cited sources alone, it is entirely possible that the analyst might develop a set of obsolete objectives.

The objectives of a program frequently shift with the passage of time; the longer a program has been in operation, the greater the chances that such a change has occurred. The objectives used should be those that the program agency personnel agree are current.

The analyst should get most of the information needed to formulate objectives by interviewing program agency personnel. In addition to the agency head and appropriate division directors, the analyst should also interview first-line supervisors and program workers to learn their perspective and to find out whether they are familiar with existing objectives. While analysts should develop their own specific questions for the interviews, Figure 7 lists some suggested questions that can form the basis for an effective interview.

Figure 6. CLIENTELE GROUP CLASSIFICATION. The analyst should know what population or clientele are affected by program goals and objectives. This classification scheme offers some assistance in developing profiles on population or clientele groups.

The analyst should always have specific questions composed in advance for these interviews. This helps to assure that all necessary information is obtained and to avoid wasting the time of program personnel with inefficient often offensive "fishing expeditions." The analyst should, of course, be prepared to diverge along a promising line of inquiry that emerges during the interview.

The analyst's list of program objectives should be presented to the agency head for discussion and approval before being transmitted to top management. While some jurisdictions may wish to do so, it isn't necessary to have objectives approved by elected officials. Many public administrators feel that overall guidance by elected officials in the form of goal statements is an adequate level of involvement.

An implicit assumption in program evaluation is that the objectives are practical. If objectives are too easy to attain they offer no real incentive for the program staff to strive for greater ach~ievement. It is probably best to set objectives that make program personnel reach a bit. On the other hand, care must be taken not to set objectives that are too ambitious lest employees become frustrated by unreasonable performance targets and cease trying their best.

Analysis of an ongoing program can raise some special problems in establishing objectives. Since the objectives of most programs shift over time, the analyst must be careful not to accept "prepackaged" objectives set down when the program started without some investigation of their relevance. Since analysis is intended to shape the future conduct of program activities, objectives that describe past practices may hamper a thorough search for alternatives. The analyst should make sure that the objectives, criteria, and clientele groups are what local policy makers intend them to be for future program operations.

Figure 7. SUGGESTED INTERVIEW QUESTIONS. These questions should help the analyst to gather information about a program's objectives during interviews with agency personnel.

Step 4 - Establishing Evaluation Criteria
Once objectives have been adopted, evaluation criteria can be formulated. Evaluation criteria are used directly to measure progress toward objectives. They are like corollaries to the objectives that answer the question. How can we measure progress toward this objective? The analyst will frequently find that there is more than one evaluation criterion for measuring progress toward each objective. As a general rule, it is always better to have too many criteria than not enough.

Evaluation criteria should be established with several factors in mind:

Criteria should be service- or people-oriented;
Criteria should reflect explicit performance targets;
Criteria should nor be subjective;
Criteria should indicate relative accomplishment or degrees;
Criteria should cover all important attributes or aspects of the program; and
Criteria should be acceptable to workers supervisors, and managers.

Evaluation criteria should be identified without initial concern about how or whether they can be measured. There are often ways to at least partially measure the more qualitative or subjective criteria by using ratings rankings and other procedures. For example, at first glance citizen perception may appear difficult or impossible to measure, but a sample survey can usually supply the needed data. Citizen perceptions are important criteria for virtually every government program since they measure the degree of public satisfaction with government service delivery. Appendix A to this technical Guide contains further material on surveys. Figure 8 presents several examples of evaluation criteria incorporating the above principles.

Figure 8. EVALUATION CRITERIA. This example shows evaluation criteria for the objectives developed in Figure 5. The criteria adhere to the guidelines presented in Step 4.

As used here, there is no right or wrong value for criteria. Fire deaths per 1,000 population can be compared with figures from other jurisdictions and national averages, but no accepted standard exists. Evaluation criteria are intended only as quantifiable indicators upon which to base judgments; the criteria themselves do not provide any answers. Thus, in order to make criteria useful from a management perspective, the jurisdiction must set performance targets for each one.

Unless such targets are currently being set as part of the management process, first-time evaluations will have to depend more on value judgments than will later evaluations. Part of the analyst's job should be to establish performance targets for each evaluation criterion in coorporation with program agency personnel, so that these personnel will have more precise direction and management will have more specific performance indicators.

POSTCRIPT
The preparatory tasks discussed in this chapter are common to both program evaluation and analysis. Chapter III "Program Evaluation." discusses those tasks that specifically apply to evaluation. Chapter IV. "Program Analysis" discussed those tasks that specifically apply to analysis. Chapter V. "Communication and Follow-Up." integrates these two discussions.

CHAPTER THREE Program Evaluation

INSERT DIAGRAM HERE

This chapter describes steps necessary to complete a program evaluation, assuming that the program has been selected, the project scope defined, the project team selected, and the goals, objectives, and evaluation criteria formulated. Task 5 presents three specific designs that are practical for state and local government and discussed the situation for which each design is best suited. Task 6 involves collection of the data necessary for evaluation. Detailed discussions address such important concerns as determining data availability collecting the data, and verifying the accuracy of the data. Task 7 covers the steps necessary to examine the data and draw conclusions about program performance. Information on preparing a written report and implementing the evaluation recommendations is included in Chapter V. For a discussion of the principles of evaluation, refer to Appendix A.

TASK 5- SELECTING AN EVALUATION DESIGN
All evaluations are basically some form of comparison. Whether comparing a group of people who received special treatment (such as in a drug rehabilitation program with a similar group who did not receive treatment, or comparing the actual accomplishments of a program with its performance objectives comparison is still the key to evaluation. An evaluation design provides the framework for making comparisons. Researchers have developed many different evaluation designs - (1) planned vs. actual, (2) time trend, and (3) before vs. after program comparison. The discussion of each design includes a description, step-by-step procedures, application considerations, and tips and cautions regarding use.

Design #1: Planned vs. Actual Performance
This design compares the actual program performance for a given time period with planned performance. It can be used for virtually all ongoing programs that have not been consciously changed during the evaluation period. The planned vs. actual design has the advantage of providing a natural lead-in to program analysis, since areas of substandard performance are identified by the evaluation. The procedural steps for the use of this design are:

1. Set performance targets for each evaluation criterion.
2. Collect data on criteria for the evaluation period.
3. Compare the actual data with the targets.
4. Estimate the effects of, or at least identify, any nonprogram factors that might have an impact on evaluation criteria.

Most local government requests for evaluation information concern ongoing operations. Decision makers want to know the effectiveness of street patching, sanitation, or fire prevention programs. The question this type of evaluation asks might be phrased. How well are we performing basic services? This need for information can be contrasted with the desire to know the effectiveness of special or experimental programs, such as drug or alcohol treatment programs. In this case, the question might be phrased. Is this program worth continuing? This distinction is important, as it underlines the need to tailor the evaluation to the specific needs of those requesting evaluation.

In this design, performance objectives for a given time period are compared against actual performance for the same time period. In order to make use of this very basic design, the evaluators must be able to identify objectives for the program and then measure progress toward them by use of the evaluation criteria. If performance targets have been set previously, this design will give precise and useful results. However, if performance targets have not been previously established, the evaluation must be handled differently. The analyst can still establish, the evaluation must be handled differently. The analysis can still establish performance targets for the past time periods being evaluated, but care should be taken not to make the first-year evaluations seem punitive because it is not fair to judge a department head's managerial ability against a set of criteria he did not know existed at the time of program performance.

There are several purposes for using this design for first-round evaluations: (1) to get a general assessment of program effectiveness and efficiency. (2) to establish explicit performance targets for future time periods. and (3) to identify some specific program areas that need improvement.  Application of program analysis techniques for these purposes should improve future program operations.  Positive aspects of the program identified during evaluation should be highlighted as part of the written report to lessen the punitive or negative image that many people attach to program evaluation.

This design implicitly assumes that the targets set are reasonable. Targets that are too easy to reach do not challenge program personnel to provide true measures of accomplishment. Targets that are too high will discourage program personnel and may give management a distorted view of agency performance. Targets must be set with the participation and cooperation of program agency personnel as described in Chapter II.

Ideally, performance targets should be set through the use of work measurement procedures. Work measurement is a technique that allows equitable time standards to be established for many jobs. For additional information on the application of work measurement techniques to State and local government operations.
Contact:
    Subscriber Services,
    Public Technology, Inc.
    1140 Connecticut avenue, N.W.
    Washington, D.C. 20036
    (202) 452-7700

Design #2: Time Trend
This design measures the effects of a program change. Evaluation criteria are selected and data collected to establish past performance trends. These trends are then compared with conditions observed after the program change. The design differs from the previous design in that it does not require the establishment of performance targets but relies entirely on actual performance measures. This design is best used to evaluate a program change rather than overall program effectiveness. It can also be used to evaluate new programs aimed at changing specific conditions, if preprogram data on these conditions are available. The procedural steps for the use of this design are:

1. Collect data on each of the evaluation criteria for several measurement periods (years, quarters, or months) prior to the program change.
2. Collect data on each of the criteria for one or more measurement periods after the program change.
3. Using graphic techniques, compare data to see if values for the criteria show a divergence from the preprogram trends.
4. Identify and estimate the effects of any non program factors that might have an impact on evaluation criteria.

The thinking behind the design is quite simple. If a program has shown a reliable performance trend in the past, then it is possible to get a good indication of the success of a specific program change by observing if there is a significant change in the trend after implementation. The analyst must be careful about drawing conclusions regarding the efficiency of the program based on changes in the values of evaluation criteria. While improved program performance probably indicates a relative increase in efficiency (assuming constant personnel resources), the program may still be relatively inefficient when compared with work measurement standards. Without such standards for comparison the analyst can only judge apparent changes in efficiency in relation to past performance.

A good example of the use of this design in a local government setting can be found in an evaluation of a change in refuse collection vehicle routing. A jurisdiction might change its refuse vehicle routing, in an attempt to conserve gasoline by cutting down on wasted double trips on certain street, and to cut back on overtime by evening out crew work load. Data on gasoline usage and overtime hours worked are usually available. Graphic comparison of these figures for several previous years with the data for the same criteria after the new routing is implemented should give a clear indication of the success of the program change in reaching its objectives. Of course, a conclusion based on these data could be invalid if the jurisdiction began using a new type of collection vehicle at the time of the routing change, or if there were some other variable besides the routing change that was introduced.

Evaluation of a long-standing program generally requires data for at least four previous years to establish a statically valid trend. Data summarized by quarters may be taken in this situation to look for possible seasonal fluctuations, such as in a recreation program or a snow removal activity. Data should then be gathered for one or more intervals after program implementation. The more post-change data available, the more certain it is that the program effects are permanent and not just a short-term reaction.

An issue in the use of this design is the consistency of evaluation criteria and data sources over time. If the criteria require data normally gathered by the jurisdiction the analyst should attempt to make sure that there were no significant changes in the way the data were gathered or recorded during the past performance intervals.

For example, a number of years ago police departments across the country began adopting a standard FBI classification system for reporting crimes. The differences in definitions and reporting procedures caused radical differences in some jurisdictions statistics for certain types of crimes. While most changes of this nature are less dramatic the analyst must make sure that data definitions, remain constant during the intervals evaluated. The most likely sources of problems will be programs for which, or all, of the data is composed of subjective ratings.

If the program data do not how a clear trend before the program change, then it may be possible to check the results of the evaluation using a nonequivalent control group. This approach was used in the evaluation of a Connecticut highway-speed crack down program. Since precrackdown data on automobile deaths were unstable, the evaluators could not be sure that the crackdown was responsible for the reduction in traffic fatalities. Data were collected from adjoining states for the same time period and plotted on the same graph. When the adjoining state data showed no equivalent decrease in fatalities evaluators had much greater confidence in their in their conclusion. Figure 9 shows the graphic display of the data for the Connecticut example.

Figure 9. TRAFFIC FATALITIES TIME TREND COMPARISON.
The graph, which displays traffic fatality data for a number of years, was constructed to evaluate the effectiveness of speeding crackdown program initialed by the state of Connecticut in 1955. Since the data did not establish a clear trend prior to the crackdown program, the evaluators could not be sure that the program was responsible for the reduction in traffic deaths. Addition of data from adjoining states indicated that the program was probably responsible for the reduction in traffic deaths. (Source: Campbell, Donald T., and H.Lawrence Ross. Law and Society Review. Vol.III, No. 1, August, 1968)

INSERT FIGURE 9 HERE

Design #3: Before program vs. After Program
This design consists of measuring criteria values just prior to the implementation of a program and then obtaining values for the same criteria after implementation or completion of the program. This design does not seek to establish a trend for the criteria but merely to take a "snapshot" of conditions before and after a specific change. Before vs. after works best to evaluate a program of short duration and limited scope. This design is appropriate when conditions in the program have been stable for some time and are expected to remain stable in the future unless altered by the program initiative. It is generally more effective if the evaluation can be planned prior to implementation of the program change in case special data are required on preprogram conditions. The procedural steps for the use of this design are:

1. Collect data on the criteria reflecting conditions prior to the program's introduction.
2. Collect data on the value of the criteria immediately after program completion or an appropriate period after program introduction.
3. Identify and estimate the effects of any nonprogram factors that might have an impact on the evaluation criteria.

This design was used in the evaluation of a special intensive street cleaning program implemented in Washington, D.C. The program extended over a nine-week period. No other major changes were expected that would affect the postprogram values of the evaluation criteria. Neighborhood cleanliness was measured just before and just after the program using a visual inspection procedure and a citizen survey. The "before" program conditions were believed to be typical and not of a seasonal nature. Since the data needed to evaluate the program were not normally available, the visual inspection procedure had to be devised before the program was initiated so that preprogram data could be collected. This evaluation is fully documented in the following publication:

How clean is our city: A Guide for Measuring the Effectiveness of Solid Waste Collection         Activities by Louis H. Blair and Alfred I. Swartz. The Urban Institute. Washington, D.C.,           1972.

The before vs. after design assumes that the values for the evaluation criteria just before program initiation accurately reflect preprogram conditions: For this reason, the analyst must be careful to avoid using this design for a program with significant seasonal fluctuations or at least to compensate for the fluctuations. This is the simplest of the three designs and one that is currently in use by some State and local governments: however it is also the design that has to be used with the most caution. The design itself provides no means to distinguish nonprogram factors causing the differences, or lack of differences, in the pre and postprogram values of the criteria. Consequently, the analyst must take considerable care to identify possible nonprogram influences to protect the validity of the evaluation.

A more detailed discussion of this problem will be presented in the data evaluation section of this chapter (Task 8). The validity and credibility of the evaluation can be enhanced if the design ???? in conjunction with the time trend design. Before vs. after program comparison should be used by itself only as a last resort.

TASK 6 - DATA COLLECTION
The sixth task in the evaluation process is usually the most time-consuming and expensive -collecting the data needed to conduct the evaluation. There are four major steps in this task: (1) Identify the necessary data, (2) determine data availability, (3) collect existing data, and (4) verify the accuracy of the data.

Step 1- Identifying the Data
Identifying the data involves determining what statistics or indicators are required to measure the criteria identified earlier in the evaluation process. In many cases, the criteria themselves will be statistical measures. An illustration of this can be seen using the example of fire service criteria presented in Chapter II where the objective was a 50 percent increase in public awareness of fire dangers this year. The associated criteria were:
(a) Number of fire safety demonstrations performed.
(b) Public response to fire safety questionnaire.
(c) Number of fire hazards reported by the public.

Criteria (a ) and are specific statistical measures. Criterion (b) actually represents several statistics, since analysis of the survey questionnaire responses would probably yield separate figures on overall awareness of hazards, and on awareness of specific types of hazards. The analyst should study each criterion and ask what data would be needed to quantify the criterion. The analyst should not be concerned at this point with whether the data are easily available, since a thorough check of this point is the next step. If no single data source seems sufficient it may be necessary to identify several data sources that indirectly measure aspects of the criteria.

Step 2- Determining Data Availability
Once the analyst has determined what data are necessary, the second step is to determine how much are available. At least a preliminary survey of data availability should have been done during the project selection process to ensure the feasibility of the project. The methodology outlined here for determining data availability is considerably more detailed than that used for preliminary data surveys.

As a matter of practicality, for small evaluations the analyst may well determine data availability and begin collection at the same time. For most evaluations, it will be desirable to keep these steps separate since the absence of required data may cause the analyst to formulate a new strategy for data collection. It is not always necessary to obtain data for every criterion of a multiple-criteria objective. Using the fire prevention example, it would not be absolutely necessary to obtain data for all three of the criteria to be able to make a sound evaluation of program effectiveness. Each piece of data would provide an additional indicator of program effectiveness, but even without all of the data, valid conclusions could still be drawn about the program.

The analyst would be well advised to prepare a worksheet to use during data identification and collection. Such a worksheet would have the specific program objective at the top of the page, a list of the applicable criteria, and the data required to measure each. Additional information could be added indicating the availability and specific location of the data. A sample of such a form using the first protection example is shown in Figure 10. There are numerous types of data, but for our purposes only three will be discussed in detail: (1) existing records and statistics, (2) client perception surveys, and (3) special data collection techniques.

1. Existing Records and Statistics. The analyst should begin the data search by examining the existing records of the jurisdiction, starting with those of the program agency. The partially completed data availability worksheets with the data requirements identified should be shown to the program agency liaison person. The agency liaison should be able to determine quickly whether the agency has the required data and help the analyst figure out the best way to collect them.

Some evaluations will require data from several agencies since the program being evaluated involves more than one agency. For example, an evaluation of police effectiveness would probably require records from the courts. Obtaining the cooperation of several agencies can be quite difficult, especially if the evaluation effort does not affect or benefit them directly. Such situations require experience and skill on the part of the evaluation team leader and underscore the importance of top-level management support for the evaluation. It is the analyst's job to locate the necessary data, but the team leader's help will often be needed to gain access to them. Some general suggestions that may prove helpful in locating data are presented in Figure 11.

2. Client Perception Surveys. If the data identification process revealed a need for data on citizen perceptions of service delivery, the analyst will probably have to turn to sources other than existing records. The analyst should determine whether a survey has recently been completed either on a jurisdiction wide basis or in the specific program area of the evaluation. A survey conducted within the past year can be considered current. The analyst should examine the questions and responses to determine if the necessary data can be obtained from the survey. If the survey is too old or none has been conducted, then consideration must be given to initiating a new survey.

The experience of several jurisdictions that have used surveys in program analysis indicates that small, narrowly defined surveys yield the most productive results. For example, a short (3-6 questions) survey on citizen satisfaction with plastic trash bags, or a specific recreation program, yields results that are easy to interpret and involves relatively little effort to prepare and administer. Such surveys are also easier for citizens to respond to than a long survey that asks their perception on a wide range of government programs or issues. The analyst may be able to use statistics on citizen complaints or service requests to gauge citizen perceptions on specific services.

3. Special Data Collection Techniques. Once the data availability worksheet has been completed, the analyst must study it carefully to see if sufficient data are available to make a valid evaluation. This will be a particularly sensitive decision for objectives that can only be measured by one or two criteria. As a rule of thumb, data should be available on more than half of the criteria to ensure the validity of the evaluation. This rule of thumb must be used very cautiously for some criteria can be more vital to an evaluation than others; therefore, it also matters which criteria can be measured. To retain the community impact emphasis of the evaluation, it is necessary to give most weight to those criteria that measure citizen perceptions and direct effects on the program clientele groups.

Figure 10. DATA AVAILABILITY WORKSHEET. This is a suggested form to be prepared by the analyst to determine the availability of the data needed to conduct an evaluation. The information shown in the sample applies to the fire prevention example originally presented in Chapter II.

Figure 11. DATA LOCATION. Below are some suggested sources for the types of data often required for program evaluations.

The analysis described above will enable the analyst to determine whether the evaluation can be completed with the available data. There will be many instances when additional data will be necessary, and even more instances when additional data can add greatly to the validity and utility of the evaluation. This is a key decision point in an evaluation because, if some of the necessary data are lacking, a determination must be made whether to : (1) continue the evaluation with available data. (2) take the necessary time and effort to gather additional data from scratch, or (3) scrap the evaluation for lack of sufficient data.

If the first decision is reached the analyst may conclude that the lack of data requires limiting the scope of the evaluation. If this limitation is deemed significant by the team leader, then management and/or elected officials should be apprised of the specifics and asked to approve the new scope or to direct that additional data be generated to perform the evaluation as originally planned. If the analyst and team leader decide there is sufficient information and that it is impractical to gather the needed data, they should document their findings and present them to management.

When a reduced evaluation scope will not provide management with the type of information needed for decision making, it is necessary to generate data from scratch. The specific data should already have been identified, so that the first job should be to determine exactly how to go about collecting them. The analyst and team leader should decide whether the data can be collected: (1) by adding one or more data items to records routinely kept by the government. (2) by establishing new records and procedures, or (3) by using a special technique, such as a citizen survey. After this decision is made, the analyst should prepare a work plan that clearly states the specific data needed, the methodology to be employed, the time period to be covered, the calendar time required, the personnel time required, the estimate cost of data collection, and the impact the collection effort will have on the schedule for the evaluation as a whole. Once the impact on the project is known, the new work plan should be submitted to top management and elected officials for their consideration to ensure that all understand and approve the scope of the evaluation. The main point to keep in mind is that the need to collect data from scratch, whatever the reason, will have a significant impact on the duration and cost of the evaluation.

Step 3- Physically Collecting the Data
Once the data requirements have been identified and availability ascertained the team leader, analyst, and agency liaison person should meet to decide the best way actually to collect the data. As mentioned earlier, there are three main sources for evaluation data: (1) existing records and statistics, (2) client perception surveys, and (3) special data collection techniques.

1. Existing Records and Statistics. Data from existing records and statistics can usually be collected most efficiently by program agency personnel. The people who handle the records on a day-to-day basis are the extract the data quickly, since they do not need a "get acquainted" period. Using program agency personnel to do the time-consulting physical work of data collection can also free the evaluation analyst for involvement in several evaluation projects simultaneously.

Several things must be done, however, before program agency personnel can be turned loose on a data collection problem. First, the analyst should spot-check the accuracy of the data is important to accuracy. If the data are guesses or estimates by field personnel rather than "hard" data provided by program clients or some reliable form of measurement, then the validity of the data may be seriously questioned. A full discussion of data accuracy will be presented in Step 4 of this task.

Second, the analyst must provide the agency personnel with clear concise directions. The analyst must be able to tell agency personnel exactly what data are needed and the specific time span to be covered. The analyst should also provide worksheets for recording the data so that they are collected in a consistent manner. It may also be possible to lay out the worksheets so as to facilitate later analysis of the data. The analyst and the agency liaison person should meet with the employees who will be doing the actual data collection and discuss the reason for the data collection, the significance of the evaluation and the collection worksheet and special instructions. After answering questions, the analyst may find it beneficial to spend a few minutes working with employees as they put the worksheets to use for the first time.

It is also wise for the analyst to spot-check data accuracy during the data collection by examining a sample of the source records and comparing them with the worksheets prepared by the agency personnel. To facilitate these checks, analysts should have the agency personnel forward worksheets to them on an "as completed" basis, perhaps once a week.

Evaluations that require data from several agencies can cause the analyst difficulty in actually collecting the data and/or in coordinating the efforts of several groups. The example of a police effectiveness evaluation used earlier will help illustrate the point. To get a complete picture of police performance, data are likely to be needed from the prosecutor and/or court system on indictment and conviction rates, and perhaps accident statistics from the traffic engineering department. The prosecutor's office may not perceive any immediate benefit to that agency from the evaluation and therefore may be reluctant to take an active part in the project. The experience, tact, and political expertise of the evaluation team leader can often greatly improve cooperation. The team leader may be able to persuade the agency