Program Evaluationand Analysis

U.S. DEPARTMENT OF HOUSING AND URBAN DEVELOPMENT
Office of Policy Development and Research
Washington. D. C. 20410

PUBLIC TECHNOLOGY, INC.

Washington, D.C. 20036

under Contract #2237

CHAPTER III: PROGRAM EVALUATION ....
    Task 5-Selecting an Evaluation Design .......
        Design #1: Planned vs. Actual Performance .
        Design #2: Time Trend .................
        Design #3 Before Program vs. After Program
    Task 6-Data Collection ...................
        Step 1-Identifying the Data ...............
        Step 2-Determining Data Availability ......
            1. Existing Records and Statistics ........
            2. Client Perception Surveys ............
            3. Special Data Collection Techniques ....
        Step 3-Physically Collecting the Data .....
            1. Existing Records ant Statistics ........
            2. Client Perception Surveys ............
            3. Special Data Collection Techniques ....
        Step 4-Verifying the Accuracy of the Data ..
            1. Clerical Errors .....................
            2. Subjective Judgment Errors ...........
            3. Methodological Errors ...............
    Task 7-Synthesizing the Data ..............
        Step 1-Organizing the Data ..............
        Step 2-Making Comparisons .............
            Design _#1: Planned vs. Actual ..........
            Design #2 Time Trend ...............
            Design #3: Before Program vs. After Program .............
        Step 3-Checking the Validity of Evaluation Results ...............
        Step 4-Drawing Conclusions and Making Recommendations ........
    Postscript ................................

CHAPTER IV: PROGRAM ANALYSIS ........
    Task 8-Developing Program Alternatives .....
        Step 1-Generating Program Alternatives ....
        Step 2-Screening Alternatives ............
    Task 9-Data Collection ...................
        Step 1-Estimating Service Demand ........
        Step 2-Collecting Cost Data .............
        Step 3-Collecting Effectiveness Data ......
        Step 4-Collecting Feasibility Information ...
    Task 10-Synthesizing the Data .............
        Step 1-Organizing the Data ..............
        Step 2-Comparing the Data ..............
        Step 3-Drawing Conclusions .............

CHAPTER V: COMMUNICATION AND FOLLOW-UP..................
    Task 11 Communicating the Results ........
        Step 1-Preparing a Draft Report .........
        Step 2- Reviewing the Draft Report.........
        Step 3-Transmitting the Final Report ......
    Task 12-Follow-up Obligation .............
        Step 1-Forming an Implementation Team ..
        Step 2-Preparing an Implementation Workplan ......................
        Step 3-Monitoring Implementation Progress...
Summary ................................

APPENDIXES .............................

APPENDIX A: PRINCIPLES OF EVALUATION

APPENDIX B: SAMPLE SURVEYS ...........
    Potential Benefits .........................
    Selected Documentation ....................
    User Experience.............................
    Adaptation Considerations ..................

APPENDIX C: PROGRAM EVALUATION AND ANALYSIS STUDIES ....

FIGURES
    1 The Program Evaluation and Analysis Process
    2 Project Selection Paper ....................
    3 Orientation Questions .....................
    4 Workplan Outline ........................
    5 Effective Objectives .......................
    6 Clientele Group Classification ...............
    7 Suggested Interview Questions ..............
    8 Evaluation Criteria .......................
    9 Traffic Fatalities Time Trend Comparison .....
    10 Data Availability Worksheet ................
    11 Data Location ...........................
    12 Table Comparison of Evaluation Criteria .....
    13 Bar Chart Comparison of Evaluation Criteria .
    14 Graph Comparison of Evaluation Criteria .....
    15 Sources for Program Alternatives ............
    16 Brainstorming ...........................
    17 Alternative Summary .....................
    18 Sources for Demand Estimates .............
    19 Sample Cost Estimation Worksheet ..........
    20 Cost Estimation Principles .................
    21 Feasibility Guidelines .....................
    22 Cost Comparisons ........................
    23 Report Outline ...........................
    24 Suggested Review Questions ................
    25 San Diego County Report Review System .....
    26 San Diego County Office of Program Evaluation(OPE)
            Implementation Workplan.........
    27 City of San Diego Action Plan ..............
    28 Invalidity Factors........................
    29 Contacts: Surveys ........................

City, and State governments are finding it increasingly difficult to balance their budgets as rising costs and service demands continue to outstrip revenue. This situation increases the importance and difficulty of making resource allocation decisions for public programs. Such decisions can be considerably improved if local officials have better information on the effectiveness of existing and proposed programs.

To help provide this type of assistance to local and State governments. the Department of Housing and Urban Development. Office of Policy Development and Research has provided financial and technical support to Public Technology, Inc. (PTI). to develop this procedural guide on the we of program evaluation and program analysis. The guide is based on research funded by the HUD Office of Policy Development and Research. plus other work done by city. county, and State governments and by universities.

PTI's Technology Exchange Program. which is partially sponsored by the Office of Policy Development and Research's Division of Product Dissemination and Transfer. prepared this guide. The Technology Exchange engages in five basic activities that were employed in developing this guide:

Problem Description. The Program actively encourages the participation of city, county, and State government representatives in defining common, high, priority problems.
Information Collection. The Program works systematically to acquire information about innovative techniques applicable to the problems defined by State and local government practitioners.
Product Engineering. The Program works to translate, synthesize, and assemble promising ideas. concepts. methods. and procedures into war-oriented products.
Package Production. The Program then produces the communication tools needed to support transfer activities including an Executive Summary. a Management Report. Technical Reports. and a Training Package.
Information Dissemination and Technical Assistance. The Program works to actively encourage awareness. interest. evaluation. and adoption of the innovative techniques by city. county. and State governments.

The Technology Exchange Program does not engage in research and development. but rather works to pull together available information about existing techniques and to place this information in the hands of the State or local practitioner in an easy-to-use form. Thus. this guide does not purport to present new research: it is a synthesis of existing techniques. a guide to what is now known about program evaluation end program analysis.

This Technical Guide is part of a package of tools designed to assist State and local governments the complete communication package consists of four parts:

Program Evaluation and Analysis: An Executive Summary-A brief overview written for elected officials and chief executives that describes the rationale for using program evaluation and analysis to support the decision-making process and covers the package organization, content, and scope.
Program Evaluation and Analysis: A Management Report for State and Local Governments-A somewhat longer document. written for chief executives and senior administrators, that explains how to plan. organize. staff. implement. and monitor a program evaluation and review effort and outlines the existing techniques described in the Technical Guide.
Program Evaluation and Analysis: A Technical Guide For State and Local Governments--A procedural guide to the tools and techniques of program evaluation and analysis. written for administrators. analysts. and other staff who will directly conduct program evaluations and analyses. The Guide describes each task and step in detail. presents selected documentation, and provides options for adapting many steps to local conditions.
Program Evaluation and Analysis Training Program--Regional workshops for State and local government practitioners interested in implementing the concepts or techniques documented in the Management Report and Technical Guide. conducted by PTI staff personnel. plus on-site technical assistance to a limited number of jurisdictions.

State and local government officials may address inquiries concerning any of the above to:

Program Evaluation
Public Technology, Inc.
1140 Connecticut Avenue. N.W.
Washington. D. C. 20036
(202) 452-7700

This Guide was with financial assistance from the Division of Product Dissemination and Transfer of the Office of Policy Development and Research. The Division of Community Development and Management Research provided technical assistance.

The specifications for this package and the necessary policy. technical. and editorial guidance for its preparation were provided by a User Requirements Committee comprising the following members:

Leroy Whiting
( Chairman )Assistant Executive
Director Model Cities Chicago
Committee on Urban Opportunity
Chicago,Illinois

Richard Heiwig
Assistant City Manager
Dayton. Ohio

Thomas C Kelly
County Manager
Volusia County. Florida

Frank Kirk
Director of Local Government Affairs State of Illinois
Springfield, Illinois

Howard McMahan
Justin Industries. Inc.
Fort Worth Texas
Formerly City Manager.
Oklahoma City, Oklahoma

James E. Phelps
Budget and Research Director
Long Beach, California

Honorable Paul Soglin
Mayor
Madison, Wisconsin

William 0. Talley
City Manager
Anaheim, California

Dean Vanderbilt
Management Services Director
Dallas, Texas

Significant contributions to this Guide were also made by: Gary Brown. Director of Evaluation. Winston Salem. North Carolina: Camille Cates. Director of Intergovernmental Relations. Sunnyvale. California: Donald Fisk. Director. and Harold Kosakoff. Deputy Director. Office of Program Evaluation. San Diego County. California: Rack Fukuhara. International City Management Association; Charles E. Hill. Budget and Research Director. and Patrick Manion. Management Assistant. Phoenix. Arizona: Charles Kirchner. Special Assistant to the Director. Department of Local Government Affair Stan: of Illinois: Dave Knapp. Assistant Director of Financial Management. City of San Diego. California: Emmett Moten. Director, Policy Planning and Analysis. blew Orleans. Louisiana: John Niles. District of Columbia Office of Budget and Management Systems: John Tow. Management Analysis Supervisor. Long Beach, California: and Victor Swyden. Councilman. Kansas City. Missouri. A special word of acknowledgment is also due for Heather Aveilhe. Han Siegel and Harley Fitts of the Department of Housing and Urban Development for their active and positive participation in this project.

The material presented in this Guide is a staff synthesis of perspectives of User Requirements Committee members: materials taken from published research and other printed sources: and visits to leading-edge jurisdiction such as Kansas City. Missouri: Multnomah County, Oregon: New Orleans. Louisiana: Phoenix. Arizona: San Diego County. the City of San Diego. and Sunnyvale. California and Winston Salem. North Carolina

Some of the material in this Guide was developed from two publications of the Urban Institute:

Practical Program Evaluation for State ant Local Government Officials. by Harry Hatry. Richard E. Winnie and Donald M. Fisk. 1973.

Program Analysis for State and Local Governments by Harry Hatry. Louis Blair. Donald Fisk and Wayne Kimmel. 1976.

In some instances, excerpts from these documents have been used verbatim. or with some modification. We gratefully acknowledge the cooperation and guidance of Harry Hatry and Jane Woodward of the Urban Institute.

The PTI Technology Exchange Program is managed by C. Nelson Hoy. The staff members responsible for this document are: Ken Steil, Project Director: David Pearl and Walter Webb, Editors: and Marcia House, Project Secretary.

This Technical Guide is written or county manager. administrative assistants or interns and management or budget analysts responsible for conducting program evaluations or analyses. The guise assumes that such users have certain basic analytical skills but no specific experience with program evaluation and analysis. Within this context the Guide documents a process for addressing whether a particular government program is producing the desired result and for determining the most effective ant efficient way to allocate resources for improved future performance.

PROBLEM OVERVIEW
State and local governments are finding it increasingly difficult to balance their budgets because costs are increasing at a faster rate than revenues. This situation results in large pan from inflated costs for labor materials and equipment: public demands for expanded. improved. or additional services; employee demands for higher pay. shorter hours, or additional fringe benefit: lower tax revenues due to a depressed local economy: delays in real estate reassessments: and public resistance to higher taxes coupled with inflation.

Public administrator have several alternatives available to them in dealing with this squeeze: ( I ) improve effectiveness (2) improve efficiency. (3) decrease expenditures. (4) increase tax revenues or (5) combine two or more of these alternatives. Program evaluation and analysis are management-oriented tools designed to help public administrator with the first two alternatives -( 1) improve program effectiveness and (2) improve program efficiency.

TECHNOLOGY OVERVIEW
Program evaluation and analysis are two closely related technologies that can be used together or separately. Program evaluation measures program effectiveness. It involve reviewing objectives reviewing evaluation criteria. collecting data. synthesizing data, drafting reports and implementing result. Program analysis concern the design of new programs or redesign of old ones. It involves selecting programs. defining problems. defining or redefining objectives establishing evaluation criteria. generating program alternatives estimating costs ant effectiveness, studying feasibility drafting reports, ant implementing results.

The two technologies presented here are complementary. Evaluation provides information on the impact of existing government efforts and highlights areas that need improvement. Analysis can then be employed to help determine the most effective form for these improvements to take. A~. . program improvements have been implemented, evaluation is once again needed to assess their success and the cycle begins again. The cycle can also begin with program analysis used to determine the best way to institute a totally new program, followed by evaluation of the program activities. While evaluation ant analysis can be used separately the payoff is greatly increased by using both processes together.

For the purposes of this Guide. the two technologies can be understood as comprising a single process. Figure I presents a schematic diagram showing the major tasks in the program evaluation and analysis process. For simplicity and clarity. the diagram does no' attempt to show all of the possible decision points and feedback loops in the evaluation/analysis process. For example. difficulties in cats collection (Task 6 or 9) could easily force a revision of the project workplan (Task 2).

Program evaluation and analysis can be viewed as:

A structured process for staff personnel to follow in situations where the public administrator needs better information for decision making. A structured process for determining whether a partially program is producing desired results or effects. rational method for designing a new program or redesigning an old one to efficiently produce desired results or effects, and
A method for helping program manatee diagnose and make improvements in their operations.

Program evaluation ant analysis attempt to answer questions about the effectiveness and efficiency of government programs by identifying quantifiable indicators of program performance. Naturally, some programs (such as street maintenance, refuse collection and vehicle maintenance) lent themselves quite readily to this approach while others (such as recreation. library services. and welfare programs) are much more Difficult to quantify. Social action programs often have vague or ill-defined goals ant objectives that span many years. While evaluation ant analysis can provide meaningful information on such programs. these types of programs will usually require greater experience and technical capabilities than many jurisdictions will have when they first undertake program evaluation ant analysis. Successful evaluation ant analysis depends on:

The existence of or the ability to formulate meaningful goals objectives. ant evaluation criteria for public programs; The ability to measure program effectiveness through the collection and interpretation of data: The willingness of public officials to support the process by basing resource allocation decisions on the results of evaluation and analysis; and The commitment on the part of local officials to implement the recommendations of evaluation and analysis projects

POTENTIAL BENEFITS
Program evaluation and analysis offers several potentials benefits to city and country governments:

Elected officials, chief executives, ant public administrators benefit by having better information to aid their decision making, thus giving them greater confidence in those decisions:

Program agency personnel have the opportunity to ex- amine and influence the future direction of their program well as benefit from an outside view of the program that will help them gain a fresh perspective:

The public benefits by receiving more effective and efficient government services for their tax dollar: and all parties benefit from the valuable insights into government programs gained from examination of a program and its basic premises.

Figure 1. THE PROGRAM EVALUATION AND ANALYSIS PROCESS. The diagram below summarizes the 12 major tasks in the program evaluation and analysis process. Note that both processes have common beginning end ending tasks and that the process is cyclical. To maintain clarity. the diagram does not attempt to show all of the possible decision points and feedback loops in the process.

ORGANIZATIONAL REQUIREMENTS
A program evaluation or analysis project. as outlined in this Technical Guide can be undertaken by a single analyst working either full. or part-time. Jurisdictions new to evaluation ant analysis are not likely to assign more than one staff to an initial effort.

The analyst should be a generalist with analytical ability-analytical ability being defined as the ability to ask the right questions. A college degree in business administration. industrial engineering. public administration or any number of fields is very helpful but not essential. as is some experience in government operations background in statistics is valuable. The analyst should also be inquisitive, resourceful and open-minded enough to ask the type of questions that assume nothing is given. Above-average verbal and written communication, skills are very important. The analyst bean the responsibility of doing most of the actual work of data collection synthesis and report writing. A Food candidate might be a budget or management analyst, a planner. or an administrative intern.

Even when a project is formally assigned to a single individual. project success requires broader involvement and cooperation. Top management supervision ensures that the evaluation or analysis effort effectively addresses management needs. Good liaison with the program agency secures the assistance of agency staff. The expertise of specialists in various departments supplements the generalized skills of the analyst.

Recognizing these needs this Technical Guide presents program evaluation and analysis in a project management framework. That is to say. once a program has been selected for evaluation or analysis. the conduct of the evaluation or analysis is viewed as a work project in and of itself. As such. the evaluation or analysis project has its own organization. objectives. and staffing requirements.

While the analyst will probably possess most of the skills required. most jurisdictions will find that it is more efficient to use a team approach. A team or multi-disciplinary approach will usually be beneficial because: ( I ) it allows the best talent available in each functional area to be assigned as needed, (2) it helps to train additional personnel in program evaluation and analysis principles and techniques and (3) it helps to foster increased cooperation among involved government personnel. Neither evaluation nor analysis can be successfully conducted without close cooperation from program agency personnel, so management should lay the necessary groundwork.

A project team consists of group of people assembled for the explicit purpose of evaluating or analyzing a government program. The group disbands once its mission is accomplished. While the size and composition of this group or "team" will vary with the complexity of the project and will probably even vary somewhat over the duration of the project, two key people will be required in addition to the analyst: The team leader and the agency liaison.

The team leader should be someone with a good track record, have a good general undemanding of governmental operations, and have the ability to work with and motivate a variety of personalities. If team leaders are not part of a central evaluation staff, then they should represent the management function of the jurisdiction and have authority that extends across departmental lines.

The team leader is responsible for the management of the project in accordance with the approved workplan and for interfacing with lop management and elected officials. The team leader parcels out specific assignments within the team and runs external political interference for the team. A good candidate might an assistant city manager. budget director. or assistant to the mayor.

The agency liaison should have a good grasp of all agency operations. access to the department head. and the ability to work with others. He or she will be responsible for helping the team leader and analyst hammer out the details of the workplan. providing access to agency personnel and information, providing substantive guidance on program purposes and background, and keeping the department head informed of the study progress. A good candidate would be an assistant department head or administrative assistant.

Smaller jurisdictions may combine the functions of team leader and analyst in one person. In larger jurisdictions, the team leader may actually supervise the work of several teams conducting several evaluation and analysis projects. The Guide assumes that the team leader and analyst will pet form the work required for program selection, and that the agency liaison will be selected during project workplan preparation. After the workplan has been approved, additional personnel will be added to the team as required. Depending on the scope and time frame of the project. the analyst is the only person who may be full-time on the project.

TECHNICAL GUIDE APPROACH
The Guide presents the program evaluation and analysis process as a series of tasks, with specific steps laid out for each task. Not all evaluations or analyses will require every step specified in this Guide in exactly the level of detail provided. However, each task and step represents a concern that must be dealt with in some way.

For example, the task concerned with generating program alternative during program analysis involves consulting many information sources and thoroughly screening alternatives. Analysis of a small program may not warrant consulting every information source and using every screening technique described in the Guide. but the basic purpose of seeking innovative ways to accomplish program objectives must be recognized and accomplished. Toward this end, the Guide describes alternative levels of effort for as many of the steps as possible.

While discussion of tasks and steps is employed, this form of presentation does not necessarily mean that the Guide is a how-to-do-it manual in the strictest sense. Thus, the major intent of the Guide is to describe an overall program evaluation and analysis approach. The tasks and steps are guides to the techniques that seem the most appropriate for jurisdictions with little or no formal evaluation and analysis experience. As a jurisdiction gains experience and expertise, more sophisticated techniques, such as mathematical modeling, can be used to evaluate and analyze more complex and less well defined programs. Consequently, this document should be viewed as a detailed primer and reference work on program evaluation and analysis, with the understanding that a jurisdiction will probably outgrow some of the techniques presented here as experience is gained.

CHAPTER ORGANIZATION
This Technical Guide is organized into four major chapter and three appendices. Chapter 11. "Preparatory Tasks," discusses how to select the appropriate programs for evaluation and analysis, how to define the project scope and prepare a project workplan, how to select the project team members and draw upon outside resources, how to define or redefine program goals and objectives, and how to establish criteria for measuring program performance (Tasks 14 in Figure 1)

Chapter III. "Program Evaluation," discusses how to elect the proper evaluation design, how to collect the necessary information and data, and how to synthesize this information and data and draw conclusions about program results (Tasks 5-7 in Figure 1).

Chapter IV. "Program Analysis," discusses how to develop alternative program approaches, how to estimate costs for each alternative, how to estimate effectiveness for each alternative. how to assess feasibility for each alternative. and how to analyze the options available Tasks 8-10 in Figure 1).

Chapter V. "Follow-Up Procedures," discusses how to prepare a draft report on program evaluation or analysis, how to review the draft report with all interested parties, how to communicate the findings to top management, and how to organize and monitor an implementation effort (Tasks 11 and 12 in Figure 1).

Appendix A. "Principles of Evaluation," introduces the theory of evaluation, including a presentation of the three major types of evaluation designs, and a discussion of the sources of invalidity in program evaluation.

Appendix B. "Sample Surveys," briefly describes sample surveys, discusses their use in the evaluation and analysis process, provides references to selected documentation that will provide assistance in actually conducting a survey, and provides contacts with jurisdictions that have used surveys.

Appendix C. "Program Evaluation and Analysis Studies," provides a functionally organized list of evaluation and analysis projects that have been conducted by five local governments and cites persons to contact in each jurisdiction for additional information about those studies.

Program evaluation and analysis are closely related technologies that can be used separately or together. Both begin with preparatory tasks including: (1) selecting critical programs, (2) preparing a workplan. (3) completing a project team. and (4) defining goals and objectives and establishing performance criteria. Since these procedures are common to both techniques, they will be discussed only once.

Task 1 involves several possible ways to select evaluation and analysis projects, including some specific tips for selecting a jurisdiction's first project. Task 2 covers planning the work of a specific evaluation or analysis project, such as defining the problem, establishing project scope, and estimating time and manpower requirements. Task 3 includes identifying and organizing the people needed to perform the various project jobs. Task 4 stresses the importance of and provides a methodology for defining or identifying goals, objectives, and performance criteria in program evaluation and analysis.

The analyst is deeply involved in all four tasks described in this chapter, and the analyst is complemented by a team leader, serving in an administrative capacity. An agency liaison is chosen early in Task 2-Preparing a Workplan. Subsequently. skills needed for the project can be identified. and other members of the project team chosen.

The tasks discussed in this chapter will lay the groundwork for the specific steps of evaluation or analysis. In most cases several possible levels of effort are described so that the level of detail and precision can be tailored to meet local requirements. Chapter III provides the same type of guidance for the tasks of evaluation, while Chapter IV discusses analysis. Chapter V again deals with concerns that are common to both techniques, the communication of project results and the implementation of recommendations.

TASK 1-SELECTING CRITICAL PROGRAMS
    Obviously, no jurisdiction has the resources to evaluate or analyze all of its programs. Therefore, it is important to determine what programs get priority treatment. There are several ways of making this selection, depending upon local conditions, but the first two steps, preparing an initial list of candidate programs and screening the list, are common to all selection methods. Beyond these common steps, a more structured and detailed approach to project selection is presented in Step 3-Preparing Additional Information on Remaining Candidate Programs. The amount of structure in the selection process will normally depend on such factors as the size of the jurisdiction, the resources committed to program evaluation and analysis, and whether elected officials are to be involved in program selection. As a general rule, the more people involved in selection, the greater the need for a formalized structure.

Step 1-Preparing the Initial List
    Selecting programs begins with the chief executive or his staff preparing a list of candidate program. Most jurisdictions have an unwritten list of problem programs that immediately come to mind during' any discussion of possible areas for improved performance. The best way to formalize such a list is to examine the most common sources for program suggestions:

Operating Agencies - Frequently, department heads and other agency employees will be acutely aware of programs that need study and improvement. Such programs are often mentioned in agency budget requests as initiatives for new programs or          expansion of existing programs.

Staff Agencies - Budget and research, planning, and the chief executive's staff personnel frequently have a depth and breadth of knowledge of government programs.

Elected Officials - Councilmen and other elected officials are often aware of critical programs and make them either campaign issues or subjects for legislative initiatives.

Community Groups - Service clubs, improvement associations and special interest groups may be particularly concerned with and draw public attention to certain programs.

Citizen Surveys - Many jurisdictions are beginning to rely on sample surveys of citizen perceptions to identify areas of concern, either in specific programs or across the full spectrum of government activities. Once the candidate list has been written, the        screening step can begin.

Step 2-Screening the Program List
       The list of candidate programs compiled in Step 1 should be examined carefully to select the most important and appropriate ones. This is best accomplished by applying the following selection criteria to the list:

Timing - Is the program approaching a decision point, such as the budget or expiration of key legislation? Is there time for the analysis to be done before decisions must be made?

Scope - Is the program significant enough in impact or amount of government resources involved to merit the effort?

Performance - Does there appear to he substantial room for improving program performance? Past evaluation results are          particularly helpful here.

Capabilities - Are the dollar and personnel resources available to Reform the analysis? Would the analyst require the use of    outside expertise?

Data Availability - Do sufficient data exist to undertake the analysis, and can needed data be gathered within the time               available

Political Feasibility - Does the program have such strong support from special interest groups (labor unions, citizens' associations. etc.) that a chance in operations is unlikely, regardless of analysis results?

Applicability - Does the program lend itself to measurement? Can reasonable estimates be made of the effectiveness of current operations or future alternatives?

A jurisdiction embarking on an initial formalized evaluation or analysis project should naturally be concerned about the success of this first project. The experience of several jurisdictions indicates that several additional criteria should be applied to increase the probability of selecting a successful first project. These criteria are:

Completion Time Span-The first project should have a relatively short time span, probably 3 - 4 months maximum. Local decision makers may lose interest in program evaluation and analysis if the initial project takes too long to complete.

Payoff-A special effort should be made to select a project that will produce easily visible benefits.

Program Perceptions-The first program selected should be one that has a positive or neutral image to decision makers. Selection of a program that is viewed unfavorably will only strengthen the misconception that evaluation and analysis are negative processes intended to criticize and embarrass operating agencies. It is helpful if the program agency head is favorably disposed to the conduct of the study.

A positive response to all of the above criteria means that the project is relatively safe in terms of the probability of producing positive, visible results. As a jurisdiction gains experience and confidence in evaluation and analysis, local officials will probably wish to attempt more challenging projects for which success is less sure but the potential payoff greater.

For example, a good first project might be a street maintenance program that is readily quantifiable, relatively narrow in scope and objectives and appears to have some room for improvement. As the local staff gains experience from such projects, local officials may make the conscious decision to evaluate or analyze more challenging programs, such as local recreation and social action programs.

While all of the selection criteria listed above are important, practitioners unanimously agree that the issue of timing is by far the most important. Since the basic purpose of evaluation and analysis is to provide information for decision making it is vital that the results be available when a decision needed. From a practical standpoint, this often means that the time when the results are needed coupled with knowledge of the available personnel resources, will frequently determine how rigorous an evaluation or analysis can be. Phrased another way there is almost always enough time to perform some level of evaluation or analysis to aid decision maker

This should not be interpreted as encouragement for less rigorous and therefore potentially inaccurate, studies: it merely recognizes the necessity for evaluation and analysis to conform to the real needs of decision makers. A more detailed discussion of time frame and scope tradeoffs will be presented in Task 2-Preparing a Workplan.

A further aid to selecting evaluation and analysis projects is the provision of time and cost estimates early in the selection process. Naturally the actual time, manpower, and cost can vary considerably according to the final scope of the project but most managers and elected officials find it useful to have tough estimates to aid their deliberations. These initial estimates may be expressed simply as ranges for calendar time, manpower and cost. For example, the analyst may estimate that an evaluation of a street maintenance program will take 3-4 months, involve 1-2 man-months of analytical time, and cost $4,000 to $7,000 to obtain meaningful results. As mentioned above, tradeoffs within these categories can be made but the estimates give decision makers some idea of the relative magnitude of the respective candidate studies.

In addition to the factors discussed above, decision makers may wish to select a program for evaluation to confirm their impression that it is performing well. If the evaluation verifies their opinions, elected officials or top management may be able to silence critics of an effective although unpopular program.

The availability of program evaluation results is an additional criterion in determining the subject of program analysis. Evaluation examines the past performance of a program to determine program effectiveness and efficiency. Evaluation results are not a deciding factor in the fate of a program: however, they are a clear indication of program performance and should focus attention on areas that need improvement.

For several reasons, the availability of evaluation results should carry considerable weight in the selection of programs for analysis. First, the fast selection of a program for evaluation indicates a high level of concern on the part of local decision makers, if the selection was based on local priorities rather than Federal requirements. Second, evaluation results point to major problem areas and analysis can suggest improvements. Third, the evaluation effort lays the groundwork for analysis by identifying objectives and criteria and familiarizing analysts with agency operations and data sources. Also, communication and cooperation links with the program agency developed during evaluation can be maintained and expanded. When program operations are favorably evaluated, it is often a waste of resources to conduct a subsequent analysis: however, in such cases analysis may indicate how the program might address the needs of different client groups, or incorporate different activities.

The above two steps may provide enough information for many jurisdictions to select evaluation and analysis projects. However, if more precision or the involvement of more people is desired in program selection, then the procedure outlined in the following step is in order.

Step 3-Preparing Additional Information on Remaining Programs
Local decision makers will find it very helpful to have additional information in a uniform format and level of detail about each of the programs that survived the screening. A good vehicle for accomplishing this purpose is a project selection paper. Such a paper is a written presentation that attempts to identify and describe the main features of a program. The paper provides an extremely useful starting point for evaluation and analysis, as well as serves as a project selection tool, since it requires a careful definition of problems addressed by the program.

A suggested outline for such a paper is presented in Figure 2. A paper following such an outline would require several hours to several days to prepare, depending on the size and complexity of the program and the amount of background knowledge possessed by the writer. The paper should be no longer than two typewritten pages to facilitate use by top management and elected officials.

The project selection papers should be submitted to the mayor, council, or other policy makers for deliberation and final issue selection. Depending on the level of effort committed by decision makers, several evaluations or analyses might be conducted simultaneously. The project selection papers should be transmitted to the study team leader or analyst for preparation of a project workplan (Task 2).

Alternative Selection procedures
The structured process outlined above can be followed for the first several rounds of projects or until the decision makers and department heads become sold on the value of evaluation and analysis. If a full-time evaluation and analysis unit has been established, the jurisdiction may wish to allow departments to bring projects directly to the attention of the evaluation staff. The evaluation director then selects the projects deemed to be the most fruitful, turning down or postponing only those less fruitful projects that cannot be covered with existing resources. Such a less-formal procedure will foster a greater feeling of cooperation and confidence in the operating departments, since they request assistance themselves rather than have evaluation and analysis imposed on them from outside.

As will be seen later, the proper working relationship between operating departments and the evaluation staff is very important in maximizing the benefits of evaluation and analysis. This less-formal approach would of course still allow the council, mayor, or manager as appropriate to mandate evaluation and analysis of politically important programs.

The experience of at least one jurisdiction that initially followed such an informal approach indicates that more structure may be needed to select programs as requests for evaluative and analytical help increase. Winston-Salem, North Carolina, has recently instituted a Management Information System Committee that screens evaluation requests as one of its functions. The committee, composed of the city manager and key department heads, meets monthly to decide which evaluation projects should be added to the work load of the evaluation staff and what priority each project should receive. This allows considerable flexibility in the screening and scheduling of the evaluation work load.

Figure 2. PROJECT SELECTION PAPER. Below is a suggested content outline for a document that can provide additional information in selecting evaluation and analysis projects. A separate paper should be prepared for each candidate program. The completed paper should be no longer than 2 typewritten pages.

TASK 2-PREPARING A WORKPLAN
The preparation of a workplan is a very important Ask in both evaluation and analysis. In addition to providing guidance for the project team, the workplan serves as a vehicle for assuring that all concerned understand the precise scope of the project before work begins. Six steps are involved in workplan preparation: (1) defining the problem that the program addresses, (2) orienting the analyst, (3) establishing project scope, (4) identifying work steps, (5) estimating time and manpower requirements, and (6) approving the workplan.

It is not absolutely necessary to prepare a workplan as detailed as the one described in this task, but it is vital that consensus on the scope of the project be obtained before work begins. It is also important that the analyst and local decision makers recognize and make full use of the workplan preparation and approval process to discuss tradeoffs between completion date, scope, personnel and dollar resources, and technical precision of the project.

Some of the initial for the evaluation or analysis is actually begun during workplan preparation. The analyst must become somewhat familiar with the organization, mission, and background of the program agency, as well as conduct a preliminary survey of the data routinely kept by the agency in order to prepare an accurate workplan. The time spent on such activities during workplan preparation is not wasted; in fact, the time required to conduct the project is usually reduced by at least as much time as was devoted to those activities.

Step 1-Defining the Problems Addressed by the program
The first step in preparing the workplan is to clearly define the problems addressed by the program. This step is often overlooked since the problems often seem obvious. Experience has shown, however, that original problem statements are often vague, incomplete, or misleading. For example. a city began an analytical study to determine how to substantially increase productivity in records microfilming program because the capacity of existing storage facilities was being taxed. The problem was initially perceived as being one of selecting, purchasing, and installing the most appropriate additional microfilm hardware.

Careful examination of the problem soon revealed that the actual problem was much larger and more complex than originally stated. The city did not know what information it was storing, what information needed to be stored, or what frequency or mode of access to the stored information was required. Analysts eventually determined that a lengthy, detailed records management program study was needed to solve the problem that was initially thought to be a microfilm hardware problem. Obviously, such a discovery can have a significant impact on workplan preparation. The best way to clarify problems is to discuss them with elected officials, management, the program agency head, and several program staff members.

Step 2-Orienting the Analyst
Once the basic problems addressed by the program have been defined, the analyst's second step is to become familiar with the program. As a matter of courtesy, the analyst and/or team leader should always begin by contacting the head of the program agency, stating the purpose and asking what procedure the department head prefers the analyst to use in contacting operation personnel. It is appropriate to suggest that the department head name someone from the agency to serve as agency liaison on the project team.

The orientation process may take from one day to several months, depending on the size of the agency, the size and background of the study team, and the complexity of the study. Depending upon the issues and the situation, the analyst might be looking for preliminary answers to general questions such as those presented in Figure 3.

Figure 3. ORIENTATION QUESTIONS. The analyst may wish to seek the answers to questions such as those below to become oriented to the program agency being evaluated or analyzed.

Step 3- Establishing Project Scope
After the analyst becomes familiar with the program, the third step is to establish a scope for the proposed program evaluation or analysis. One approach to this task is to formulate several very specific questions to be answered by the study. Another approach is to establish specific objectives for the evaluation or analysis project. Obviously, these questions or objectives relate directly to what top management needs to know to make its decisions. The project scope should be stated in writing.

It is important to establish the scope before work begins so that all concerned will understand precisely what issues will be addressed and to what level of detail. Establishing this information will forestall after-the-fact misunderstandings about what the study was supposed to accomplish.

Initial guidance on defining the scope should come from the program selection process. If the project was selected by elected officials or top management, the analyst must try to determine the decisions they hope to make about the program and then tailor the scope to provide the kind of information needed to support such decisions This may not be easy ant may involve some educated guesses on the part of the analyst, even after interviewing elected officials and top management. The accuracy of these guesses will be verified by presenting the completed workplan to appropriate officials for approval before actual work begins (Step 6).

Depending on local circumstances, an effective project scope can range from broad to detailed. For example, a jurisdiction that does not have goals and objectives established for its programs might undertake a broad study to accomplish the following :

Once an evaluation of this scope has been accomplished, subsequent evaluations and analyses of program activities can be more rigorous. In all cases the statement of project scope should clearly indicate which activities are to be examined and in what level of detail.

An example of a more detailed scope can be seen in the following excerpt from the San Diego County evaluation of the County general relief welfare program:
1. Effectiveness Questions and Concerns

Step 4 - Identify Elements of the Workplan
The fourth step in this task is for the analyst to determine the work that will be necessary to complete the study. While there may be slight variation, the following major elements will always be included in the workplan:

Step 5 - Estimating time and Manpower Requirements
The fifth step is to estimate how much time and manpower will be required to perform each of the workplan elements identified in Step 4.

Very little concrete guidance can be given in estimating the time or effort for the above tasks because the time will vary considerably according to program size and complexity and the manpower available to the project team. Each task can take from one day to severe, months, depending upon the specific circumstances. Analysts should however, be able to develop a realistic workplan by reading through the entire process presented sensed in this Technical Guide so that they have a clear understanding of what is involved in each of the elements outlined above and can make careful estimates of the specific situation. The value of preparing a workplan should be increasingly obvious, especially for first time evaluations and analyses. Analysts will undoubtedly begin to get a feel for the time and effort involved as they gain experience, but a workplan should still be prepared to ensure that nothing is overlooked.

One extremely important precaution about making firm time commitments for the study. If existing data are inaccurate, the analyst may have to formulate a plan for developing data from scratch. This problem will be addressed in greater detail in Task 6, but the analyst should be aware now than this problem could occur since it obviously can seriously affect the work schedule.

Step 6-Approving the workplan
The information developed in the preceding five steps should be incorporated into a single written document. A suggested outline for an evaluation or analysis workplan is presented in Figure 4. The completed workplan should be submitted to elected officials or top management as well as to the program agency, so that all parties understand and agree on the scope of the study before it begins. It is during this step of the process that tradeoffs with respect to scope, time span, manpower, and technical rigor are normally made.

Management or elected officials may be willing to sacrifice the answer to one or more effectiveness or efficiency questions in order to have the project completed at an earlier date: or they may wish to apply additional resources to obtain more information. Whatever changes are made, it is important that the agreed upon scope be committed to writing to avoid after the fact misunderstandings about what the project was supposed to accomplish.

Due to unforeseen circumstances, it may be necessary to alter the scope of the project after work commences. For example, the project team may discover that the data needed to complete a part of the project are not available, or that some available data are inaccurate. When circumstances dictate a change in project scope, it is important that the team leader discuss the problem with management and/or elected officials to arrive at a new understanding of what is to be accomplished. Local officials may be satisfied with the reduced scope, or they may mandate that the project be postponed until the difficulties can be resolved, or they may wish the project terminated.

Whatever the case, the project team should not make unilateral decisions that change the scope of the project. The project workplan plan must represent a contract with management and/or elected officials if evaluation and analysis are to pin or retain credibility with local decision makers.

Figure 4. WORKPLAN OUTLINE. This outline covers the topics suggested for either a program evaluation or analysis workplan.

TASK 3-COMPLETING THE PROJECT TEAM
Preparation of the workplan should give the analyst a fairly clear idea of the skills required to conduct the study. It is not necessary to have a central staff with a wide range of specialized skills, since personnel with needed expertise can be borrowed on a temporary, part-time basis from other departments or can be recruited from outside government on a volunteer basis.

This task comprises three steps: (1) Identifying skills, (2) obtaining the appropriate personnel, and (3) briefing the project team.

Step 1- Identifying the Required Skills
The analyst should study the methodology section of the project workplan and make a list of the skills required to conduct the various steps. While the specifics will vary somewhat from one project to another, several basic skills are required for all studies. These skills will usually be divided among a team leader, an agency liaison, and the analyst, as specified in Chapter I.

In addition, many studies will require some form of technical expertise that neither the team leader nor the analyst possesses. Budget analysts or accountants may be needed to analyze the cost of performing certain tasks so that the costs can be related to results. Statistical expertise may be required to analyze evaluation criteria values. Specific program-related expertise is often required.

A good way for the analyst to approach this step is to prepare brief (two or three sentences) written descriptions of each of the perspectives required. These descriptions can then be used to determine which specific individuals should be added to the project team. It is possible that one person can provide two or more of the perspectives contained in the descriptions. Here, a note of caution to the analyst-try to avoid predetermining the solution by your selection of technical experts.

For example, the use of computer specialists on a program analysis project team will virtually guarantee that the alternatives proposed and selected will involve the use of a computer. While this may indeed turn out to be the best alternative, the analyst must still be careful not to inadvertently narrow the range of options by the selection of too narrow a set of perspectives.

Step 2. Obtaining Appropriate Personnel
Once the necessary skills have been identified the next step is to find individuals with those skills. Most of the personnel will be available within the government, either in the program agency or in staff agencies such as finance or planning. Most of the specific types of program-related expertise will have to come from within the program agency itself. If, for example, you are studying a housing rehabilitation program, then code enforcement and housing rehabilitation within the program agency should provide most of the expert rise. However, it may still be possible to get outside assistance from a civil engineer in the public works department or from housing specialists in State or Federal agencies.

In addition to these personnel, it may be necessary in some cases to augment government personnel with outside resource people such as a consultant to help familiarize the team with a specialized subject matter or to perform a specific task as a complex statistical analysis. Although it is possible to contract with a consulting firm to perform an entire evaluation or analysis, this approach provides a jurisdiction with little internal capacity for evaluation and analysis. Furthermore, a jurisdiction that has not performed several studies itself will probably experience difficulty in communicating with and providing guidance to a consultant.

An additional possible source of expertise is a local college or university. Academic personnel have often been used to help design a questionnaire or to conduct a survey to measure client perceptions of program performance. Statistical and industrial engineering skills, as well as business skills, can often be found in local colleges. However, it is best to use academic personnel in narrowly defined roles rather than to allow them to conduct the entire study since some academicians have a tendency to concentrate on aspects that are of interest to themselves rather than on the practical needs of local decision makers.

Some communities have made good use of local talent on study groups. and there is no reason why this source of expertise cannot be tapped for evaluation and analysis studies. Specific technical expertise is sometimes provided by local firms as a community service. There are even some instances in which a local firm has made a standing agreement to provide this expertise on an as-needed basis.

Such arrangements work well as long as the company keeps a firm commitment to provide the services when they are needed by the government rather than when persons can be spared by the company. In fact, the question of time availability is the biggest drawback to using personnel resources other than government employees or contractor. Community volunteer function well if given adequate guidance by the government and if they provide their services in the appropriate time frame. Such volunteers usually have full-time jobs and they can devote only evening and weekend time to the project. Ideally. the government may be able to find retired people with the necessary skills who are willing to volunteer their time to fit the study schedule. Several jurisdictions have found it useful to maintain community volunteer talent banks listing available expertise and experience.

In conducting program analysis it is important to create a project team that will generate new innovative, and practical approaches to program alternatives. A special effort should be made to include people who bring a fresh perspective to the problem-people who are known to be open-minded and creative. New employees who come from private industry or another jurisdiction
might provide such a perspective. If all the team members are intimately familiar with the program area, they may be too close to the problems and traditional approaches to offer useful alternatives.

Step 3 - Briefing the Project Team
After specific personnel have been located the team leader should call a meeting of the project team. Each member should be provided with a copy of the project workplan. The team leader should discuss the study objectives and methodology and explain what pan each person will have in the study. Questions about timing and possible conflicts with other duties should be ironed out at this point. Often, members of the project team can make suggestions to improve the workplan. Such suggestions should be incorporated as long as they do not change the scope of the project or adversely impact the completion date. If either of these conditions occur, the team leader must seek approval of the changes from appropriate local decision makers.

Specific assessment and discussion should also take place regarding the impact of project work on the regular tasks and responsibilities of the team members. One way to ensure willing cooperation of team members is to assure them that provisions will be made to get their regular work done if a conflict should arise. Obviously, such assurances will not always be possible, but caution should be used to make sure that evaluation and analysis project work does not become a burdensome extra responsibility for team member.

TASK 4- ESTABLISHlNG GOALS OBJECTIVES AND EVALUATION CRITERIA
With the project team selected, the next major step is to investigate the program's goals, objectives, and evaluation criteria. A program goal is a broad statement of intended accomplishments or a description of a general condition deemed desirable. Goal setting should be primarily the responsibility of elected officials and public administrators. A program objective is a specific, well defined and measurable condition that must be attained in order to accomplish a stated pal. Objective setting should be primarily the responsibility of public administrator and their staffs. or operational personnel. Evaluation criteria are the actual instruments used to measure progress toward objectives. Evaluation criteria are normally formulated by project personnel.

Ideally, all government programs should have goals and objectives explicitly stated as part of the program planning process. In reality, very few government programs have explicit, meaningful goal and objective statements. This poses a problem in program evaluation, since there is no clearly stated direction with which actual performance can be compared. Often Often and objectives are scattered throughout program documentation. In such cases, the analyst must identify goal and objective statements and phrase them clearly.

Many times, the program documentation contains much of the information to compose goals and objectives. In such instances, the evaluator is best advised to assist program personnel, management, and elected officials to establish meaningful goals and objectives to be used in future evaluations and to guide program activities. In the latter case, evaluative effort should not be as rigorous and critical as in cases where goals and objectives clearly exist and are recognized by the program staff.

Since program analysis concerns future program activities, it is always appropriate to establish goals and objectives for a new program and redefine them for existing programs. It will usually be necessary to establish evaluation criteria for all programs, even those that already have effective goals and objectives.

The above discussion should make it obvious that the analyst's role in goal and objective setting will not be the same in every project. The analyst may need to do nothing more than make sure that the set of clearly stated goals and objectives is the most current available or he may need to participate in a full-blown goal- and objective-setting process. In most cases, he will assemble and restate goals and objectives drawn from program documentation and other sources. The steps outlined in this task are designed to provide some guidance to the analyst for each of the cases discusses above. The four major steps are (1) Review program material, (2) define program goals, (3) define objectives, and (4) establish performance criteria.

Step 1-Reviewing Program Material
The first task for the analyst is to review source material relating to the program and to get a general idea of the overall purpose behind the program. Some suggested sources for leads to program goals ant objectives are:

Budget document-The program agency's annual budget request and justification will often have statements of program goals and objectives. Such statements may appear under other names, such as program purpose or program scope.

Program personnel-Perhaps the most important source is the program agency personnel themselves. Their knowledge of program operations and history as well as access to records containing policy memoranda etc., make them the prime source. This activity should be the first concern of the analyst during the agency orientation task of either evaluation or analysis.

Enabling legislation-Many jurisdictions include an indication in their budgets as to the legal basis for tine program. Examination of the charter or applicable statutes will frequently give insights as to the intent and scope of the program.

Policy messages of elected officials-Such documents as"state of-the-city" messages frequently provide insight into what elected officials perceive to be the functions of various key programs.

Expressions made by legislators, citizen groups or individual citizens at hearings before a local council or in the press-Testimony before committees considering a bill to create, expand, abolish, or evaluate a program may contain useful discussions of both explicit and implied objectives.

Minutes of boards and commissions-Many local government programs have some policy-wide oversight body to give overall direction. The minutes or annual reports of such bodies will frequently provide insights into the goals of the program

Study of these sources may revert clear goal and objective statements that fit the characteristics listed in Steps 2 and 3 in which case the analyst should proceed to develop evaluation criteria as outlined in Step 4.

Step 2- Defining Program Goals
A goal statement should describe in general terms something to be accomplished. A goal statement should be written with several factors in mind:

A goal covers long time spans relative to objectives:
A goal can be either intangible or tangible:
A goal should be people- or community oriented:
A goal should not predetermine the details of program activities;
A goal should reflect the direction daired by the general public, elected officials, and public administrators- not staff personnel: and
A goal should be expressed as a desired outcome or condition to be achieved rather than as an action or process.

Examples of program goal statements include:
Traffic Engineering-Safe, efficient, and convenient movement of people and goods.
Fire Department-The highest level of public physical safety with the resources available.
Economic Development-Economic opportunities for persons who have not enjoyed economic equality.

Many programs have several related activities, each of which may have one or more subgoals. For example, a fire department will usually have separate organizational activities for fire suppression, training, fire prevention, ambulance services, and administrative support. Subgoals for these activities might be:

Fire Suppression-Rapid suppression of fire.
Fire Prevention-Reduction in incidence of fires.
Training-More effective and efficient fire department personnel.

As the examples show the subgoals support the general program goal and address a segment of the program mission.

If the analyst cannot find or derive goal statements such as those listed above, it will be necessary to establish goals from scratch. Ideally, goal setting should be directed by the chief administrative officer of a jurisdiction with the direct input of elected officials. As a matter of practicality the analyst may find it more efficient to draft goal Statements in conjunction with the agency head and present these draft goals to the chief administrator and elected officials to stimulate discussion.

The chief administrator and elected officials can be expected to take a greater interest in the goal-setting process as they begin to grasp the importance of goals and objectives in the management of government programs. This means that the goal-setting discussions may be relatively brief for the first several programs studied but may increase in length and intensity for subsequent programs. The analyst should keep this factor in mind, as it can affect the length of time necessary to conduct a study ant therefore should influence workplan preparation.

Step 3-Defining Program Objectives
Once consensus has been reached on the more general goal statement, the analyst's next job is to review, redefine, or define specific and measurable objectives. As a matter of practicality, much of the groundwork for the formulation of objectives will have been done during the development of goal statements. The analyst may even wish to develop the goals and objectives at the same time and to present both to the agency head, top management, and elected officials through the procedure presented above. This consolidated effort will work best when there appears to the analyst to be little question or disagreement on the goal statements as drafted. However, when the program goals seem to be controversial, the analyst should make sure that the goals are agreed upon before attempting to develop objectives.

An objective should describe something to be accomplished in specific, well-defined and measurable terms. Objectives are derived from goals by, first, formulating a strategy for reaching the goal and, second, establishing one or more objectives necessary to make this strategy work. In the case of the fire department example, the subgoals represent an expression of the chosen strategy. That is, in order to achieve the overall program goal of maintaining public physical safety, the strategic elements of fire prevention, suppression, training, and medical assistance are necessary. Specific objectives are then developed for each subgoal.

An objective should be written with these factors in mind:

An objective is something that must be accomplished in order to achieve a goal:
An objective is not a program or project function, activity, task, or step:
An objective should not predetermine in any fashion the solution to a problem or way to do something:
An objective should relate to the needs of groups of citizens or the community as a whole:
An objective should explicitly consider unintended or negative effects:
An objective should be achievable within a specific time frame: and
An objective should be expressed as a desired outcome or condition to be achieved rather than as an action or process.

Figure 5 presents Several examples of effective objectives that follow the above guidelines. If effective objectives cannot be found in or derived from program documentation the analyst will have to develop them from scratch.

In developing objectives, the analyst should take into consideration the effects the objectives have on various population or clientele groups. Different groups may be affected by a program in different degrees. It is important to identify such groups and to collect data reflecting program impacts on them. An "average" crime rate or "average" family income for a jurisdiction will not adequately reflect possible major differences that may exist among segments of the population. The following points should be considered:

Each program will have some groups that are intended beneficiaries; i.e., clients of the service.
Each program is likely to significantly affect certain other groups that are not intended beneficiaries. These effects may be detrimental or beneficial.
The citizens of the community or state considered as a whole often make up a category that should be explicitly identified.
In some cases, future citizens may be an important group to consider explicitly because their interests are closely related to the program.

Figure 5. EFFECTIVE OBJECTIVES. Below are examples of program objectives determined according to me guidelines presented in Step 3.

The analyst will find that the preparation of a clientele group profile will help to develop objectives that are people-oriented by creating a picture of the group that is the target for the program. Figure 6 contains a suggested list of characteristics for inclusion in such a profile. Most of this information can be obtained from census data. Each program is likely to have at least some unique clientele groupings.

It is important that the program objectives be developed in close cooperation with program personnel, especially for programs of long standing, because the analyst is developing the standards against which programs will be measured and it is only fair that everyone agree on the essentials at the outset. Also, should an analyst attempt to develop objectives from the other cited sources alone, it is entirely possible that the analyst might develop a set of obsolete objectives.

The objectives of a program frequently shift with the passage of time; the longer a program has been in operation, the greater the chances that such a change has occurred. The objectives used should be those that the program agency personnel agree are current.

The analyst should get most of the information needed to formulate objectives by interviewing program agency personnel. In addition to the agency head and appropriate division directors, the analyst should also interview first-line supervisors and program workers to learn their perspective and to find out whether they are familiar with existing objectives. While analysts should develop their own specific questions for the interviews, Figure 7 lists some suggested questions that can form the basis for an effective interview.

Figure 6. CLIENTELE GROUP CLASSIFICATION. The analyst should know what population or clientele are affected by program goals and objectives. This classification scheme offers some assistance in developing profiles on population or clientele groups.

The analyst should always have specific questions composed in advance for these interviews. This helps to assure that all necessary information is obtained and to avoid wasting the time of program personnel with inefficient often offensive "fishing expeditions." The analyst should, of course, be prepared to diverge along a promising line of inquiry that emerges during the interview.

The analyst's list of program objectives should be presented to the agency head for discussion and approval before being transmitted to top management. While some jurisdictions may wish to do so, it isn't necessary to have objectives approved by elected officials. Many public administrators feel that overall guidance by elected officials in the form of goal statements is an adequate level of involvement.

An implicit assumption in program evaluation is that the objectives are practical. If objectives are too easy to attain they offer no real incentive for the program staff to strive for greater ach~ievement. It is probably best to set objectives that make program personnel reach a bit. On the other hand, care must be taken not to set objectives that are too ambitious lest employees become frustrated by unreasonable performance targets and cease trying their best.

Analysis of an ongoing program can raise some special problems in establishing objectives. Since the objectives of most programs shift over time, the analyst must be careful not to accept "prepackaged" objectives set down when the program started without some investigation of their relevance. Since analysis is intended to shape the future conduct of program activities, objectives that describe past practices may hamper a thorough search for alternatives. The analyst should make sure that the objectives, criteria, and clientele groups are what local policy makers intend them to be for future program operations.

Figure 7. SUGGESTED INTERVIEW QUESTIONS. These questions should help the analyst to gather information about a program's objectives during interviews with agency personnel.

Step 4 - Establishing Evaluation Criteria
Once objectives have been adopted, evaluation criteria can be formulated. Evaluation criteria are used directly to measure progress toward objectives. They are like corollaries to the objectives that answer the question. How can we measure progress toward this objective? The analyst will frequently find that there is more than one evaluation criterion for measuring progress toward each objective. As a general rule, it is always better to have too many criteria than not enough.

Evaluation criteria should be established with several factors in mind:

Criteria should be service- or people-oriented;
Criteria should reflect explicit performance targets;
Criteria should nor be subjective;
Criteria should indicate relative accomplishment or degrees;
Criteria should cover all important attributes or aspects of the program; and
Criteria should be acceptable to workers supervisors, and managers.

Evaluation criteria should be identified without initial concern about how or whether they can be measured. There are often ways to at least partially measure the more qualitative or subjective criteria by using ratings rankings and other procedures. For example, at first glance citizen perception may appear difficult or impossible to measure, but a sample survey can usually supply the needed data. Citizen perceptions are important criteria for virtually every government program since they measure the degree of public satisfaction with government service delivery. Appendix A to this technical Guide contains further material on surveys. Figure 8 presents several examples of evaluation criteria incorporating the above principles.

Figure 8. EVALUATION CRITERIA. This example shows evaluation criteria for the objectives developed in Figure 5. The criteria adhere to the guidelines presented in Step 4.

As used here, there is no right or wrong value for criteria. Fire deaths per 1,000 population can be compared with figures from other jurisdictions and national averages, but no accepted standard exists. Evaluation criteria are intended only as quantifiable indicators upon which to base judgments; the criteria themselves do not provide any answers. Thus, in order to make criteria useful from a management perspective, the jurisdiction must set performance targets for each one.

Unless such targets are currently being set as part of the management process, first-time evaluations will have to depend more on value judgments than will later evaluations. Part of the analyst's job should be to establish performance targets for each evaluation criterion in coorporation with program agency personnel, so that these personnel will have more precise direction and management will have more specific performance indicators.

POSTCRIPT
The preparatory tasks discussed in this chapter are common to both program evaluation and analysis. Chapter III "Program Evaluation." discusses those tasks that specifically apply to evaluation. Chapter IV. "Program Analysis" discussed those tasks that specifically apply to analysis. Chapter V. "Communication and Follow-Up." integrates these two discussions.

This chapter describes steps necessary to complete a program evaluation, assuming that the program has been selected, the project scope defined, the project team selected, and the goals, objectives, and evaluation criteria formulated. Task 5 presents three specific designs that are practical for state and local government and discussed the situation for which each design is best suited. Task 6 involves collection of the data necessary for evaluation. Detailed discussions address such important concerns as determining data availability collecting the data, and verifying the accuracy of the data. Task 7 covers the steps necessary to examine the data and draw conclusions about program performance. Information on preparing a written report and implementing the evaluation recommendations is included in Chapter V. For a discussion of the principles of evaluation, refer to Appendix A.

TASK 5- SELECTING AN EVALUATION DESIGN
All evaluations are basically some form of comparison. Whether comparing a group of people who received special treatment (such as in a drug rehabilitation program with a similar group who did not receive treatment, or comparing the actual accomplishments of a program with its performance objectives comparison is still the key to evaluation. An evaluation design provides the framework for making comparisons. Researchers have developed many different evaluation designs - (1) planned vs. actual, (2) time trend, and (3) before vs. after program comparison. The discussion of each design includes a description, step-by-step procedures, application considerations, and tips and cautions regarding use.

Design #1: Planned vs. Actual Performance
This design compares the actual program performance for a given time period with planned performance. It can be used for virtually all ongoing programs that have not been consciously changed during the evaluation period. The planned vs. actual design has the advantage of providing a natural lead-in to program analysis, since areas of substandard performance are identified by the evaluation. The procedural steps for the use of this design are:

Most local government requests for evaluation information concern ongoing operations. Decision makers want to know the effectiveness of street patching, sanitation, or fire prevention programs. The question this type of evaluation asks might be phrased. How well are we performing basic services? This need for information can be contrasted with the desire to know the effectiveness of special or experimental programs, such as drug or alcohol treatment programs. In this case, the question might be phrased. Is this program worth continuing? This distinction is important, as it underlines the need to tailor the evaluation to the specific needs of those requesting evaluation.

In this design, performance objectives for a given time period are compared against actual performance for the same time period. In order to make use of this very basic design, the evaluators must be able to identify objectives for the program and then measure progress toward them by use of the evaluation criteria. If performance targets have been set previously, this design will give precise and useful results. However, if performance targets have not been previously established, the evaluation must be handled differently. The analyst can still establish, the evaluation must be handled differently. The analysis can still establish performance targets for the past time periods being evaluated, but care should be taken not to make the first-year evaluations seem punitive because it is not fair to judge a department head's managerial ability against a set of criteria he did not know existed at the time of program performance.

There are several purposes for using this design for first-round evaluations: (1) to get a general assessment of program effectiveness and efficiency. (2) to establish explicit performance targets for future time periods. and (3) to identify some specific program areas that need improvement. Application of program analysis techniques for these purposes should improve future program operations. Positive aspects of the program identified during evaluation should be highlighted as part of the written report to lessen the punitive or negative image that many people attach to program evaluation.

This design implicitly assumes that the targets set are reasonable. Targets that are too easy to reach do not challenge program personnel to provide true measures of accomplishment. Targets that are too high will discourage program personnel and may give management a distorted view of agency performance. Targets must be set with the participation and cooperation of program agency personnel as described in Chapter II.

Ideally, performance targets should be set through the use of work measurement procedures. Work measurement is a technique that allows equitable time standards to be established for many jobs. For additional information on the application of work measurement techniques to State and local government operations.
Contact:
    Subscriber Services,
    Public Technology, Inc.
    1140 Connecticut avenue, N.W.
    Washington, D.C. 20036
    (202) 452-7700

Design #2: Time Trend
This design measures the effects of a program change. Evaluation criteria are selected and data collected to establish past performance trends. These trends are then compared with conditions observed after the program change. The design differs from the previous design in that it does not require the establishment of performance targets but relies entirely on actual performance measures. This design is best used to evaluate a program change rather than overall program effectiveness. It can also be used to evaluate new programs aimed at changing specific conditions, if preprogram data on these conditions are available. The procedural steps for the use of this design are:

The thinking behind the design is quite simple. If a program has shown a reliable performance trend in the past, then it is possible to get a good indication of the success of a specific program change by observing if there is a significant change in the trend after implementation. The analyst must be careful about drawing conclusions regarding the efficiency of the program based on changes in the values of evaluation criteria. While improved program performance probably indicates a relative increase in efficiency (assuming constant personnel resources), the program may still be relatively inefficient when compared with work measurement standards. Without such standards for comparison the analyst can only judge apparent changes in efficiency in relation to past performance.

A good example of the use of this design in a local government setting can be found in an evaluation of a change in refuse collection vehicle routing. A jurisdiction might change its refuse vehicle routing, in an attempt to conserve gasoline by cutting down on wasted double trips on certain street, and to cut back on overtime by evening out crew work load. Data on gasoline usage and overtime hours worked are usually available. Graphic comparison of these figures for several previous years with the data for the same criteria after the new routing is implemented should give a clear indication of the success of the program change in reaching its objectives. Of course, a conclusion based on these data could be invalid if the jurisdiction began using a new type of collection vehicle at the time of the routing change, or if there were some other variable besides the routing change that was introduced.

Evaluation of a long-standing program generally requires data for at least four previous years to establish a statically valid trend. Data summarized by quarters may be taken in this situation to look for possible seasonal fluctuations, such as in a recreation program or a snow removal activity. Data should then be gathered for one or more intervals after program implementation. The more post-change data available, the more certain it is that the program effects are permanent and not just a short-term reaction.

An issue in the use of this design is the consistency of evaluation criteria and data sources over time. If the criteria require data normally gathered by the jurisdiction the analyst should attempt to make sure that there were no significant changes in the way the data were gathered or recorded during the past performance intervals.

For example, a number of years ago police departments across the country began adopting a standard FBI classification system for reporting crimes. The differences in definitions and reporting procedures caused radical differences in some jurisdictions statistics for certain types of crimes. While most changes of this nature are less dramatic the analyst must make sure that data definitions, remain constant during the intervals evaluated. The most likely sources of problems will be programs for which, or all, of the data is composed of subjective ratings.

If the program data do not how a clear trend before the program change, then it may be possible to check the results of the evaluation using a nonequivalent control group. This approach was used in the evaluation of a Connecticut highway-speed crack down program. Since precrackdown data on automobile deaths were unstable, the evaluators could not be sure that the crackdown was responsible for the reduction in traffic fatalities. Data were collected from adjoining states for the same time period and plotted on the same graph. When the adjoining state data showed no equivalent decrease in fatalities evaluators had much greater confidence in their in their conclusion. Figure 9 shows the graphic display of the data for the Connecticut example.

Figure 9. TRAFFIC FATALITIES TIME TREND COMPARISON.
The graph, which displays traffic fatality data for a number of years, was constructed to evaluate the effectiveness of speeding crackdown program initialed by the state of Connecticut in 1955. Since the data did not establish a clear trend prior to the crackdown program, the evaluators could not be sure that the program was responsible for the reduction in traffic deaths. Addition of data from adjoining states indicated that the program was probably responsible for the reduction in traffic deaths. (Source: Campbell, Donald T., and H.Lawrence Ross. Law and Society Review. Vol.III, No. 1, August, 1968)

INSERT FIGURE 9 HERE

Design #3: Before program vs. After Program
This design consists of measuring criteria values just prior to the implementation of a program and then obtaining values for the same criteria after implementation or completion of the program. This design does not seek to establish a trend for the criteria but merely to take a "snapshot" of conditions before and after a specific change. Before vs. after works best to evaluate a program of short duration and limited scope. This design is appropriate when conditions in the program have been stable for some time and are expected to remain stable in the future unless altered by the program initiative. It is generally more effective if the evaluation can be planned prior to implementation of the program change in case special data are required on preprogram conditions. The procedural steps for the use of this design are:

This design was used in the evaluation of a special intensive street cleaning program implemented in Washington, D.C. The program extended over a nine-week period. No other major changes were expected that would affect the postprogram values of the evaluation criteria. Neighborhood cleanliness was measured just before and just after the program using a visual inspection procedure and a citizen survey. The "before" program conditions were believed to be typical and not of a seasonal nature. Since the data needed to evaluate the program were not normally available, the visual inspection procedure had to be devised before the program was initiated so that preprogram data could be collected. This evaluation is fully documented in the following publication:

How clean is our city: A Guide for Measuring the Effectiveness of Solid Waste Collection Activities by Louis H. Blair and Alfred I. Swartz. The Urban Institute. Washington, D.C., 1972.

The before vs. after design assumes that the values for the evaluation criteria just before program initiation accurately reflect preprogram conditions: For this reason, the analyst must be careful to avoid using this design for a program with significant seasonal fluctuations or at least to compensate for the fluctuations. This is the simplest of the three designs and one that is currently in use by some State and local governments: however it is also the design that has to be used with the most caution. The design itself provides no means to distinguish nonprogram factors causing the differences, or lack of differences, in the pre and postprogram values of the criteria. Consequently, the analyst must take considerable care to identify possible nonprogram influences to protect the validity of the evaluation.

A more detailed discussion of this problem will be presented in the data evaluation section of this chapter (Task 8). The validity and credibility of the evaluation can be enhanced if the design ???? in conjunction with the time trend design. Before vs. after program comparison should be used by itself only as a last resort.

TASK 6 - DATA COLLECTION
The sixth task in the evaluation process is usually the most time-consuming and expensive -collecting the data needed to conduct the evaluation. There are four major steps in this task: (1) Identify the necessary data, (2) determine data availability, (3) collect existing data, and (4) verify the accuracy of the data.

Step 1- Identifying the Data
Identifying the data involves determining what statistics or indicators are required to measure the criteria identified earlier in the evaluation process. In many cases, the criteria themselves will be statistical measures. An illustration of this can be seen using the example of fire service criteria presented in Chapter II where the objective was a 50 percent increase in public awareness of fire dangers this year. The associated criteria were:
(a) Number of fire safety demonstrations performed.
(b) Public response to fire safety questionnaire.
(c) Number of fire hazards reported by the public.

Criteria (a ) and are specific statistical measures. Criterion (b) actually represents several statistics, since analysis of the survey questionnaire responses would probably yield separate figures on overall awareness of hazards, and on awareness of specific types of hazards. The analyst should study each criterion and ask what data would be needed to quantify the criterion. The analyst should not be concerned at this point with whether the data are easily available, since a thorough check of this point is the next step. If no single data source seems sufficient it may be necessary to identify several data sources that indirectly measure aspects of the criteria.

Step 2- Determining Data Availability
Once the analyst has determined what data are necessary, the second step is to determine how much are available. At least a preliminary survey of data availability should have been done during the project selection process to ensure the feasibility of the project. The methodology outlined here for determining data availability is considerably more detailed than that used for preliminary data surveys.

As a matter of practicality, for small evaluations the analyst may well determine data availability and begin collection at the same time. For most evaluations, it will be desirable to keep these steps separate since the absence of required data may cause the analyst to formulate a new strategy for data collection. It is not always necessary to obtain data for every criterion of a multiple-criteria objective. Using the fire prevention example, it would not be absolutely necessary to obtain data for all three of the criteria to be able to make a sound evaluation of program effectiveness. Each piece of data would provide an additional indicator of program effectiveness, but even without all of the data, valid conclusions could still be drawn about the program.

The analyst would be well advised to prepare a worksheet to use during data identification and collection. Such a worksheet would have the specific program objective at the top of the page, a list of the applicable criteria, and the data required to measure each. Additional information could be added indicating the availability and specific location of the data. A sample of such a form using the first protection example is shown in Figure 10. There are numerous types of data, but for our purposes only three will be discussed in detail: (1) existing records and statistics, (2) client perception surveys, and (3) special data collection techniques.

1. Existing Records and Statistics. The analyst should begin the data search by examining the existing records of the jurisdiction, starting with those of the program agency. The partially completed data availability worksheets with the data requirements identified should be shown to the program agency liaison person. The agency liaison should be able to determine quickly whether the agency has the required data and help the analyst figure out the best way to collect them.

Some evaluations will require data from several agencies since the program being evaluated involves more than one agency. For example, an evaluation of police effectiveness would probably require records from the courts. Obtaining the cooperation of several agencies can be quite difficult, especially if the evaluation effort does not affect or benefit them directly. Such situations require experience and skill on the part of the evaluation team leader and underscore the importance of top-level management support for the evaluation. It is the analyst's job to locate the necessary data, but the team leader's help will often be needed to gain access to them. Some general suggestions that may prove helpful in locating data are presented in Figure 11.

2. Client Perception Surveys. If the data identification process revealed a need for data on citizen perceptions of service delivery, the analyst will probably have to turn to sources other than existing records. The analyst should determine whether a survey has recently been completed either on a jurisdiction wide basis or in the specific program area of the evaluation. A survey conducted within the past year can be considered current. The analyst should examine the questions and responses to determine if the necessary data can be obtained from the survey. If the survey is too old or none has been conducted, then consideration must be given to initiating a new survey.

The experience of several jurisdictions that have used surveys in program analysis indicates that small, narrowly defined surveys yield the most productive results. For example, a short (3-6 questions) survey on citizen satisfaction with plastic trash bags, or a specific recreation program, yields results that are easy to interpret and involves relatively little effort to prepare and administer. Such surveys are also easier for citizens to respond to than a long survey that asks their perception on a wide range of government programs or issues. The analyst may be able to use statistics on citizen complaints or service requests to gauge citizen perceptions on specific services.

3. Special Data Collection Techniques. Once the data availability worksheet has been completed, the analyst must study it carefully to see if sufficient data are available to make a valid evaluation. This will be a particularly sensitive decision for objectives that can only be measured by one or two criteria. As a rule of thumb, data should be available on more than half of the criteria to ensure the validity of the evaluation. This rule of thumb must be used very cautiously for some criteria can be more vital to an evaluation than others; therefore, it also matters which criteria can be measured. To retain the community impact emphasis of the evaluation, it is necessary to give most weight to those criteria that measure citizen perceptions and direct effects on the program clientele groups.

Figure 10. DATA AVAILABILITY WORKSHEET. This is a suggested form to be prepared by the analyst to determine the availability of the data needed to conduct an evaluation. The information shown in the sample applies to the fire prevention example originally presented in Chapter II.

Figure 11. DATA LOCATION. Below are some suggested sources for the types of data often required for program evaluations.

The analysis described above will enable the analyst to determine whether the evaluation can be completed with the available data. There will be many instances when additional data will be necessary, and even more instances when additional data can add greatly to the validity and utility of the evaluation. This is a key decision point in an evaluation because, if some of the necessary data are lacking, a determination must be made whether to : (1) continue the evaluation with available data. (2) take the necessary time and effort to gather additional data from scratch, or (3) scrap the evaluation for lack of sufficient data.

If the first decision is reached the analyst may conclude that the lack of data requires limiting the scope of the evaluation. If this limitation is deemed significant by the team leader, then management and/or elected officials should be apprised of the specifics and asked to approve the new scope or to direct that additional data be generated to perform the evaluation as originally planned. If the analyst and team leader decide there is sufficient information and that it is impractical to gather the needed data, they should document their findings and present them to management.

When a reduced evaluation scope will not provide management with the type of information needed for decision making, it is necessary to generate data from scratch. The specific data should already have been identified, so that the first job should be to determine exactly how to go about collecting them. The analyst and team leader should decide whether the data can be collected: (1) by adding one or more data items to records routinely kept by the government. (2) by establishing new records and procedures, or (3) by using a special technique, such as a citizen survey. After this decision is made, the analyst should prepare a work plan that clearly states the specific data needed, the methodology to be employed, the time period to be covered, the calendar time required, the personnel time required, the estimate cost of data collection, and the impact the collection effort will have on the schedule for the evaluation as a whole. Once the impact on the project is known, the new work plan should be submitted to top management and elected officials for their consideration to ensure that all understand and approve the scope of the evaluation. The main point to keep in mind is that the need to collect data from scratch, whatever the reason, will have a significant impact on the duration and cost of the evaluation.

Step 3- Physically Collecting the Data
Once the data requirements have been identified and availability ascertained the team leader, analyst, and agency liaison person should meet to decide the best way actually to collect the data. As mentioned earlier, there are three main sources for evaluation data: (1) existing records and statistics, (2) client perception surveys, and (3) special data collection techniques.

1. Existing Records and Statistics. Data from existing records and statistics can usually be collected most efficiently by program agency personnel. The people who handle the records on a day-to-day basis are the extract the data quickly, since they do not need a "get acquainted" period. Using program agency personnel to do the time-consulting physical work of data collection can also free the evaluation analyst for involvement in several evaluation projects simultaneously.

Several things must be done, however, before program agency personnel can be turned loose on a data collection problem. First, the analyst should spot-check the accuracy of the data is important to accuracy. If the data are guesses or estimates by field personnel rather than "hard" data provided by program clients or some reliable form of measurement, then the validity of the data may be seriously questioned. A full discussion of data accuracy will be presented in Step 4 of this task.

Second, the analyst must provide the agency personnel with clear concise directions. The analyst must be able to tell agency personnel exactly what data are needed and the specific time span to be covered. The analyst should also provide worksheets for recording the data so that they are collected in a consistent manner. It may also be possible to lay out the worksheets so as to facilitate later analysis of the data. The analyst and the agency liaison person should meet with the employees who will be doing the actual data collection and discuss the reason for the data collection, the significance of the evaluation and the collection worksheet and special instructions. After answering questions, the analyst may find it beneficial to spend a few minutes working with employees as they put the worksheets to use for the first time.

It is also wise for the analyst to spot-check data accuracy during the data collection by examining a sample of the source records and comparing them with the worksheets prepared by the agency personnel. To facilitate these checks, analysts should have the agency personnel forward worksheets to them on an "as completed" basis, perhaps once a week.

Evaluations that require data from several agencies can cause the analyst difficulty in actually collecting the data and/or in coordinating the efforts of several groups. The example of a police effectiveness evaluation used earlier will help illustrate the point. To get a complete picture of police performance, data are likely to be needed from the prosecutor and/or court system on indictment and conviction rates, and perhaps accident statistics from the traffic engineering department. The prosecutor's office may not perceive any immediate benefit to that agency from the evaluation and therefore may be reluctant to take an active part in the project. The experience, tact, and political expertise of the evaluation team leader can often greatly improve cooperation. The team leader may be able to persuade the agency to cooperate by showing the agency head how his or her agency will benefit.

In the above instance, the team leader may be able to convince the prosecutor that the evaluation may produce results pointing to the need for police officers to build cases on more solid evidence, thus making the prosecutor's job easier. If such a line of reasoning fails to persuade the agency head, the team leader may be able to gain cooperation by offering the resources of the team to help the agency head solve an operational problem in return for voluntary assistance with the evaluation. A management mandate ordering the agency to cooperate should be sought only as a last resort, since the resulting hard feelings often lead to unfavorable agency perceptions of the evaluation process.
An evaluation such as the one outlined above also raises the issue of confidentiality of personnel data. Some agencies may refuse the evaluation team access to individual records on this basis. In such instances the evaluation team may be able to examine the records in question by limiting access to a single designate analyst who will work solely on the agency premises. In other cases, the agency may be willing to aggregate key information about a group of individuals so that no one person can be identified. While it is always preferable to examine the data first-hand, there may be instances in which aggregation of data must be accepted.

2. Client Perception Surveys. Client perceptions are becoming increasingly important data sources for evaluations as governments seek to measure various program impacts on the people they serve. The most prevalent tool for measuring client (citizen) perceptions is the survey.
Surveys are tools for questioning selected samples of the general public. They may involve mailing questionnaires to respondents, leaving questionnaires at respondents' homes and retrieving them at a later date. Also interviewing respondents in person, or interviewing respondents over the telephone. Surveys provide feedback on respondent perceptions, desires, needs, preferences, priorities, opinions, and experiences.

The primary benefit that surveys offer is the capacity to elicit the views of numerous individuals many of whom would not otherwise participate in the program evaluation process. Thus, survey information can be more representative of the public at large than information obtained through other kinds of public involvement efforts. Surveys also offer the following benefits:

Survey responses can be readily analyzed to determine underlying patterns and relationships, including trends over time.
Survey can focus on specific respondent groups and/or specific issues or objectives of interest to the user jurisdiction.
Survey can identify the rationale behind respondent answers.
Surveys can gather information about people's perceptions, desires, and opinions unavailable from other sources.
Surveys can reduce the sense of isolation or alienation felt by many respondents.

It is important for the analyst to realize that a properly prepared and analyzed survey is a very useful and powerful tool, but one that requires a considerable amount of calendar time. A simple reliable survey may take several weeks to complete, and several months is a more realistic estimate for many surveys. Although a detailed explanation of the conduct of sample surveys is beyond the scope of this Guide, an appendix has been included to provide guidance. Appendix B contains decide when a survey is appropriate, (2) prepare and administer the survey instrument (questionnaire), and (3) analyze and present the results. References are also provided to jurisdictions that have practical experience in the use of sample surveys.

Methods other than surveys are available to measure citizen perceptions. Regular meetings of improvement associations service clubs, and other service organizations can provide a forum for the airing of perceived problems. While such input is not necessary representative of the entire community, the analyst can discern useful information through careful questioning and listening. Such techniques may be necessary when the time or resources are not available to conduct a survey. Extreme caution is urged in the use of information obtained in this way, because of its lack of precision and objectivity.

Citizen complaints or service requests are not normally used as indicators of citizen perceptions because few jurisdictions have made any effort to handle them systematically. Additionally, such information is obviously selective since only dissatisfied clients use this avenue of communication. However, at least one jurisdiction, Kansas City, Missouri, has made an effort to use complaint data. Through the city's "action center,"service requests are recorded, channeled to the appropriated department for action, and followed up with a postcard to the citizen asking for an evaluation of the city's response to the request. Complaint data and citizen ratings are summarized monthly for the operating departments and the city manager's office. Such a system allows the administration to get a rough barometer of feeling toward specific services by tracking complaints. City council members also use the monthly summary figures of complaints as rough performance indicators for the various departments.

3. Special Data Collection Techniques. Often, data must be generated from scratch. Perhaps the most common way to do this is to add one or more data items to forms currently in use by the agency. Such efforts are usually relatively low-cost, since the only additional expense is redesigning and reprinting the appropriate forms. The chief drawback is that collection of the information will require at least one program interval ( a month , a quarter , or a year), thereby delaying the evaluation for that time period.

A more involved variation of the above is when information must be added later to records already on hand, as in the example of collecting additional information on clients already served by a program. Such activities are very difficult to conduct because the participants must first be located and then persuaded to cooperate. Such after-the fact data collection techniques should be used only when alternatives are exhausted.

In some situations, subjective ratings by professionals may be appropriate for evaluating program effects. This approach may be most useful in social service fields. For example, professional social workers could use subjective ratings to measure changes in family and community functioning attributed to social welfare programs. Rating scales might cover; family relationships and family unity, individual behavior and adjustment , car and training of children, economic practices, social activities, home and household practices, health conditions and practices relationship to social workers, and community resource use. Explicit directions must be provided for use of each rating on a scale. Ideally, the rating system should enable a group of professionals, observing the same conditions to arrive at the same rating.

A pretest is highly desirable to see if different professionals using specified procedures would in fact give reasonably similar ratings. When using such a rating scale, individuals should not be asked to rate themselves on their own effectiveness in providing a service. Raters should be selected who do not have a personal interest in the outcome.

For meaningful program evaluation, three factors should be standardized; the characteristics evaluated by professionals, the rating scale applied to these characteristic, and the conditions under which the ratings are made. In the family functioning example, the professionals are given guidance on the aspects of family functioning to be rated. Each aspect is rated according to a standard descriptive scale. For instance, one aspect "sibling relationships." would be assessed on the basis of criteria for each grade on the scale:

Inadequate: There is conflict between children resulting in physical violence or cruelty which warrants intervention....
Marginal: Emotional ties among children weak... rarely play together...
Adequate: Positive emotional ties and mutual identification....

The actual rating is made by first-hand observation of the family by the social workers.

This method requires professionals who are competent to make judgments about the particular situations and who can be impartial in their appraisals. Also, if a grading scale is not readily available, considerable time and effort will be needed to establish an acceptable rating system. The costs of making ratings could be large because of the time required for each observation and the specialized personnel involved. However, if such ratings can be provided as part of the regular jobs of employed professionals, the actual out-of-pocket costs to a government may be small.

In some situations, as time passes, raters may deviate from the rating scale. Periodic checks and retraining in the use of the scale can alleviate this. For example, during the Washington, D.C. "Operation Clean Sweep," checks of a sample of inspector ratings using the street cleanliness rating scales indicated that inspectors tended after a time to compress the scale; i,e.. To give fewer extreme ratings. To correct the problem, the inspectors were exposed to the photographic rating scale. While the tendency to compress the scale may not be as pronounced with more highly trained professionals, it is still a situation that the analyst must guard against.

This method of data collection is basically subjective and normally should be used in conjunction with more objective measurements. For example, the number of reported difficulties in school for client-family children could supplement professional ratings to measure child adjustment.

If none of the data sources above seems to fit the evaluation the analysts are free to develop measures and sources of their own as long as the accuracy of the approach can be verified. As an example, an evaluation of the Fairfax County, Virginia road maintenance program was aided by the use of a device called a " roughometer" that measured inches of roughness per mile. The evaluation team verified the accuracy of this approach by showing a high correlation between citizen perceptions of roughness and readings taken by the roughometer on the same sample of streets. There are many less dramatic examples of analysts making creative uses of field observation techniques by measuring emergency equipment response time or making special counts of participants in recreational activities. The point is that the evaluation team should not restrict itself to the approaches presented in this Guide.

Step 4-Verifying the Accuracy of the Data
One of the most frequently overlooked aspects of program evaluation is verifying the accuracy of the data. While treated here as a separate step for emphasis, the discussion of the previous step correctly suggests that data accuracy should be verified during data collection. In this way, the analyst can take actions to correct or improve the data immediately, rather than initiate a second collection effort later. There are three major types of data inaccuracies-clerical errors, subjective errors, and methodological errors.

1. Clerical Errors. Clerical errors are one of the most common sources of inaccuracy. Such errors (transposed digits recording the wrong figure, etc.) frequently occur when data are transferred from original source documents to summary reports or data collection worksheets. Clerical errors can be detected by checking a sampling of the data collection worksheets against the original source documents. If more than 10 percent of the sample entries are incorrect, the analyst can take one of several remedial actions.

If more than one person has been recording the data in question, the analyst should try to determine whether the high error rate is uniform among all collectors or is found only in the work done by one or more individuals. The employee completing each worksheet can be identified by a code on the sheet itself. If the high error rate is restricted to one or more individuals, the analyst can either review collection procedures with those individuals and stress the importance of accuracy to the employees and their supervisor, or request that a more accurate employee be assigned to recollect the same data. Should the high error rate prove to be uniform among all collectors, the analyst should review the collection procedures with all employees and appropriate supervisors to determine whether the worksheets are poorly designed or the data collection procedures incomplete or confusing.

If data collection accuracy does not improve, analysts may want to consider collecting the data themselves or finding another way to measure the criterion in question. Another remedial course is to postpone the evaluation while improved data collection procedures are developed. This will usually mean postponing the evaluation for one program period (one month to one year). Naturally, the earlier in the evaluation process this determination can be made, the fewer dollar and personnel resources will be wasted on an incomplete effort.

2. Subjective Judgment Errors. Data involving subjective judgments will require more involved accuracy checks than outlined above. When dealing with subjective ratings such as those provided by inspectors or social services counselors, the analyst must make an effort to determine the accuracy of the rating system. This is accomplished by examining the rating scale to determine how clear and comprehensive the descriptions are of the various rating categories. In addition, the analyst should attempt to determine how much training the field personnel have had in the use of the scale and how often the training is reviewed. The review question can be significant, since experience has shown that extended use of a subjective scale often result in "compressed" ratings; i.e.. Fewer ratings toward the extremes of the scale. Periodic reviews of the scale with supervisors can help alleviate this tendency.

The analyst may also find it useful to examine the turnover rate among field personnel, since high turnover often results in inconsistent ratings over the evaluation of the ratings by getting several people independently to apply the rating scale to the same situation or site at the same time.

3. Methodological Errors. Of the data collection techniques mentioned, surveys are most prone to methodological error. The analyst should review the survey instrument (questionnaire) for possible bias, the sample selection method, the size of the sample, the degree of training given to surveyors, and the methods used to analyze responses. The references found in Appendix B should provide the information needed to make most of these determinations. No survey can be 100 percent accurate. What the analyst should watch for are instances in which opinions or results are not clear-cut on a specific question and there is some evidence of significant inaccuracy in the survey. Management should be caution that does not have a high degree of reliability. Data from flawed surveys can still be used, but with due caution.

Another type of methodological error can sometimes be avoided by double-checking of the analyst's thought processes. It is very easy to get so involved in what you are doing that relatively simple errors go unnoticed. For example, an evaluation director reported that one of his associates was deeply involved in establishing criteria and collecting data on the effectiveness of fire suppression services. The analyst hit on the idea of using the percentage of a building that was consumed by fire as a criterion for effectiveness of the fire department. The evaluation director hastened to point out that since the fire department had no control over how long a building had been burning before an alarm was turned in, and that a building might well be fully engulfed before the department was notified, the proposed criterion was neither fair nor valid. There is a much better chance of avoiding such errors if the work of an analyst is checked by at least one other analyst.

It is generally inadvisable to continue the evaluation with data errors greater than 10 percent. If an evaluation is continued under such circumstances, the analyst should be sure to identify clearly resulting distortions in the evaluation report. Managers must understand that they cannot place the same degree of confidence in evaluations with questionable data as in evaluations with highly reliable data.

In summary, five major options can be pursued if key data are discovered to be inaccurate: (1) The evaluation team can seek other, perhaps less direct, ways of getting acceptable data. (2) Improved procedures can be adopted for collecting the data and the evaluation postponed until new, reliable data can be gathered. (3) the evaluation team can seek to improve the quality of the data by such methods as clear supervision of the collection effort, or the use of better collection forms. (4) The evaluation can be continued with the clear warning that management should be cautious in using the data in question for decision making. (5) The evaluation can be canceled as infeasible. While the most suitable option will depend on the specifies of the situation, analysts will probably feel the most confident with the second option, where practical. The important point is to recognize that inaccurate data can badly undermine the credibility of an evaluation, and the analyst should guard against this problem.

TASK 7 - SYNTHESIZING THE DATA
This task involves three major steps: (1) Organizing the data, (2) making comparisons, and (3) drawing conclusions. The initial synthesis should be incremental: once conclusions have been reached regarding one objective, the next objective can be considered, and so on until all objectives have been analyzed.

Step 1 - Organizing the Data
The analyst must assemble in one place all of the data collected during the previous task. The data collection worksheets for the various criteria should be placed together with the data availability worksheet for the appropriate objective. Using the previous fire service example, this would require three sets of data collection worksheets, one for each criterion, and a data availability worksheet for each of the three objectives. If it has not been done by the data collectors themselves, the analyst should summarize the data for each criterion. The analyst should then check the data availability worksheet against the data summary sheets to make sure that all of the available data are assembled. Next, the analyst can begin to find the proper way to organize them.

As mentioned in the section on selecting an evaluation is comparison. The emphasis in analyzing the data will be on organizing them so that the correct comparisons can be made. There are several approaches to arranging the data for comparison, and each will be described in turn.

Perhaps the simplest technique for comparing data is the use of a table. A table showing the title of the criteria, the measured values for the evaluation period, the planned target values or values from other time periods, and a percentage of accomplishment may help illustrate gross relationships between criteria. A table will probably not be of use when the data for the criteria are in different forms, and as when some of the data are survey responses or subjective ratings and the rest are raw statistics. Figure 12 illustrates such a table.

Graphs and charts are important comparative techniques. They:

Show Relationships. Graphs can show data relationships over time to illustrate (where appropriate) historical sequences. New data can be added to a continuous graph whenever received. Such a graph not only shows the past but the present as well. Also, future trends may be spotted by the alert user.

Permit Study of Data. Graphs permit the study of data regularities and irregularities. A careful scrutiny of the data points on the graph and the relationships between these points may reveal meanings that could not be observed otherwise. Thus, even though tabular presentation contains all the individual data points (values) that can be plotted on a graph, relationships and trends are not as readily discernible from tabular presentation as from graphs.

Figure 12. TABLE COMPARISON OF EVALUATION CRITERIA. The table shows the target values, the actual measured values, and the percentage of accomplishment for each of four unspecified criteria.

Suggest New Ideas. When information is viewed in chart form: the data points may suggest new relationship and idea, or may suggest connections between seemingly unrelated bits of information that the user may know from previous experience, reading, or general knowledge.
Efficient Use of Information. Charts and graphs represent a refinement of verbal description by limiting the presentation of information to fundamental relationships by following generally accepted procedures of charting.
Raid Visual Impression. Charts can provide a quick visual impression of norms and standards with which present results can be compared. For example, if the average number of students per teacher is considered the norm, and if national or state norms are available, then local year-by-year data can be plotted against the norms and/or data. Interpretation is thus facilitated.
Simplification of Complex Ideas. While a degree of risk always exists with over-simplification, there is much to be gained from proper simplification. If a chart presents a number of relationships in easy-to-understand and difficult-to-misunderstand terms, the user will benefit from the added clarity.
Achievement of Standardization. When related sets of data are presented in a standard graphic format, the user spends less time analyzing the graphic format and more time interpreting the data itself.

Narrative descriptions can also be used to present comparisons. For each criterion to be compared, the description should address the same specific points. The analyst must carefully think out the key questions that must be answered about the program and make a written list of questions. Each narrative description should answer the questions in the same order and level of detail. Also, surveys will almost always require a narrative interpretation of the results as well as a display of the summarized response data.

In some instances, when good evaluation criteria have been developed and accurate data are available, the pattern may be easy to see with little or no manipulation of the data. In most cases, however, some thought and effort will be required to clearly establish the message of the data. The analyst may have to use two or more of the above techniques at the same time to help discern hidden relationships and patterns. One of the major reason: Why Chapter II stressed the need for multiple evaluation criteria and objectives was to help establish a clear and reliable pattern during analysis. If only one objective of one criterion is used, the analyst and decision maker cannot have complete confidence in the results, especially when the data do not show clear-cut patterns. Similarly, when only two criteria are used and the date values for one criterion indicate program failure, considerable doubt exists about true performance. However if four or five criteria are used and the values for several of them point consistently toward success or failure, then a greater degree of certainty can be associated with the evaluation.

Figure 13. BAR CHART COMPARISON OF EVALUATION CRITERIA. The bar chart shows the percentage of planned performance actually accomplished for four criteria.
INSERT FIGURE 13 HERE

Step 2 - Making Comparisons
The above general principles and techniques will now be applied specifically to the three evaluation designs presented in Task 6 at the beginning of this chapter.

Design #1: Planned vs. Actual. The comparison to be made in this evaluation design is between the planned values and the actual measured values for the evaluation criteria. A common and effective way to make this comparison is to express the relationships as a percentage. Percentages for all of the criteria for an objective can be displayed in either a simple table or in a bar chart. The table would have vertical columns for brief description of the criteria, the target value, the actual measured value, and the percentage of accomplishment (actual measured value divided by the planned value). An example of such a table was shown in Figure 12. The bar chart would show only the percentage of accomplishment for each evaluation criterion. An example of this technique is shown in Figure 13.

Data arrayed in this fashion will often enable the viewer to draw rapid, accurate conclusions. In the example in Figure 13, it is obvious that the level of accomplishment for Criterion #3 is considerably below that of the other criteria. There may be any number of reasons for the difference, but the important point is that such a chart immediately highlights what should be matter of concern. A percentage of accomplishment of 85 percent or better is generally considered acceptable but a jurisdiction could easily adopt a higher standard assuming that the targets are reasonable.

Figure 14. GRAPH COMPARISON OF EVALUATION CRITERIA. This graph displays the trend in the value of a specific criterion-- fire-related deaths per 1,000 population. The vertical dashed line indicates when a program change was implemented.
INSERT FIGURE 14 HERE

Design #2: Time trend. The comparison to be made in this evaluation design is between the trend shown for the evaluation criteria values prior to program implementation and the value for the criteria after implementation. The criteria values should be displayed on a graph to facilitate analysis. Time should be displayed on the horizontal (X) axis and the criteria measurement scale on the vertical (Y) axis. Actual measured values for the criteria should be plotted for several time periods prior to program implementation and at least one time period after implementation and at least one time period after implementation. A vertical dashed line or other indicator should be drawn on the graph to mark the time of program implementation. The analyst looks for a discernible change in the trend of the graph line that coincides with the start of, or change in the program being evaluated. An example of such a graph is shown in Figure 14.

In the example shown in Figure 14, a change in fire protection procedures was implemented in January, 1975. As can be seen, the trend from previous years was definitely toward more deaths per 1,000 population, but the implementation of the new procedures appears to have reversed the trend.

It is possible to estimate how much difference the program change made by statistically projecting the criteria values without the program change and plotting this value on the graph. The difference between the projected value and the actual value is an indication of how much effect the program change really had.

Extreme caution is urged in using this technique. Since accurate results require that the evaluation team have access to someone with skills and experience in making statistical projections. Projections made on a haphazard basis may seriously distort the analysis. This technique should not be used until the jurisdiction has become practiced at program evaluation and even then should be used with caution.

Design #3: Before Program vs. After Program. The comparison in this design is between the actual value of the evaluation criteria measure immediately before a program is initiated or a program change introduced, and the actual value measured at some later time. This is the simplest comparison of the three and can be made by comparing the raw values for the criteria, or by using a bar chart showing two bars, one for preprogram data and the other for postprogram data. The difference between the two values can be tentatively considered the program impact, subject to reservations that will be discussed below. This type of analysis differs from the previous design in that no attempt is made to establish a trend prior to implementation. It is possible to use this design for ongoing evaluations by continuing to gather data for several time periods after program implementation to see if the change caused by the program continues, increases, or decreases.

Step 3- Checking the Validity of Evaluation Results
In order to draw conclusions about the program, the analyst must first investigate any factors external to the program that may have influenced the data. None of the designs discussed in this Guide guards against the possibility of events outside the program affecting the measured values of evaluation criteria. Such events have either a negative or positive effect on the program. If other possible factors are suspected, the analyst must make an effort to prove or disprove their impact.

An example of this problem can be seen in an evaluation of the Indianapolis Police Fleet Plan. The plan involved patrolmen taking marked patrol cars home and using them a personal cars. The plan was tested in an effort to reduce certain crimes by having more marked police cars on the streets. Increased police morale and a greater feeling of security were also objectives. Before and after program data were gathered and summarized. A trend was established for specific crimes that the program was expected to affect, including auto theft. Graphs indicated that most criteria showed an improvement after implementation of the Plan with a particularly dramatic decrease in auto thefts.

In the course of analyzing the data, someone realized that the program start coincided with the introduction of a new automobile ignition lock system by automobile manufacturers. Consequently, it was possible to hypothesize that the reduction in auto thefts was due to another cause. Comparison of auto theft statistics for surrounding areas and on a nationwide basis indicated that all other areas continued to show an increase in auto thefts, thereby strengthening the conclusion that the reduction in auto thefts in Indianapolis was attributable to the Police Fleet Plan.

The analyst must make an effort to conduct just such a search for corroborating evidence to ensure the validity of the program evaluation. If it is not possible to prove or disprove the effect of a nonprogram factor, the analyst should discuss this factor or event with program agency personnel and program clients and then draw conclusions.

Step 4- Drawing Conclusions and Making Recommendations
On the basis of the evaluation criteria investigated the analyst must determine the relative success or failure of various aspects of the program. Each objective should be considered in turn and the relevant criteria measurements examined. Multiple criteria for any given objective are more reliable than a single criterion in providing indications as to whether the objective has been realized. Similarly, consistent achievement of multiple objectives provides strong evidence of the achievement of the program goals. The analyst must consider the evidence and draw conclusions that can be stated in writing and verbally.

It should be pointed out that data tabulation analysis and display may strongly suggest conclusions and clearly show relationships, but the final decision is to the relative "success" or "failure" of the program is still a matter of judgment. In most cases, the apparent success must be weighed against the costs in order to reach the "bottomline." Analysis is not a substitute for decision making; it merely seeks to provide objective information upon which to base a decision. Complete reliance on quantitative answer is as wrong and misleading as complete reliance on instinct.

In addition to drawing tentative conclusions about the program as a whole, and about its various component parts, the analyst should gather together any ideas that may have emerged during the study concerning how current program activities could be improved. Such recommendations would not basically alter a program, but rather streamline present program operations, possibly through forms modification or procedural changes/ ,other questions about a program would be best addressed through program analysis.

POSTSCRIPT
The next-to-the-last step in the evaluation process is communication the results to all interested parties. A written report usually serves this purpose along with an oral presentation, when appropriate. Since the communication process is very similar for both program evaluation and program analysis, Chapter V discusses the importance of the communication process and presents instructions on preparing a draft report, reviewing the draft with interested parties, preparing a final report and transmitting it to decision makers. Chapter V also discusses the importance of the final step in the process - implementing evaluation recommendations. The chapter covers methods of organizing and monitoring the all-important implementation effort.

This chapter describes the tasks necessary to complete a program analysis. The assumption is made that the preparatory steps outlined in Chapter II (selecting a project, preparing a work plan selecting a project team, and establishing goals, objectives, and evaluation criteria) have been completed. Analysis begins with identifying possible alternative ways of accomplishing the goals and objectives of a program. After the most promising alternatives are identified, data on the cost effectiveness, and feasibility of each alternative are gathered. The final task covered in this chapter is synthesizing the data and drawing conclusions about the alternatives. Instructions for communicating results to decision makers will be presented in chapter V along with information on organizing and monitoring the implementation process.

Figure 15. SOURCES FOR PROGRAM ALTERNATIVES. The list below discusses some of the sources that an analyst should investigate to develop ideas for program alternatives.

telephone interviews so that he or she is sure to cover all relevant points in the least amount of time. It may be well worth considering taking a day or more and actually visiting the jurisdictions that appear to have the most promising approaches.

At this point, the analyst should be in the best position to formulate innovative alternatives based on information provided by the other sources. Time should be set aside for the analyst, and perhaps other members of the team, to consider all of the information gathered up to this point and to give concentrated thought to the formulation of alternatives that fit the program goals and objectives. These alternatives should also be screened by the procedure outlined above.

The generation of program alternatives and their subsequent screening should be viewed as an iterative progress that continues as new information is gathered and absorbed. The number of fruitful iterations will vary with the project, but most analyses will require at least two rounds to produce three to six good alternatives. The written descriptions of these most promising alternatives become the source documents for the next program analysis task, data collection.

TASK 9 - DATA COLLECTION
The analyst must collect data and information regarding four separate aspects of each alternative: (1) service demand, (2) cost, (3) effectiveness, and (4) feasibility. Each of these will be discussed as a separate step.

Step 1- Estimating Service Demand
The first type of information needed by the analyst will be data that indicate the probable demand for the service to be provided by the alternative. Obviously, the amount of service will have a direct impact on the cost and an indirect impact on the effectiveness and feasibility of the alternative.

There are two types of service demand to be considered (1) expressed demand, which is based on past usage of the service, and (2) latent or hidden demand, which is the demand that would occur if services were better publicized, more convenient, or more economical to use. The analyst must make an attempt to estimate both types of demand, even though the estimates for latent demand may be imprecise. Sources of information that can be tapped to estimate service demand are shown in Figure 18. These sources will generally provide information on the current demand for services. In order to make these estimates meaningful for program analysis, the analyst will have to project future demand. The analyst should examine data for the past several years to see if a trend exists and to make a projection based on that trend.

An effort must be made to determine if latent demand will be a significant factor for the program. The analyst must be especially careful in estimating latent demand since it is much more difficult to quantify accurately. For example, if a survey indicates that 1,000 additional people would use a service if the cost were lower, the analyst might assume that the number of additional people who would actually use the service is something less than 1,000, since many people will give a positive survey response as long as it does not commit them to a course of action. On the other hand, the number may turn out to be greater than 1,000, since the new service users may encourage their friends or neighbors to take advantage of the service. The point is that latent demand is very difficult to estimate.

It is probably not possible to arrive at a single estimate for required service level, especially for new programs for which the demand is difficult to estimate accurately. In such instances, the analyst should make several estimates based on a range of service levels. This will enable the analyst to see and show decision makers how sensitive the cost and effectiveness projections are to fluctuations in demand. The analyst should also keep in mind that estimates are to be made for more than one year, since decision makers will want to know both the short-term (first year) and long-term (recurring) costs and effects of the alternative. Once the demand has been estimated, the analyst is ready to collect the additional data required to complete the analysis.

Figure 17. ALTERNATIVE SUMMARY. Below is a suggested content outline for a brief (two typewritten pages) document that summarizes the important data about a program alternative. A separate summary should be prepared for each alternative.

Figure 18. SOURCES FOR DEMAND ESTIMATES. Below are some suggested sources for data that the analyst can use to estimate the future demand for a particular service.

The analyst should begin by preparing a cost worksheet for each alternative. The worksheet should have space for general cost elements or groups, such as personnel costs, equipment and supplies, contractual services, and capital expenditures. Specific cost items are then listed under each element. This exercise will identify all of the specific data items to be costed. Figure 19 illustrates a cost estimation worksheet; the analyst may wish to prepare sheets that more closely fit the local accounting system.

In almost all cases, the current program, extended at its present service level, should be one of the alternatives costed out. This will give decision makers a baseline to use in considering the other alternatives. This is also a good starting point since it will probably be the easiest alternative to estimate, and the experience should make costing other alternatives easier. Cost estimates will generally be the most difficult to make for those alternatives that show the greatest divergence from present practices. Figure 20 contains a list of principles that the analyst should consider when making cost estimates.

Once the cost estimates have been determined for each alternative, the next step is to make the actual cost estimates. Several approaches to cost estimating are discussed below. They can be used separately or together.

Unadjusted current data applied to the future. This costing approach is primarily applicable to costs that are not expected to change significantly. As an example, the latest salary and employee benefit scales might simply be used to estimate future personnel costs, or current data on the number of personnel or staff-hours required to perform a specific task might be used to estimate the future requirements for that task.

There are severe limitations to this approach, particularly if demand for the service changes or if technological improvements in equipment are anticipated. If, for example, a government's emergency rescue vehicles are expected to become more complex in the future (e.g., equipped with more automatic monitoring or telecommunications devices and emergency equipment), higher costs per vehicle might be anticipated. There also might be higher maintenance costs and additional training costs for operation. In this current data to predict future costs.

Price level changes may of course also affect the future cost of program components even if nothing else changes. This problem is discussed later.

Vendor estimates. Certain programs may involve equipment or facilities for which price quotations can be obtained form a seller or builder. If the quotes are for already existing items or such items with minor modifications, the prices should be accurate. However, a firm commitment is not always implied in the estimate. Production costs might be higher than anticipated.

Engineering estimates. As new programs or program activities are proposed, other cost estimating techniques are needed. The major technique currently in use is for technical experts - government employees or consultants - to prepare cost estimates for new program components. This procedure calls for an appropriate expert to break the program into as many component parts as possible and make dollar estimates for each component based on experience. One difficulty in such estimates is that if many program alternatives and variation are examined, the time require for the estimates may be substantial.

Figure 19. SAMPLE COST ESTIMATION WORKSHEET. Specific items should be listed under the appropriate category. Different categories can be created to conform to local accounting systems.

Statistical estimation. Predicting future costs, especially for programs with new and perhaps unusual characteristics, is a very difficult task. Statistical analysis or data on past performance can often effectively supplement expert judgment in predicting future costs of performance.
The use of statistics can be very simple or very complicated. The simpler techniques of statistics are familiar. For example, to derive a figure for the fuel and maintenance cost of police cars used in a traffic control program, the previous year's costs for all traffic control police cars can be divided by the number of cars to obtain an average cost per car. Assuming no price increases or significant changes in the nature of the police cars to be used, the average cost per car could then be used to estimate the cost of proposed alternative programs involving any number of police cars of the same type.

Uniform cost factors. Certain types of costs will be regularly considered in program analysis. Examples might include fringe benefit rates for various classes of employees, overhead costs (processing payroll, accounting, purchasing, etc.), per unit costs for such things as vehicle usage and building maintenance. These uniform cost factors can be calculated and updated regularly to assure uniformity and accuracy in cost estimating.

The analyst must make several additional decisions. For example, the analyst must decide how to deal with possible price changes. This problem is particularly important if some of the alternatives are more likely than others to be affected by future price increases. Estimating future changes in general price levels is difficult. The rate of increase will usually vary for each cost element, payroll, construction project, piece of equipment etc. A concern is that the price level estimates could be self-fulfilling-that contractors, unions, or other groups could become aware of the estimates and would be unlikely to settle for less than the estimated increases. Perhaps such projections should be made only when it appears clear that the program choice could be significantly affected by price changes. If the analyst or team leader decides to adjust for price changes, the same adjustments should apply to all alternatives and probably to all analyses. These adjustments should be noted in the cost worksheets and final report.

Uncertainty is another special problem in program cost analysis. Cost and effectiveness estimates are seldom precise, especially when they extend beyond the next budget cycle. For unfamiliar alternatives, estimates to within 10 to 25 percent will often be as accurate as can be expected. The magnitude and likelihood of cost changes may affect final program decisions and should, if significant, be assesses carefully. Techniques can range from simply labeling estimates as "reasonably accurate" or "highly uncertain" to more elaborate techniques that attempt to estimate quantitatively the likelihood and size of the uncertainty. Finally, there is a tendency in cost analysis to attempt to be overly precise in situations that do not warrant it. This can be both wasteful and misleading. Analysts should roughly estimate the magnitude of the costs involved, decide how much precision is necessary, and then adjust cost analyses to those dimensions.

The final estimate for each cost item should be entered in the appropriate place on the cost worksheet, but the scratch sheets and other back-up materials should be saved. Since these working papers will facilitate adjustment during the analysis and provide documentation on questioned items. Once the estimation work-sheet has been completed with all of the necessary cost figures, the analyst is ready to move on to the next data collection step.

Step 3-Collecting Effectiveness Data
In addition to estimating the cost of program alternatives, an attempt must be made to estimate the effectiveness of alternatives in accomplishing program objectives. The analyst should begin by examining the goals, objectives, and particularly the evaluation criteria are the yardsticks used to measure effectiveness. Estimates of future program effectiveness are difficult to make, particularly for new programs. The analyst must seek to determine the impact of each program alternative on each evaluation criterion. As an illustration, take a fire suppression program with the criterion response time in minutes for both fire and rescue equipment. In estimating the effectiveness of the various alternatives, the analyst would have to consider what impact each alternative would have on response time. Similar estimates must be made for each of the evaluation criteria. While estimating effectiveness is far from an exact science, several approaches are useful.

Unadjusted Projections. Future performance can be estimated on the basis of data on pat program performance. This approach assumes that conditions will not change substantially in the future. For example, if the criminal apprehension rate for the past year is 20 percent, this figure can be used as an estimate for a future year of the same program. This approach is certainly simple, but it is probably overused. This assumption of stability is questionable in many, if not most, cases. Unfortunately, information on existing programs is often unavailable.

Adjusted Projections. Past performance data can be adjusted by estimating likely effects on performance of changes in future conditions. There are many ways to do this, such as using time series data. Rather than using only performance data for the past year, an analyst takes an average of several years to compute a projection line based on recent trends. For example, if the apprehension rate for the past year is 20 percent, and if in prior years the rate had been gradually increasing, a higher apprehension rate would be projected. The assumption is that the trend over a number of years is a more reliable indicator of the future than a single year's data.
Time alone should not be considered an adequate explanation for future conditions in most situations. Changes in the overall population, in the client mix (such as age, sex, income, race, and residential location (such as new housing and transportation) can affect alternative program performance. While the effects may be complex, the analyst can often identify certain key conditions that are changing. Once these changes are identified, their effects can be projected into the future and used to modify estimates of the program's effectiveness. For example, an examination of solid waste landfill disposal alternatives would require, in part, projections of changes in the number of households and of waste generated per household. This would yield an estimate of future demand for solid waste disposal by residential units. This estimate could then be added to estimates of waste from other sources and be compared with present disposal capacity and the capacity of other landfill options to determine how effective each is in handling projected future needs.

Experience of Other Jurisdictions. If a proposed alternative has been tried by another government. Useful data from that government's experience may be available. Unfortunately, such data are likely to be inadequate, since governments seldom make explicit provision for collecting evaluative information. Also, analysts should be cautious when using published reports, since such reports can be mainly public relations documents or may not have been based on systematic program evaluation. For example, a recent examination of a computerized system for allocating police resources by geographic area and time of day indicated that, despite the apparent belief that response time data had not been collected and could not have been analyzed to support that belief.
Performance reports prepared shortly after the initiation of a program should be considered with caution. A program generally requires six to 12 months, and often longer, before its operation stabilizes and negative or unintended effects can be detected.
Even if good evaluative information from other jurisdictions is available, it does not remove the need for an independent analysis. The attractiveness of any alternative depends in part on the conditions of the particular state or local jurisdiction.

Figure 20. COST ESTIMATION PRINCIPLES. The analyst should keep the principles below in mind when making cost estimates for alternatives.

Vendor Estimates. In some circumstances, vendors can provide performance estimates for equipment-oriented alternatives, or at least performance information on the equipment itself. Vendor estimates, of course, are usually optimistic: they are also likely to be limited to the narrowly focused equipment specifications and not to the variety of impacts--especially negative ones--that may occur when the equipment is used by fallible human beings in less than ideal working environments.

Synthesized Estimates. In some cases, the analyst has to synthesize an estimate from known facts about the alternative, or even use estimates based on anticipated characteristics of the proposed system. This is particularly so with alternatives that involve new technologies or procedures for which appropriate comparison data are not available. For example, a new solid waste disposal system might be crudely assessed by using data from the design and the technical specifications to estimate the amount of waste the system could handle on a daily basis and the amount of pollution it would yield. Analysts should also assess whether the estimated effectiveness is likely to remain the same or change significantly in the years following introduction. If significant changes seem likely, estimates of the amount of the change should be made. This type of analysis is filled with certainties, however. More technologies rarely perform, at least at first, as well as anticipated.

Expert Judgment. When none of the previous methods can be employed, expert judgment may be appropriate. Experts may be government personnel or persons outside the particular government who have extensive experience in the program area. Their judgments can be used for making direct estimates of an alternative's effectiveness or for estimating the future values of various factors needed for effectiveness calculations. A systematic procedure for making judgments should be used, and the judgments should be documented and substantiated as well as possible. Sophisticated approaches for soliciting expert opinion, such as the Delphi technique which uses anonymous opinions of a number of experts to refine progressively a specific projection* can sometimes be helpful. But these sophisticated approaches tend to be time-consuming and relatively expensive; they are probably justifiable only if the program is very important to the government.
    Experts could be asked merely to rank the relative effectiveness of alternatives in terms of a particular criterion. For example, analysts might assess the degree to which various probation and parole service approaches would lead to a reduction in recidivism. At the very least, experts might rank each proposed approach as "better," "worse," or "about the same" as the existing approach. More useful for analysis would be estimates of the degree of success.

"Simulated Adversary Process." In this approach, each major alternative is assigned to a different team, and each team then builds as strong a case as possible for its assigned alternative, probably using some of the techniques already described. This approach is appropriate primarily when analysts are dealing with programs that have varying impacts on different groups in the community. In such cases, the approach can provide government officials with broadened perspective of the pros and cons of the various alternatives.

Trial Period. Finally, if sound estimates are not obtainable, and if government officials believe that a particular alternative has considerable potential but that uncertainties are too great for a full-scale commitment, the government might undertake a limited trial to provide more reliable information on the new program. This approach is most appropriate in cases where a limited, short-term program is feasible and where only small initial investments are needed for manpower and capital additions. The trial approach has another advantage when past experience is not available: A trial is likely to detect unintended, perhaps negative, program effects.
    One example of the trial approach is in crime control, where it is extremely difficult to predict the effectiveness of various manpower allocation or patrol strategies such as team policing. In such cases, the government might undertake a one-year trial of a specific strategy in particular neighborhoods to obtain information on the strategy's effectiveness.
    Many difficulties are associated with this approach, however. For example, many programs cannot be adequately evaluated on the basis of a one-year experience. Start-up problems might temporarily lower performance, producing inaccurate indications of long-term performance. On the other hand, special attention paid to the program might result in better short-term performance, yielding misleading indications of long-term performance.
    If a government undertakes the trial approach. It should provide for a systematic program evaluation. The trial should be designed realistically, and critical evaluation criteria should be identified in advance. Trials are often conducted without adequate concern for the need for performance data. As a result, the government is likely to have very little information about program effectiveness by the end of the trial period.

As with estimating costs, it may be appropriate to use more than one of the above approaches to estimate the effectiveness of a given alternative. The analyst should indicate on the worksheets which approach or approaches were used so that it is possible for someone else on the project team to check the logic of the estimation process and the accuracy of the work. Effectiveness estimates should be expressed as expected values for the program evaluation criteria. The rationale used in making these effectiveness estimates should be carefully documented for three reasons: (1) so that the logic and accuracy can be checked by others, (2) to guide future analyses by inexperienced personnel or to serve as a "refresher" for experienced program analysts, and (3) as a check against actual performance of implemented alternatives to establish the accuracy of the estimating process, and hopefully, spot ways to improve estimation.

A summary sheet should be prepared for each alternative, listing the appropriate effectiveness estimates. The analyst is now ready to collect the fourth and last kind of data required for program analysis.

Step 4--Collecting Feasibility Information
A fourth and vital step in the data collection process is the collection of information that provides indications of the feasibility of implementing the various alternatives. There is a very real and understandable danger of devising alternatives that seem logical on paper, but overlook practical realities. While it is often difficult to quantify implementation problems, the analyst must make an effort to identify and describe them in a way meaningful to decision makers.

Many factors determine the feasibility of program alternatives. The analyst should consider the questions shown in Figure 21 to help isolate the pertinent feasibility factors.

The importance of these questions for a given alternative may vary with each individual case. In some cases, it is possible for the analyst to devise a weighting scheme, but such a scheme is not suggested here since examination of the questions Should make it obvious that the relative importance depends largely on the local situation. Care should be taken not to allow a weighing scheme to add a sense of false precision.

Probably the best procedure to use in assessing implementation feasibility is to review these questions with personnel from the appropriate operating agencies and perhaps with someone from the mayor's, council, or city manager's staff who would be sensitive to the political issues. This review should identify the pertinent factors, which should be described in sufficient detail in the analysis report.

During the study of significant implementation problems, the analyst or other project team members may be able to identify specific changes to an alternative that would ease implementation. This does not mean that the original alternative, even if it involves one or more major obstacles to implementation, should be eliminated from consideration. Decision makers should be allowed to examine both the original alternative and suggested variations to determine whether the implementation problems are serious enough to warrant the variation. The analyst should carefully examine the estimates already made for any modified alternative to make sure that the full impact of the modification is accounted for.

The analyst should prepare a written narrative for each alternative summarizing the feasibility factors and what changes might facilitate implementation. The narrative should answer the following questions:

What are the major practical barriers to implementation?
How can these barriers be overcome?
Are there any factors that make this alternative particularly attractive?

This narrative summary will be used to prepare the program analysis final report.

Figure 21. FEASIBILITY GUIDELINES. The analyst should use the questions below to guide the determination of the feasibility of implementing program alternatives.

TASK 10-SYNTHESIZING THE DATA
Once service demand has been estimated, cost and effectiveness data gathered, and feasibility information summarized, the next step is to combine the four types of information and draw conclusions.

It is best to keep in mind at the outset that it is neither likely nor desirable that the analyst will focus on a single alternative. The purpose of program analysis is to provide decision makers with program alternatives or options and sufficient information to assess accurately the tradeoffs. Analysts will probably arrive at conclusions regarding the relative merits of alternatives and will present these conclusions as part of their final report, but they are not expected to provide "the answer" to the problem. The purpose of the analysis process, then, is to draw together information that shows how effectively each of the alternatives meets the program objectives, at what cost, and with what possible implementation obstacles.

The three major steps in this task are: (1) Organize the data, (2) make comparisons, and (3) draw conclusions.

Step 1-- Organizing the Data
The first step in data synthesis is to consolidate and organize all of the data for an alternative. The analyst should collect five items for each alternative: (1) the narrative description of the alternative, (2) estimates of service demand, (3) the cost worksheet, (4) a worksheet summarizing the effectiveness estimates, and (5) a narrative summarizing the implementation feasibility factors. Once the analyst has compiled all of the data for all of the alternatives, he is ready to begin making comparisons.

Step 2--Comparing the Data
A series of comparisons will be required, examining the relative costs of the alternatives, the relative effectiveness, and the respective feasibility summaries.

Simple tables will generally be the best way to present the data for comparison. Cost data for three years of a program can easily be compared with the help of a table such as the following:

Alternative	1st Year Cost	2nd Year Cost	3rd Year Cost	Total Cost
#1	$83,000	$75,000	$80,000	$238,000
#2	$83,000	$60,000	$68,000	$211,000
#3	$120,000	$50,000	$54,000	$334,000
#4	$105,000	$95,000	$102,000	$302,000
#5	$50,000	$35,000	$40,000	$125,000

This table will tell very little by itself, but together with similar tables comparing other aspects of the alternatives, it will help to establish the relative merits of the alternatives. A table similar to the above but comparing the various cost categories for each alternative should be useful to both the analyst and decision makers. Such a table would show which alternatives involve high capital costs, which involve high personnel costs, and which require high expenditures for contract services.

Most decision makers will be particularly interested in labor costs since personnel and related costs are usually the most difficult costs to reduce in late budgets. Figure 22 is an illustration of such a table using the same data as above.

Another table that should facilitate analysis is a comparison of the effectiveness estimates. Such a table might look like this:

Alternative	Number of Fire Safetly Demonstrations	Number of Fire Hazards Reported	Percent of Public Scoring 70 or Better on Questioning
#1	450	2,000	50
#2	100	1,200	25
#3	200	2,500	40

Step 3--Drawing Conclusions
Careful study of the above tables comparing cost and effectiveness indicates that alternative #2 seems to offer the most favorable ratio of effectiveness to one, assuming that the effectiveness criteria all carry equal weight.

The analyst might specify such a conclusion in a project report as long as he points out that decision makers may attach greater importance to some of the criteria than others and therefore should concentrate their attention on those alternatives that score well on those criteria.

Figure 22. COST COMPARISONS. This table shows a cost comparison of five program alternatives by cost category. Such a table would allow decision makers to determine which alternatives would be more labor-intensive or capital expenditures intensive, for example. The personnel category includes salaries, fringe benefits, and personnel-related costs such as uniforms and training expenses.

Some alternatives may yield intangible, or at least unquantifiable, benefits. Benefits such as greater administrative control, special service to a particular clientele group, fewer citizen complaints, or an increased oneness to citizen participation are virtually impossible to quantify, yet may count heavily with local decision makers. The analyst should make every effort to identify such benefits and point them out in the summary of conclusions regarding each alternative.

Finally, the analyst should examine the most promising alternatives to determine which are the most feasible to implement. On the basis of the forgoing analysis, the analyst should identify one or two of the alternatives that are superior to the others. These are the alternatives that will be prominently featured in the analysis report. The analyst should keep in mind, however, that the final decision as to which (if any) alternative to adopt remains with management and elected officials, after they have weighed the comparative data prepared by the analysis team. Preparation of a report that facilitates this decision-making process, and information on implementing a program alternative, are covered in
Chapter V.

This chapter deals two major tasks; (1) communicating the results, and (2) following up program evaluation and analysis with implementation projects. Perhaps the most frequently overlooked aspect of the evaluation and analysis process is the communication of results. Many analysts, even though they recognize that effective presentation is critical to the acceptance and use of their findings, do not really know how to prepare effective reports and presentations. The painstaking effort involved in identifying objectives, establishing evaluation criteria, and gathering and analyzing data is wasted if the findings are not presented in a way that makes them understandable and useful to decision makers. One evaluation director has estimated that he spends as munch as 50 percent of this time communicating and discussing results with department heads, the city manager, and operating agency personnel. While a certain amount of personal contact will always be necessary and may even be desirable, a well-written report can reduce redundant explanations. An effective oral presentation to decision makers can increase the probability that the study will affect the decision process.

Effective communication ensures that project efforts are used to best advantage. The project team can contribute further if they participate in the implementation of project recommendations. The responsibility and involvement of he team should not end with the presentation of a report. Valuable insights and information gained during a study can be brought to bear on the implementation effort. It is absolutely vital that the project team be prepared to develop a plan to implement those study recommendations that are acceptable to local decision makers; evaluations or analyses that do not lead to program changes are a waste of valuable resources.

TASK 11 - COMMUNICATING THE RESULTS
There are three major steps in communication of results: (1) preparation of a draft report, (2) review of the draft version by various parties, and (3) transmission of the final report to decision makers.

Step 1 - Preparing a Draft Report
As mentioned earlier, evaluation and analysis studies are conducted to provide decision makers with objective information about program effectiveness. This information is usually communicated in a written report.

Whatever the format it is important that evaluation and analysis results be written, since written reports make the information available to all concerned and reduce the chance for misunderstandings. Written documents are also less likely to be ignored by decision makers than are verbal reports. A further advantage to written reports is that they documents studies for use later when evaluating program progress or changing program direction. Also, evaluation and analysis reports can by of value to other jurisdictions in that such reports document possible program alternatives and provide insights on what has and has not worked in other communities. Appendix C contains a listing of the program evaluation and/or analysis studies that have been conducted by five jurisdictions that have expressed a willingness to share documentation. The reports are grouped in functional categories and a contact person is supplied for each jurisdiction.

The report should be written with its target audience(s) clearly in mind. Program evaluation and analysis reports often have more than one target audience. One audience is made up of elected officials chief executive officers, department heads, and community leaders. This audience is primarily interested in the information needed for decision making. A second audience is the department administrator and middle-level managers responsible for the program in question, and whose concerns are more specific. This audience is interested in how the study was conducted, what observations were made, and what conclusion were reached. There is sometimes a third audience of analytical personnel from other agencies or jurisdictions who are interested in the specific steps and techniques employed in the study, as well as in reviewing actual data for information or comparative purposes. The analyst should define the target audiences for the report and confer with the project leader on the subject before actually writing the report.

Once the target audiences are identified, the analyst is ready to begin outlining report(s) that provide the information needed by the target audiences. A modular approach is suggested here including an executive summary, a management report, and a technical report. Even if three physically separate documents are not prepared division of the report into the above sections will accomplish much the same purpose. At least tow jurisdictions - Phoenix, Arizona, and San Diego County, California - are currently reporting good results from the routine preparation of a separate executive summary for evaluation and analysis reports. A suggested outline for such a report is presented in Figure 23.

If the analyst prepares separate documents for the different target audiences, it will be necessary to modify the outline somewhat. The analyst should add the material outlined for the problem statement and program description ( in section I ) to the beginning of a separate management report to make it a coherent document. The material contained in such an expanded management report should be added to the beginning of the technical report if it is expected to be a stand-alone document.

The analyst should first prepare the management and technical sections of the report as outlined above. This should ensure that the analyst has a clear idea of what the report contains when he is ready to write the executive summary. It is important that attention be given to the report's style, which should be brisk, clear, and concise. As a general rule, the reports should be written for the least sophisticated among the target audiences since a sophisticated reader will normally forgive simplistic writing while a relatively unsophisticated reader will almost always get lost in complex writing. Since most jurisdictions do not have a technical writer or editor available in a staff capacity, the analyst will have to turn elsewhere for help in achieving the desired style and tone. Two books that can provide guidance for the analyst are:

Figure 23. REPORT OUTLINE. Below is a suggested outline for an evaluation or analysis report. The outline lends itself to preparing one consolidated report, or two or three separate smaller reports for various target audiences.

Both books discuss organization, tone, style, and graphic aids. Since both evaluation and analysis reports will make use of graphs and charts to display data, the analyst will find the following books useful in preparing graphic aids:

Effective Graphic Communication
by Norbert Lloyd Enrick.
Princeton, New Jersey: Auerbach Publishers, 1972.

Handbook of Basic Graphics: A Modern Approach by Cecil H. Meyers.
Belmont, California: Dickenson Publishing . 1970

Effective Business Report Writing
by Leland Brown.
Englewood Cliffs, New Jersey: Prentice-Hall, 1963.

How to Write a Report Your Boss Will Read and Remember
by Raymond V. Lesikar.
Homewood, Illinois: Dow Jones-Irwin.Inc., 1974.

It is important that program evaluation report present a balanced picture of the program. A report concerned only with unsatisfactory performance will serve to heighten the commonly held impression that evaluation per se is negative. The analyst must make sure that positive performance aspects are prominently mentioned in the report and subsequent presentations.

Step 2 - Reviewing the Draft Report
Several kinds of review are necessary to polish and finalize an evaluation or analysis report. First, the report should be reviewed by the project team leader to make sure the analyst has not left out important material or made misleading statements, by another analyst as a check on methodology and accuracy, and by someone who has not been involved in the study to make sure that the report is clear, understandable, and free of potentially confusing jargon. If the jurisdiction has someone with editorial experience on the staff, such a person can correct grammar, style, and tone.

Second, and most important the draft report should be reviewed by program agency personnel and, in some circumstances, by affected community groups and labor unions. The draft report should be transmitted to the head of the program agency, along with the analyst's suggestions for specific agency personnel who should be involved in the review. The analyst should also specify what procedure and format the program personnel should use in making their comments, as well as set a reasonable deadline. Care should be taken to allow enough time for the agency to conduct the review: the amount of time required will vary with the size and complexity of the report.

The analyst should obtain political clearance form elected officials or the chief administrator before releasing draft reports to persons outside the government. This precaution is to avoid the possibility that study results will be leaked to the media before politically sensitive language can be adjusted and key government personnel become familiar with the report results. Organized community groups should be included in the review process when the program under study has an appreciable interest to such groups.

If program agency employees belong to a labor union, then both union representatives and the government's labor relations officials should also participate in the review. This review is vital to spotting important omissions, errors, misinterpretations of data, faulty methods, poor logic, or unsubstantiated conclusions. Also, since in the case of evaluations it is the efforts of agency personnel that are being "graded." such personnel should have the courtesy of advance knowledge of the findings and an opportunity to respond to statements they feel are unfair or inaccurate.

It is obvious from the above discussion that the project team should allow several weeks for the review process. The analyst should first compile a list of persons or organizations to review the draft report and review that list with the project team leader. The analyst will find that most people can provide much more useful input if they are given some guidance. Consequently, the analyst should draw up a list of several key questions for reviewers to guide their efforts. While such a list must of necessity, be tailored to the specific study and to the viewpoint of the reviewer. Figure 24 presents a list of questions that may help the analyst formulate appropriate questions.

A copy of the draft report and a list of appropriate questions should be sent to each reviewer along with a final date for returning comments. The team leader should analyze the review comments and decide which warrant changes in the report. The most important comments will usually be those coming from the program agency. A decision must be made by the team leader as to how the review comments will be handled. Some jurisdictions include review comments in the final report. San Diego County places agency review comments at the end of each section of the report, printed on a different color paper from the rest of the report. The Office of Program Evaluation (OPE) response to the agency comments is also given in these addendum sections. The OPE staff feels that this approach helps to maintain their credibility with local officials, since the original report keeps its integrity.

An example of the San Diego County approach to handling review comments is shown in Figure 25. Another approach is to append program agency comments to the report as a separate document or section. Either way, the project team leader may wish to respond to the agency comments, or indicate what changes were made to the report as a result of agency review.

Step 3--Transmitting the Final Report
Once the report has been finalized, it is time to present the results to the local decision makers. The exact protocol will depend on local precedents, but it is generally desirable to allow officials several days to review the written report before it is presented orally. In some cases, local officials may not want the project team to make an oral presentation, but a presentation is usually desirable, especially for the first evaluation or analysis project undertaken by a jurisdiction. A presentation gives the project team an opportunity to emphasize points the team considers important, and it gives decision makers an opportunity to ask questions and seek clarification.

The presentation will, of course, be based largely on the executive summary and management report sections of the written report. The presentation should include only that information contained in the write report, although details omitted from the report might be mentioned in response to specific questions. The project team leader should obtain direction from top management concerning the specific issues, topics, or questions that should be addressed in the presentation.

On the basis of that direction, the team leader should organize the presentation with the analyst and other members of the project team, and specific responsibilities should be assigned to members of the project team. The team leader may wish to involve as many of the team members in the actual presentation as possible, since it gives the workers exposure to the thinking of decision makers and also allows team members to take credit for their efforts.

Figure 24. SUGGESTED REVIEW QUESTIONS. The analyst will find that most people find it helpful to have specific questions to guide their report review. Below is a list of suggested questions. Naturally, the analyst will have to compose his own list that is specific to the project and the perspective of the individual reviewer.

Some caution must be exercised, though, to keep the presentation from becoming a disjointed "show-and-tell" session. The team leader may wish to consider having the analyst, or some other team member, prepare visual aids so that the graphics will be consistent and so that someone can build expertise in preparing graphics.

Additional detailed information on planning the strategy of presentations, organizing the material, developing the presentation, and improving individual effectiveness in oral presentations can be found in the following references:

Effective Presentations
by Edward Hodnett,
West Nyack, New York: Parker Publishing Company, 1967.

Presenting Technical Ideas: A Guide to Audience Communication
by W. A. Mambert,
New York: John Wiley and Sons, 1968.

The emphasis of both the report and oral presentations should be on stimulating officials to select courses of action add to commit resources for implementation. Program evaluations and analyses that are academic exercises and do not affect the decision-making process are a waste of everyone's time.

TASK 12--FOLLOW-UP OBLIGATIONS
Perhaps the most vital task in the evaluation and analysis process is the task that converts all of the project team efforts into program improvement--the implementation of approved recommendations. All of the tasks, beginning with the selection of the program to be evaluated or analyzed, have been oriented toward this end. It is all too easy for the project team to get so involved with the mechanics of the study that the production and presentation of the project report becomes the final goal, rather than just one step toward the true goal-the improvement of government services. The project report has been structured to encourage local officials to choose a course of action based on the findings of the project team. That same team should now be concerned with converting that decision into actions. This task encompasses three major steps: (1) forming an implementation team, (2) preparing a work plan, and (3) monitoring progress.

Figure 25. SAN DIEGO COUNTY REPORT REVIEW SYSTEM.
The sample below illustrates how the San Diego County Office of Program Evaluation (OPE) handles program agency comments on draft reports. Pages such as this printed on different colored paper from the rest of the report, are placed at the end of each report section on which the program agency wishes to comment. Space is provided for OPE response.

Step 1 - Forming an Implementation Team
Once local officials have decided which recommendations or program alternatives should be implemented, an implementation project leader should be appointed and should begin detailed planning for the implementation of that decision. This project leader may be the same individual who managed the preceding study.

The first major step is the formation of an implementation team. The team should consist of representatives of the program agencies involved and the analyst who worked on the evaluation or analysis. In some instances, experts from outside the government and representatives of community groups should also be included. Lead responsibility should be given to someone from the agency most involved in the program, with the project analyst serving in an advisory capacity. The inclusion of the analyst is important for three reasons: (1) The analyst has spent considerable time studying the program and this depth of knowledge should not be wasted, (2) Involvement by the analyst will foster the idea that the program evaluation or analysis staff is interested in cooperating to improve government operations and is not " out to get" the agency. (3) The analyst can see first-hand the impact of his recommendations and learn from mistakes. Staff limitations may curtail the amount of time the analyst should at least monitor the progress of the team.

Step 2 - Preparing an Implementation Work Plan
The second major step is the preparation of a work plan covering implementation tasks and steps. The work plan should include a description of all tasks necessary for implementation, a chart showing duration and timing of those tasks and detailed estimate by task of the personnel, dollar, and equipment resources required. A sample of the type of scheduling chart used by the San Diego County Office of Program Evaluation for their implementation work plan is shown in Figure 26. Each recommendations should be shown as a major task, with the key steps necessary to implement the recommendation shown as subtasks.

The City of San Diego uses a slightly different format and approach. Rather than show a time-line, the city's work plan lists each recommendation and subtask along with a target date for completion, and the name of the individual responsible for carrying out that part of the implementation. This approach means that top management mandates various middle management personnel to carry out specific duties by a specified date. Such an approach is relatively simple to monitor and lets each actor know his responsibilities. An expert from a City of San Diego "Action Plan" is shown in Figure 27.

Whatever method is chosen, a work plan will not only provide team members with a clear picture of where they are going, but also provide management with a yardstick to measure progress. See Chapter II for a more detailed discussion on preparing workplans.

Figure 26. SAN DIEGO COUNTY OFFICE OF PROGRAM EVALUATION (OPE) IMPLEMENTATION WORK PLAN. Below is an excerpt from the type of chart that the, San Diego County OPE includes in its implementation work plan to show the timing and duration of the work required to implement each recommendation. The numbers in parentheses are the number of weeks that the subtasks will take.

WORK PLAN FOR THE IMPLEMENTATION OF CHANGES TO THE GENERAL RELIEF PROGRAM AT PILOT LOCATION
INSERT FIGURE 26 HERE

Figure 27. CITY OF SAN DIEGO ACTION PLAN. The excerpt below illustrates how the city of San Diego allocated responsibility for implementing evaluation results. Each task represents a recommendation from the evaluation report. The target completion date and responsibility for each recommendation are clearly shown.

Task	Target Date	Person Responsible
2. Equipment Division storerooms access be restricted to storeroom personnel where and when they are available in compliance with Division instructions	90 Days after Auto Parts Manager position is filled	Keith
a. Prepare memo to Division personnel.	9/22	Keith
b. Assign responsibility for compliance	9/22	Keith
3. Equipment Division Service Writers, whenever possible, take over the preparation of the Buy out request forms and searching parts catalogs for parts numbers	9/22	Keith
a. Notify appropriate Division personnel of action.	9/22	Keith
4. Equipment Division prepare and submit PC-1 to Personnel Department to create Auto Parts Manager Positions	9/22	Trousdale, Oxe, Silbernagel

Once implementation has begun, it should be the team leader's responsibility to track the progress of the effort. The work plan should provide several milestones to measure progress. As major milestones are reached, the implementation team leader should report to local decision makers whether the implementation is proceeding according to schedule and whether the implemented actions are actually having the effect expected by management. In order to accomplish the second objective of progress monitoring, the analyst will have to measure program evaluation criteria on an ongoing basis, in essence performing a mini-evaluation. The second aspect of progress reporting should be of key interest to the analyst, since it should provide feedback on the accuracy and practicality of the analysis. The reporting process should continue until implementation is complete at which time management should begin to make plans for a formal evaluation of the program.

Both the implementation team and management should be aware that implementation may not proceed strictly according to the workplan. Implementation is usually fraught with practical difficulties. For example, a clerical operation was analyzed using industrial engineering techniques to determine the amount of work each employee should produce each day under new procedures. The analyst who made the calculation forgot to include accepted industrial engineering factors for nonproductive time, thus inadvertently creating greater expectations for operational improvement than were actually realized.

Other types of common problems, such as under estimating the time required to change procedures or train personnel can easily delay or reduce expected benefits. It may be necessary for the implementation team to devise and seek management approval of revised recommendations. Effective implementation requires flexibility, and can have tremendous educational benefits for analysis personnel.

SUMMARY
Program evaluation and program analysis are closely related processes aimed at providing State and local government officials with improved information on program effectiveness for use in making resource allocation decisions. Evaluation provides information on the impact of existing government efforts and highlights areas that need improvement. Analysis can then be employed to help determine the most effective form for those improvements to take. After program improvements have been implemented, evaluation is once again. The cycle can also begin with program analysis used to determine the best way to institute a totally new program, followed by an evaluation of the program activities. While evaluation and analysis can be used separately, the payoff is greatly improved by using both processes together.

The successful use of evaluation and analysis depends on:

The existence of, or the ability to formulate meaningful goals, objectives, and evaluation criteria for public programs; The ability to measure program effectiveness through the collection and interpretation of data; The willingness of public officials to support the process by basing resources allocation decisions on the information presented to them and The commitment on the part of local officials to implement the recommendations of evaluation and analysis projects.

Evaluation designs are of three major types: (1) experimental, (2) quasi-experimental, and (3) nonexperimental. A useful design for any particular study must allow the analyst to determine what effects are attributable to program activities, and what effects result from other influences. External factors that are not taken into account are a source of invalidity. The most relevant of these sources are presented in Figure 28.

The ideal evaluation design guards against all possible sources of invalidity. The only designs that fit this description are experimental designs. Experimental designs involve setting up a program as if it were a scientific experiment. That is, a target group is established of "clients" (people, precincts, work teams, etc.) that are alike in as many respects as possible. Members of this target group are assigned randomly to be part of a control group or a program group. Each group is measured according to pertinent evaluation criteria. The program group is exposed to the program in question and the control group is not. At the end of the program, or at some interim point, both groups are again measured against evaluation criteria. The two groups' preprogram and postprogram scores are then compared, and the difference, if any, is attributed to the program.

Figure 28. INVALIDITY FACTORS. The factors below are the major sources of invalidity that can affect the results of an evaluation. (Source: Van Maanen, Jr.. The Process of Program Evaluation. Washington, D.C.: National Training and Development Service Press, 1973.)

An example of the use of such a design can be seen in the testing of high-intensity street lighting to combat crime. Because the jurisdiction does not know if the new lighting will in fact reduce crime and/or traffic accidents, officials are reluctant to commit the considerable amount of money required without better evidence. A number of geographic areas within the jurisdiction are identified that have highly similar characteristics (crime rate, land use, population density, traffic patterns, family income, etc.). Some of these areas are then randomly designated to receive the new street lights. Data on crime and automobile accident rates for all of the areas are gathered and compared at six months and again a year after the installation of the new lights. If no other causes can be identified, then any significant change in the crime or accident rates is attributed to the new lights.

Obviously, this design poses some serious problems. Many local government programs are not experimental: that is, they are ongoing programs of regular services (street patching, recreation programs, fire prevention, etc.) that cannot be denied to a segment of the population merely to provide a basis for evaluation. Even when the type of program lends itself to the use of this design, the jurisdiction leaves itself open to potential political problems if community groups feel that they are being used as guinea pigs or are being arbitrarily denied improved services. Experimental designs can be used only for programs that have not yet been implemented because of the need for a control group unaffected by the program. Critics also claim that experimental designs require holding the program constant rather than improving it on a continuing basis.

As the above brief description should indicate, experimental designs are the most expensive and time-consuming to use. Experimental designs are, however, the most precise and can be useful in certain limited circumstances to State and local governments, such as in the street lighting example given above. Since their use is the most limited, no experimental designs have been documented in this Guide.

The next type of evaluation design is the quasi-experimental type. As the name suggests, quasi-experimental designs, but have many of the same features. Quasi-experimental designs do not require rigorous comparison group selection but instead use comparison groups that are closely matched in many characteristics with the program group.

The third category of evaluation designs is the non-experimental type. Nonexperimental designs do not use any kind of similar group for comparative purposes. Instead, such things as planned performance or the trend of program evaluation criteria over time are used for comparative purposes.

It is conceivable that a jurisdiction, after gaining some experience with program evaluation, may wish to undertake an evaluation project that would involve a design more appropriate to its special circumstances than the designs presented in this chapter. Accordingly, the references listed below are provided so that a jurisdiction that has gained some experience in program evaluation can have access to the full range of available designs. These references also apply additional technical information on the three designs presented in this appendix.

Evaluation Research
by Carol H. Weiss,
Englewood Cliffs, New Jersey: Prentice-Hall. 1972.

Experimental and Quasi-Experimental Designs for Research
by Donald T. Campbell and Julian C. Stanley.
Chicago, Illinois: Rand McNally, 1966.

Evaluative Research
By Edward A. Suchman.
New York: Russell Sage Foundation, 1967.

Appendix B
SAMPLE SURVEYS

Surveys are tools for questioning selected samples of the general public. They may involve mailing questionnaires to respondents, leaving questionnaires at respondents' homes and retrieving them at a later date, interviewing respondents in person, or interviewing respondents over the telephone. Surveys provide feedback on respondent perceptions, desires, needs, preferences, priorities, opinions, and experiences. This information can be used to augment the following program evaluation and analysis activities:

Formulating program goals and objectives.
Choosing among alternative programs to meet these objectives, and
Measuring program accomplishments and results.

Potential Benefits
The primary benefit that surveys offer is the capacity to elicit the views of numerous individuals, many of whom would not otherwise participate in the evaluation and analysis process. Thus, survey information can be more representative of the public at large than information obtained through other kinds of public involvement efforts.

Surveys also offer the following benefits:

Survey responses can be readily analyzed to determine underlying patterns and relationships, including trends over time.
Survey can focus on specific respondent groups and/or specific issues or objectives of interest to the user jurisdiction.
Surveys can illuminate the rationale behind respondent answers.
Surveys can gather information about people's perceptions, desires, and opinions unavailable from other sources.
Surveys can reduce the sense of isolation or alienation felt by many respondents.

Selected Documentation
The following five documents can provide valuable assistance to jurisdictions interested in administering surveys:

An introduction to Sample Surveys for Government Managers by Carol H. Weiss and Harry P. Hatry.

Obtaining Citizen Feedback: The Application of Citizen Surveys to Local Governments by Kenneth Webb and Harry P. Hatry.

Survey Research by Charles H. Backstrom and Gerald H. Hursh.

Citizen Involvement/ Communication Manual
Chapter VIII. "Surveys" by the City of Lakewood, Colorado.

Survey Manual for Comprehensive Urban Planning: The Use of Surveys and Sampling Techniques in the Planning Process by Jerome R. Saroff and Alberta Z. Levitan.

An Introduction to Sample Surveys and Obtaining Citizen Feedback, both published by The Urban Institute, promote the use of surveys in government on a regular scheduled basis. The books provide background information valuable in deciding whether or not to administer a survey. Topics discussed include:

Potential applications of survey techniques; Organization of a sample survey; The relative merits of building an in-house capacity to administer surveys versus hiring outside consultants; The advantages and disadvantages of different survey modes, including cost information; and Pitfalls commonly encountered in administering surveys.

Survey Research is a nontechnical manual dealing with personal interview surveys, but the material presented is equally useful for other survey modes. Government personnel should consult Survey Research for instruction in the following requisite survey steps:

Selecting a representative sample and an adequate sample size;
Developing effective personal interview introductions:
Developing effective questions;
Identifying necessary demographic questions;
Designing effective questionnaires;
Training, briefing, motivating, and equipping interviewers;
Coordinating fieldwork; and
Coding responses.

Citizen Involvement/ Communication Manual, Chapter VIII. "Surveys," presents guidelines for preparing, conducting, and reporting on mailed, drop and pick-up, personal interview, and telephone surveys. The Lakewood guidelines cover the following subjects:

Deciding whether a survey is needed and feasible; Writing and sequencing questions; Selecting a survey sample; Pretesting a questionnaire; Coding questionnaires and analyzing survey results; and Preparing a report on survey results.

Survey Manual for Comprehensive Urban Planning presents a case study of a personal interview survey in Providence, Rhode Island, and a methodology for designing surveys, collecting data, and analyzing the results. Government personnel should consult this manual to select a survey design and an appropriate sampling method.

User Experience
Although many jurisdictions use surveys on an ad hoc basis to evaluate specific programs, few administer regularly scheduled surveys. The following jurisdictions survey their citizens on a regular basis:

St. Petersburg, Florida, just completed its third annual personal interview survey of citizen perceptions about municipal service effectiveness.

Metro Nashville-Davidson County, Tennessee, completed two annual personal interview surveys of citizen perceptions about municipal service effectiveness and switched to quarterly telephone surveys in 1976.

Dayton, Ohio, has been using annual personal interview surveys for several years to provide the City Council with information about community priorities during the budget-setting process.

New Orleans, Louisiana, has completed one extensive personal interview survey and is planning biennial telephone surveys to identify neighborhood needs and problems.

Dallas, Texas, has conducted three annual citizen surveys to measure citizen satisfaction with the quality and effectiveness of city services.

In addition to the above general surveys, Winston Salem, North Carolina, and Phoenix, Arizona, have made extensive use of surveys to evaluate specific programs.

Adaptation Considerations
Jurisdictions considering the use of surveys should review An Introduction to Sample Surveys for Government Managers, by Weiss and Hatry, and Obtaining Citizen Feedback: The Application of Citizen Surveys to Local Governments, by Webb and Hatry, and contact other jurisdictions experienced in their use. For more information about how actually to administer a survey, they should review Survey Research, by Backstrom and Hursh: Citizen Involvement / Communication Manual. Chapter VIII. "Surveys," by Lakewood. Colorado; and Survey Manual for Comprehensive Urban Planning, by Saroff and Levitan. Figure 29 provides information on how to obtain these documents and whom to contact in other jurisdictions.

Jurisdictions considering the use of surveys should be ????of the following relationships between sample size, survey mode, survey accuracy, survey cost, and the need for trained, experienced personnel:

Special expertise in using sample, questionnaires, and interviewers is needed to minimize the bias present in survey results;

Consulting survey specialists used in place of staff expertise can triple survey costs:

Cost increase with increasing sample size;

Personal interview surveys are more expensive than telephone surveys, and both are more expensive than drop and pick-up and mailed surveys;

Drop and pick-up and mailed surveys are more easily biased than personal interview and telephone surveys; Survey results are only as accurate as the respondents answers, which reflect their knowledge, memories, and motivation, so responses to questions requiring special knowledge should be interpreted carefully to avoid mistaking opinions or guesses for facts;

Survey representativeness is affected by the number of citizens who object to being interviewed because they fee it is an invasion of privacy, or, especially in communities which are surveyed to often, a waste of time; and

The results of surveys dealing with a single specific service or program are easier to interpret and apply than the results of more general surveys.

One of the most difficult information sources for a local government to tap is the work done by other jurisdictions. Information on the program alternatives considered or adopted by other jurisdictions can be of valuable assistance to a program analysis project. Evaluation reports from other jurisdictions can sometimes help the analyst identify effective evaluation ???. In an effort to stimulate this type of information change, five governments have volunteered their of completed program evaluation studies. Short description of each study are listed by functional areas contact person for each jurisdiction is noted at the of the listing. The analyst is surged to contact appropriate jurisdiction by phone to get a more detail description of the study scope before requesting a copy of the study report.

FOR INFORMATION ABOUT:
Specific studies conducted in Winston-Salem, North Carolina, contact:
Gary Brown
Evaluation Director
Room 817
NCNB Building
Winston-Salem, North Carolina 27102
(919) 727-2653

Specific studies conducted in Phoenix, Arizona, contact:
Charles E. Hill
Budget and Research Director
Room 801
151 West Washington
Phoenix, Arizona 85003
(602) 262-6721

Specific studies conducted in San Diego County, California, contact:
Donal Fisk
Acting Director
Office of Program Evaluation
1600 Pacific Highway
San Diego, California 92101
(714) 236-4053

Specific studies conducted in the City of San Diego, California, contact:
David Knapp
Assistant Director of Financial Management
202 "C" Street
San Diego, California 92101
(714) 236-6060

Specific studies conducted in Long Beach, California, contact
James E. Phelps
Budget and Research Director
Room 207, City Hall
Long Beach, California 90802
(213) 436-9041

	Alternative	#1	#2	#3	#4	#5
	Personnel	$ 70,000	$ 55,000	$ 40,000	$ 15,000	$ 25,000
1st Year Cost	Capital Equipment	$ 10,000	$ 20,000	$ 65,000	$ 20,000	$ 17,000
	Contact Services	----	----	$ 10,000	$ 65,000	$ 5,000
	Other Expenditures	$ 3,000	$ 3,000	$ 5,000	$ 5,000	$ 3,000
	Total	$ 83,000	$ 83,000	$ 120,000	$ 105,000	$ 50,000

	Personnel	$ 75,000	$ 58,000	$ 45,000	$ 20,000	$ 30,000
	Capital Equipment	----	----	$ 2,000	----	$ 2,000
2nd Year Cost	Contact Services	----	----	----	$ 72,000	----
	Other Expenditures	----	$ 2,000	$ 3,000	$ 3,000	$ 3,000
	Total	$ 75,000	$ 60,000	$ 50,000	$ 5,000	$ 35,000

	Personnel	$ 80,000	$ 63,000	$ 49,000	$ 22,000	$ 35,000
	Capital Equipment	----	$ 1,000	$ 2,000	----	$ 2,000
3rd Year Cost	Contact Services	----	----	----	$ 79,000	----
	Other Expenditures	----	$ 4,000	$ 3,000	$ 1,000	$ 3,000
	Total	$ 80,000	$ 68,000	$ 54,000	$ 102,000	$ 40,000

	Grand Total	$ 238,000	$ 211,000	$ 224,000	$ 302,000	$ 125,000

Description	Jurisdiction
POLICE
Citizen attitudes toward police	Winston-Salem, North Carolina
Survey of police attitudes and job satisfaction	Winston-Salem, North Carolina
Police Department program analysis and review	Phoenix, Arizona
Analysis of city police department costs vs. county sheriff costs	Long Beach, California
Development of MBO structure	Winston-Salem, North Carolina
Refined analysis of crime data	Winston-Salem, North Carolina
FIRE
Fire department program analysis	Phoenix, Arizona
Productivity and salary evaluation	Phoenix, Arizona
Evaluation of fire inspection operation	Winston-Salem, North Carolina
Fire condition information system	Winston-Salem, North Carolina
PUBLIC WORKS
Street maintenance program analysis	Phoenix, Arizona
Equipment division parks management evaluation	City of San Diego, California
Posted street sweeping analysis	Long Beach, California
Construction inspection and clerical work load evaluation and analysis	Long Beach, California
Optimal retirement and replacement periods for large equipment	Winston-Salem, North Carolina
Impact of special clean-up and rodent control programs	Winston-Salem, North Carolina
Uncontainable refuse service policy analysis	Phoenix, Arizona
Productivity improvement, equipment division	City of San Diego, California
RECREATION AND PARKS
Parks division program analysis	Phoenix, Arizona
Needs analysis for bike paths	Winston-Salem, North Carolina
User and nonuser surveys of various recreation services	Winston-Salem, North Carolina
Evaluation of proposed expansion of golf course concession	Long Beach, California
Evaluation of several recreation centers	City of San Diego, California
Productivity improvement, parks division	City of San Diego, California
HOUSING
Impact of housing inspections	Winston-Salem, North Carolina
Development of system to measure impact and work outputs of inspection operations	Winston-Salem, North Carolina
HUMAN RESOURCES
Evaluation of Food Stamp program	San Diego County, California
Alcohol detoxification evaluation	San Diego County, California
Evaluation of three alcohol care and treatment programs	San Diego County, California
Head Start program evaluation	San Diego County, California
General Relief welfare evaluation	San Diego County, California
LEAP program analysis (Community Action Agency)	Phoenix, Arizona
COMMUNITY AND ECONOMIC DEVELOPMENT
Hotel market data study	Long Beach, California
Economic Development Corporation evaluation	City of San Diego, California
Impact of community development program	Winston-Salem, North Carolina
Impact of Urban renewal, rehabilitation, and relocation	Winston-Salem, North Carolina
PERSONNEL
Evaluation of personnel policies	San Diego County, California
Personnel department program analysis and review	Phoenix, Arizona
TRANSPORTATION
Traffic engineering program analysis and review	Phoenix, Arizona
Bus-user survey to determine satisfaction and potential change	Winston-Salem, North Carolina
Non Bus-user survey to examine market potential	Winston-Salem, North Carolina
FINANCE
PPB systems evaluation	Phoenix, Arizona
Bicycle license study	Long Beach, California
Marina and launching ramp fee study	Long Beach, California
MISCELLANEOUS
Court program analysis and review	Phoenix, Arizona
Library program analysis and review	Phoenix, Arizona
Planning department Environmental Studies Division	Long Beach, California