Office of Juvenile Justice and Delinquency Prevention. Evaluating Juvenile Justice
Programs: A Design Monograph for State Planners. Washington, DC: Prepared for the
U.S. Department of Justice, Office of Juvenile Justice and Delinquency Prevention by
Community Research Associates, Inc.; 1989.  pp. 50-56.

View entire document

Building Evaluation Into a Program RFP

One of the best ways to insure that evaluation information will be available for funded programs, especially information that is useful to you and others, is to require production of that information as "deliverable." If you know you want to evaluate the program, then you know that you want to collect basic monitoring information. Why not include a requirement to collect such information in the published program RFP or as a condition of program funding? While you may believe that programs collect that information anyway, and don't want to place unnecessary bureaucratic burdens on your program personnel, there are some good arguments for being more aggressive for producing evaluation information in the RFP and funding processes. These include:

• Potential grantees or funding recipients need to know that you are serious about evaluation; that the information they produce is important to you and others. Often, they rightly complain not about producing evaluation information, but about producing information that they feel will not be used by anyone.

Other Considerations

• Placing evaluation information needs in a program RFP forces you to think seriously about evaluation design and information needs. Once you resolve the issues regarding research questions, data, and research design for yourself, they will be more easily communicated to program administrators and evaluators.
• Documenting your information makes the collection of the information routine, and strengthens your ability to demand it. It is difficult to introduce request for information to program administrators or evaluators midway through a project. Likewise, it is hard to ignore information needs that are in writing and that have been agreed to. Once a data collection effort is started it quickly becomes a natural organizational process.
• It is important for program administrators to know that the program will be evaluated if you intend to do so. Explaining this in the RFP puts them on notice and eliminates the element of surprise for staff that can occur when outside evaluators come knocking.

Even if you are not accustomed to documenting evaluation plans in program RFPs, or to requesting evaluation information from your grantees, these points argue for doing so. By making your plans known, and being as specific as possible about your expectations, you increase your chances of conducting and completing a useful program evaluation.

Including evaluation plans in a juvenile justice program RFP is not difficult. The following suggestions are offered as a means of doing so.

• State your plan to evaluate the program at the outset. Most RFPs begin with a summary of the project or program to be funded. This statement should clearly state that program evaluation will take place and who will conduct the evaluation, and that certain information requirements will have to be met as part of the program activities.
• Stress that program goals must be clearly stated in the proposal, as well as objectives or action plans for achieving the goals and a demonstrated link between program operations, goals, and objectives. While it may not be the case that program "success" will be measured by goal attainment (often this need not be the case)m to begin any evaluation the evaluator must have some notion of what the program developers had in mind in the way of goals, objectives, and operations at the time of the program's creation. This information must be made explicit in the program proposal.
• Include a section on evaluation information. To the extend that you can be specific, identify in a separate section the types of evaluation program monitoring information you expect to be collected. This may be as brief as a list of information items, and a stipulation that the details will be worked out soon after the award is made, or as complete as a list of information items with definitions, formats, data sources, reporting schedules, and so on. The import thing is to identify in as much detail as possible, the data collection responsibilities the grantee assumes. This serves to stress the seriousness of the evaluation effort and to put potential grantees on notice.
• Include a section on evaluation plans. Regardless of the extent program administrators involvement in the actual evaluation or whether you plan to use outside evaluators, it is important to discuss the ways in which the evaluation information provided will be used. For example, if you are monitoring for your information and for the SAG's information, explain that the data will not be widely circulated. If you plan on conducting a comparative monitoring or evaluation effort, explain that others will be collecting the same information, and that the feedback will be useful to program administrators. Similarly, if you plan to produce a widely circulated program description monograph, let that be known. It is also important to delineate the many values of the evaluation information. It will help funders and program administrators make decisions, it will help educate the public and legislature and perhaps reduce myths surrounding the program, it will help others replicate their work, and so on.

If you plan on involving an outside evaluator, there are two additional important pieces of information to include in the REP:

(1)  The statement that an outside evaluator will be involved.

(2)  The statement that the evaluation effort will be a collaboration between the program, your office, and the evaluator. It is important for the staff to know that they will have an opportunity to educate the outside evaluator and assist in the interpretation of data, and that your office is interested enough in the evaluation to be involved. Even in the most routine programs, the presence of an outside evaluator often requires the involvement of someone at your level to keep things on a positive note and to smooth out any potential trouble areas. The expected evaluation products should be discussed to provide potential grantees with a clear understanding of why the information is needed and how it will be ultimately used.

Preparing an Evaluation RFP

If you are going to use an outside evaluator, more than likely you will need an evaluation Request for Proposals (REP) to solicit proposals for the project. Preparing evaluation RFPs is much like preparing any other RFP-it must be detailed and specific regarding the activities and products you will pay for, schedules, responsibilities, and the like. In this section, we will review some important guidelines for preparing an evaluation RFP, paying particular attention to the qualities that set them apart. We recommend, however, that the reader also consult a more thorough reference on preparing requests for proposals, and obtain a good example to follow.

A comprehensive request for proposals will contain at lease five key sections:

(1)  A statement of objectives that states clearly and concisely the project to be funded and the need for the evaluation effort.

(2)  A section containing general information of applicants. This section normally contains technical and logistical details that applicants must know.

(3)  A specifications section that explains in detail the entire evaluation project-purpose, timelines, mandatory deliverables, products, and so on. This is the most important section of the RFP.

(4)  A section outlining the detailed information requirements that must be contained in a proposal.

(5)  A section explaining the RFP evaluation and selection process.

In most states and other jurisdictions, there are a host of sections and subsections that are required for an RFP. These outline specific information requirements e.g., financial disclosure statements, affidavits of various kinds, minority business enterprise information, required clauses for termination of contracts, and the like. Too numerous to mention here they can be obtained from the appropriate procurement office in your state or local government. Again, obtain a copy of a successful RFP that has passed the procurement office standards if possible. Following is a review of the five sections mentioned above.

Objectives of the RFP

This section serves as the introduction for the reader of the RFP, usually potential applicants or grantees. It states your intent to solicit proposals for the evaluation of one or more specific programs, provides the RFP schedule-when proposals are due, when the evaluation process begins and ends, and such. It is usually brief, and may include other sections such as a brief statement that a formal proposal evaluation process will take place, and the makeup of the proposal evaluation committee, information about a pre-proposal conference in which potential applicants meet to obtain other information from your office, if applicable, and a statement that final approval of the award is subject to review from a higher authority such as the state budget office, perhaps, or the procurement office.

General Information of Applicants

This section normally contains many subsections with logistical and other information that is required in all RFPs. The following is a list of the types of information that should be required. It is not comprehensive since these requirements vary by jurisdiction. You may even wish to present them as section topics under "General Information for Applicants."

(1)  Names and phone numbers that applicants can call with questions and inquires about the RFP.

(2)  A statement that your office reserves the right to amend or cancel the RFP process at any time, and that all applicants will be notified of such changes.

(3)  A statement that your office reserves the right to require any applicant to make an oral presentation to clarify the proposal.

(4)  A statement that your office will not assume any of the proposal preparation costs.

(5)  Suggest a method for submitting multiple proposals, or limit each applicant to only one.

(6)  Requirement of financial disclosure by applicants, according to the relevant statute.

(7)  Requests for certifications of various kinds required by low such as anti-bribery affidavit, non-collusion certificate, public information act notices, procurement affirmation, and minority business enterprise certification.

(8)  A statement explaining the conditions under which your office may terminate any contract made under this RFP.

(9)  A reference to the relevant statutes and procedures regarding disputes or protests that arise.

(10)  References to any federal laws or guidelines that apply.

Most of this information is standard for RFPs, though the language and detail varies. They often appear tedious, bureaucratic, and irrelevant, but they serve important purposes-the protect you and your superiors from a number of liabilities, and they provide objectivity and fair competition for the procurement process.

Specifications

This is one of the most import sections of the RFP. In it you explain the program(s) to be evaluated, and the evaluation you would like to have completed by the grantee. This section should contain the following subsections:

• Background information. A review of the program(s) to be evaluated, including the following information-historical detail, goals and objectives, program activities and administration, relevant findings and statistics to date (if applicable), the federal initiative (if applicable), and reference to similar programs.
• Type of evaluation. Unless you are suggesting that the applicant devise the entire evaluation plan, you should explain the type and level of evaluation that you desire whether monitoring, comparative monitoring, process or outcome evaluation. Again reference to similar evaluations might be wise.
• Information requirements. The type of evaluation desired will automatically suggest certain evaluation information requirements whether monitoring information, process measures, and/or outcome measures. These should be explained in detail.
• Evaluation products. This section should explain how the evaluation information should be packaged. You may specify a final report (or two reports-a non-technical summary, and a technical document), a idea, a training seminar, a presentation to relevant audiences and so on
• Timelines. This section should contain detailed information about important dates and time-specific components relating to the evaluation-program start date, periodic review dates, important meeting dates, funding period, the treatment process time, follow-up periods, etc. You should also mention the time period of acceptance, the time after delivery of final products that you will review the products before making final payment.
• Product review. This section should mention the review process that the final products delivered will undergo-the persons involved, the schedule, the criteria for acceptance.

Two important points bear stressing here:(1) The quality of the proposals you receive will depend on the quality of this section. A well thought-out evaluation process will usually receive good proposals, while one that is poorly thought-out will not provide solid enough information, resulting in proposals that are not responsive to your evaluation needs.

A well thought-out evaluation process will usually receive good proposals, while one that is poorly thought-out will not provide solid enough information, resulting in proposals that are not responsive to your evaluation needs.

(2) Applicants will be judged on how they respond to the issues presented in this section. If program and evaluation details are not clearly stated, the preview process may steer you in the wrong direction. You may find yourself forced to choose a proposal you know is not the best.

Finally, a word on mandatory versus desirable specifications. A good way to judge proposals and to get the most from applicants is to specify mandatory specifications which are elements that an evaluation plan must have to be considered at all and desirable specifications which are elements that you would like to see developed as part of the evaluation plan, but that you leave up to the applicant.

For example, you may specify a process or short-term outcome evaluation as mandatory, but specify an end process outcome evaluation plan as desirable. This may result in the applicant giving the extra effort in terms of design and deliverables. Or, you may specify a final report as mandatory, Or, you may specify a final report as mandatory, and cite other deliverables as desirable such as a small public information brochure, a formal presentation to staff and the SAG, and so on. If the applicant knows the proposal will be evaluated on the extent to which desirable elements are delivered, you increase your chan ces of getting the most for your money.

Information Required in the Proposal

This section is nearly as important as the previous one. It specifies for the applicant exactly the information that should be contained in the proposal; it may even go as far as providing a suggested outline. It should be stated clearly that a proposal that does not contain all the information required will not be considered.

In addition to a detailed explanation of the evaluation program according to the Specifications section, the following should be required from applicants:

• A detailed discussion of the evaluation management plan, listing the people who will work on the project and including their updated resumes, project management, and the responsibilities of each person.
• A detailed budget should be required, accounting for every dollar by spending category including salaries, fringe, overhead, indirect, supplies, contractual, travel, and so on.
• Applicants should be required to submit examples of prior work in program evaluation, and references to past employers.

Evaluation and Selection

It is important, and fair, to provide applicants with precise information on how their applications will be evaluated. This section should outline the proposal evaluation process, the persons who will evaluate them, the evaluation criteria, and weights or mathematical calculations that will be applied, and how and when the final selection will be announced.

Some considerations in this area include:

• Will technical criteria, such as the logic and complexity of the research design be weighted the same as cost criteria (how much the evaluation will cost) or will technical criteria dominate?
• How much influence, will the delivery of desirables versus mandatory specifications have on the evaluation process?
• Will the evaluation be qualitative or quantitative? You may devise an elaborate scoring scheme, with appropriate weights, and have the resulting scores drive your decision, or you may have each of the evaluation committee members read the proposals and make written recommendations with reasons provided. You may combine a qualitative and quantitative approach. Procurement officials tend to like quantitative approaches, since they avoid subjectivity in the evaluation process, but you may find them restrictive.

An evaluation RFP process is recommended if you plan on spending more than a nominal amount of money on evaluation, if competition for such business is high, and especially if you want to maintain an environment of objectivity regarding your evaluation program. You may have no choice in the matter if your state of jurisdiction mandates an RFP process

Additionally, the RFP should provide an indication of the dollar amount you plan to allocate for the project. Otherwise the applicants may propose evaluation projects that vary widely in scope and methods. If you wish to spur competitiveness in the RFP process without revealing how much money you plan to spend on evaluation, you may provide information regarding program costs, or suggest a cap on evaluation project costs and indicate the project cost will be a significant factor in the proposal evaluation.