Planning to Evaluate an Information Sharing/Integration Initiative? What Are Some Challenges?

Difficulties in sharing information across multiple stakeholders require that evaluators address implementation issues before conducting usability and outcome evaluations.

Outdated mainframes, incompatible network systems, rules for information sharing, and varying definitions and standards among users can be obstacles to the successful implementation of information sharing/integration initiatives. An evaluator must be confident that information can be successfully shared between all potential users before attempting to assess the usability and impact of the information sharing/integration initiative or its outcomes. Hence, careful study of the information sharing/integration initiative implementation process must be conducted by the evaluator prior to conducting usability and outcome evaluations. Doing so will decrease the chances of obtaining misleading results or unusable data.

It is important to measure long-term outcomes, but doing so may be problematic.

Most studies of information sharing/integration initiatives have either measured implementation or usability. This work has provided valuable knowledge for understanding how to properly implement the systems. However, increased effort is needed to show whether these systems are accomplishing their goals. Outcome measures that address long-term outcomes relating to service, programmatic, operational, financial, and return on investment factors should be developed to determine if the new information sharing/integration initiative systems are making a significant impact concerning the stated short- and long-term goals and objectives.

It must not be assumed that if short-term objectives are accomplished, long-term goals and objectives will be reached. Though it may be difficult to determine if the new system is the cause of reaching a long-term objective, such as increased public safety, links between objectives like increased efficiency in processing information and public safety must be demonstrated. Measuring long-term goals such as increased public safety should not be overlooked even though the accomplishment of a short-term objective like increased efficiency in information processing has been reached. Evaluation is needed to demonstrate whether the accomplishment of short-term objectives is linked to the accomplishment of long-term goals/objectives.

Rigorous evaluations of information sharing/ integration initiatives may be difficult to design and implement.

Since this is a relatively new area of study, there are few prior outcome evaluations to assist in developing and implementing appropriate research designs. Studies have examined the impact of the information sharing/integration initiative, such as impact on job performance, impact on productivity level, and accuracy and efficiency of the information sharing/integration system. A common approach to these studies has been the use of questionnaires both before and after using the new initiative. Although this method can indicate improvements with the implementation of a new data sharing/integration initiative, it does not permit one to determine if these changes are attributable to the new system or some other factor(s). A before/after design with a comparison group (e.g., another jurisdiction/agency with similar characteristics not implementing the new system) would yield stronger evidence for the effectiveness and efficiency of the new system. An even stronger approach would include random assignment to the treatment or comparison group. An example would be randomly assigning 10 police precincts where half implement the new information sharing/integration initiative while the other half use the old approach. Because of the nature of information sharing/ integration initiatives, this may be a difficult research design to implement within one jurisdiction. The best time to attempt this would likely be in the pilot-testing phase of a new initiative.

| | |
| | |