Chapters 10-15

CHAPTER 10
MEASURES OF COMMUNITY COLLABORATION

Many individuals and organizations involved in addressing violence against women have heard about collaborative community-wide responses to ending such violence and in many communities across the country, efforts are under way to establish such systems. But what does coordinated community response really look like, and how do you know it when you see it? Furthermore, what is coordinated community response meant to achieve, and how does one measure this?

To most observers, the term "community-wide collaboration" conveys the fairly obvious, if somewhat vague, notion of different members of a community coming together to address a common problem. Clearly, there is more to community collaboration—or at least successful collaboration—than this. Christiansen et al.(1997) ¹ characterize collaboration as involving interdependence and ongoing give and take; solutions emerging through participants' dealing constructively with differences; partners working beyond stereotypes to rethink their views about each other; and stakeholders assuming joint ownership of decisions and collective responsibility for future decisions. They believe that collaboration is an ongoing process, not something done once and finished. Partners in collaboration are always looking at how things are going, and talking through how to keep improving them.

It is important to keep in mind that there is no single ideal of community collaboration. What makes sense for one community may not work in another. There are, however, a variety of scenarios with common elements that appear to be effective. These elements or factors can provide us with models against which we can measure efforts in most communities.

The remaining sections of this chapter identify (1) the basic elements needed to start the process of community collaboration, (2) intermediate and system level outcomes associated with successfully establishing community collaboration, and (3) the ultimate outcomes of such collaborations. As noted above, collaboration is a never-ending process. Each success builds on the last one. This means that the process is circular—commitment (the first essential ingredient for a collaborative community-wide response) leads to system changes that lead to renewed commitment, etc. Similarly, each system change that is accomplished feeds a willingness to increase commitment for the next change. This complexity reflects the dynamic and fluid nature of establishing a community-wide collaborative response to violence against women.

It should also be noted that the greater the number (and longevity) of factors present in a given community, the more likely it is that the collaborative effort is truly entrenched within that community and that the effort will survive the withdrawal of a major source of financial support such as VAWA STOP grants. We have provided three tables (Tables 10.1, 10.2, and 10.3), each focused on one possible goal of a collaborative effort. Each table provides specific objectives under each goal, measures of these objectives, and recommended data collection procedures. Please keep in mind that this is not an exhaustive list. Those who are closest to the communities and are actually assessing the collaborations may have other measures in mind and/or have a better idea of what is realistic in terms of data collection.

Elements of Community-Level Collaboration

Commitment

The foundation of any type of community collaboration is commitment. This includes an understanding that (1) violence against women is a community problem that must be addressed by the community; (2) community-wide approaches are necessary to provide all victims with a full range of options and services free from unnecessary hassles and difficulties, to hold perpetrators accountable for their actions, and to create public confidence and trust in community institutions; and (3) changes are (almost certainly) necessary in all service units separately, and in the way they interact with each other, to promote excellent care and continuity for victims and to hold perpetrators accountable.

Organizing Structure

Coordinating bodies are a critical part of a community's efforts, providing a forum for identifying problems and developing solutions on an ongoing basis. When faced with the challenge of developing an integrated community system, it is important that the organizing body have some legitimized power or authority to make recommendations and/or decisions. This means that the committee or group is, at least, the recognized structure for interunit communication, for feedback about one's own unit's performances, for establishing priorities among the community-wide tasks to be done, and for public outreach and dissemination about its goals and activities.

Organizing structures can take many forms, but some of the most common are: a committee or task force; a nonprofit, nongovernmental body that assumes coordinating responsibility; or a small, informal network of two or more agencies that plan and implement activities together. As Table 10.1 illustrates, in addition to noting the structure of organizations, it is also useful to gather information such as the length of and reasons for their existence.

Committees. A coordinating committee or task force may employ a coordinator—full- or part- time—whose primary responsibilities are to organize all efforts of the committee, serve as the central contact for its members, and facilitate communication among members. If funding is not available for a full-time position, a suitable member from the community may serve as coordinator on a part-time or voluntary basis. In addition to aiding the work of the task force, the presence of a coordinator may help to ensure continuity in a politically changing environment. Committees (and their subcommittees) vary greatly in their frequency of meeting, generally ranging from once a month to one or two times a year. Meetings provide a forum for updating members on progress toward set goals and the development of new issues.

Nonprofit Organizations. Another possible structure is a nonprofit organization that takes responsibility for some or all of the community's collaboration efforts. Possible scenarios include an organization that was created for the sole purpose of coordinating the community response or an already-existing organization that takes control of one or two specific projects.

Others. In communities where there is no formal coordinating structure, a number of agencies representing different areas (e.g., nongovernmental victim services and law enforcement agencies) may choose to work together to develop collaborative projects. In others, there may be more than one coordinating mechanism.

Formation of Additional Structures. In some cases, new VAW coordinating structures appear. They are formed for various reasons, which may include the realization that certain needs are not being met by the initial coordinating body, disagreement over objectives or means of achieving them, or simply the need to extend the reach of existing structures.

Composition of Committee/Group

Diverse Membership. Diversity of membership is important because it is either difficult or impossible for one person or agency to solve a problem as multi-faceted as violence against women. The challenge of collaboration is to blend the backgrounds of people from several agencies into workable relationships that are mutually beneficial. As noted in Table 10.1, the types of participants in a collaborative effort may vary depending on the specific goals of the community response, but should consist of representatives from different sectors, possibly including: battered women's and sexual assault services, law enforcement, prosecution, civil and criminal courts, health care agencies, child welfare agencies, batterer intervention programs, the corporate/business sector, clergy and religious institutions, and other opinion leaders.

It may also be important to have a combination of experienced/founding members and new members, since older members may be inclined to move on to other projects. The presence of new members will also indicate that there is continued community support and interest in the committee's efforts.

Continuity of Personnel. One of the keys to a successful collaboration is having some or all of the same individuals involved throughout the entire effort. Collaborations improve over time when participants become familiar with each other's strengths and weaknesses and learn to work as a team. Personnel changes occur for a variety of reasons, including work reassignment, termination, promotion, retirement, and, in some cases, a decision not to participate. Any disruption can have adverse effects. At a minimum, changes delay planned activities, and in the worst case the collaboration may dissolve.

Especially vulnerable are collaborations in which a dynamic leader is no longer available—what might be termed the "vanishing advocate" problem. Some collaborations are formed because an individual has taken a problem on and has forged a coalition to seek solutions. Without that leader, the effort folds.

Level of Participating Staff. Some successful groups comprise primarily high-level staff, while others achieve their goals with broad-based community and grassroots support. Whatever the level of participating staff, it is important to consider whether they are in a position within their agency or community that enables them to make changes and influence policy. They should also have the backing or commitment of their own agency to cooperate with the community system. Another factor that should be noted is whether or not those who are in power in the relevant agencies are supportive of ongoing collaborative efforts. (Suggestions for possible ways to collect these data are mentioned in Table 10.1.) For those found to be critical or ambivalent toward the group's efforts, an important initial goal should be to get them on board.

Activity Level/Involvement of Participants. In some groups, the bulk of the work is done by one or two powerful members, while in others it is shared by many. Measures used to gauge the level of member activity might include the number of tasks initiated and completed by each member, the proportion of the member's overall time that he/she dedicates to collaborative efforts, and how many meetings he/she attends.

Another issue is the degree to which participants are allowed to become involved. For example, one might ask who in the group is leading activities and making important decisions and whether or not all members have the opportunity to participate equally.

Levels/Sources of Support

Some collaborative VAW efforts enjoy support from all levels of government, while others are forced to work from the ground up, creating grassroots networks to inform elected and appointed officials and lobby for change. Some initiatives, for example, are formed when states pass laws affecting violence against women or governors or mayors make this issue a priority in their administrations. In other situations, the occurrence of a particularly tragic event or overwhelming frustration with the system prompts service agencies and/or victims to organize a community response from the grassroots level.

Both levels of support are equally important—albeit for different reasons—and should be measured. A high-level elected official such as a governor can help make a collaboration strong by directing state funds and legislative support toward the collaboration's activities. This also happens at a local level when a mayor or other city official uses a portion of the city's funds for this purpose. But due to the changeable political climate, collaborations should not become solely dependent on any one source of support. If they enjoy widespread public support, their activities will continue even after the loss of a strong political ally. For instance, even though grassroots organizations generally are unable to provide the steady funding streams that the government can, they may offer other benefits to the collaboration, such as media attention and lobbying efforts.

Community Collaboration: System-Level Outcomes

Communicating Effectively

Among Participants. Having any communication among agencies where there was none, more communication where some already existed, or improved or more effective communication that leads to greater understanding and trust are all indicators of an established community response. Communication may be measured based on frequency, type, level, basis, and nature of contact, as mentioned in Table 10.2.

While regular meetings are one way of keeping participants informed, it is also important to create and maintain connections among individuals. Informal dialogues, memoranda of understanding, reports, and e-mail transmissions can also contribute to effective communications. Ultimately, better communication among agencies and sectors may evolve into arrangements (both formal and informal) for different types of agencies or agencies in different geographic areas to share administrative or management information system databases.

Informal Networks. In addition to communication among formal committee/group members, participants should create their own informal communication networks. For example, even if the coordinating body is composed primarily of criminal justice members, there should be connections established with other community groups. Although a little more difficult to measure than formal networks, these connections are important because even if the coordinating body disbands or loses its coordinator, many of the connections will remain. One example would be to hold a monthly lunch with a brief program presented by a different agency each time.

Feedback from Community. Groups may or may not incorporate community feedback in their efforts. Two possible areas to measure noted in Table 10.2 are (1) whether meetings and materials are available to the general community and if so, how often, and (2) the frequency with which community feedback is incorporated into decisions and/or projects.

Developing a Shared Vision

The myriad agencies/organizations involved in addressing VAW may have different views on one or more of the following issues: fundamental principles of intervention, roles of each component, merits of collaboration, necessity of public accountability, and goals of reform. Despite differences of opinion, they should be able to develop a common vision and mission. It should be acceptable for parties to have different aims, as long as they are not conflicting and fit under an overarching common purpose. In short, everyone needs to have the same understanding of the overall goals of the effort, and all collaborative activities should be working toward those goals.

Developing a common vision should result in fewer misperceptions about each member's roles/responsibilities. Each person should play a role in the collaborative effort and understand what that role is. Individual committees and task forces should have clearly defined goals and purposes. Other indicators of a shared vision include developing a strategic plan, a mission statement, prioritizing tasks, joint planning, and joint implementation (see Table 10.2 for ways to measure shared vision).

In addition, agencies should produce compatible protocols or guidelines for practice in each component. Services provided by different agencies should be as seamless as possible. For example, police and victim advocates may share a jointly developed protocol that specifies (1) that both will go on all calls; (2) once on the scene, the responsibilities of the law enforcement officer and responsibilities of the victim advocates; and (3) the responsibilities of each for follow-up and keeping the victim informed. Another example might be a protocol developed jointly by law enforcement and prosecution, describing the types of evidence police could collect that are of most value to prosecutors and the kind of feedback from prosecutors that would be most valuable to police.

Establishing Systems of Conflict Resolution and Evaluation

Beyond communication, there may be systems or mechanisms in place for the following: identifying problems and developing solutions, evaluating practice efficacy and incorporating community input (public accountability) and other forms of feedback, and monitoring adherence to adopted standards/practices. These systems/mechanisms should be known and be seen as legitimate, reliable, and timely. Table 10.2 suggests various ways to measure these systems.

Developing Trust and Mutual Respect

An important aspect of a successful collaboration is trust and mutual respect among participants. Some collaborations may "click" from the start, especially when the groups have worked together successfully in the past. But this is rare; most take time to build. Collaborative relationships usually move through several early steps: expression of interests, verification of each other, trust building, joint decision making, and small successes. One indication that mutual respect and trust are present is that the group has formed a flexible working arrangement, eliminating the hierarchy that normally characterizes relationships with participating agencies.

Gaining the trust and respect of other group members may itself be a reward for individual participants. When their expertise has been recognized, their ideas have been considered— perhaps even accepted—and their influence has led to solutions to a problem, they may gain a sense of personal satisfaction. The collaboration itself may then be rewarded by improved performance by the individual.

Engaging in Joint Activities and Co-location

As mentioned in the Shared Vision section, a healthy sign of open coordination among different agencies is when they decide to embark on joint activities such as joint training of staff or designing new joint policies or protocols. In some cases, coordinating agencies may decide to co-locate services (e.g., victim service units within police departments and prosecuting attorney's offices). Alternatively, they may decide to respond together as when a victim advocate accompanies police officers on 911 calls. Similar arrangements can be made in other settings such as hospital emergency rooms and courts.

Reporting

Agencies involved in coordination should have a way to report performance, activities, and accomplishments. Ideally, this would be a formal report published on a regular basis. Public information would promote continued support for the initiative from the general public, the private sector, and the government, and may inspire other agencies to undertake similar efforts. As mentioned in Table 10.2, both direct and indirect results of publishing reports should be noted.

Funding

Often community-wide collaboration efforts can lead to changes in various funding mechanisms. In some states or localities, collaborative organizations have succeeded in establishing new sources of funding (e.g., through new marriage license taxes or line items in city budgets). In other cases, collaborative organizations may be given the authority and responsibility for distributing funds within the community, which increases their opportunities to support and foster cross-agency or cross-jurisdictional collaborative projects and other efforts.

Community Collaboration: Ultimate Outcomes

Creating Permanent Policy and Practice Changes

One strong indication that a community's collaborative efforts have become entrenched is the existence of permanent policy changes. In some communities, permanent changes in the law and agency policies or procedures may occur as a result of collaboration efforts and will remain even if the original committee disbands or loses steam.

Treating Victims and Perpetrators Consistently

Coordinated community response should lead to more consistent treatment of both victims and perpetrators. The same services and options should be available to victims of all racial, linguistic, geographic, or other characteristics, irrespective of the point at which they enter the system (e.g., a 911 call to the police, an emergency room, child protection, etc.). Similarly, perpetrators should be held accountable in a consistent manner by everyone they come into contact with, such as law enforcement officers, prosecutors, judges, treatment providers, and probation and parole officers. Specific examples of how perpetrators might be treated consistently include: not reducing criminal charges; including participation in batterer and chemical dependency treatment programs and compliance with protection orders in sentences; rigorous enforcement of probation and parole conditions; having systems in place that allow one to track compliance with treatment programs and probation and parole terms; and coordinating terms and conditions with other legal cases (both civil and criminal) involving the perpetrator, victim, or other family members.

Creating More Options and Resources for Victims in the Justice and Human Service Systems

Effective coordinated community response should produce more options and less duplication of services for victims in the justice and human service systems. Because sexual assault and domestic violence victims often face a variety of difficulties and unique circumstances, there should be flexible solutions. For example, domestic violence victims may be offered a choice between staying at a temporary group shelter or receiving financial assistance in order to obtain a more permanent residence. A community may make various types of counseling, including group and individual therapy, available to victims of sexual assault and allow them up to two years to enroll in such services. Standards may be imposed to improve the quality and appropriateness of batterer intervention programs or sex offender treatment programs, and higher quality may lead to higher utilization and more resources. New options should also include more culturally sensitive responses to VAW on the basis of race, language, religion, culture, class, kinship networks, perspectives of the efficacy of the legal process, physical and/or mental disabilities, and urban/rural status.

Changing Public Knowledge of VAW and Reducing Unmet Need

Finally, effective community response should result in an increased level of awareness and understanding of violence against women by the public at large. A variety of public education strategies tailored to differing segments of the population (teens, immigrants, etc.) can be employed and the effects measured by individual surveys on levels of understanding and tolerance about violence against women. Chapter 11 discusses community attitudes toward violence against women further and offers specific tools for measuring these attitudes.

The general public should also be aware of the services available to victims and their families. Increased knowledge and use of services by those that need them would indicate that the collaborative efforts had reached community members, who may in turn become interested in joining the groups' efforts.

CHAPTER 11
MEASURING CHANGE IN COMMUNITY ATTITUDES,
KNOWLEDGE OF SERVICES, AND LEVEL OF VIOLENCE

Many STOP grants have as either intermediate or ultimate outcomes the goal of making communities safer and more supportive of women victims of violence. Steps toward this goal can take many forms. Many projects try, through outreach, thoughtful location of services, education of professionals and the general public, and public service announcements to increase women's knowledge of the services that are available to help them deal with domestic violence and sexual assault. These projects want women to know that help is available if they need it, and where to find it. Other projects try to reduce the level of support for violence against women by reducing myths and attitudes that blame women for their own victimization, minimize the harm done by violence, discourage women from trying to improve their situation, and provide excuses for male perpetrators of violence. In efforts to change community behavior, projects may work to enlist support from community members other than the "usual suspects" from the justic systems and victim advocacy. These other community allies and partners can do many things to make your community safer for women. Finally, many projects want to know what the true level of violence against women is in their community, and most projects hope that the ultimate effect of all their efforts will be to reduce the overall level of this violence. This chapter provides a brief look at some ways that these goals might be measured. The chapter is far from exhaustive, but it does offer some resources for measuring the achievement of these goals.

Knowledge of Service Availability

It would be hard to imagine a program offering services and supports for women victims of violence that was not interested in whether the community knew about its existence. Therefore any service might profit from doing some investigation into whether its name rings any bells among members of the larger community. The issue of community knowledge is especially important for new services, and for services that are trying to reach new populations. Since one of the major VAWA objectives is to extend services to previously underserved populations, many projects have been funded with this objective. All of these projects should be making an effort to learn whether members of their target community have heard of them.

What to Ask About

Basically, you want to know about recall and about recognition. "Recall" means that when you ask someone "Do you know anywhere you could go to get help if you were raped?" the person says, "Yes, I would go to the ABC Crisis Service." This answer indicates they recall the name and purpose of your agency, and do so without prompting or help from you. Even if people cannot recall the name of your agency, they may still recognize it when asked a question such as "Do you know what ABC Crisis Service does?" Or, you could ask "Which of these agencies would you go to if you had just been raped and wanted help?" and then give the person a list of agencies that includes ABC Crisis Service and see whether she pick out ABC Crisis Service as a place to go for this type of help.

With people who already know about your program, there are several questions you can ask to get important feedback. You can ask whether they know your location, your hours, whether you speak their language, and whether they could get to your agency using the transporation available to them. You could then ask them how they learned about your program, whether they have ever used your services, or whether they know anyone who has. Follow-up questions could ask for feedback about the program and its services, such as whether they have heard that you treat women with dignity and sympathy.

Sample questions include the following (answered "yes" or "no"):

• Do you know of a specific place or program in your community:
  • That can help if a family member is being violent to someone else in the family?
  • That offers information and assistance to someone who has just been raped?
  • Where someone could talk about how they're feeling about a sexual assault that happened some time ago?
  • That will help you get a restraining order to keep someone who has been harassing you away from you?
  • And so on.
• For each "yes" answer, you could then ask:
  • Do you know the name of that agency? What is it?
  • How did you hear about that agency?
  • Do you know where it is?
  • Do you know what kinds of people use that agency?
  • And so on.
• Continue making up questions to get the answers you want. Be sure that the questions are worded very simply.

How, When, and Whom to Ask

There are many different places where you could ask these questions. Each place will let you learn about perceptions of your program from a different group of women. Different places may require you to use different formats, some of which will be very formal while others will be very informal, such as group discussions. Here are a few of the places and formats you might be able to use to get information about how well your program is known in the community:

• Focus groups with women who use a variety of generic service agencies (e.g., community health centers, welfare offices, employment centers, child care centers, cultural or recreational centers). These are very informal situations, in which you pick between 4 and 6 questions on which you want feedback, and keep the group focused on providing answers to those questions. Do not try to treat a focus group as a group interview; you will not get quantitative data from a focus group. But you will get a lot of feedback about your program. Also, you might want to be prepared to find out that few of them have ever heard of your program, and spend time exploring what it would take to spread the word.
• Exit interviews or questionnaires in generic service agencies such as health, social services, or welfare agencies, or in other VAW-related locations such as the police department. Interviews could be formal or informal. Questionnaires have to be written, of course, and should be short (1 page is best).
• Surveys of community members. This is the most formal approach. You would have to develop a formal, written survey containing the questions to which you want answers. Then you would have to do one of the following: find a firm to conduct the survey, recruit volunteers and organize several nights during which you call households at random, find an existing household telephone survey from which you can buy some time and attach your questions, or develop other methods yourself.

Attitudes toward Violence Against Women

The causes of rape, other sexual assaults, and domestic violence are many and complex. One causal factor may be community support for or tolerance of violence against women. Attitudes such as acceptance of sex-role stereotypes, belief in rape myths, and beliefs that some circumstances justify battering help to create a climate that gives abusers and rapists a justification for their actions, and is hostile to victims of domestic violence and rape. A wide variety of Americans, including typical citizens, police officers, and judges, have been shown to hold beliefs that can be used to justify rape (Burt, 1980; Feild, 1978; Mahoney et al., 1986) and domestic violence (Broverman et al., 1970; Pagelow, 1981).

In addition to supporting violence against women, attitudes may also have the effect of creating an environment that is hostile to victims. Rape-supportive attitudes are partly responsible for low levels of rape reporting (Russell, 1982) and for a blaming-the-victim attitude that makes it difficult for victims to seek help and recover from their assaults (Ehrhart & Sandler, 1985). Belief in rape myths leads to a strict definition of rape and denies the reality of many actual rapes (Burt & Albin, 1981), which makes it difficult to prosecute rapists and support victims. Rape victims are often victimized twice—once from the actual assault and a second time when they encounter negative, judgmental attitudes from the police, courts, and family and friends (e.g., Deming & Eppy, 1981; Weis & Borges, 1975; Williams, 1984).

Myths and stereotypes about victims of domestic violence often make it difficult for these victims to seek help and to improve or sever relationships with their abusers. As mentioned earlier, police are less likely to arrest offenders if they believe that domestic violence is a "family issue" or that victims will refuse to press charges (Pagelow, 1981). And negative attitudes—such as apathy or hostility—that are often experienced by abused women when they reach out for help, may actually help perpetuate abuse by stigmatizing women and making it harder to leave abusive relationships (Stark, Flitcraft, & Frazier, 1979). Because they contribute to the perpetuation of violence against women, recognizing and measuring community attitudes can play an important part in evaluating the effectiveness of some STOP-funded projects.

Measuring Community Attitudes

A number of quantitative measures can be used to examine attitudes toward violence against women. Some, such as the Attitudes Toward Women Scale (Spence & Helmreich, 1972) measure general attitudes about the roles and rights of women, while others, including the Rape Myth Acceptance Scale (Burt, 1980) and the Inventory of Beliefs about Wife Battering (Saunders, et al., 1987) assess attitudes specifically related to rape and domestic violence. These measures can be used in a variety of ways, depending on the preference of the grantees and the intended target of their programs.

The most practical and least expensive way to implement the measures is not to survey the broad community, but to focus on those that could be considered a "captive audience," such as students, potential jurors, or members of civic or religious organizations. For example, if a prevention program has given presentations to students in a particular high school, grantees may wish to survey students in that high school and in a school without the prevention program and then compare the results.

Some communities might have the capacity to conduct surveys with a broader range of community members. If you are trying to change attitudes in the whole community, it is much more convincing to conduct a community-wide survey than to use "captive audiences," many of whom will surely have been exposed to your message. One way to do this, if your state has a periodic statewide poll, is to connect with the organization that conducts it and add a few questions about community attitudes toward violence against women. Another way is to find a sponsor who will support a special-focus survey of your particular community.

Rape Myths

Researchers in the field of rape have argued that these widely accepted myths—defined as prejudicial, stereotyped, or false beliefs about rape, rape victims, and rapists—support and promote rape. Rape myths are part of the general culture and people learn them in the same way they acquire other beliefs and attitudes—from parents, friends, newspapers, books, and television (Burt, 1991). Some of these myths include (1) women enjoy sexual violence; (2) sex is the primary motivation for rape; (3) women are responsible for rape prevention; (4) only bad women are raped; (5) women falsely report rape; and (6) rape may be justified (Lottes, 1988). Belief in rape myths is significant because it "provides men with a structural position from which to justify sexually aggressive behavior" (Marolla & Scully, 1982). For example, Scully and Marolla (1984, 1985a, 1985b) found that the myths that women both enjoy and are responsible for their rape were used by convicted rapists to excuse and justify their crimes.

RAPE MYTH ACCEPTANCE SCALE

Another possible measure:

BUMBY COGNITIVE DISTORTIONS SCALE

Domestic Violence Myths and Stereotypes

Popular myths and stereotypes abound on the subject of domestic violence victims and batterers. The public and many professional groups often hold negative attitudes toward battered women (Dobash & Dobash, 1979; Gelles, 1976; Straus, 1976). Victims are often thought to be (1) masochistic; (2) weak; (3) "seeking out" the batterers; or (4) somehow at fault. Batterers are excused because they are "sick" or their force is perceived as justified because of the wife's behavior (Greenblat, 1985; Pagelow, 1981). One way in which domestic violence myths are harmful is that they are often believed by the people responsible for aiding domestic violence victims: social workers, judges, health professionals, and law enforcement officers. For example, a common perception among police officers is that domestic violence victims are too weak-willed to press charges. This is particularly damaging in cases where officers base their decision to arrest an abuser on whether or not they believe the case will ever reach court (Bates & Oldenberg, 1980).

Several scales have been developed to measure attitudes toward battering, including the Attitudes to Wife Abuse Scale (Briere, 1987) and the Inventory of Beliefs about Wife Beating by Saunders and his colleagues (1987).

ATTITUDES TO WIFE ABUSE SCALE

INVENTORY OF BELIEFS ABOUT WIFE BEATING

Adversarial Sexual Beliefs and Hostility Toward Women

Adversarial sexual beliefs and general hostility toward women are attitudes that contribute to violence against women. Adversarial sexual beliefs refer to the feeling that men and women are adversaries in their sexual relationships with one another—that relationships are exploitative and each party is manipulative and not to be trusted (Burt, 1980). Hostility toward women is related to adversarial sexual beliefs, but goes beyond sexual interactions to asses a more general feeling of animosity and distrust of women in all aspects of relationships and interactions. Check (1984) expanded Burt's Adversarial Sexual Beliefs Scale (Burt, 1980) to create a measure of general hostility toward women and found that men who scored high on the hostility scale tended to (1) have traditional sex-role beliefs; (2) believe in rape myths; (3) admit to using force in their attempts to get women to have sex with them, and say they would be likely to do so again; and (4) use high levels of physical punishment when rejected by women and given the opportunity to retaliate.

ADVERSARIAL SEXUAL BELIEFS SCALE

HOSTILITY TOWARD WOMEN SCALE

Sex-Role Stereotypes

Traditional sex-role socialization or sex-role stereotyping, which views women as having a lower social status and lesser rights than men, appears to play an important role in violence against women. People who hold these beliefs—for example, feeling that a woman should get married and raise a family, be a virgin when she marries, and never contradict her husband in public—are more likely to be tolerant of and/or engage in domestic violence (e.g., Eisikovits, Edleson, Guttmann & Sela-Amit, 1991; Kincaid, 1982; Koss et al., 1985, in Parrot). In one study by Finn (1986), traditional sex-role attitudes were found to be the most powerful predictor of attitudes supporting marital violence. Sex-role stereotypes also play a role in rape and sexual assault. In their analysis of sexually aggressive and non-aggressive men, Koss and Dinero (1988) found that sex-role stereotyping was causally related to sexual assault in that "the more sexually aggressive a man has been, the more likely he was...to accept sex-role stereotypes."

ATTITUDES TOWARD WOMEN SCALE

Masculinity Ideology

Also playing a role in support of violence against women is masculinity ideology. Masculinity ideology, which refers to beliefs about the importance of men adhering to culturally defined standards for male behavior, may be responsible for some problem behaviors—particularly in adolescent men. In their 1993 study, Pleck, Sonenstein, and Ku found that a higher score on the Masculinity Ideology Index was associated with, among other things, twice the odds of being sexually active and of ever forcing someone to have sex. And in a study of 175 college males, Mosher and Anderson (1984) found that a measure of macho personality with three components—callous sex attitudes towards women, a conception of violence as manly, and a view of danger as exciting—was significantly correlated with a history of self-reported sexual aggression against women.

MASCULINITY IDEOLOGY

"Beyond the Usual Suspects"

Most STOP projects will focus their efforts on the civil and criminal justice system agencies, and on providing services and advocacy to women victims of violence. These are "the usual suspects." But some STOP projects are trying to bring more players into the effort to end violence against women. The previous section on public attitudes covered one approach to involving others, namely, trying to reduce beliefs and attitudes in the community that minimize perceptions of harm, blame women, and justify acts of violence. This section addresses efforts to involve particular local actors—preferably people with influence who are opinion leaders—as allies and partners in ending violence. Possibilities include the following (all of which have occurred in at least some communities):

• Business and corporate leaders—might (1) change the coverage of their employee health insurance policies, (2) establish special assistance opportunities through their employee assistance programs (EAPs), or (3) institute security procedures to protect women against harassment and stalking. In addition, they could (4) play a prominent role in a local anti-violence council, (5) organize a campaign to have several corporations "adopt-a-victim-services-program" (as has actually happened in at least one state, where corporations give ongoing financial and other support to "their" domestic violence shelter), (6) donate computer hardware and know-how to develop efficient databases, (7) fund a community telephone survey to establish levels of violence, and so on.
• Clergy—might (1) develop training sessions for other clergy that set anti-violence actions in a convincing doctrinal context and recruit more voices against violence, (2) preach against violence and for the rights of women to be violence-free; (3) get training to be supportive counselors when women approach them with a problem related to violence; (4) serve on a community-wide anti-violence council, and so on.
• Doctors and therapists—might (1) obtain training that would help them offer support and assistance to women victims of violence and, at a minimum, not make things worse, (2) develop into better screeners to identify more women victims of violence, (3) influence the practices of emergency rooms and other treatment settings that they control.
• Substance abuse treatment professionals—there is a tremendous overlap in treatment populations between batterers and men attending substance abuse treatment (especially those ordered to do so as a consequence of drunk driving and drunk and disorderly offenses). These treatment professionals could develop special sessions in which they deal with issues of battering and its relationship to substance abuse in the lives of treatment participants.
• Journalists—might (1) become familiar with the latest thinking about violence against women, (2) learn what is happening in their community to combat this violence, and (3) regularly publicize the issue with article on programs, community activities such as Take Back the Night marches, successful survivors, reformed batterers, and so on.
• Politicians—in addition to making public statements opposing violence against women, the main things politicians can do are (1) put money in city, county, and state budgets to fund services; (2) update laws and ordinances to conform to the latest legislative recommendations of anti-violence advocates; (3) mandate VAW-related training for all new recruits/professionals; and (4) mandate regular in-service training for all professionals dealing with women victims of violence, focusing on the latest in VAW-related legal and other developments (e.g., best practices).

For any project with objectives that include the development of community partners and allies, an evaluation will need to document the goals and the project's success in reaching them. Process analysis techniques will be the ones to use in this effort. Goals should be specified in terms of who (which allies are targeted), how many, how (what approaches will be used, and when), and for what (what do you want them to do). Goals can be documented by examining written project materials and by conducting interviews with key staff. Success in reaching the goals can be documented by interviewing key project staff, but that is only the beginning. It is essential that you also do the first bullet below, and that you consider the remaining bulleted suggestions:

• Interview the people who were targeted to see what they think of the campaign, what they believe their activities have been, and so on.
• To the extent that the community partners and allies have produced anything concrete, these products should be collected and described. Such products might include revised personnel policies, changing patterns of EAP use, newspaper articles, expanded budgets, expanded contributions to service agencies, new screening procedures in place, new policies in the emergency room, and so on.
• Other possible indicators to examine if they are relevant to your goals include the following:
  • The number of community leaders—such as business people, politicians, and clergy—speaking out against domestic violence and sexual assault.
  • The number of volunteers at rape crisis centers, domestic violence shelters, and hotlines.
  • Support from the school system for adolescent violence prevention programs.
  • Media coverage that portrays rape and domestic violence in a victim-supportive manner.

Measuring Community Levels of Violence Against Women

Ultimately, everyone involved in STOP wants to see reductions in levels of violence against women. However, documenting success on this goal usually falters on the shoals of inadequate measurement. Police statistics are known to underrepresent levels of violence, and are also biased (meaning that they miss particular types of women victims, as well as generally not getting all the incidents). The direction of bias may vary from community to community depending on local attitudes and practices, but some type of bias will virtually always be there. Records from victim service agencies and hotlines also undercount and have biases, although the biases are often quite different from the biases of police data. National surveys such as the National Crime Victim Survey undercount, are biased in various ways, and cannot be used to reveal what is happening in particular communities because their sample sizes are only adequate for national estimation.

There are no easy solutions to this dilemma, but that does not mean that there are no solutions. We recently came across a state-level effort that seems reasonably priced and reasonably successful in getting respondents. Further and most important, it obtained estimates of lifetime and recent victimization that seem in line with what one might expect (rather than being grossly low). We thought it would be worth sharing this resource with users of this Guidebook, because some of you may want to replicate this study in your own state or community.

In 1996, the Michigan Department of Community Health, using funding from the Centers for Disease Control and Prevention, developed and conducted a telephone survey of Michigan women to determine the lifetime and recent prevalence of violence in their lives. ¹ The violence examined included physical or sexual violence or threats of violence from strangers, boyfriends or dates, other acquaintances, current intimate partners including husbands and live-in partners, and ex-partners. Reports were limited to experiences since age 16. The structure and portions of the questionnaire content were based on an instrument that Statistics Canada used in its first national survey of violence against women in 1993. The telephone survey itself was conducted by the Gallup Organization, using random digit dialing.

Results represent the general population of women ages 18-69; 1848 women completed the interview. There is some bias in the survey respondents toward currently married women, employed women, and higher income women, as might be expected from a telephone survey methodology. Weighting was used to produce the final results, using age, race, and educational status as factors in the weighting to adjust for the sample biases and obtain representative data.

To give you a flavor of the results and their reasonableness, we quote (with permission) from information distributed at a presentation of this study in Indianapolis, at the 1997 Annual Meeting of the American Public Health Association. The study found:

• 58% reported experiencing some type of violence (physical, sexual, or threat) by a man since age 16.
• 12% reported at least one form of violence during the past year.
• 38% reported experiencing physical violence by a man, while 40% reported some form of sexual violence.
• Over one-third (38%) of women who have ever had a partner reported having experienced violence by a partner.
• Nearly half the respondents said they had experienced some type of violence by a non-partner since age 16.
• 27% reported that a non-partner either raped or attempted to rape them. 38% reported some type of unwanted sexual acts by a non-partner.

These results, and others available from the study team (see footnote 2) will suggest to anyone who has tried to do this type of survey that this team did something right. Anyone interested in conducting a similar survey would be wise to discuss methodology with this study team. In particular, important things to discover are (1) how they conducted telephone screening that identified women and established a safe and secure time for the interview, and (2) how they introduced the study as a whole and each section of the questions to elicit the level of self-disclosure that they achieved. At the time this Guidebook went to press, no final report was available for this study, but you should be able to get it soon from the study team, and it will undoubtedly make interesting reading.

Addendum: Additional Reading

Bates, V., & Oldenberg, D. (1980). Domestic violence and the law. Paper presented at the annual meeting of the Pacific Sociological Association, San Francisco.

Broverman, I., Broverman, D., Clarkson, F., Rosenkrantz, P. & Vogel, S. (1970). Sex-role stereotypes and clinical judgements of mental health. Journal of Consulting and Clinical Psychology, 34, 1-7.

Burt, M. (1980). Cultural myths and supports for rape. Journal of Personality and Social Psychology, 38, 217-230.

Burt, M. (1991). Rape myths and acquaintance rape. In A. Parrot & L. Bechhofer (Eds.), Acquaintance Rape: The Hidden Crime. New York: Wiley & Sons.

Burt, M., & Albin, R. (1981). Rape myths, rape definitions, and probability of conviction. Journal of Applied Social Psychology, 11, 212-230.

Check, J. (1984). The Hostility Toward Women Scale. Unpublished doctoral dissertation, University of Manitoba.

Deming, M., & Eppy, A. (1981). The sociology of rape. Sociology and Social Research, 65, 357-380.

Dobash, R. E., & Dobash, R. (1979). Violence against wives: A case against patriarchy. New York: Free Press.

Ehrhart, J., & Sandler, B. (1985). Myths and realities about rape. Washington, DC: Project on the Status and Education of Women.

Eisikovits, Z. C., Edleson, J. L., Guttmann, E., & Sela-Amit, M. (1991). Cognitive styles and socialized attitudes of men who batter: Where should we intervene? Family Relations, 40, 72-77.

Field, H. (1978). Attitudes toward rape: A comparative analysis of police, rapists, crisis counselors, and citizens. Journal of Personality and Social Psychology, 36, 156-179.

Finn, J. (1986). The relationship between sex role attitudes and attitudes supporting marital violence. Sex Roles, 14 (5/6), 235-244.

Gelles, R. (1976). Abused wives: Why do they stay? Journal of Marriage and the Family, 38, 659-668.

Kincaid, P. J. (1982). The omitted reality: Husband-wife violence in Ontario and policy implications for education. Concord, Ontario: Belsten.

Koss, M. & Dinero, T. (1988). Discriminant analysis of risk factors for sexual victimization among a national sample of college women. Journal of Consulting and Clinical Psychology, 57, 242-250.

Koss, M., Leonard, K., Oros, C., & Beezley, D. (1985). Nonstranger sexual aggression: A discriminant analysis of the psychological characteristics of undetected offenders. Sex Roles, 12, 981-992.

Lottes, I. (1988). Sexual socialization and attitudes toward rape. In A. Burgess (Ed.), Rape and Sexual Assault II. New York: Garland Publishing.

Mahoney, E., Shively, M., & Traw, M. (1986). Sexual coercion and assault: Male socialization and female risk. Sexual Coercion and Assault, 1, 2-8.

Marolla, J., & Scully, D. (1982). Attitudes toward women, violence, and rape: A comparison of convicted rapists and other felons. Rockville, Md.: National Institute of Mental Health.

Mosher, D., & Anderson, R. (1984). Macho personality, sexual aggression, and reactions to realistic guided imagery of rape. Typescript.

Pleck, J., Sonenstein, F., & Ku, L. (1993). Masculinity ideology and its correlates. In S. Oscamp & M. Costanzo (Eds.), Gender Issues in Contemporary Society. Newbury Park, CA: Sage.

Russell, D. (1982). The prevalence and incidence of forcible rape and attempted rape of females. Victimology, 7, 81-93.

Scully, D., & Marolla, J. (1984). Convicted rapists' vocabulary of motive: Excuses and justifications. Social Problems, 31, 530-544.

Scully, D., & Marolla, J. (1985a). Rape and vocabularies of motive: Alternative perspectives. In A. W. Burgess (Ed.), Rape and Sexual Assault: A Research Handbook. New York: Garland Publishing.

Scully, D., & Marolla, J. (1985b). "Riding the bull at Gilleys": Convicted rapists describe the rewards of rape. Social Problems, 32, 252-263.

Stark, E., Flitcraft, A., & Frazier, W. (1979). Medicine and patriarchal violence: The social construction of a 'private' event. International Journal of Health Services, 9, 461-493.

Straus, M. (1976). Sexual inequality, cultural norms, and wifebeating. Vicitmology: An International Journal, 1, 54-76.

Weis, K., & Borges, S. (1975). Victimology and rape: The case of the legitimate victim. In L.G. Schultz (Ed.) Rape Victimology. Springfield, Ill: Charles C. Thomas.

Williams, J. (1984). Secondary victimization: Confronting public attitudes about rape. Victimology, 9, 66-81.

CHAPTER 12
MEASURING PERCEPTIONS OF JUSTICE

Many STOP projects are trying to change the ways that justice system agencies treat women victims of violence. These changes range from interpersonal behavior such as respectful listening through significant changes in skills and procedures applied to cases involving violence against women. Examples of the latter are better evidence collection, better communication with victims about the status of their case, better prosecution strategies, better monitoring and supervision of offenders, more attention to victims' wishes and circumstances in case disposition and sentencing, and so on.

Most, if not all, of the changes just described should affect how victims feel about participating in the justice system, and whether they feel that "justice has been done" in their case. Therefore, you may want to include measures of perceptions of justice in your evaluation, to let you know whether system changes have produced a better feeling about the system among victims. Your evaluation can measure perceptions of justice by the victims or parties in a legal action. You can also develop a rating of the extent to which the justice intervention is consistent with the rights of victims.

Perceptions of Justice

The first step in measuring perceptions about justice is to decide whether you want to measure (1) distributive justice, (2) procedural justice, or (3) both.

Distributive Justice

Distributive justice refers to perceptions of the outcome of police, court or other justice procedure (regardless of how those outcomes were arrived at). In rating the justice of an outcome, research indicates that people base their perceptions on how they answer two questions:

• Did the punishment fit the crime? Measures would assess the extent to which the disposition of the case (or police call, hearing, or other intervention) resulted in a punishment that fit the offense.
• Is the outcome likely to reduce subsequent offenses? Measures would assess the extent to which the disposition of a case, incident, or complaint reduces future harm to the victim or society.

Procedural Justice

Procedural justice refers to how the system acted, how it processed a case regardless of the outcome that resulted. It involves several components, including the consistency, fairness, and appropriateness of rules and procedures, and how those in authority treated the participants.

• Consistency, fairness, and appropriateness of rules and procedures used in handling the case or grievance. In general, research finds that people think something is fair or appropriate when they are able to tell their side of the story and they also perceive that all parties in a case or grievance have had a chance to tell their side of the story. This aspect of fairness has been measured by asking about the extent to which the parties have control of the process or input into the process (Shapiro & Brett, 1993; Tyler, 1994).
• How participants are treated by those in positions of authority during the process of handling the case or grievance. Two factors are particularly important in determining whether someone will feel that procedures are just (1) how/whether the intervention process/rules are explained to participants; and (2) how participants are treated during the intervention. Items used to measure this aspect of procedural justice usually ask people what they think of the third-party decision maker (e.g., the judge, police officer, prosecutor)—are they understanding, are they neutral/impartial, are they trustworthy, do they show consideration to those involved in the case (Tyler, 1989, 1994)?

Somewhat surprisingly, ratings of procedural and distributive justice appear to be independent of each other. Participants may rate the outcome as just, but not the procedure—or the procedure fair, but not the outcome. When individuals are forced to choose between the two, some research indicates that the higher ranking is given to achieving a just outcome (Wagstaff & Kelhar, 1993).

The following items and scales draw on the work of Tyler (1989), but include some new wording and answer categories. The scales ask separately about police and courts and can be extended to other justice agencies such as prosecutors' offices.

Procedural fairness items use 5-point scales and include the following:

• Were the procedures used by the authorities (police, courts) very fair (1), somewhat fair (2), neither fair nor unfair (3), somewhat unfair (4), or very unfair (5)?
• Was your case treated very fairly (1), somewhat fairly (2), neither fairly nor unfairly (3), somewhat unfairly (4), or very unfairly (5)?

Perceptions of fair personal treatment (by police/courts/others) use "yes" and "no" as answer categories; items include the following:

• Were you angry?
• Were you frustrated?
• Were you pleased (with the authorities)?

General fairness of treatment by (police/courts/other) items, rated on 5-point scales, include the following:

• Do you think (police/courts/other) usually treat people and handle their problems very fairly (1), somewhat fairly (2), neither fairly nor unfairly (3), somewhat unfairly (4), or very unfairly (5)?
• Do the (police/courts/other) treat people fairly almost always (1), most of the time (2), some of the time (3), not very often (4), or never (5)?

Distributive justice (the fairness of the outcome) items, also on 5-point scales, include the following:

• Was the outcome of your case very fair (1), somewhat fair (2), neither fair nor unfair (3), somewhat unfair (4), or very unfair (5)?
• Did you receive what you deserved from the (police/courts/other)? Strongly agree (1), Agree (2), Neither agree nor disagree (3), Disagree (4), Strongly disagree (5).
• If you were to deal with the (police/courts/other) in the future, do you think you would be treated very fairly (1), somewhat fairly (2), neither fairly nor unfairly (3), somewhat unfairly (4), or very unfairly (5)?

Protection of Victim Rights

You can develop a rating of justice by asking women victims of violence to tell you the extent to which they have been treated according to the principles contained in the Victims' Bill of Rights that is the law in many states. This Bill of Rights lists certain legal remedies and protections that should be extended to victims. Not all states have this legislation, but your state may. You may also want to include rights accorded victims in other states. Rights mentioned in these bills include, among others:

• The right to restitution for financial losses;
• The right to have the person who caused the harm detained, tried, and sufficiently punished;
• The right to have relevant evidence presented in court;
• The right to be protected from further harm or reprisal;
• The right to be treated with respect, dignity, and fairness;
• The right to address the court at sentencing;
• The right to timely notification of proceedings;
• The right to notification that the perpetrator has been released or escaped from custody;
• The right to confer with the prosecution;
• The right to costs of medical or psychological treatment for injury inflicted during a crime;
• The right to confidentiality of records;
• The right to the prompt return of property seized as evidence; and
• The right to notification of all legal remedies.

This list can serve as a starting point for a scale of the justice victims receive when a crime (sexual assault or domestic violence) occurs by asking victims whether their rights were observed in these ways. The list can be tailored to the program you are evaluating. You can further clarify or define these items and/or add other rights to the list. Answer categories can be yes/no or ratings of satisfaction using five-point scales similar to those used above. However, items in a single scale should all use the same number of answer categories (two or five).

Addendum: Readings on Perceptions of Justice

Casper, J.D., Tyler, T.R., & Fisher, B. (1988). Procedural justice in felony cases. Law & Society Review, 22, 403-507.

Feld, B.C. (1990). The punitive juvenile court and the quality of procedural justice: Disjunctions between rhetoric and reality. Crime & Delinquency, 36, 443-466.

Folger, R., Cropanzano, R., Timmerman, T.A., Howes, J.C., & Mitchell, D. (1996). Elaborating procedural fairness: Justice becomes both simpler and more complex. Personality and Social Psychology Bulletin, 22, 435-441.

Gilliland, S. W. (1994). Effects of procedural and distributive justice on reactions to a selection system. Journal of Applied Psychology, 79, 691-701.

Lind, E.A., & Tyler, T.R. (1988). The Social Psychology of Procedural Justice. New York: Plenum Press.

Miller, J.L., Rossi, P.H., & Simpson, J.E. (1991). Felony punishments: A factorial survey of perceived justice in criminal sentencing. The Journal of Criminal Law & Criminology, 82, 396-422.

Shapiro, D.L., & Brett, J.M. (1993). Comparing three processes underlying judgments of procedural justice: A field study of mediation and arbitration. Journal of Personality & Social Psychology, 65, 1167-1177.

Stroessner, S.J., & Heuer, L.B. (1996). Cognitive bias in procedural justice: Formation and implications of illusory correlations in perceived intergroup fairness. Journal of Personality & Social Psychology, 71, 717-728.

Tyler, T.R. (1994). Psychological models of the justice motive: Antecedents of distributive and procedural justice. Journal of Personality & Social Psychology, 67, 850-863.

Tyler, T.R. (1989). The psychology of procedural justice: A test of the group-value model. Journal of Personality & Social Psychology, 57, 830-838.

Vingilis, E., & Blefgen, H. (1990). The adjudication of alcohol-related criminal driving cases in Ontario: A survey of crown attorneys. Canadian Journal of Criminology, 639-649.

Wagstaff, G.F., & Kelhar, S. (1993). On the roles of control and outcomes in procedural justice. Psychological Reports, 73, 121-122.

CHAPTER 13
MEASURING THE IMPACT OF TRAINING

By Heike P. Gramckow, Ph.D., Jane Nady Sigmon, Ph.D.,
Mario T. Gaboury, J.D., Ph.D. ¹, and Martha R. Burt, Ph.D.

It is hard to imagine any training in the area of violence against women that does not seek, at a minimum, to increase knowledge, heighten sensitivity toward victims, and enhance skills that will help trainees contribute to efforts to protect victims. It does not matter whether those attending training are police officers, prosecutors, judges, victim adovcates, hospital personnel, or probation officers—these basic goals will be the same even when the content of the knowledge and skills to be imparted will differ. Thus every STOP-funded training project will need to review this chapter, because every trainer needs feedback indicating whether the time and effort invested in training has any payoff in learning and in changed attitudes and behavior.

Logic Model of a Training Project

Underlying all impact evaluation is a common concept described in more detail in Chapter 6. Determining impact requires comparing the conditions of individuals who have experienced an intervention (e.g., a training program) with individuals who have experienced something else. We identify the impact of training by comparing data on participants and non-participants, or by measuring participants before and after an intervention (and also possibly while the intervention is in progress), or by other methods of comparison.

This chapter addresses issues in evaluating a particular type of project (rather than offering outcome measures relevant to a variety of projects), so it is a good place to practice developing a logic model. Exhibit 13.1 shows a schematic logic model, without a lot of detail. Column A contains background factors which, in this case, will be a variety of characteristics of the trainees (which characteristics will depend on the content of training). Column B shows characteristics of the training itself, which should be documented with the techniques of process evaluation. This is an essential step; process evaluation has inherent value in illuminating whether the training operated as intended, and is necessry to understand the results of an outcome evaluation.

Column C identifies some common external factors that can enhance or undermine the impact of training. As with background factors, which ones are important in your evaluation will depend on the content of training, the origins of the trainees, and the circumstances that prevail in the trainees' home agencies and home community. Column D identifies the goals or outcomes of training, which are described in greated detail throughout this chapter. Note that the data source anticipated for Columns B and C is process analysis.

Selecting Outcomes to Evaluate

The ultimate goal of VAWA-supported training is improved outcomes for women victims of violence. However, few training evaluations will be able to take this long-term view. The more immediate and intermediate outcomes such an evaluation can measure are those that facilitate or promote the ultimate goal. Immediate outcomes include new knowledge, changed attitudes, and new skills. Intermediate outcomes may be changes in the behaviors of those who received training, putting their new learning and attitudes into practice. More long-term but still not ultimate outcomes may include changes in police or court procedures, the adoption of new policies that affect the treatment of victims by criminal justice or allied professionals, or the development of a coordinating council that connects all community services for victims. The contents, and sometimes the structure, of training will be shaped by which of goals the training aims to achieve. Likewise, any evaluation must structure its measures to be able to assess the different goals and activities of the training program.

Avoiding Inappropriate Outcomes

Any evaluation of training should cover only outcomes that the training itself can be expected to accomplish. When the training's main goal is to increase the knowledge of its participants, then the evaluation should concentrate on assessing knowledge acquisition. While a number of other outcomes may result from the training, they cannot be the focus of an evaluation if the training itself did not posit that its goals included these other outcomes. It would be an odd training program that wanted only to convey knowledge and did not also hope or expect that participants would use the new knowledge in some way (i.e., that they would change some behaviors as a consequence of what they have learned). But suppose for the moment that this is the case. Then an evaluation would stick to assessing knowledge. However, when the training program's logic model includes the expectation that new knowledge will lead to different behaviors, and the program's training curriculum is set up to encourage changed behaviors and give people time to practice them, then the evaluator should ask whether the training participants use the new behaviors once they are back on the job.

Evaluators also need to be careful not to expect training programs to achieve ambitious goals that clearly require more than just training. When police officers participate in training that teaches them how to work closely with victim advocates in order to offer better protection to women who have been assaulted, we would look for increased and improved coordination between police officers and appropriate victim advocates. But can we also expect to see that female victims of domestic assaults are actually better protected? The latter may be a long-term goal of the training effort that very likely requires the availability of additional resources (e.g., shelter, protection orders, electronic warning systems). While the training may actually teach police officers how to access these resources, it cannot make them available in each community. When training alone cannot, realistically, achieve a stated goal, then the hoped-for impact cannot be expected to occur and should not figure centrally in an evaluation of training.

Picking the Right Outcomes

Developing the right measures for goal achievement becomes more involved the more comprehensive the goals of training are. Usually the results of training that aims only to provide knowledge are relatively easy to measure. For example, when the training curriculum includes a session on new legislation that changes the situations in which protective orders may be issued or a session on new techniques for gathering evidence in sexual assault cases, participants should be assessed on their ability to describe the situations or techniques presented in training. When the training also has the goal of providing the skills to apply the new law or the new techniques and includes sessions for mastering these skills, then the evaluation should include an assessment of how much the skills are used in practice.

Specifying and collecting the right evaluation measures for assessing skill acquisition is more difficult than for knowledge acquisition, because one must collect evidence of changed behavior. This may come through self-report from a follow-up questionnaire, from the testimony of others (e.g., interviewing co-workers and supervisors about the trainee's use of the new skills), from actual observation, or from analysis of case records (assuming that the records contain adequate information). Obviously, it is more expensive and time-consuming to gather these types of information to document behavior change (or lack of it) than it is to give trainees a paper-and-pencil knowledge test on the last day of training. However, the importance of what you learn from your increased investment usually more than makes up for the additional trouble.

If training is to achieve its goals, the substantive information actually provided in the training session must be linked to those goals. Suppose that the goal of a training program is to increase the number of protection orders that officers obtain and the training provides law enforcement officers with knowledge of the legal requirements for obtaining protection orders, but does not teach them how and where to obtain the protection order. Then, considering the content of the training sessions, the desired result is not likely to occur, or if this result occurs, it is not likely to be a direct result of the training.

Also, the training must be implemented well enough that its critical elements have been delivered, and to the right people. If, for example, one section of a training program is not taught, you cannot expect to see the same results as if trainees had received the entire program. The missing component has to be identified and accounted for since it is only logical to assume that only a completely implemented effort can have the expected impact. If a training component is missing this may actually be an explanation for why the assessment results indicate that the program did not have the predicted impact.

Levels of Evaluation for Training Programs

You will need to gather several types of information before you can see whether a training is successful. First, you need information from the trainers about the goals and the content of training, to identify what was intended and what was delivered. Second, you need baseline information from the trainees about their level of knowledge, attitudes, and skills before the training. Finally, you need information from the trainees after the training to identify changes as a result of the training. In addition, training sessions related to violence against women will often include victims, survivors, and advocates as participants or presenters. These participants bring a unique perspective to a training session. They should be asked to give their perceptions of how well the other participants "got the message," especially on the issues of attitudes and sensitivity toward victims.

A very widely accepted approach to evaluating training programs has been developed by Kirkpatrick (1996), who outlines four levels of measurement to address the range of potential training effects. Although many training evaluations focus on only one or two of these levels (reaction assessment and learning), he argues that to truly assess the outcome of training, one should gather data to measure all of the following:

• Reactions: an on site measurement of immediate "customer satisfaction" with various aspects of the training program;
• Learning: an assessment of changes in attitudes, increased knowledge, and/or improved skills;
• Behavior Changes: an assessment of changes in participants' performance or ability to apply learning; and
• Results, or Problem Impact: an assessment of the impact or applied results of the training.

This model can be applied to any training regardless of the setting or topic. Each level of assessment can provide valuable information for developing, evaluating, and revising training curricula. However, assessment strategies such as the type of information to be gathered, timing of the assessment, and the method of gathering information are different at each level.

Table 13.1 shows Kirkpatrick's four levels of evaluation, reaction, learning, behavior change, and problem impact (in our case, impact on VAW), and how they apply to the common training goals of knowledge and attitude change, skill acquisition, behavioral intentions and behavior change, organizational change, and impact on victims and on violence. It also shows the most relevant timing for evaluation at each level, and the importance of taking direct and indirect external influences into account. Immediate assessments are those done during a training session or just before everyone goes home. Short-term follow-up assessments are done after trainees are back on the job for a while, but usually within the first two or three months, possibly up to six months, after training. Long-term follow-up assessments can be done months or even years after the training. Timing will depend on how long you think it will take for a particular impact of training to occur. Changes in trainees' own behavior might be expected to occur relatively quickly (within the first few months), whereas changes in colleagues' behavior might take up to a year to catch on, and institutional change could take several years (although there would be signs along the way, such as resource commitment, planning committees, protocol development committees, etc.). We discuss each level of evaluation shown in Table 13.1, including the measures that are most commonly used in relation to each training goal for each level.

Level 1: Reaction Assessments

The most basic level of impact evaluation focuses on capturing immediate, on-the-spot reactions to different aspects of the training, providing a measure of "customer satisfaction." However basic, it is an important method of obtaining information that will help training organizers improve training programs for future presentations.

The information gathered usually asks for the trainee's judgment about the content of each session, the materials used, the instructor's ability to present the material, the relevance of the material to the trainee's job, and the training facility. Examples of "reaction assessment" questions, to be answered on a scale from 1= strongly disagree through 5 = strongly agree follow:

1. The content of the session was relevant to my professional needs.
2. Overall, the information provided in the session was practical.
3. The time allocated to the subject was adequate.
4. The session enhanced my knowledge of the subject.
5. The instructor was very good at getting his/her points across.
6. The room where the session was held was adequate (big enough, warm/cool enough, appropriate arrangement of tables and chairs, easy to see the board/screen/flip-chart, etc.).
7. I learned a lot that I needed to know.
8. The session changed the way I will think about (_________) in the future.

All evaluation at the level of reaction assessments is immediate. One can assess reactions to the presentation of materials, regardless of what the materials were trying to do (that is, whether they were trying to convey knowledge, change attitudes, change behavior, etc.). One can assess trainee's beliefs that they learned something new, their perceptions that efforts to change their attitudes or feelings about a topic succeeded or failed, their beliefs that they have acquired new skills, and their intentions to change their behavior once they get back on the job.

Even with reaction assessment, background and external factors (Columns A and C of your logic model) can play a role. To understand how these factors might affect reactions to training, you would also gather information about the ones you thought might make a difference and analyze your results against these factors. Some background and external factors will need to be collected through questionnaires completed by trainees (including their application for training), while others will be obvious to anyone who attended the training. For reaction assessments, the major background and external influences one would want to guard against are:

• Aspects of the trainees' own background, prior exposure to the field and the material, and work situation; and
• Major disruptions in training (e.g., a power failure meant that two-thirds of the sessions were canceled; an outbreak of Legionnaire's Disease meant that half of the participants were sick or getting sick; the training materials were not delivered until the most important sessions were finished).

Since a reaction assessment gathers immediate perceptions and feelings, it is usually conducted during the training and immediately prior to the trainees' departure by distributing and collecting training evaluation questionnaires to be completed by all participants. These questionnaires can be done for each session, for each day, and for the training program as a whole, depending on the level of detailed feedback you want to receive. A session questionnaire should assess immediate reactions to content and presentation of each individual session, including the adequacy of opportunities to practice new learning if that is one of the goals of training. A combination of forced-choice (e.g., yes/no, agree/disagree) and open-ended questions that give people space to write in comments will facilitate participant feedback. A daily questionnaire should assess reactions to the balance of types of sessions (plenary, didactic, practicum, coalition-building, etc.), order of sessions, quality of presenters, and so on. When the training extends beyond one day, an overall training evaluation questionnaire should be used to gather participant assessments of the training program as a whole, the quality of written materials, format, schedule and organization of the course, trainers, and facilities. If the session is one day or less, this information should be requested on the daily questionnaire, which will serve also as the overall questionnaire. In an effort to encourage full and honest participation, such evaluations are typically submitted without names or other identifying information.

While one usually gets many completed questionnaires from a reaction assessment, it provides only limited information on the effect of the training. Its findings generally do not provide information about goal achievement, because a reaction assessment cannot capture even the most basic impacts such as the amount of learning or attitude change that occurred. Nor can it register how much of the newly acquired knowledge actually stays with the participant over time or how well it can be and has been applied in the field.

Reaction assessments are very much like the little card one is often requested to complete in a restaurant to indicate how much one liked the services and food. You can indicate being pleased with or displeased with the facility, the waiter, and the food, but there is no opportunity to add comments when the food did not agree with you several hours later. Some feedback of this immediate type is important but it cannot give the whole picture, nor does it capture the most important potential impacts of training. Immediately following training, participants may have the feeling that they learned much and may indicate this in a questionnaire administered on site. This result reflects the participants' perceptions of their learning, but the reaction assessment does not actually test that learning, nor does it assess whether the learning is applied on the job. Thus, a reaction assessment cannot adequately evaluate whether a training program accomplished any of its goals (beyond merely having gotten through the training day).

Level 2: Learning Assessments

An essential component of a training evaluation is an assessment of what participants learned. New knowledge and new attitudes are always primary training goals, and are also the basis on which trainers expect that trainees may change their behavior in the future.

Learning can be measured by assessing changes in participants' knowledge, attitudes, or skill levels from immediately before to some time after a training program. Such pre- and post-test measures of how well participants understood the material covered in the training program provide an objective indication of whether learning took place. Ideally, pre- and post-test assessments would be conducted with a group of training participants and with a comparison group who did not participate in training. By examining the differences between the two groups, evaluators can assess on the post-test whether participation in the training program improved participants' performance. However, few training programs have the funds and staff to use comparison groups. As a result, learning assessments are generally limited to those who participate in the training.

As mentioned earlier in this chapter, for training related to violence against women in particular, attitude change and increased sensitivity to victims must be an essential goal. There are some general points of sensitivity that should be included in any STOP-funded training evaluation, including the following:

• Behavior and language should reflect a belief in what the woman is telling you;
• Treat the woman with respect;
• Don't make disparaging remarks or jokes;
• Show by your behavior and remarks that you take her situation seriously;
• Protect the woman's privacy, to the extent possible in the situation.

Questions assessing attitudes can be constructed to fit the specific work situations of the trainees. For example, in a police training focused on new laws specifying mandatory arrest of batterers, important attitudes to address in both training and evaluation are those which have been shown to reduce the likelihood that police will make an arrest (Ford, 1987). These include ideas about the seriousness and criminality of battering, whether some women are "unworthy" of police protection, whether women have acted on their own behalf, and whether battering is a "private matter." One can assess these attitudes on 5-point scales (e.g., 1=agree strongly, 2= agree somewhat, 3=neither agree or disagree, 4= disagree somewhat, 5=disagree strongly; or that a factor, if present in the situation, 1=strongly favors arrest, 2=somewhat favors arrest, 3=irrelevant to arrest, 4=somewhat against arrest, 5=strongly against arrest). The first group of items shown below uses the agree-disagree scale; the second group uses the favors-against arrest scale:

• Misdemeanor battery is too minor an offense to deserve arrest.
• A man does not deserve to be arrested for slapping his wife.
• If a woman does not leave a man who beats her she should not expect the police to protect her.
• It takes two to fight so one cannot blame the man alone when he hits his wife.
• It is best to arrest both parties in order to let the court decide who is at fault.
• The woman is highly agitated [as a reason not to arrest].
• The police had been to the residence before [as a reason not to arrest].
• The woman went out drinking against her husband's wishes.
• The woman does not want to prosecute.

The same types of attitudes are also relevant to prosecutors' decisions not to prosecute, judges' decisions to reduce sentences or waive punishment, and other actors' decisions about whether or not to support a battered woman. Parallel attitudes affect the decisions of police, prosecutors, judges, juries, and potential support people in cases of sexual assault. Attitude scales related to both sexual assault and battering are described in Chapter 11, where you can see other examples of attitudes that are important to include in training and to measure because they make a difference for people's behavior toward women victims of violence.

To assess learning, the content of a pre- and post-test evaluation instrument must be completely consistent with the training course content. The questions on the questionnaire should match the most important elements of the information that the training organizers were trying to convey. The most typical format is a combination of true/false and multiple choice questions that are based on material covered in the course. It is best to include both fact-based questions and questions that require the application of information to a problem situation. If attitude change was one of the training goals, questions should cover current beliefs related to the topics about which trainers hoped to change attitudes. If skill acquisition is a training goal, paper-and-pencil measures may not be enough to measure training success. In addition to participants' self-reports that they learned new skills, evaluators may wish to set up situations in which they can actually observe trainees to see whether they are using the skills. This can be done during training in practice sessions and also is good to do in the participants' regular job situation at some time after training. Finally, paper-and-pencil assessments toward the end of a training program can ask participants about their behavioral intentions—what do they intend to do when they get back to their jobs, how is it different from their usual behavior, and how do they plan to introduce these new behaviors into the workplace.

Timing is a critical dimension of learning assessments. You will want to collect data before training to assess "pre" levels of knowledge, attitudes, and skills. You will also certainly want to collect data at the end of training, to provide the easiest form of "post" evaluation data. However, bear in mind that "after" training covers a lot of time. It is very important for a training evaluation to try to include some assessment "down the road," as the effects of knowledge and attitude change are known to wear off quite rapidly if they are not quickly reinforced with behavior change. So a good evaluation of training should try to assess learning retention two or three months after training, and should also track whether by that time participants have changed their behavior in any way that fulfills the behavioral intentions they stated at the end of training.

Since the measurement of learning is linked to the topic presented in a course, few measures can be applied to every training evaluation. We can, however, give an example in which judges were taught the contents and implications of a state's victim rights legislation. Measures of increasing levels of learning and how they can be established for this subject matter cover the following (Bloom, 1956):

• Knowledge: Training participants are able to quote the rule/legislation that victims have the right to speak at the sentencing hearing.
• Comprehension: Participants are able to summarize the rule/legislation that states that victims have the right to speak at sentencing hearings.
• Application: Participants see a video of a standard sentencing hearing and examine all methods that were followed and discover opportunities that were missed, that would have allowed the victim to speak.
• Analysis: Participants can illustrate how the victim may be allowed the opportunity to be heard in a sentencing hearing based on a scenario in which, for example, a victim who is severely brain damaged and, as a result, unable to speak.
• Synthesis: Participants devise a new rule for victim participation when victims are unable to speak for themselves.
• Evaluation: Participants are able to assess the adequacy of methods used to allow victims the right to be heard at sentencing hearings.

If the purpose of a training program is to increase the skills of participants in a particular area, a performance-based pre- and post-test evaluation may be utilized on site to collect immediate information about the effectiveness of the training and to provide immediate feedback to the trainees. Evaluators using this approach can