Abstract
Evaluation of universal dating violence prevention programs has rapidly expanded in the past two decades. Many programs demonstrate change in attitudes supportive of dating violence, and a few show evidence of behavior change; however, detailed analysis of process and fidelity of program implementation is generally neglected. An important goal of prevention research is to identify successful initiatives that can be replicated and disseminated in the field. The purpose of the current case study is to document the implementation process of a middle school–based dating violence prevention curriculum in economically disadvantaged urban neighborhoods. Particular attention is given to the school context, such as the process of school and teacher recruitment, the program model, and classroom implementation of the dating violence prevention program in four areas: teacher training, student outcomes, program fidelity, and student engagement. Nine health and physical education teachers from six urban middle schools participated. Results describe effective strategies to secure school participation and engagement, and provide evidence regarding methods to train health and physical education teachers in low-income, urban neighborhoods. Furthermore, classroom observations demonstrate that teachers successfully implemented the five-lesson curriculum, which resulted in positive student outcomes to prevent dating violence. This case study represents an important step in deepening our understanding of the mechanisms of program delivery.
Evaluation of universal dating violence prevention programs has rapidly expanded in the past two decades. Initiatives often target all students in the school population and take place in the school context during regularly scheduled classes, after school, or in the community (e.g., Avery-Leaf, Cascardi, O’Leary, & Cano, 1997; Foshee et al., 2004; see Leen et al., 2013, for review). Many programs demonstrate change in attitudes supportive of dating violence (see Leen et al., 2013; Ting, 2009, for reviews), and a few show evidence of behavior change (e.g., Foshee et al., 1998; Foshee et al., 2004; Wolfe et al., 2009). However, detailed analysis of process and fidelity of program implementation is generally neglected. An important goal of prevention research is to identify successful initiatives that can be replicated and disseminated in the field. Without an understanding of factors that enhance or impede implementation, dissemination of effective programs will be limited (Elliott & Mihalic, 2004). The primary purpose of this article is to describe the process of recruitment and implementation of a universal, school-based dating violence prevention program.
The literature on implementation of dating violence prevention programs is limited. We identified two studies, which reduced rates of dating violence and provided information about the program implementation process (Foshee et al., 1998; Foshee et al., 2004; Wolfe et al., 2009). In one study, teachers reported covering approximately 90% of curriculum material in the Youth Relationship Project, which focuses on healthy dating and peer relationships (Wolfe et al., 2009). In another program, Safe Dates, health teachers were reported to have completed 91% of curriculum activities (Foshee et al., 1998). These data suggest that changes in student behavior were likely due to the interventions, which were implemented with a high level of protocol adherence. However, teachers’ delivery of program materials was not directly observed in either study. Teachers self-report about program adherence, while useful, may not accurately characterize the implementation process, that is, teachers may be motivated to characterize their program delivery in a favorable light to please investigators and school administrators. Moreover, Foshee et al. (2004) noted, “From anecdotal reports we know that the program was not always used in its entirety” (p. 621). Thus, teachers may have overestimated their adherence to the protocol.
Reliable data about fidelity are important to inform practitioners about the conditions under which desired prevention outcomes may be achieved (Moore et al., 2014). Fidelity is generally defined as the degree to which delivery of an intervention adheres to a specific protocol (Mowbray, Holter, Teague, & Bybee, 2003). Other factors may also affect implementation success, such as the quality of teacher delivery (e.g., enthusiasm, demonstrations of content mastery) and student responsiveness and engagement. It is entirely possible that a teacher could show excellent fidelity, but the students may be disengaged and inattentive or teachers may undermine their efficacy through lack of facility with curriculum lessons. Analysis of these factors can further understanding as to whether the program or teacher training must be amended to increase efficacy. Thus, definitions of program fidelity should include adherence to proscribed curriculum components as well as other aspects of program delivery, that is, teacher enthusiasm and student responsiveness.
Another important facet of school-based dating violence prevention program evaluation is the process of securing school administrative and teacher support. Lack of school buy-in will undermine implementation and diminish the intended impact of a program (Rohrbach, Graham, & Hansen, 1993). Conversely, collaborative relationships with school administrators and teachers can enhance program impact and point toward successful strategies to facilitate dissemination to other school communities. It is also important to characterize the community where prevention programs are evaluated to determine the extent of a program’s generalizability from the environment where it was tested.
Despite the importance of these factors, they typically do not receive attention in the dating violence prevention literature. The purpose of the current study is to document the implementation process of a middle school–based dating violence prevention curriculum in economically disadvantaged urban neighborhoods. Adapting Saunders, Evans, and Joshi’s (2005) process evaluation model, we described
the school context where the program took place, including the process of school and teacher recruitment;
the program model, including its theoretical justification, stated objectives, and delivery methods; and
the implementation of the dating violence prevention in four areas: teacher training, student outcomes, program fidelity (broadly defined to include teachers’ adherence to the curriculum protocol and quality of implementation), and student engagement.
Method and Results
School Context
As part of a cooperative agreement with the Centers for Disease Control and Prevention (CDC), a school-based primary prevention program to deter dating violence in Philadelphia was evaluated from 1998 to 2000. Institutional Review Boards at Medical College of Pennsylvania (MCP)-Hahnemann University, School District of Philadelphia and the CDC each approved the study protocol. At the time of this study, schools in Philadelphia were organized into neighborhood clusters and these clusters were centralized through one district office. The School District Central Office and cluster leaders granted initial approval for this project. This proved to be a crucial step in gaining access to specific middle schools, as school administrators were receptive to learning more about the project knowing that it had already been vetted by central administration.
School neighborhoods that were economically disadvantaged and had high crime rates relative to citywide violent crime were recruited for participation (n = 8). Data from the Philadelphia school district office indicated that more than 97% of students were African American, in large schools (school enrollment: M = 837, range = 509-1,011 students), and economically disadvantaged. A majority of students (70%-100% across schools) received Temporary Aid to Needy Families and were eligible for reduced or free lunch. Violent crime in school neighborhoods was 1.5 times the citywide average (http://cml.upenn.edu/).
School and teacher recruitment
Recruitment meetings were held with school principals, guidance counselors, and health/physical education teachers. Participation of these key stakeholders facilitated engagement and support, particularly among teachers who would be called upon to integrate the dating violence prevention curriculum into existing health classes. Importantly, during these meetings, we addressed administrator and teacher concerns regarding the appropriateness of the topic for middle school students, loss of class time from mandated curriculum requirements, plans for program continuation post project completion, and dissemination of results. Specifically, we shared research findings to substantiate the high rates of dating violence among adolescents nationwide. We partnered with the schools to identify elements of the core curriculum that the dating violence prevention program satisfied and emphasized the value of training existing teachers for program continuity. Finally, we explained ethical protections regarding confidentiality, which eased concerns that school identity would be revealed when results were disseminated. Two of the eight schools declined participation because they did not perceive dating violence to be an appropriate topic for middle school. The remaining six agreed to participate. One half of schools were randomly assigned to receive the program and the other half served as a wait list control.
Program Model: BRIGHT Program
Cascardi and Avery-Leaf (1998) developed the Building Relationships in Greater Harmony Together (BRIGHT), which consists of five daily lesson plans designed for delivery in 40-min health education classes. Theoretically relevant, empirically supported, and modifiable correlates of dating violence are targeted in this program. Each lesson includes a comprehensive rationale, instructional strategies and troubleshooting tips, a set of lesson objectives, planned activities, discussion points, and handouts. See Figure 1 for a detailed summary.
BRIGHT program overview.
The theoretical framework of this program emphasizes background characteristics that may predispose individuals for aggressive behavior against a dating partner, such as violence in one’s family of origin and attitudes supportive of dating violence as well as situational factors that may increase the likelihood of dating violence in specific circumstances, including jealousy and relationship conflict (e.g., Riggs & O’Leary, 1996). Features of this model have been supported by research demonstrating that physical dating aggression is consistently associated with history of maltreatment (e.g., Chiodo et al., 2012; Narayan, Englund, Carlson, & Egeland, 2014), past dating violence (e.g., Boivin, Lavoie, Hébert, & Gagné, 2012), verbal/psychological aggression with a dating partner (Cano, Avery-Leaf, Cascardi, & O’Leary, 1998; Murphy & O’Leary, 1989), and attitudes supportive of dating violence (e.g., Ali, Swahn, & Hamburger, 2011; Kernsmith & Tolman, 2011; Temple, Shorey, Tortolero, Wolfe, & Stuart, 2013). In sum, the BRIGHT program treats courtship aggression as a multidetermined phenomenon, is sensitive to gender inequities that may foster violence, while recognizing that males and females occupy roles as aggressors and victims. The program also emphasizes attitude change and skill enhancement to decrease aggressive expression with a dating partner.
BRIGHT lessons are primarily built around a fictitious teen couple, whose relationship is characterized by jealous and control behaviors and escalates to physical dating aggression over time. The case is used to illustrate key concepts such as relationship rights and responsibilities, common teen stresses, anger management and communication skills, and help-seeking resources. Prior research on this program demonstrated that it is associated with decreases in attitudes justifying the use of dating violence as a means to resolve conflict (Avery-Leaf et al., 1997) and jealous, controlling behaviors and physical aggression against a dating partner 3 months post program completion (Cascardi, Avery-Leaf, O’Leary, & Slep, 1997).
Implementation
Teacher training
Two doctoral level clinical psychologists conducted 9 hr of teacher training for nine middle school health and physical education teachers (four female and five male). Participation was voluntary and confidential, and teachers were compensated for their time at customary professional development rates. Each teacher signed an informed consent form agreeing to be observed and evaluated while teaching the BRIGHT program.
A significant portion of training included modeling for the teachers how to teach the BRIGHT program to develop basic knowledge and mastery of the program. Teachers were given answer keys for class worksheets and activities, as well as detailed lesson plans and troubleshooting tips. As discussion of teen dating violence can quickly escalate in intensity and diverge from the ostensible training purpose, a highly structured training agenda was used to maintain order and focus.
Dating violence, and relationships more generally, can be emotionally charged topics, and teachers must be prepared to handle their personal reactions to the material. Thus, a critically important element of teacher training was engaging teachers in self-reflection about their own relationships, communication, and emotional management skills. To this end, teachers completed a communication style quiz and role-played various anger-inducing scenarios to discuss how their individual experiences may affect their delivery of program content. These exercises were provocative and uncomfortable for some teachers. However, their value cannot be underestimated. Despite some degree of discomfort, all teachers reported appreciating the opportunity to reflect on their relationships and identified areas for continued personal growth and improvement.
Teachers completed anonymous surveys at the end of training to provide feedback about the BRIGHT program and the impact of training. Overall, teachers reported that the program should be taught in all health classes and was easy to read and follow, with clear teaching objectives and concepts. Teachers also believed that the video, worksheets, and activities would engage the students. There was some concern that the BRIGHT program had too much material per session and that the content may exceed students’ reading comprehension, and efforts were made to adjust the program length in accord with these concerns.
BRIGHT program student outcomes
The program was implemented across 2 years, with one cohort of youth (N = 899) receiving a single exposure to the five-session BRIGHT program during health class, and another cohort (N = 465) receiving a second exposure to a five-session booster version of BRIGHT, which reinforced principal concepts and skills. Youth in each of these cohorts were compared with students in the control schools (N = 1,319). A passive parental consent procedure was used. All participating students gave assent. Youth completed a pre-and post-program survey in the week before and after each administration of the BRIGHT program. Results indicated that a single dose of the BRIGHT program was associated with endorsement of fewer myths about dating violence, for example, jealousy is a sign of love. Attitudes justifying verbal and physical aggression, jealous and control tactics, as well as intentions to use physical aggression against a dating partner decreased significantly only after a second exposure to the BRIGHT program. Neither youth in the control condition nor those who received a single exposure to BRIGHT showed attitude or intention to change (Cascardi, Avery-Leaf, & O’Brien, 1999).
Program fidelity
Program fidelity was broadly defined to include teachers’ adherence to the curriculum protocol and the overall quality of implementation and effectiveness in engaging students. Two doctoral level evaluators trained for 1 month to achieve agreement on fidelity rating scales before observations began. Both raters observed the same 40 class sessions across nine teachers (4-5 sessions per teacher) and resolved any discrepant ratings through discussion. Teachers were not informed about which classes the raters would observe to minimize effects on teacher behavior.
Adherence to curriculum protocol
Overall, 66% of session objectives and 76% of session activities were fully covered across all five lessons of the curriculum. Although a majority of lesson objectives and activities were completed, teachers were generally unable to complete all session objectives and activities. Only 33% of teachers completed all objectives and 40% completed all activities. Session-by-session analysis revealed that few objectives or activities were missed: one (of four) objectives and no activities in Session 1, no objectives and one (of five) activities in Session 2, no objectives or activities in Session 3, and one (of five) objectives and no activities in Session 4 were omitted. Two teachers who were not engaged in the teacher training showed low commitment to curriculum implementation. Repeat visits were required to complete observations for these teachers as on two occasions they did not hold their scheduled health class.
Quality of implementation
Overall quality of implementation was measured by teachers’ presentation styles in eight domains: volume, articulation, pace relative to students’ needs, enthusiasm, ability to engage students in program activities, encouragement of on-task behavior, respect and display of professional behavior conducive to curriculum implementation (e.g., refraining from sexist jokes or minimizing violence). Each domain was rated on a 3-point scale: 1 = “very poor,” 2 = “fair,” and 3 = “very good.” In the domain of articulation, “very good” meant that the teacher demonstrated clear/distinct pronunciation of words, so that he or she could be understood for most or all of the class; a rating of “fair” was given if the teacher demonstrated clear/distinct pronunciation of words, so that he or she could be understood for some of the class; and “very poor” when the teacher’s enunciation of words was poor and he or she could not be understood for most of the class.
Overall, teachers demonstrated a high level of preparedness and engagement with the curriculum material that was conducive to learning, as evidenced in their classroom delivery (see Table 1). The vast majority of teachers spoke loudly and clearly, and maintained a consistent teaching pace relative to students’ needs across almost all classes. Teachers also demonstrated ease, freedom, and comfort with the dating violence curriculum material and exhibited eagerness, devotion, and excitement while teaching the lesson. Professional demeanor and respect for students was the norm; teachers did not endorse gender stereotypes, sexist or violence jokes or make inappropriate self-disclosures. Teachers also frequently encouraged on-task behavior and engaged students in discussion, activities, and video material.
Teacher Delivery.
Student Engagement
Student engagement was based on students’ responsiveness and observed level of attentiveness and involvement in class discussion, which was rated on a 3-point scale: 1 = very poor, students were not involved or did not participate for most of the class; 2 = fair, students were involved and participated for parts of the class; and 3 = very good, students were actively involved and participated for most of the class. In addition, four student behaviors: bored or distracted, fidgeting, bell behavior (e.g., leaving class before dismissed), and disengaged (e.g., putting head on desk), that may have affected their attention in class were rated as present or absent.
In more than one half of classes (55%), students sustained attention and actively participated in class discussion and virtually all students remained seated until class dismissal (92% of classes). However, students’ behavior also disrupted the educational process in a variety of ways. For at least some period of the class, a majority of students appeared bored and distracted or fidgeted (92% of classes). In less than one half of classes, students exhibited disengaged behaviors, such as placing their head on the desk (45% of classes).
Discussion
This study reported the results of a multipart process evaluation for a social learning and skills-based program to prevent dating violence. Overall, this process evaluation demonstrated successful recruitment of schools and teachers in urban, high crime, and economically disadvantaged neighborhoods; provided a detailed program model; documented methods for effective teacher training; and gave evidence that well-trained health and physical education teachers successfully implemented a five-lesson curriculum to prevent dating violence. A particularly unique aspect of this effort was the collection of data based on direct observation of teachers’ delivery of the BRIGHT program in the classroom and students’ response to it.
The success of school-based programs requires buy-in from key stakeholders as well as their participation in the planning process (Elliott & Mihalic, 2004; Nation et al., 2003; Payne, Gottfredson, & Gottfredson, 2006). In this study, approvals at each level of school administration increased individual school’s willingness to meet with the project team. Receptivity to the program was most enthusiastic when initial and follow-up meetings included the principal, the lead health and physical education teacher, and school guidance counselors. This team approach encouraged a cohesive and collaborative attitude, and maximized engagement and success of the program effort. Other important keys to success were ensuring program continuity in the absence of the research project, providing data to convince school personnel of the need for a program focused on dating violence prevention in middle school, and identifying the ways in which the BRIGHT program met core state curriculum standards and requirements.
Training, in general, is most effective when trainers possess content expertise, skills for managing group processes, and rely on role-plays and small group discussions (Berkhof, van Rijssen, Schellart, Anema, & van der Beek, 2011; Khamarko, Koester, Bie, Baron, & Myers, 2012; Pearce et al., 2012). Results from this study suggest that important qualifications for trainers include content knowledge about dating violence prevention, prior teaching experience, and expertise in using nonthreatening strategies to encourage teachers’ reflection on their interpersonal skills and biases that might affect their ability to teach about dating violence prevention. Due to the sensitive nature of this topic, it is very important that teacher trainers similarly understand their own limitations associated with personal relationships and possess strong communication skills for the purposes of demonstrating lesson exercises as well as facilitating discussion that has a high potential for strong emotional reactions. For example, there were times during training when teachers disagreed with curriculum content or demonstrated discomfort or unwillingness to self-reflect. The trainer must be equipped to gently encourage engagement and also recognize when teachers are not comfortable or willing to prepare for effective delivery of the BRIGHT program. In this study, two teachers seemed unwilling to use the BRIGHT program and did not consistently hold class as scheduled. As schools deliberate using a dating violence prevention curriculum, one important consideration is the degree to which all selected teachers share a high level of enthusiasm and willingness for personal reflection as well as respect and support for the topic.
An important contribution of this study was direct observation of what took place during lessons of the BRIGHT program. A detailed program model enabled comparison of teachers’ performance with the program contents. Teachers were not privy to the observation schedule, so they could not prepare beforehand for observation sessions. Thus, classroom observations likely represent a genuine representation of how class typically occurred. Classroom ratings clearly demonstrated that teachers possessed mastery and comfort with the BRIGHT program and used a variety of strategies to engage students in the material. More than two thirds of lesson objectives and activities were completed. This rate of completion is lower than what has been reported previously (Foshee et al., 1998; Wolfe et al., 2009); however, in prior reports completion was based on teacher report and not direct observation. It is possible that self-report may yield higher completion rates due to teachers’ desire to present themselves in a positive light or inaccuracies in recollection, problems commonly associated with self-report methods (Kazdin, Esveldt-Dawson, & Loar, 1983; van de Mortel, 2008). This is the first study of which we are aware to provide data based on objective observations of teacher behavior. These results suggest that it is important to observe directly what takes place in the classroom and not rely solely on teacher report. However, we did not collect information on teachers’ reports about lesson completion, so it remains uncertain how teacher report and direct observation compare. Both methods used simultaneously will likely yield the most complete picture of lesson delivery.
Analysis of specific lessons showed that teachers had some difficulty completing all components of the program. Despite these challenges, however, the overwhelming majority of teachers were rated as having an appropriate pace relative to students’ needs, demonstrating ease and comfort with the material, as well as very good enthusiasm and engagement of students. Taken together, this suggests that the quality of lesson delivery was high overall, even if certain aspects of the program may not have been fully addressed. In sum, these data indicate that the observed positive student outcomes, that is, changes in attitude and behavioral intention, were likely linked to the program and also point toward the importance of teacher proficiency to achieve successful implementation. Training is a key component to successful implementation that cannot be minimized or eliminated.
Despite teachers’ apparent mastery and enthusiasm for the BRIGHT program, and change in students’ attitudes and behavioral intentions, observation of student behavior indicated that students were not uniformly responsive to the program. While teachers appeared to use the BRIGHT program in a competent and engaging manner, most students were bored or inattentive for at least some portion of the class. This observation may not be unusual for middle school students. However, it is important to understand which aspects of class were particularly uninteresting, so that the lessons can be improved. Another troubling observation was the proportion of students who put their heads on the desk. While the precise meaning of these behaviors is unclear, we make several inferences. As a majority of youth never engages in dating aggression or desists as they approach the end of high school (Orpinas et al., 2012, 2013), it is possible that these youth were the ones who appeared disinterested or bored. Alternatively, for some students, the BRIGHT program content may have been distressing and they put their heads on their desk or otherwise appeared bored and distracted as a way of coping with upsetting information. Although we did not have data about students’ emotional reactions to the program or the reasons for any potentially distressing reactions, it is possible that the program elicited negative emotional responses. Universal prevention programs may not be appropriate for youth who find the material emotionally upsetting. Instead, it may be beneficial to tailor prevention efforts to youth who may be at risk for emotional distress, such as those who have prior trauma or repeated dating violence experiences, in a more sensitive manner. Finally, it is also possible that the developmental profile of middle school students, such as self-consciousness about bodily changes associated with puberty, and apprehension about dating may make mixed sex classes uncomfortable and led to the observed student behavior (National Middle School Association, 2003; Simmons, Blyth, Van Cleave, & Bush, 1979). Same sex classes may be more appropriate for this age group. Future research should address students’ emotional responses to the content of dating violence prevention programs to address these potential concerns and their implications for optimal prevention strategies.
Limitations
The current study was implemented in a very challenging setting: high crime neighborhoods in an inner city. Other limitations of the current study include the small number of teachers trained and the small number of classes observed, which may be addressed through replication studies. The inability to correlate teachers’ reports about training and observations of their classroom implementation posed another limitation. Future research should relate training effectiveness to teachers’ classroom performance. Furthermore, students were not asked about their classroom experiences with the BRIGHT program, so some of the observational data are difficult to interpret, for example, students who put their heads on the desk during class. Future research would benefit from additional qualitative data from students about their perceptions of the program content as well as their emotional reactions to the material.
Conclusion and Implications
In conclusion, this case study goes beyond the typical descriptive approach to offer a unique exploration of several key aspects of program implementation. Substantial resources on many fronts have been poured into the study of adolescent dating violence and how to remediate this problem. Although we know that school-based universal primary prevention efforts are promising, and therefore worth pursuing, very little is known about the separate components of such interventions to inform program development and other refinements. The current study represents an important step in deepening our understanding of the mechanisms of program efficacy.
Implications from this work emphasize the importance of securing project approval at the district level, which smoothed the process of achieving stakeholder buy-in among schools and teachers. At the school level, identifying ways to integrate the prevention program contents with mandated curriculum requirements facilitated enthusiasm for the project. Although teachers achieved competence and comfort with the BRIGHT program during training, teachers had some difficulty implementing all aspects of the program. Therefore, expanding and/or intensifying training might be the best way to maximize efficacy of program. Other promising approaches in other fields are a two-phased coaching model, where teachers are initially trained in a universal program and then receive individualized coaching sessions tailored to specific areas of weakness (Becker, Darney, Domitrovich, Keperling, & Ialongo, 2013) and continuous professional development (LaChausse, Clark, & Chapple, 2014). Future research on dating violence prevention should devote greater attention to evaluating training strategies and expanding qualitative assessment of students’ response to curriculum material.
Footnotes
Authors’ Note
The findings and conclusions of this study are those of the authors and do not necessarily represent the official position of the funder.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research and/or authorship of this article: This study was funded by the National Center for Injury Prevention and Control, Centers for Disease Control and Prevention, Cooperative Agreement US4/CCU314265-01.