Adapting a men’s behaviour change program to online delivery using a developmental evaluation approach

Abstract

Men’s behaviour change programs constitute an important family violence intervention strategy. In 2020, the COVID-19 pandemic and associated restrictions brought increased risk of family violence and service demand. When face-to-face groups became impossible under restrictive public health orders, the Men’s Behaviour Change Program provided by Relationships Australia Victoria was delivered using videoconferencing technology. We describe a developmental evaluation approach to the adaptation of the Men’s Behaviour Change Program to online delivery. Developmental evaluation can inform innovation in complex environments and emphasises learning and adaptation at program development phase. We demonstrate the application and utility of this approach to the rapid adaptation of social services for online delivery. Quantitative and qualitative data were collected from participants, facilitators and external observers and subjected to regular collaborative review to inform practice development. The results point to a steep learning curve, during which facilitators and observers identified practices to support effective online delivery of the Program. The evaluation indicated that, following adaptations, Program objectives were achievable using this method of delivery and positive dynamics remained possible in online groups. This finding is consistent with the current No to Violence position that online men’s behaviour change programs have an important and ongoing role in the specialist family violence services sector.

What we already know

• Men’s behaviour change programs are an important family violence intervention informed by a variety of state and national guidelines.

• Developmental evaluation is an appropriate evaluation technique in complex and adaptive environments.

The original contribution the article makes to theory and/or practice

• Effective practices for delivering online men’s behaviour change programs are identified.

• The utility of developmental evaluation in adapting community services to online delivery is demonstrated.

Introduction

Family violence is a widespread problem in Australia, and a major risk factor affecting the health and wellbeing of women (Australia’s National Research Organisation for Women’s Safety 2018). COVID-19 and associated restrictions created strong potential for an increase in family violence, with millions of Australians required at various stages of the pandemic to stay home except for essential outings. In May 2020, two thirds of women who had experienced physical or sexual violence from a partner during the pandemic reported the violence having started or escalated in the three months since the onset of COVID-19 (Australia’s National Research Organisation for Women’s Safety, 2021).

Men’s behaviour change programs are family violence interventions aimed at holding men accountable for violent and controlling behaviour, challenging the beliefs underlying this behaviour, and thus improving the safety of women and children (O’Connor et al., 2020). Australia’s National Research Organisation for Women’s Safety has produced several studies relating to evaluation readiness, program quality, outcomes and effectiveness (Bell & Coates, 2022; Day et al., 2019). These reviews have found that evidence for men’s behaviour change programs effectiveness in general is piecemeal but growing, and identify elements of effective practice (Bell & Coates, 2022; Day et al., 2019; see also O’Connor et al., 2020).

Relationships Australia Victoria is a community based, not-for-profit organisation delivering a wide variety of services from 18 centres across Victoria and additional outreach locations. Relationships Australia Victoria is one of the largest providers of men’s behaviour change programs and family safety support services in Victoria and has delivered the Men’s Behaviour Change Program since 1993. Though the Program has employed various formats in this time, it was always delivered in person prior to 2020.

When the outbreak of COVID-19 in March 2020 forced face-to-face groups to cease under public health orders restricting interpersonal contact, the Men’s Behaviour Change Program at Relationships Australia Victoria shifted to online delivery. Many social services across Australia shifted to videoconferencing or telephone service delivery during ‘lockdowns’, coinciding with increased demand for services (Joshi et al., 2021). Though there is robust evidence for the use of technology in health and medicine, evidence for the effectiveness, benefits and risks of tele-practice is still emerging in family and relationship services (Joshi et al., 2021).

Given the demand for and importance of the Men’s Behaviour Change Program in enhancing the safety of women and children, particularly during COVID-19, combined with limited evidence to guide the adaptation of programs for online delivery, an evaluation strategy was embedded simultaneously with the transition to videoconferencing. A developmental (or adaptive) evaluation approach was adopted to inform ongoing adaptations and continuous improvement in a dynamic and complex environment (Fagen et al., 2011; Patton, 2011). This included the systematic collection of data aligned with key program objectives (Patton et al., 2015). In this article we present the data collected to inform this approach, and the resulting adaptations to the Men’s Behaviour Change Program. In so doing we demonstrate the application and utility of developmental evaluation to the rapid adaptation of social services for online delivery.

Considerations relating to online delivery

The Men’s Behaviour Change Program at Relationships Australia Victoria is a 20-week program informed by Family Safety Victoria Minimum Standards (Family Safety Victoria, 2018), the No to Violence Implementation Guide (No to Violence, 2018a), the National Plan to Reduce Violence against Women and their Children 2010–2022 (Council of Australian Governments, 2019), and National Outcomes Standards for Perpetrator Interventions (Council of Australian Governments, 2015). The curriculum is informed by key theories, including socio-political and gender equality theories; practice elements, including solution-focused therapy, psychoeducation, the Transtheoretical Model, and group work; and a program logic (Bell & Coates, 2022; Lee et al., 2009; Mann et al., 2002; Richards et al., 2004; Webster & Flood, 2015). The focus is on accountability and responsibility of participants, enhancing women and children’s safety, promoting readiness for change, and monitoring men’s use of coercive control, abuse and violence (O’Connor et al., 2020). The majority of participants attend voluntarily.

Prior to the onset of the pandemic, and at the time of the evaluation, there was limited evidence to support the online delivery of MBCPs (No to Violence, 2018b). Concerns about online interventions related to compliance, engagement and retention of participants, and the ability to maintain MBCP principles and practice considerations in an online environment (No to Violence, 2018b). However, COVID-19 restrictions coinciding with increased scope for family violence during lockdowns required service responses to innovate rapidly in order to keep MBCP participants engaged. During COVID-19, Family Safety Victoria, (2020), No to Violence, (2022) released service guidelines and proposed principles for blended programs, respectively, which informed the evaluation design. More recently, resulting from the experience of the sector facilitating online program delivery during COVID-19, No to Violence (2022) has endorsed the view that online or hybrid men’s behaviour change programs with additional safety measures are appropriate, can maintain best practice standards, and are likely to remain a permanent contribution to the sector.

Adapting the Men’s Behaviour Change Program to online delivery allowed Relationships Victoria Australia to maintain a focus on risk, safety and accountability during the pandemic. An impromptu pilot commenced in March 2020 when a group that had commenced face-to-face prior to the pandemic was selected to trial online delivery by video for the latter part of the curriculum. Lessons from this pilot were compiled and adjustments made to the curriculum and delivery of the online program using developmental evaluation principles, as described below. Subsequently, commencing in September-October 2020, a further six Men’s Behaviour Change Programs were delivered entirely online by four Victorian centres that normally offer face-to-face groups.

Aims and method

Developmental evaluation applies evaluative thinking and evidence to the development and implementation of innovations in complex environments (Patton, 2011). It is particularly well suited to the adaptation of a program to respond in a crisis (Patton, 2011). A set of guiding principles provides direction and is adapted to specific contexts and situations, making developmental evaluation a distinct evaluation approach (Patton et al., 2015). The aim is to provide program developers with timely feedback that can be used to adapt and improve practice by asking evaluative questions, applying evaluative logic and gathering data to support program development. The more specific aim of the developmental evaluation described here was the rapid adaptation of the existing, face-to-face Program to produce an effective online program.

The underlying objective of the Men’s Behaviour Change Program, regardless of mode of service delivery, is to support the safety of women and children by holding perpetrators to account for their harmful behaviours, attitudes and beliefs, and by driving positive behaviour change. In keeping with this aim, and to align with Family Safety Victoria (2020) COVID-19 service guidelines and principles proposed by No to Violence at the time, the focus of the developmental evaluation was the extent to which the overall program logic, safety and risk management, efficacy and engagement were maintained as the program shifted to online delivery.

Key principles of developmental evaluation include co-creation, utilisation focus, timely feedback, innovation, complexity, systems thinking, evaluation rigour and developmental purpose (Patton et al., 2015). These principles make developmental evaluation an appropriate and common approach to support program adaptation and innovation (Gamble, 2008; Patton et al., 2015; Preskill & Gopal, 2014). The online delivery method and evaluation approach were developed and implemented simultaneously and in consultation with key stakeholders to inform and support the change process. This was an iterative and collaborative process where the evaluator was embedded within the program development team, with a high degree of engagement from the group.

Survey development and review

A developmental evaluation informs program adaptation using concurrent data collection. Surveys were used to collect data from Men’s Behaviour Change Program participants, co-facilitators, external observers and ex-partners, respectively, for the duration of the evaluation and online program development. As the online delivery method was evolving and curriculum content is different each week, groups were invited to complete surveys after each session. This was important to understand the efficacy of various techniques employed, and whether the specific curriculum aims for each week were achieved.

We followed the Guidelines for the Ethical Conduct of Evaluations (Australasian Evaluation Society, 2013). Program participants provided written consent for their participation in the evaluation and to the use of their de-identified responses in external reporting. Practitioners (including co-facilitators) also provided permission for their responses to be used in this way. No to Violence (who provided external observers, as discussed below) were consulted on the preparation of this article. The role of the observers was discussed in sessions by practitioners and transparent to all participants.

Surveys were designed in consultation with a team of managers, practice specialists and group facilitators, adopting a utilisation-focused approach to ensure uptake of the findings and investment in the process itself (Patton, 2008). This team met weekly to review results from these surveys and to design program adaptations in response. The four surveys conducted as part of this evaluation were constructed and administered online using Microsoft Forms.

Data to support developmental evaluation

The data gathered as part of the developmental evaluation are described below. These data informed the development of the online Men’s Behaviour Change Program and informed modifications to ‘usual’ practice. Although responses evolved as the online program was refined over the course of the evaluation, key findings and themes are summarised. Challenges identified along the way, and resulting practice adaptations, are then described in the sections that follow.

Men’s behaviour change program participants

Starting with the pilot group that shifted to online delivery in the latter half of 2020 (from week three of the curriculum), men participating in the Men’s Behaviour Change Program were invited to complete a mix of quantitative items and open questions allowing for short, qualitative responses. Between July and December 2020, a total of 99 anonymous responses were received from 60 men (the total number of men who participated in the Program during this period) from seven online groups (a mean of 5.4 responses relating to each week of program content).

Overall, Men’s Behaviour Change Program participants were positive about online videoconferencing as a mode of delivery. When asked to rate its suitability for the program on a scale of 1 (‘not at all’) to 10 (‘completely’), the mean rating was 8.6 over the duration of the survey, with 85% of responses rating the suitability of this method highly (eight or above). Qualitative data indicated that men were open to participating in this way (‘It was really useful and also a different experience’) and appreciative, in the context of pandemic restrictions, that the program could continue by any means (‘[Videoconferencing] is a new and efficient way in this difficult time’). This was particularly so for those with some experience of face-to-face sessions (i.e. the pilot group).

Participants were also asked to rate their engagement with (a) the program, (b) facilitators and (c) other participants in comparison to engaging face-to-face. A majority of responses across these questions (50%–57%) indicated that engagement with the online program, facilitators and other participants ‘stayed the same’, compared with face-to-face groups that respondents had experienced in the past. Up to a quarter of responses (20%–26%) indicated that engagement was ‘somewhat better’, and a small minority (4%–8%) rated it ‘somewhat worse’. Positive comments most often related to the convenience of attendance without travel, or to more equal contributions from participants due to smaller classes (see Adaptations) and taking turns speaking online (‘More personal due to group size’), but also to feeling ‘safer’ and ‘more relaxed’ when participating from home. This accords with responses to the remote delivery of other relationship services during the pandemic (Heard et al., 2022). Negative comments most often related to the loss of informal interactions around the formal content, with some feeling less connected to other participants and facilitators (‘Interaction isn’t the same’).

When asked how they felt following each session, responses indicated that participants experienced positive feelings (e.g. ‘enlightened’, ‘energized’, ‘optimistic’, ‘relieved’ and ‘confident’), as well as feelings of being challenged by the program (‘reflective’, ‘ashamed’ and ‘regret[ful]’). Some felt ‘anxious’, ‘stressed’ or ‘nervous’ about implementing changes to their behaviour. Many participants described feeling better informed about the effects of family violence after the sessions (‘I was surprised as I didn’t know much about the impact of violence on children’).

Participant responses further suggested that insights gained were interpreted on a personal level (‘confident that I can have a positive change on myself and the relationship with my family’) and that meaningful engagement with the content of the program therefore remained possible under remote delivery. Most men could summarise a key lesson from each session, with responses indicating that the online program was successful in conveying the principle of individual accountability, and in motivating clients towards respectful relationships (‘You need to understand the damage you caused to your family members in order to be able to rebuild a strong relationship with them’).

Facilitators

Men’s Behaviour Change Program sessions are led by one female and one male facilitator, to manage group dynamics and model appropriate gendered relationships. It was important to obtain both facilitators’ perspectives to verify the fidelity of the program and the extent to which core program principles were maintained. Following each session, facilitators were invited to complete questions on the delivery of the week’s content and its translation to the videoconferencing platform. A total of 51 responses were received to the co-facilitator survey; 42 from 10 self-identified facilitators (five male, five female) who ran groups between July and December 2020, with a further nine provided anonymously.

Again, a mix of quantitative and qualitative items was used in the co-facilitator survey. Facilitators were asked to assess how well each session succeeded in meeting weekly program objectives on a scale of 1 (‘not at all’) to 10 (‘completely’), and to rate on a five-point scale (‘strongly disagree’ to ‘strongly agree’) how well they felt able to achieve objectives for the session. Qualitative responses were sought about challenges faced in delivering each session and suggested adaptations to subsequent online sessions.

Data from the facilitator survey indicated that facilitators encountered a steep learning curve at the outset of online delivery. A strong theme in early qualitative feedback was the development of and improvement in the delivery of the online program (‘It is evolving and getting better as it goes’). A large part of this process was the facilitators gaining confidence in their use of the online platform and delivering the program in a new and different way (‘Each week my confidence is growing and delivery is becoming more comfortable’).

Once facilitators gained experience (and adaptations were made; see below), most were positive about their capacity to engage participants online (‘Really enjoying the way the men engage in this group…. finding this a great way to deliver a service’), although one observed the additional work required to maintain engagement (‘The energy of being in the physical space is not there and it required extra effort for the facilitators to keep energy up’). Facilitators further indicated confidence that online Men’s Behaviour Change Program ‘worked well’ in terms of participation and effectiveness.

Facilitators gave online sessions an average rating of 8.2 on the ten-point scale over the duration of the survey and agreed in the great majority of responses that they felt able to achieve the objectives for the session, address accountability, manage risk and safety, and engage participants. All facilitators agreed they felt able to deliver the content as intended using the online format (see Figure 1).

Figure 1.

Facilitator Survey Responses (n = 51): ‘Thinking about this week’s group session, do you feel you were able to…?’.

External observers

No to Violence, Australia’s peak body for organisations and individuals working with men to end family violence, was consulted throughout the duration of the pilot to establish the integrity of the online Men’s Behaviour Change Program, and supplied an external observer to attend each session and provide feedback and recommendations. Observers were invited to respond to each session using a survey as well as in debrief sessions with facilitators. The external observer survey was anonymous to encourage frank feedback. A total of 27 responses were recorded from across the seven online groups.

No to Violence observers provided detailed, balanced and neutral advice drawing on their experience in the men’s behaviour change program field. Over the survey period, a great majority (96% in each case) agreed or strongly agreed that facilitators effectively delivered the Program content online and effectively engaged participants (Figure 2). Similarly, 93% agreed or strongly agreed that the session they attended was suitable in this format. In each case, the small remainder were ‘undecided’.

Figure 2.

External observer survey responses (n = 27).

Observers’ qualitative survey responses reflected a process of addressing initial challenges presented by moving the Men’s Behaviour Change Program online, which they ultimately considered an effective delivery method. Their comments supported facilitators’ reflections to the effect that facilitators became more comfortable and adept at online delivery with experience (‘done more seamlessly this week … facilitators are becoming more comfortable with using the functions of [videoconferencing software]’). Positive feedback from observers indicated that many aspects of the program were successfully adapted to online delivery, such that desired program objectives were achieved:

Tonight's session demonstrated a few strengths and reinforces the good potential of online groups. For example, it was demonstrated that clients can be guided to showing and expressing empathy…. This achievement suggests that more practical activities should be explored to understand their effectiveness in an online space.

The men were demonstrating emotions that were similar to a group closing in face-to-face. This observation suggests that the emotional transference that occurs in face-to-face is also able to occur in an online space.

One observer felt that online delivery offered unexpected benefits:

By having everyone's faces up close, facilitators are better able to monitor participant's reactions to content and other participants.… monitoring how men are travelling is easier, including if they are confused about the learning and seeming engaged or not.

Overall, observers felt that group dynamics were maintained well in the online environment:

The group is quite collaborative and supportive … [men] were happy to be in the online space.

Family safety contact

Delivery of the Men’s Behaviour Change Program includes access to a Family Safety Contact service for (ex-)partners or family members of participants. The Family Safety Contact service seeks to support victim-survivors and establish safety. As such, it was important to seek the perspectives of victim-survivors during the transition to online delivery of the Men’s Behaviour Change Program on the efficacy of this method. During the initial pilot, four (ex-)partners engaged with the Family Safety Contact service, and each was invited to complete a brief survey. However, only one response was received. Given the added complexities of the pandemic, and the sensitivities of collecting data from victim-survivors, this component of the evaluation was discontinued for the remainder of 2020 as Family Safety Contact practitioners focused on building rapport and assessing risk.

Challenges experienced during online delivery

The majority of Men’s Behaviour Change Program responses (n = 78 or 79%) indicated no significant problems participating in the online program. The remainder indicated some anxieties around the use of videoconferencing in anticipation (‘I was worried my… app won’t work’) and some experiences of technology not working well (‘Camera on laptop wouldn’t work’, ‘Connection issue’). Facilitators noted similar challenges. In general, however, technical problems were minor or easily addressed; participants’ and facilitators’ confidence in using equipment and software was the greater challenge to begin with but improved over time.

Some participants found it challenging to participate from home with family around during COVID-19 restrictions (‘Finding quiet space’; ‘I have to do the course from the neighbouring room to my partner’). Facilitators identified additional challenges relating to online delivery, including managing the flow of discussion without the same ability to read participants’ body language, and needing to review learning tools (such as the display of previous work) that did not translate well to the online platform.

Facilitators reported being most conscious of challenges in supervising Men’s Behaviour Change Program participants, in accordance with the principle of keeping men in view. Similarly, the problem of men attempting to participate whilst engaging in other activities on the same screen or on another device was raised several times by No to Violence observers during the pilot. One observer linked this to the inevitable delays in getting the technology ‘right’ for all participants in each session:

Whenever the group discussion pauses for technology issues, some members take that opportunity to do something else, e.g., open a browser, check the phones.

This issue led to a dissenting observer view on the effect of online delivery on the group dynamic (‘Group dynamics are hindered in an online space due to unique temptations that are not available in face-to-face, e.g., being on phones hidden from view, browsing multiple screens’).

Adaptations and implications for online men’s behaviour change program development

Numerous practice recommendations resulted from the collaborative learning process that underpinned the developmental evaluation during the online Men’s Behaviour Change Program pilot and subsequent online programs offered in 2020.

In keeping with the developmental evaluation principles, recommendations were discussed with team members following weekly sessions and adaptations implemented concurrently with the evaluation as online Men’s Behaviour Change Program delivery continued in 2020. Consequently, facilitators made numerous changes to the delivery of the program to suit the online platform and to address associated challenges. Ongoing data collection ensured that changes were effective. Key recommendations and resulting program adaptations are discussed below and are also set out in Appendix 1 according to the challenge addressed by each.

Mitigating risk

In response to the risk of family members’ exposure to sessions under remote delivery, particularly under lockdown conditions, participant information and consent forms were updated for online delivery. These required that partners or family members living in the same home must consent to men’s participation in online sessions, and specified that children cannot be in participants’ care during sessions.

Mitigating time pressures

Due to the time-consuming nature of establishing and adjusting each participant’s technical setup in each session, combined with reduced opportunity for informal interaction around the formal content, both participants and facilitators noted a need for smaller classes and/or more time for online sessions. This was an early recommendation from the pilot program. Consequently, content was condensed, and sessions lengthened from 60 to 90 minutes, while group size was reduced from 15 to 8–12 participants for online groups, also improving individual engagement (see below).

Telephone check-in was conducted with participants individually before each session, allowing ‘more time in the session for group discussion and content delivery’, according to a facilitator. Men were asked to complete some activities prior to the session; similarly, the evaluation survey, initially completed in-session, was emailed to participants to release class time.

Mitigating technological challenges

Recommendations and resulting adaptations included testing each participant’s equipment and advising on setup (lighting and positioning) during check-in, to avoid interruptions due to discussing or correcting setup during sessions; and having a printed sheet with participants’ contact details to assist in reconnecting if technology failed.

Enhancing participant engagement

Based on learnings from the pilot, facilitators shifted from being together in a room to each being on a separate screen, which improved participants’ ability to see their faces, and vice versa. An observer noted that ‘This separation meant that facilitators did not need to wear masks and were clearly visible (headshot) like all the other participants. This change appeared to benefit engagement greatly’.

Modifications were made to content in response to feedback from participants, facilitators and observers about which activities translated well to the online platform and which did not. Role plays were reduced while self-completion activities, interactive whiteboard components and video clips were included or increased. Instructions for activities were shared on screen to avoid repetition in the group, to save time and avoid losing participants’ attention in these moments.

Following discussion with stakeholders, facilitators trialled and documented the use of online ‘breakout rooms’ ‘to effectively manage group participation’ and ‘discuss deeper the issues’; subsequently, these were judged ‘an effective setting for enhancing the main group content’. Observer feedback provided support for the use of platform features to assist online delivery (‘the breakout room did provide for an experience that was not too dissimilar to face-to-face’).

Facilitating supervision

Supervision challenges identified during the evaluation resulted in the modification of guidelines to require upper body visibility (hands and torso) to deter men from engaging in other activities. Participant agreements were updated to prohibit the use of other devices or browser windows during sessions, to reduce distractions and maintain engagement.

Limitations

Surveys offered to online groups in 2020 as part of the developmental evaluation were designed specifically to inform the transition of the Men’s Behaviour Change Program to videoconferencing technology. Prior evaluation surveys with face-to-face groups concerned program outcomes, and the data are not comparable. The lack of material comparing client outcomes from tele-practice with outcomes from face-to-face delivery is noted in the literature (Joshi et al., 2021), as is the lack of evaluation based on program logics (O’Connor et al., 2020). Future evaluation instruments will ask the same questions of Men’s Behaviour Change Program participants, (ex-)partners and practitioners experiencing both modes of delivery, to facilitate direct comparison of outcomes from online and face-to-face groups, and so that both methods can be assessed against a common program logic. Greater numbers progressing through online groups will also allow the calculation and comparison of valid completion rates.

The literature identifies a lack of perspectives of victim and survivor (ex-)partners of Men’s Behaviour Change Program participants (Bell & Coates, 2022). Since this evaluation, a routine electronic survey for participants engaged with the Family Safety Contact service has been successful in collecting data from Family Safety Contact clients. This information allows the organisation to assess (ex-)partners’ and children’s wellbeing and feelings of safety, and will inform future and ongoing Men’s Behaviour Change Program evaluations.

Conclusion

Developmental evaluation was well suited to the rapid adaptation of the Men’s Behaviour Change Program to online delivery, given the strong need for programs to continue under the circumstances of the pandemic. The evaluation of the pilot documented a steep initial learning curve, which helped facilitators and observers identify and implement practices to support effective online delivery of the Program. As online sessions continued throughout 2020, the evaluation informed ongoing adaptations that ultimately resulted in the successful delivery of the Men’s Behaviour Change Program using videoconferencing.

Overall, evaluation data collected concurrently with the transition to online delivery suggest that key aims and objectives of the Men’s Behaviour Change Program as informed by Family Safety Victoria and No to Violence were achieved, despite the rapid shift necessitated by the pandemic. This was evident in responses of participants, external observers and facilitators, which indicated that men participating online expressed accountability for their behaviour, were challenged in their beliefs and were motivated towards change. While the pilot group began face-to-face and had already established some level of engagement, data from subsequent groups suggest that following adaptations, a positive group dynamic remained possible even when the program was delivered online.

This article contributes to the literature about the efficacy of men’s behaviour change programs, but also to the emerging literature on tele-practice in family and relationship services (Joshi et al., 2021) and the adaptations required to support it. The evaluation supports the emerging view that with additional safety measures and strategies to mitigate the challenges of online delivery, best practice can be maintained, and videoconferencing can be an appropriate method of delivery for men’s behaviour change programs (No to Violence, 2022). Finally, as social services evolve to address complex social issues, this article demonstrates the utility of a developmental evaluation approach to support rapid program adaptation in a dynamic environment.