Abstract
Evidenced-based psychotherapies (EBPs) are primarily disseminated via in-person workshops, which have been shown to be effective in training clinicians. However, there is limited space at these workshops, they occur in limited locations, and the time and money required to travel to these workshops is limiting for many providers. An online virtual and interactive training is a potential solution to increase EBP training opportunities for providers. The present study was an initial step to address this limitation in the current research. In-person workshops were compared with live 3-D workshops conducted in the online virtual and interactive platform, Second Life. Analysis of the data demonstrated that the 706 participants, regardless of training modality, demonstrated significant knowledge gains and that the magnitude of these gains did not differ across the in-person and virtual online training modalities. Participants in the online and in-person modality also did not differ in their perceived readiness to implement the EBP after the workshop. A significant difference in training satisfaction was found between the two modalities, with the in-person training group reporting higher satisfaction ratings than the online training participants. Overall, these findings suggest that virtual online EBP training is at least as effective as in-person training.
There is a limited but growing body of research investigating best practices for training behavioral health clinicians in the use of evidence-based psychotherapies (EBPs). Traditionally, the dissemination of treatment manuals was the primary means of training, and clinicians relied primarily on self-study in preparing themselves to use EBPs. However, an extensive review of the literature suggests that this method of distributing information alone is not an effective way to train clinicians to deliver the treatments efficaciously (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005). An alternative approach is to combine the dissemination of treatment manuals with in-person training by experts in the various treatments. This type of in-person training is now the most widely used method to prepare behavioral health clinicians to administer various evidence-based treatments (Dimeff et al., 2009).
Given the frequency of its use, it is not surprising that in-person training methods have been the primary focus of research to assess the effectiveness of EBP dissemination. Beidas and Kendall (2010) conducted a critical review of 32 studies available on the effectiveness of in-person training and reported that, across studies, individuals trained in-person in an EBP increased their perceived and declarative knowledge and improved their attitude toward the EBP. The study also found that, if the training included active learning opportunities (e.g., role-playing) rather than relying exclusively on passive learning techniques (e.g., lectures), the therapist was more likely to adhere to the EBP, and have more perceived competence and skill following the training (Beidas & Kendall, 2010).
Despite increased efforts to disseminate EBPs, these treatments are still not widely implemented by clinicians with their patients. Surveys of clinicians suggest a number of barriers to EBP implementation. Notably, one common reason given by VA clinicians for not using an EBP is that clinicians have not received proper training or have not had the time to attend an EBP training workshop (Barnett et al., 2014). In a sample of 207 licensed psychologists surveyed on their training and use of cognitive processing therapy (CPT) or prolonged exposure therapy (PE) for treating posttraumatic stress disorder (PTSD), only 31% reported being formally trained, and only 9% reported using CPT or PE with the majority of their PTSD patients (Becker, Zayfert, & Anderson, 2004). Participants indicated that the usage rates were the result of a combination of a lack of training and misconceptions about the therapies (Becker, Zayfert, & Anderson, 2004). Other studies suggest that the lack of trained clinicians results in large part due to the shortage of available training opportunities for clinicians (e.g., Weissman et al., 2006).
The lack of trained clinicians in EBPs and the lack of accessible training continues to be a limiting factor in the ability to ensure the delivery of effective treatments to patients in need. A number of logistic considerations limit the number of providers who can be trained through the preferred in-person training method. The number of workshops that can be offered is limited by the availability of qualified trainers, their ability and willingness to travel to different training sites, and the availability of such sites with the requisite support. The number of individual providers who can be trained in an in-person event is limited by the size and accessibility of the venue, the ability and willingness of providers to travel to the venue, and limits on the size of a group that can be adequately trained by the trainer(s). Within the Department of Defense (DoD) and Department of Veterans Affairs (VA) for example, travel restrictions placed on DoD and VA personnel have significantly limited the number of clinicians able to be trained, despite the recognition that EBP training is needed within these populations. Recognizing that the use of in-person workshops limits the ability to disseminate EBP training, some researchers have begun to examine the use of online training modalities to determine if training delivered online can adequately address some of the limitations of in-person training. In general, online training is less expensive and enables wider dissemination of information than in-person training (Harned et al., 2014; Weingardt, 2004).
In assessing the potential use of online platforms for training clinicians in EBPs, the effectiveness of these training options needs to be examined. This is a challenge given the many variations in online training platforms. For example, online training can be delivered using two-dimensional (2-D) or three-dimensional (3-D) platforms. Examples of 2-D training platforms include webinars and similar audio/video presentations. In comparison, 3-D platforms are typically interactive virtual worlds where trainees create virtual representations of themselves and interact with a virtual classroom or another learning environment. Most existing research has used 2-D platforms, but there is some indication that 3-D formats offer increased opportunity for experiential learning, participant engagement, and more effective collaborative learning (Dalgarno & Lee, 2010). To date, no research has examined how these different online platforms might impact the effectiveness of EBP training.
In addition, online training can be delivered using synchronous (e.g., virtual classrooms, teleconferencing) or asynchronous (e.g., self-paced web-based training) approaches. To date, research into the online dissemination of EBPs has focused exclusively on asynchronous training modalities (Harned, Dimeff, Woodcock, & Skutch, 2011; Harned et al., 2014; Weingardt, 2004) and has not examined the potential use of synchronous training approaches. The few studies comparing the effectiveness of in-person and asynchronous online training for EBP treatment protocols suggest potential for this training approach. For example, Beidas, Edmunds, Marcus, & Kendall (2012) taught cognitive behavioral therapy (CBT) for anxiety in youth to clinicians using routine in-person training (i.e., didactic instruction, PowerPoint presentations, videos), in-person training that included role-plays and small-group discussion, and an online asynchronous presentation of the routine in-person training materials. Results indicated that each training approach produced small gains in therapist adherence to CBT, CBT skills, and knowledge. However, participants were more satisfied with the two versions of in-person training than the online training. A second study by Dimeff and colleagues (2015) examined differences in the outcomes of clinicians trained in dialectical behavior therapy (DBT) via a treatment manual, an in-person training, and an asynchronous online training that included online practice sessions that were interactive. They reported that clinicians trained in-person demonstrated larger gains in self-efficacy as well as greater increases in motivation to use and satisfaction with the treatment protocol. In contrast, providers trained in the online condition showed greater knowledge gains. These initial findings suggest that asynchronous online training platforms are effective at disseminating information about EBPs but may differ from in-person training in other indices of training and performance.
Synchronous online training may offer even greater opportunities for learning than the asynchronous options applied in the cited studies. For example, it has been noted that the online synchronous environment offers the potential for the real-time asking of questions and exchange of information (e.g., through chat rooms, or voice over IP protocols) that may lead to additional, or different, learning than is possible during asynchronous and even in-person workshops (Weingardt, 2004). As previously discussed, though, there is no published research directly comparing synchronous online training and in-person training for EBPs, so the effectiveness of synchronous training modality has not yet been established (Beidas et al., 2012; Dimeff et al., 2015; Harned et al., 2011; Karlin et al., 2010; McDonough & Marks, 2002; Sholomskas et al., 2005; Weingardt, Cucciare, Bellotti, & Lai, 2009).
The present study represents an initial examination of the effectiveness of an online 3-D synchronous platform for training in EBPs. Program evaluation data was used to compare the effectiveness of this online training to traditional in-person training of various EBPs. It was expected that participants in both the online and in-person workshops would demonstrate increased knowledge about and perceived readiness to use the EBP. However, it was predicted that participants who attend the in-person workshops would report greater perceived readiness to use the EBP protocol, whereas the participants in the online workshop would demonstrate greater knowledge gain. It was further hypothesized that the participant’s satisfaction with the training would be greater for the in-person than the online training.
Methods
Participants
All 706 participants who completed pre-training surveys and 780 who completed post-training surveys were behavioral health providers seeking training in EBPs for the treatment of PTSD, depression, insomnia, or the prevention of suicide. Participants voluntarily registered for and attended either an online or in-person EBP training conducted by the Center for Deployment Psychology (CDP) between May 2014 and December 2015. All participation in the pre-training and post-training program evaluation assessment was voluntary and anonymous. EBP training workshops presented by the CDP are advertised in a variety of ways, including, but not limited to, the posting of information about upcoming training on the CDP website (deploymentpsych.org), and through information disseminated via email and listservs. Participants were not compensated for their participation; they were afforded the opportunity to gain continuing education credits (depending on their discipline) following attendance.
The dataset analyzed for the study consists of data collected during standard program evaluation (quality assurance) activities at CDP. All participants were informed that any information they provided on pre-training and post-training program evaluation assessments would be used to assess the effectiveness of the current training and to enhance the effectiveness of future CDP training. No personally identifying data were collected from participants during the program evaluation process, and all survey participation was voluntary and anonymous. The number of respondents varied across training workshops ranged from 6 to 37 participants.
Procedure
Between May 2014 and December 2015, CDP collected standard program evaluation data for 31 in-person EBP workshops and eight online EBP workshops in Second Life (a 3D virtual world that uses avatars to allow people to interact with one another and the virtual environment). Participants in both training modalities participated in didactic presentations, viewed video-recorded examples of cases, engaged in small group discussions, and completed role-plays to practice techniques. The topics of the EBPs included Cognitive Processing Therapy (CPT), Prolonged Exposure Therapy (PE), Cognitive Behavioral Therapy for Insomnia (CBT-I), Cognitive Behavior Therapy for Depression (CBT-D), Cognitive Behavior Therapy for Chronic Pain (CBNT-P) and Evidence-Based Suicide Risk and Reduction. Trainers from the CDP conducted all workshops in both modalities. In-person workshops were conducted by an individual trainer or teams of two trainers with logistical support from personnel at the CDP. All online workshops were conducted by teams of two trainers with additional technical support from personnel at the CDP.
Measures
All workshop participants were provided the opportunity to complete a pre-training and a post-training survey. Pre-training surveys consisted of a 15 or 17-item (depending on particular EBP) multiple-choice knowledge assessment measure and took an estimated 10–15 minutes to complete. Post-training surveys included the same knowledge assessment items as well as items to measure participants’ self-reported readiness to use the EBP and general satisfaction with the training experience. Post-training surveys included additional items on topics such as technical difficulties, presenter ratings, and general comments, but these data were not analyzed for the present study. Post-training surveys took participants 20–30 minutes to complete. All program evaluation surveys were designed and developed internally by CDP faculty to align with the objectives and content of the EBP training and were distributed and completed using the online data collection platform, SurveyMonkey.
Prior to and following each workshop, participants completed knowledge assessments comprised of 15 multiple-choice items with five answer options, only one of which was correct. Separate assessments were developed for each EBP and items were designed to assess knowledge of the content presented during the 2-day EBP training. Knowledge assessments were unique to each topic being covered. These topics included CBT-D, CBT-I, CPT, PE, suicide reduction, and CBNT-P. Internal consistency was evaluated for each of these knowledge tests separately. These results indicated that the tests for CBT-D (Cronbach’s alpha = .72), CBT-I (Cronbach’s alpha = .70), CPT (Cronbach’s alpha = .87), PE (Cronbach’s alpha = .76), and the suicide training (Cronbach’s alpha = .73) each had acceptable levels of internal consistency. The CBNT-P knowledge test had lower internal consistency (Cronbach’s alpha = .46). This is likely due, in part, to the small number of surveys included in the analysis because there were only two CBNT-P workshops completed during the time in question.
Participants rated their perceived readiness to use the EBP with their patients after attending the EBP training using a 4-point scale (1 = Not at all ready, 2 = Somewhat ready, 3 = Mostly ready, 4 = Very ready). Participants also rated their overall level of satisfaction with each training using a 4-point scale (1 = Not at all satisfied, 2 = Somewhat satisfied, 3 = Mostly satisfied, 4 = Very satisfied). The Likert scales used to assess readiness and satisfaction were 4-point (even-numbered) to avoid neutral, mid-point responses. These data were collected to assess the effectiveness of these EBP trainings and to inform decisions regarding program updates and process improvement. As a result, it was preferable for participants to identify whether their experiences and responses were more positive or negative, rather than neutral. Reliability was not evaluated for these single item assessments.
Data Analysis
Statistical analyses were conducted using SPSS 22.0. Because all participation in survey completion was anonymous, it was not possible to match individual participant’s responses on the pre-training and post-training assessments. As a result, all statistical tests treated the data as independent samples.
The data were screened for non-normality by applying the Kolmogorov-Smirnov test. Based upon a finding of non-normal distributions and the use of four-point Likert scales for the measure of the readiness and satisfaction variables, these variables were analyzed using non-parametric statistical tests, specifically Mann-Whitney-U tests. A graphical examination of the knowledge data showed that it approximated a normal distribution even though it failed the Kolmogorov-Smirnov test. Given the relatively normal data distribution for this variable and the scale used to assess knowledge gain (percent correct), it was determined that this variable was better suited to parametric analysis. As a result, changes in knowledge were examined using a 2 (pre-workshop vs. post-workshop) X 2 (in-person vs. online training) between-subjects ANOVA.
Results
A total of 706 individuals completed pre-training surveys and 780 individuals completed post-training surveys across both the in-person and online training platforms. Of those who provided pre-training data, 141 participated in an online workshop, and 565 in an in-person workshop. Of the individuals who completed post-training data, 181 participated in an online workshop, and 599 participated in an in-person workshop. Not all participants completed all parts of the survey though, so there is some variation in the number of participants for each question, which is represented in each table. Data on participants’ profession were collected from 705 participants (the question was not included in the program evaluation for two of the earliest in-person workshops). Participants were a mix of Department of Defense (military, civilian, contractor) and civilian (Veterans Health Administration, community) providers. Of those participants reporting a profession, 317 (45%) self-identified as social workers, 250 (35%) as psychologists, 57 (8%) as licensed professional counselors, 29 (4%) as nurses, 13 (2%) as marriage and family therapists, 30 participants (4%) reported that they were in some form of training status (such as student, intern, or post-doc), and 9 participants (1%) indicated another profession (e.g., chaplain, mental health technician).
Pre- and Post-training Knowledge Scores.
Note. Pre-training and Post-training knowledge scores (expressed as percent correct) for all participants and broken out by training modality, in-person or online.
Readiness to Use EBP.
Note. Percent of responses for the item assessing readiness to use the EBP protocol with patients broken out by training modality, in-person or online. Frequency is expressed as percent of the sample in that modality.
Training Satisfaction.
Note. Percent of responses for the item assessing training satisfaction broken out by training modality, in-person or online. Frequency is expressed as percent of the sample in that modality.
Discussion
The results of the present study offer initial support for the use of synchronous online training workshops in the dissemination of evidence-based psychotherapies. Program evaluation data from 39 EBP workshops indicated that participants in online and in-person workshops showed similar, and significant, knowledge gains. This finding is consistent with multiple studies of asynchronous online training models that have reported overall increases in knowledge, with no differences across modality (Beidas et al., 2012; Harned et al., 2011, Harned et al., 2014).
Similarly, there were no significant differences between the online and in-person training participants in their ratings of self-reported readiness to deliver the EBP treatment protocol. Although there is little in the literature comparing different training modalities for this specific variable, this finding does seem to be somewhat inconsistent with previous reports that in-person training participants reported greater gains in self-efficacy, motivation to use and satisfaction with an EBP treatment protocol (Dimeff et al., 2015) than did participants who completed an asynchronous online learning experience. The synchronous online training workshops included in the present study allowed attendees to interact with other learners, get answers to questions in real time, and perhaps most importantly, complete role-playing exercises similar to those completed by attendees at the in-person workshops, and potentially different from experiences in an asynchronous training. Research directly comparing synchronous and asynchronous online training modalities is needed to understand the relative strengths and weaknesses of the two.
The present study did find significant differences between the online and in-person attendees for training satisfaction. Participants who attended in-person workshops indicated greater satisfaction with their training experience than those who attended the online workshops. The higher levels of satisfaction with in-person training observed in the present study are consistent with findings from earlier research comparing in-person workshops with asynchronous online learning tools (Beidas et al., 2012; Dimeff et al., 2015). It should be noted, however, that despite the difference observed between the two groups in the present study, more than 90% of the participants in the synchronous online workshops indicated that they were mostly or very satisfied with the workshops. Additional research is needed to understand what factors might contribute to the observed differences in satisfaction. It is very likely that the differences reflect differences in what the attendees were “used to” or expecting from a continuing education workshop. As noted above, the vast majority of such workshops are conducted in person. When online modalities are available, they are typically asynchronous learning tools or 2-D presentations (e.g., webinars, live-streamed lectures). Engaging and navigating in the virtual environment is a new experience for most workshop participants. In addition, there may be other issues that contribute to the online learning experience being less satisfactory for the learners. The potential for technical difficulties that interfere with the learning experience is present as are possible differences in the participants themselves or in the learning environments involved. Future studies will explore these and other possibilities.
Overall, these findings suggest that, like asynchronous online training, synchronous online training platforms such as Second Life are as effective as the traditional in-person training method for disseminating information about EBPs. This finding is important because if more research continues to support it than it could lead to wider and less expensive dissemination of EBPs (Harned et al., 2014; Weingardt, 2004). The results of the present study, when compared to previous research, offer a hint that synchronous training workshops may offer an approach that leads to more participant self-efficacy than asynchronous online training. However, these findings must be seen as preliminary due to methodological limitations of the present study.
The present study utilized available program evaluation data from established workshop programs. The data were collected anonymously, and therefore it was not possible to directly link an individual’s pre-workshop responses with that same individual’s post-workshop responses. This primarily affected the knowledge data and, as a result, a more conservative statistical approach, treating and analyzing the data as a between-subjects research design, was utilized. It is possible that any differences between the groups may have been more easily identified if data were matched. However, the finding of no significant differences between in-person and online training participants for knowledge gain is consistent with the outcomes of multiple similar studies (Beidas et al., 2012; Harned et al., 2011, Harned et al., 2014), lending support to the validity of this finding. The present study relied on self-reported preparedness and satisfaction ratings and on a multiple-choice assessment of knowledge gain. The lack of objective measures of the application and effective use of the information and skills covered by the EBP workshops limit the present findings to conclusions regarding dissemination of information about these protocols. Additional research examining the ability for providers to acquire skills to deliver the EBP protocols is needed.
The present study did not randomly assign participants to the in-person or online workshops. It is possible that participants selected their workshop modality (online or in-person) based upon individual preferences, differences in experience or comfort with technology, or issues with access to training locations or computer hardware/software. These differences or limitations in access may have affected certain aspects of their training experience, including variables assessed in this study such as training satisfaction. Random assignment to training modality would potentially control for any differences across the participant groups and may help us to better understand the relative strengths and weaknesses of the different approaches. On the other hand, it may be that allowing providers to select the modality that they prefer and that they think would work best for them might produce the best outcome.
Despite these limitations, the present study provides the initial evidence that online synchronous training of EBPs is possible and effective. Synchronous online platforms could contribute to the dissemination of such treatments and address some of the many barriers preventing providers from receiving beneficial training in EBPs, particularly across national or regional healthcare delivery systems. The findings that online attendees gained as much knowledge about the EBPs and identified themselves as equally ready to use the EBPs with patients as did those attending in-person workshops are promising. Further research is needed to assess the ability of providers to translate this knowledge gain and self-reported readiness into effective delivery of the clinical interventions. These promising results, combined with the greater efficiency of training delivered over the internet, particularly when the target audience is geographically dispersed, support continued exploration of synchronous online learning opportunities for training in psychotherapy protocols. Clearly, such ideas are speculative at this point and it is vital that the present findings be replicated and extended to better understand the potential role of online synchronous training in the broader efforts to disseminate evidence-based treatments.
Footnotes
Author Note
The opinions expressed in this manuscript are those of the authors and do not necessarily represent the opinions of the Uniformed Services University of the Health Sciences, the Department of Defense, or the United States Government.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) received no financial support for the research, authorship, and/or publication of this article.