Abstract
Background
This study evaluates the effectiveness of the IMPACT WV training portal as a resource for service providers and families affected by substance use in rural West Virginia.
Objective
To evaluate perceived satisfaction and knowledge impact of service provider modules such as stigma and mental health resources in rural communities.
Methods
Service providers were recruited from health centers, substance use treatment facilities, and schools across rural north central West Virginia. A total of 416 individuals registered for training modules on substance use and family support. Engagement and course completion rates were monitored, along with participant feedback via evaluations of course content and satisfaction ratings. Pre- and posttest assessments measured knowledge gains and skill improvements, utilizing descriptive statistics and nonparametric correlation tests for analysis.
Results
Completion rates varied, with the stigma-free services module at 73% and the children with special health care needs module at 37.8%. Overall evaluations indicated positive feedback, although some modules received lower satisfaction scores. Pre- and post-assessment results showed significant knowledge improvements in key areas, including neonatal abstinence syndrome and substance use.
Conclusion
This study underscores the commitment of rural service providers to enhance their competencies through the IMPACT WV training portal. It highlights the need for adaptive educational strategies to align with participant feedback, ultimately aiming to improve service provider knowledge, service delivery, and outcomes for families facing substance use disorders in rural communities.
Keywords
Introduction
Substance use continues to pose a significant challenge for families and communities, particularly in rural areas, where access to resources and support often falls short.1–5 Rural families frequently face unique barriers that exacerbate the impacts of substance use, including limited availability of mental health services, social stigma, and a lack of awareness about effective interventions.6,7 The interplay of geographic isolation, economic hardship, and cultural factors further complicates the situation, as families may struggle to find the necessary support to address substance use issues. Consequently, effective strategies for prevention and intervention are vital to promoting healthier families and communities in these underserved regions.8,9
Service providers in rural areas encounter their own set of challenges when addressing substance use among families.8–10 Many providers lack access to comprehensive training programs that encompass current research, best practices, and culturally sensitive approaches tailored to rural populations.11,12 Furthermore, the high turnover rates and burnout among providers in these regions prevent the establishment of cohesive support systems.10,13 As a result, there remains a pressing need for training opportunities that not only enhance the skills of service providers but also align with the unique circumstances and needs of rural families experiencing substance use challenges.14,15
There is a critical need for training materials, especially in the context of the ongoing opioid pandemic. Many providers are thrust into roles where they must navigate complex challenges related to substance use, often without adequate preparation or support. These materials can provide essential refreshers on substance use knowledge but also practice-based elements and a range of relevant topics, including communication strategies, coping mechanisms, and resources available for supporting loved ones struggling with substance use. 14 By equipping providers with up-to-date knowledge and practical skills, we can enhance their capacity to provide effective support, encouraging resilience and fostering a healthier environment for both caregivers and those they are assisting.
This study aimed to evaluate the development, engagement, and preliminary outcomes of the IMPACT WV training portal designed to support rural service providers and families affected by substance use, operationalizing its aims using the Reach, Effectiveness, Adoption, Implementation, and Maintenance (RE-AIM) framework. 16 The Reach component assessed the extent to which the portal engaged its intended audiences, examining registrant demographics, representation from rural and underserved areas, and awareness pathways, with key questions focusing on participation levels among target groups. Effectiveness evaluated whether the portal improved participants’ knowledge and perceived satisfaction using pre/post assessments, satisfaction surveys, and qualitative feedback to measure learning gains. Adoption examined how individuals used multiple levels of each module (where applicable). Implementation described the development and deployment process, including content design, stakeholder involvement, delivery format, platform functionality, and engagement metrics, to understand how the portal was experienced by users and developers. Lastly, Maintenance explored the sustainability and ongoing use of the portal beyond initial implementation, focusing on plans for updates and strategies for long-term maintenance, with the overarching goal of ensuring sustained impact on rural provider and family education.
Methods
Subject recruitment and eligibility
Participants for the IMPACT WV training portal were recruited through a combination of community outreach and strategic partnerships aimed at engaging service providers in north central counties in rural West Virginia (WV). These counties were specifically selected as part of a larger research program attempting to provide multiple services to families experiencing substance use with infants. Outreach efforts involved collaborating with local organizations, such as community health centers, substance use treatment facilities, schools, and social services to disseminate information about the training portal. Additionally, recruitment leveraged social media platforms, community newsletters, and targeted advertisements to reach a wider audience of potential participants. Information sessions and workshops organized in various community settings also provided an overview of the training portal and encourage enrollment. Individuals were eligible to register, and complete any online provider training modules in the portal if they provided services of any type (e.g. medical, behavioral health, community-based) to families experiencing substance use with young children. Individuals providing services in WV were included in this study specifically.
IMPACT WV overview
The IMPACT WV program is an initiative focused on supporting women and infants affected by substance use (https://impact.cedwvu.org). Designed to address the multifaceted challenges posed by substance use disorder, the program provides comprehensive care and resources for pregnant and postpartum women, ensuring both their health and the healthy development of their infants. Through a holistic approach, IMPACT WV offers services that include prenatal and postpartum care, substance use counseling, mental health support, and access to community resources. By fostering a supportive environment for women and their families, the program aims to reduce stigma around substance use, facilitate recovery, and promote healthier outcomes for both women and their children.
Training course development
Modules found within the training portal were based on feedback provided by providers in focus groups, regional meetings, and responses to online surveys. Eighty-nine providers requested select modules from the portal. Initial feedback recommended the following topics: behavioral management, child development, educational opportunities (postsecondary training for caregiver), employment, family strengthening, family support, housing, medical services, mental health services, social/parent support, stigma, and substance use.
Plan Do Study Act (PDSA) cycles were completed for quality improvement (QI) to identify remaining training gaps.17,18 Information was managed by the Learndash learning system, the domain was hosted by Nikihost with the name wvimpact.org, Cosmolabs Wordpress plug-in allowed registrations, profiles, and login form, and Qualtrics housed the evaluations. Continuing education units were approved and offered for social workers, WV State Training and Registry System, occupational therapy, physical therapy, and the WV Certification Board for Addiction & Prevention Professionals. IMPACT WV partnered with the West Virginia University Continuing Education Office for Continuing Medical Education (CME) credits for nurses and physicians.
A Clearinghouse Research Development Guideline document was created to guide the module developers. The team considered whether the topic pertained to general public, professionals, or both audiences. Authors were asked to develop slide presentation organizing background information, insert notes that would be captured for production, and sort resources based on supplemental materials, resource/references, or case studies. Modules had three learning levels: Level 101 was used for basic topic resources and progressed to Level 201, and then 301 building specific topic related skills. Course content was then reviewed by subject matter experts at an internal, state, and national level. Once reviewed and updated, voiceovers were added to the PPT slides and then developed into an accessible recording using the YouTube platform.
Additional QI was conducted using a reminder survey and evaluation questions pertaining to incomplete modules. PDSA Cycle 4 again reviewed Continued Education Units (CEU)/CME module completion by steps versus registered. Emails with a QI link was sent to 201 participants in July 2023. The web developer completed web analytics 7.1.21 to 6.30.22.
Training portal courses
The IMPACT WV training portal featured a diverse range of courses tailored to meet the unique needs of service providers working with individuals facing substance use challenges. A total of 14 courses were available, covering various topics essential for enhancing skills and knowledge in this critical field. These courses include foundational subjects such as “understanding substance use disorder,” and “effective communication strategies,” alongside practical workshops focusing on specific skills like “coping mechanisms for family caregivers.” Additionally, advanced courses focused on specialized areas such as “co-occurring disorders” and “cultural competence in service delivery.” Each course was designed to be interactive and engaging, incorporating multimedia resources, case studies, and practical exercises to enhance learning outcomes. The online training portal modules were offered free of charge, ensuring that cost did not pose a barrier to participation for service providers. To access the modules, users were required to complete a simple registration process that involved providing basic demographic information. This straightforward registration helped facilitate, track engagement, and tailor future training content to meet the needs of difference audiences. Sustaining the IMPACT WV training portal beyond its initial launch involved strategies aimed at ensuring long-term usability and ongoing engagement. Key efforts included tracking the number of repeat or returning users over time to assess continued interest and utilization.
Measures
The assessment component of the IMPACT WV portal modules for service providers involved evaluating participants’ knowledge and perceived skills both at baseline and following module completion. This process entailed administering pretests prior to engaging with the content to establish initial levels of understanding, followed by posttests after completing the modules to measure learning gains. All items were tailored to the content of each module in terms of knowledge. These items were developed by the author of the module. This resulted in baseline and post scales with different ranges across modules (see Table 1) but pertinent to within each module to compare baseline to post in this study. The comparisons between these baseline and postassessment scores provided quantitative data on the effectiveness of each module in enhancing knowledge. Demographics were captured during the registration process by each learner. These included age, gender, race, ethnicity, years of education, and county of residence.
Module participation, completion, and pass rates for online portal.
Procedures
The implementation of the IMPACT WV training portal was characterized by its flexible, self-paced, and asynchronous delivery format, which allowed participants to access training modules at their convenience, accommodating diverse schedules and learning preferences. The platform was equipped with interactive features such as progress tracking, assessments, and multimedia content to enhance engagement and facilitate learning.
Participants were granted access to the IMPACT WV training portal through a streamlined registration process designed to be user-friendly and efficient. Upon expressing interest in the materials, individuals were directed to an online registration form hosted on the IMPACT WV website. After completing the form, participants received a confirmation email that included details about their enrollment and instructions for accessing the training portal. A unique login credential was generated automatically for each participant, ensuring secure access to the platform. Additionally, an orientation session was offered via webinar to familiarize participants with the training portal's features, navigation, and available resources. All study procedures were approved by the Institutional Review Board (IRB: 2201501210) at West Virginia University.
Statistical analyses
Training portal registrants and training information from January 2019 to August 2023 was used for this study. Participant information including evaluation of each course and knowledge quiz items, where applicable, was captured within the online portal system. Participants were asked “Overall, I am satisfied with the knowledge and skill gained from the training” and rated on a Likert scale of Strongly Agree to Strongly Disagree. Each module had multiple levels based on knowledge and skill. Average scores at baseline for each level of a given module were calculated. We calculated a similar average for each module level postcompletion. This straightforward registration helped facilitate, track engagement, and tailor future training content to meet the needs of difference audiences.
Participant demographics and responses were cleaned. Participants were excluded for analyses if missing data for relevant items. We initially conducted descriptive statistics for each item and each course including mean responses, standard deviation, and ranges. Operational metrics, including course completion rates, the number of attempts per module, and time spent on each module, were systematically monitored to evaluate usability and identify opportunities for improvement. We examined participant usage of the portal by noting the number of participants who registered for each course as well as the proportion of those registered who completed the materials. Lastly, we used nonparametric correlations to identify factors associated with participant completion and higher scoring in each course. We used an alpha ≤ 0.05 to detect significance. SPSS version 29.0 was used for all statistical analyses in this study.
A priori sample size calculations were performed to ensure adequate power for detecting meaningful associations. Assuming a medium effect size (ρ = 0.30), an alpha of 0.05, and a desired power of 0.80 for nonparametric correlation analyses (Spearman's rho), a minimum of 84 participants per analysis was required. Given that our dataset exceeded this threshold across all courses, the study was adequately powered to detect statistically significant relationships among the variables examined.
Results
Reach: participant characteristics
Four hundred sixteen participants had completed registration for at least one of the online training modules during the study period. More than half (58.4%) of the participants sought continued educational units after completing a training module. In terms of participant education, 6 participants had received an associate's degree, 99 had a bachelor's degree, 65 had received a master's degree and additional licensure, 59 had a doctoral or other professional degree (MD, RN), and the remaining 187 either had obtained a General Education Development/high school diploma or did not provide that information during the registration. Thirty-eight participants self-reported Hispanic or Latino ethnicity backgrounds, 5 participants noted more than one ethnic group identification, and the remaining 373 identified as non-Hispanic. Seventy-seven percent (n = 322) of participants were female, 12.3% (n = 51) were male, 0.02% (n = 7) self-identified as gender variant/nonconforming, 0.01% (n = 4) self-identified as transgender female, 0.01% (n = 3) as transgender male, and the remaining respondents did not wish to provide. Most participants were white (n = 325; 78.1%). Remaining participants identified as American Indian or Alaska Native (n = 5; 1.2%), Asian (n = 2; 0.5%), African American (n = 39; 9.4%), Native Hawaiian (n = 1; 0.2%), or more than one race (n = 11; 2.6%). Ninety-three (22.4%) participants resided in the specific counties eligible for the larger study services. The remaining participants were from other portions of WV (n = 81; 19.5%), outside of WV (n = 9; 0.02%), or noted an international residence (n = 2; 0.01%). The mean age of the participants was 40 years (SD = 12.5) and ranged from 17 to 80 years.
Effectiveness: training portal usage
Fourteen modules were initially developed within the portal. Each module had at least two levels of information with an educational overview of the topic being the first level and building to a skills-based learning opportunity in the second and third (for some topics). Table 1 presents each module, initial participation analytics, and completion rates. All the modules listed in this table were developed for service provider audiences including medical, social services, and home visiting providers.
Several modules were more commonly sought by providers including those related to child development, stigma-free services, and substance use care (see Table 1). The most common modules within the portal were those on topics related to child development and substance use. Module completion rates ranged from 37.8% (children with special health care needs) to 73.0% (stigma). A greater proportion of participants passed the stigma modules; fewer passed the modules pertaining to neonatal abstinence syndrome (NAS). Lastly, a greater number of continued education units were obtained from the adult education and employment, substance use, and child development modules. These were more commonly obtained by home visiting providers.
Adoption: participant satisfaction
Most participants started evaluating the module postcompletion, but a significant portion did not complete all items within the evaluation (see Table 2). Of those completed, most modules received positive evaluations (strongly agree/agree) to being satisfied with the content. Less favored modules included the finance and family strengthening modules.
Participant training evaluations by module area.
Course success
Table 3 provides a cumulative average (average across levels—101, 201, 301—within a given module) each module at baseline and post. Significant comparisons were determined based on p < .05 significance level. Additional information including the number of participants who passed each level of a module is presented as well as the number of attempts they took to pass the level of each module. The lowest average score at baseline was found for the NAS, social services, and parent support modules. The highest average at baseline was for the provider education and employment module. Lowest average scores at baseline were found for the NAS, social service, and parent support modules.
Participant pass rate by module and level.
Knowledge and skill improved between baseline and post average scores for all modules. This difference was significant for the adverse childhood experiences (p < .02), addressing challenging behaviors (p < .02), children with special health care needs (p < .05), family strengthening (p < .05), NAS (p < .001), substance use (p < .01), and social service delivery (p < .001). For most modules, pass percentage rates ranged between 60% and 99%. The pass percentage rate for the substance use module was half of those taking the first level (101) but that increased to 71% and 70.8% for the 201 and 301 levels respectively. Conversely, pass percentage rates for the finance and family strengthening modules were higher in the 101 and 201 levels but dropped in the more complicated 301 levels.
Implementation: delivery format, functionality, and metrics
Table 3 provides course/module completion rates and attempts. While the time to complete each module varied from 30 min to 1.2 h, the average time per module for service providers to complete was 54.2 min.
Maintenance: sustainability and ongoing use of the portal
Plans for regular content updates were established, informed by user feedback and emerging best practices, to keep the training relevant and current. Additionally, technical support mechanisms were put in place to address potential issues promptly, and organizational buy-in was cultivated to promote ongoing use and expansion of the portal's reach.
Discussion
The findings from this study provide important insights into the characteristics, engagement, and educational outcomes of participants in online training modules related to service provision, particularly in the areas of child development, substance use, and stigma-free services. With a sample size of 416 registrants, the demographic profile indicates a predominantly female (77%) and white (78.1%) population, with many pursuing continuing education units after participating in these training modules. This suggests a strong motivation among service providers to enhance their knowledge and skills, which is promising for ongoing professional development in the field. The high levels of motivation align with literature emphasizing the importance of continuing education for improving provider competence and ultimately service quality. 19 Given that these modules were developed to fill training gaps for rural service providers, it appears they effectively addressed those needs, supporting research that highlights the necessity of tailored training to overcome barriers faced by rural providers.
When analyzing the training portal usage, it is notable that participants exhibited variable interest and completion rates across different modules, with particularly high engagement in areas like child development and stigma-free services. This aligns with current trends emphasizing holistic, stigma-reduction approaches to care, which have been shown to improve client engagement and trust, especially in underserved populations. 20 The stigma-free services module achieved the highest completion rate (73%), suggesting providers are especially interested in strategies to combat stigma—a key factor in improving health outcomes and service utilization. 21 Conversely, modules such as “children with special health care needs” had a lower completion rate of only 37.8%, indicating potential barriers to engagement that warrant further investigation. Such barriers may include content relevance, perceived applicability, or difficulty level, echoing findings from research that stresses the importance of contextualized and user-centered curriculum design to enhance learner engagement. 22 In the context of the RE-AIM framework, 23 having demographic data from 322 out of 416 registrants represents a high level of completeness overall.
Participant satisfaction ratings emerged as another critical component of engagement. Most evaluations indicated positive feedback about the knowledge and skills gained, aligning with literature that underscores learner satisfaction as a predictor of training effectiveness. 22 However, the lower satisfaction scores for modules like “finance” and “family strengthening” suggest these topics may benefit from pedagogical re-evaluation or more practical, case-based approaches to increase relevance and applicability. Incorporating systematic feedback loops, as recommended in best practices for instructional design, can lead to continuous improvement of training content and delivery. 23
The significant improvements in knowledge and skills from pre- to posttest assessments across most modules further support the training's effectiveness. Notably, modules on “neonatal abstinence syndrome” and “substance use” showed marked gains, reflecting the critical need for targeted education in these areas, which are often identified as gaps in provider knowledge. The observed decline in pass rates at higher levels of complexity, particularly in “finance” and “family strengthening,” resonates with cognitive load theory, which suggests that increasing material difficulty can impede learning if not appropriately scaffolded. 24 Future iterations should therefore incorporate additional support mechanisms, such as guided practice or mentorship, to facilitate mastery of complex topics.
Overall, this study highlights the dedication of service providers to improve their competencies through targeted training, while emphasizing the importance of adaptive educational strategies that respond to participant demographics and needs. The high interest in certification and continuing education aligns with literature emphasizing the role of ongoing professional development in maintaining high-quality services. 19 Moving forward, iterative enhancements based on participant feedback and learning outcomes will be essential to ensuring these educational initiatives remain relevant and effective across diverse rural settings. Continuous evaluation of participant outcomes and satisfaction will help refine content and delivery, ultimately leading to improved service delivery for vulnerable populations and better health outcomes. 24
Limitations
This study has several limitations that should be considered when interpreting the findings. First, the participant sample was predominantly female (77%) and white (78.1%), which may limit the generalizability of the results to other demographic groups. The overrepresentation of certain demographics could skew the outcomes and perceptions of the training portal's effectiveness. Additionally, participants were recruited from specific community organizations, introducing potential self-selection bias. Individuals who chose to participate may have had different motivations, experiences, or levels of engagement with substance use issues compared to those who did not participate, which could affect the overall evaluation of the training modules.
Moreover, the significant variability in completion rates among different modules—73% for the stigma-free services module compared to 37.8% for the children with special health care needs module—raises questions about engagement and the applicability of certain topics, thereby limiting the generalizability of knowledge gains. A further limitation is the lack of information regarding the number of participants who discontinued participation, the reasons for noncompletion, and the potential similarities or differences between those who engaged and those who did not, which restricts a full understanding of participant retention and engagement patterns. Finally, the study also relied on self-reported measures for assessing knowledge improvement and satisfaction, which can be subject to bias; participants may have overreported their knowledge and satisfaction levels to align with social desirability, inflating the perceived effectiveness of the training.
Conclusion and implications
The findings from this study offer valuable insights into participant engagement and perceived learning outcomes within the online training modules, especially in areas such as child development, substance use, and stigma reduction. A substantial portion of participants expressed motivation to pursue continuing education, reflecting a strong interest in professional development. However, the demographic data completeness represents a moderate level of detail, which limits the ability to fully assess the representativeness of this sample. Continuous evaluation of participant outcomes and satisfaction will further improve service delivery for vulnerable populations.
Footnotes
Acknowledgments
We want to acknowledge the participants who have completed the modules within the IMPACT WV online portal and offered their feedback over time.
Ethical considerations
All procedures in this study were reviewed and approved by the West Virginia University Institutional Review Board (IRB).
Consent for publication
Informed consent for publication was provided by the participants or a legally authorized representative. We have obtained the consent in writing per our IRB guidelines.
Author contributions
Lesley Cottrell, Mark Smith, and Charlotte Workman developed content for individual trainings including assessment process. Melina Danko and Jeremy Bock developed structure of the online portal and converted content into online format. All authors reviewed captured data and updated analytic plan as needed. Lesley Cottrell conducted analyses. All team members reviewed analyses and interpreted findings. All authors contributed to writing tasks in this manuscript.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This two-generational research project is supported by the Health Resources and Services Administration (HRSA) of the US Department of Health and Human Services (HHS) as part of an award totaling $1,500,000 with 0% financed with nongovernmental sources. The contents are those of the authors and do not necessarily represent the official views of, nor an endorsement by, HRSA, HHS, or the US Government.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Data availability statement
Data used in this study is available upon request as part of a larger service coordination program for rural families.