Abstract
Introduction/Objectives:
Since 2015, the rise in e-cigarette use among youth has concerned public health authorities. After peaking in 2019, usage rates have declined but remain high. In 2023, 10% of high school and 4.6% of middle school students reported current use. This report describes the implementation and evaluation of a school-based e-cigarette prevention program, CATCH My Breath (CMB), in 8 central Appalachian counties. These counties have some of the highest rates of tobacco use, poor health, and premature death in the United States.
Methods:
A total of 6217 students across 25 middle and high schools in West Virginia and Kentucky received the CMB program from 2019 to 2023. Lists of participating counties, schools, and teachers were maintained on a rolling basis over the 4 years of the project. Program reach and impact on e-cigarette knowledge and use are reported. Thirteen-item pre- and post-tests were completed electronically by students before and after each course delivery. Questions assessed knowledge about e-cigarettes, current (past 30-day) e-cigarette use, and interest in future use.
Results:
From 2019 to 2023, there were 9399 deliveries of the 4-session CMB course, primarily to middle school students. Approximately 84% reported being less likely to use e-cigarettes following the program. Significant improvements in e-cigarette knowledge and beliefs on 5 of 8 items were observed, including a significant increase in knowledge composite score (4.15-4.75, P < .001). From pre- to post-test, declines in current e-cigarette use (5.1% to 4.4%; P = .005) and peer influence (4.9% to 4.0%; P = .025) were also observed.
Conclusions:
CMB was feasible, effective, and well-received in a convenience sample of central Appalachian counties and schools. This curriculum, combined with other policy initiatives, offers hope for curbing the growing epidemic of youth e-cigarette use and nicotine dependence. The success of this project contributed to a decision by the West Virginia Bureau for Public Health and the West Virginia Department of Education to launch a joint effort to bring CMB to middle schools in all 55 counties during the next 3 school years of 2023-2026.
Introduction
E-cigarettes were introduced to the United States (US) market in 2007. By 2014, e-cigarettes were the most frequently used tobacco products among youth and young adults in the US. 1 The US Surgeon General issued a call to action for e-cigarette prevention programs in 2016. 2 Unfortunately, from 2017 to 2019, current (past 30-day) use of e-cigarettes in the US jumped from 11.7% to 27.5% among high school students and from 3.3% to 10.5% among middle school students.3,4 Public health authorities recognized this crisis and began working to develop interventions. 5
While the relative harms of e-cigarettes versus combustible cigarettes are still debated, there is no safe level of nicotine exposure for adolescents.6,7 According to one study, a single JUUL pod delivers an amount of nicotine equivalent to a pack of cigarettes, increasing the risk of rapid addiction among youth. 8 Nicotine has adverse effects on the developing adolescent brain including executive function, cognition, and attention.9 -12 E-cigarettes are associated with an increased risk of mental health issues in adolescents including depression, anxiety, suicidality, eating disorders, ADHD, and conduct disorders.11,12 Aerosolized e-cigarette fluid or “juice” contains heavy metals and carcinogens at doses with potential unknown long-term effects. 13 Furthermore, educators report a high volume of disciplinary problems which divert time from other educational priorities. 14
The 2021 Youth Behavioral Risk Survey (YBRS) revealed that 27.5% of West Virginia (WV) high school students were current users of e-cigarettes compared to 18% of high school students in the US as a whole. 15 WV middle school students were current users of e-cigarettes at a rate of 12% with no YBRS middle school comparison figure for the US as a whole. 15
The CATCH My Breath (CMB) curriculum, developed at the University of Texas School of Public Health, is the only curriculum designed for primary prevention with demonstrated effectiveness to reduce e-cigarette initiation among middle school students. 16 The program aims to improve knowledge about e-cigarettes, develop social-cognitive norms toward not using e-cigarettes, and build refusal skills. Beyond the pilot study, 16 limited evaluation of the CATCH My Breath program has been conducted. In one evaluation of the program in a private middle school, students’ knowledge about e-cigarettes significantly improved after receiving the curriculum, and this improvement was stable at follow-up 3 months later; however, students’ attitudes and susceptibility toward vaping did not improve. 17
This study reports the implementation of CMB in schools and impacts on knowledge about, use of, and interest in future use of e-cigarettes among students receiving the CMB curriculum in 8 WV and Kentucky (KY) counties. These are counties with some of the highest rates of adult smoking, premature death, poor health, and income inequality in the entire US. 18 WV has an adult smoking rate of 24%, ranging from 16% to 31% across counties; KY has an adult smoking rate of 20%, ranging from 14% to 35% across counties. 18 This is compared to a national adult smoking rate of 15%. 18 The high rates of tobacco use in this region among adults make it an especially important target for early prevention measures.
Methods
Design and Setting
The evaluation employed a descriptive design coupled with statistical analyses to assess the geographic reach of the program, baseline student knowledge about e-cigarettes, e-cigarette use, and interest in future use. A single-group, pre- and post-test design was adopted to quantify changes in knowledge, e-cigarette use, and interest in future use following completion of the CMB course.
The CMB course was delivered during 4 school years, specifically SY19-20 (2019-2020 school year), SY20-21, SY21-22, and SY22-23. The unforeseen outbreak of the COVID-19 pandemic in the Spring of 2020 caused school shutdowns and, in all but one school, delayed the start of instruction until Fall of 2020. During 2020, the national CATCH My Breath program quickly developed a video version of the curriculum that was easily loaded onto web-based instructional platforms. CMB instruction resumed in the Fall term of 2020 for both in-school and homebound students using school tablets or personal computers. All data collection was completed online, anonymized, and saved to UpMetrics secure servers. 19
For both logistical and Family Educational Rights and Privacy Act (FERPA) reasons, students remained anonymous during this project, resulting in exempt status from the reviewing IRB. Some schools delivered the CMB program every year to students across multiple grade levels. Thus, some students received the entire CMB curriculum annually over the course of 2 or 3 consecutive school years. These students could not be individually identified. For these reasons, “course deliveries” was chosen as the unit of analysis for this study, not individual students. The definition of a single course delivery was that a student received all 4 sessions of the CMB course, usually as 1 session per week, over 4 weeks.
Project funding did not allow in-school observations to measure fidelity of curriculum delivery including all 4 sessions, delivered at 1-week intervals, with peer-led small discussion groups. Teacher interviews indicated there was variation, usually in the number of course sessions and the use of peer discussion leaders. The direction of effect for these variations would be expected to diminish course impacts not increase them. For evaluation purposes, it was assumed that if a student completed a pre-test, then that student then received all 4 sessions of the CMB course. Therefore, the number of course deliveries was equal to the number of submitted pre-tests (9399).
The minimum number of unique participating students was estimated using the following procedure. First, for each participating school, all pre-tests from the semester with the greatest number of participating students were identified. Then, based on student grade level and the year the pre-test was taken, all pre-tests taken by students who would have graduated before they could take the course a second time were counted. Using these 2 criteria allowed the estimate of 6217 unique participating students.
This initiative and its evaluation were carried out by Partners in Health Network, Inc. (PIHN) and the West Virginia Alliance for Creative Health Solutions (WVACHS) based in Charleston, WV. WVACHS was primarily responsible for project evaluation.
Recruitment of Participants
PIHN used its network in an 11-county area of southern WV and eastern KY to recruit a convenience sample of schools having any or all of grades 5 through 10 to participate in the project. WV and KY state databases20,21 showed 200 such schools in these 11 counties with a combined enrollment of 40 000 students in grades 5 to 10. Half of the 200 schools were PK-5 elementary schools. Invitations via regular mail and e-mail were sent to superintendents in the 11 counties and directly to principals and health teachers in the 100 PK-8, middle, and high schools. Of those 100 contacted schools, 1, 9, 17, and 18 schools participated in the first, second, third, and fourth program years respectively. The CMB curriculum, training, technical support, and financial “wellness awards” were provided to participating schools. Financial incentives of $10 per student were awarded to schools based on the number of students who received the curriculum during each semester. To receive the school incentive, students needed to complete online pre-tests and post-tests with a post-test completion rate greater than or equal to 70% of the pre-test count.
Intervention
The CMB curriculum consists of four 40-min classroom sessions that include slide-show lectures, discussion opportunities, and peer-led small group activities. CMB was developed by the University of Texas School of Public Health. Social Cognitive Theory provides the theoretical basis for this curriculum and its delivery. 22 The curriculum is free to schools thanks to funding by the CVS Foundation. 23
Teacher Training
For schools to be eligible for incentive payments, teachers were required to complete a 45-min online training in CMB, complete an exam, and provide a copy of their certificate to PIHN.
Data Collection
The project used student pre- and post-tests adapted from longer surveys provided by the national CMB development and evaluation team. 13 The pre-test included 13 items that addressed knowledge about e-cigarettes, current use of e-cigarettes, and interest in future use of e-cigarettes. The post-test included the same 13 items plus 5 additional items asking for degree of agreement or disagreement about potential benefits of the course. The surveys were anonymous, with the demographic information limited to sex, year of birth, grade, school, and a secret nickname known only to the student. The responses were securely stored on a data platform by UpMetrics. 19 Surveys from the single participating school in SY19-20 were completed on paper and entered into UpMetrics by the research team. Starting in SY20-21, all surveys were completed by students online and loaded directly into UpMetrics. Survey data allowed measurement of counties reached, schools reached, and number of students who completed the course. Comparison of pre- and post-test data allowed measurement of knowledge gain and changes in e-cigarette use and interest in future use.
Measures
The following measures were used to assess e-cigarette knowledge, interest, and current use at baseline and after delivery of the CMB course.
Knowledge
Topics assessed ranged from e-cigarette contents (eg, “water vapor”; nicotine), age restrictions, and prevalence of e-cigarette use among peers. Each survey item is listed in Table 2. Responses were selected from a Likert scale that included “strongly agree,” “agree,” “disagree,” and “strongly disagree.”
Current use
Participants who reported using an e-cigarette, even once, in the past 30 days were classified as current users. Each survey item is listed in Table 3. Respondents were provided with 2 possible responses, “yes” and “no.”
Interest in future use
Participants were asked to answer 3 items related to interest in future e-cigarette use: (1) “Do you think you will vape in the next year?”; (2) “Are you curious about what it would be like to vape?”; and (3) “If one of your best friends offered you a vape, would you try it?” Each survey item is also listed in Table 3. Possible responses were “definitely yes,” “probably yes,” “probably not,” and “definitely not.” For analysis in Table 3, responses were collapsed into 2 categories: “yes” (including “definitely yes” and “probably yes”) and “no” (including “probably not” and “definitely not”).
Table 1 displays data groupings used for data analysis as part of this evaluation.
Data Groupings for Analysis.
Statistical Analysis
The pre- and post-test data collected through UpMetrics underwent rigorous cleaning (eg, addressing missing values), normalization (eg, addressing inconsistencies in free text entries), and validation (eg, confirming all responses are consistent with available survey choices) using Microsoft Access 24 to safeguard data integrity, enhance reliability, and ensure the meaningfulness of study findings. Statistical analyses were performed using JMP Statistical Software version 17.0 25 which facilitated comprehensive examination of the data collected. Descriptive analyses, including frequencies and percentages, provided a detailed overview of participant responses. Chi-square tests were used to assess for changes in students’ knowledge, beliefs, current e-cigarette use, and interest in future use of e-cigarettes. These tests were specifically applied to knowledge, belief, and interest statements, allowing for the detection of significant changes from pre- to post-curriculum delivery. Statistical significance was determined at the conventional threshold of P < .05, ensuring robust interpretation of the findings.
Results
Program Reach and Demographics
Over 4 school years starting in Fall 2019, CMB was implemented across 25 schools in 8 counties by 121 teachers. Some schools delivered the curriculum over multiple years, resulting in 9399 course deliveries to an estimated 6217 unique students. All reported results pertain to course deliveries. A total of 9399 pre-tests and 7008 post-tests were collected. Of the 9399 course deliveries, 48.6% were to female students, 49.1% to male students, and 2.3% provided another or no response. Ninety-six percent (96.0%) of course deliveries were to students in the target range of grades 5 to 10. Figure 1 shows counts for each grade.
Course deliveries by grade.
Psychosocial Impacts Knowledge and Beliefs
Full details of student responses for knowledge and beliefs about e-cigarettes are shown in Table 2.
Knowledge and Beliefs.
Receiving CMB was associated with strong knowledge gain by students about e-cigarettes. On all 6 survey questions that assessed factual knowledge, students made large gains.
For the first knowledge question about whether e-cigarettes contain mostly water, 39% gave a correct response of “no.” On post-tests, the correct response of “no” rose to 67% and “I don’t know” declined to 24%. A similar pattern was found for questions about whether all e-cigarettes contain nicotine (yes) and what is not a main ingredient of e-cigarette liquid (water).
There was a high percentage of correct responses to the question about whether nicotine is addictive and harmful to the body on both pre (91%) and post-tests (94%), but this gain still reached statistical significance.
Two psychosocial questions are best described as “belief” questions. The first was whether e-cigarettes are more or less harmful than combustible cigarettes. This issue is still debated among experts,6,7 although none argue that e-cigarettes are safe for tobacco-naïve children and adolescents. Over half of students responded “equally harmful” on both pre and post-tests.
The second belief question asked respondents how many high school students they thought had vaped within the past 30 days. Although the statistic cited in the course is 14%, 13 the majority of students endorsed percentages above 40% on both pre- and post-tests.
Behavioral Impacts Current e-cigarette Use and Interest in Future Use
Full details of student responses for current e-cigarette use and interest in future use are provided in Table 3. In Table 3, responses to the questions about future e-cigarette use were collapsed into 2 categories: “yes” (including “definitely yes” and “probably yes”) and “no” (including “probably not” and “definitely not”). This approach highlights the overall trends in student responses.
Current E-Cigarette Use and Interest in Future Use.
On the pre-test, 5.13% of participants endorsed current e-cigarette use, compared to 4.39% on the post-test, indicating a statistically significant decrease (P < .05). Among all pre-test responses, 8888 (94.6%) reported no current e-cigarette use. Three questions asked students about interest in future e-cigarette use. Of those students who reported no current e-cigarette use, when asked whether they thought they would vape in the next year, 2.33% chose “yes” on the post-test compared to 2.19% on the pre-test, which was not a statistically significant change. When asked whether they were curious about what it would be like to vape, 14.10% answered “yes” on the post-test compared to 14.12% on the pre-test, which was also not a statistically significant change. However, when asked whether they would try a vape if it were offered by one of their best friends, 4.02% answered “yes” on the post-test compared to 4.93% on the pre-test, representing a statistically significant decrease (P < .05).
Student Reflections About CMB
On the post-test, students were asked to rate their agreement or disagreement with 5 statements describing potential benefits of the CMB course using a 4-point Likert scale with the choices of strongly agree, agree, disagree, or strongly disagree. Responses of strongly agree and agree were combined for a final rating of agreement. Perhaps the most important findings were that 87% stated that “all middle and high school students should go through the CMB program,” and 80% stated they are less likely to vape because of the course. Full details are shown in Table 4.
Student Reflections About CMB.
Discussion
Over the course of 4 school years, this project delivered the CMB curriculum in 8 counties and 25 schools to 6217 students with a total of 9399 course deliveries. This was more than double the original program goals. This was accomplished despite the major interruption of the COVID-19 pandemic during 2020 and 2021. Crucial to this success were program adaptations introduced in SY20-21 after a first year of very slow uptake. These adaptations included a change in recruitment model from using school-based health centers to direct outreach to principals and health teachers; replacement of paper pre- and post-test instruments with real-time online test instruments; development of video versions of the curriculum for home learning during the COVID shutdown; and cash school wellness awards of $10 per student based on the number of students who received the curriculum and the completion of pre- and post-tests. Participation increased dramatically in the Fall of 2021 and continued through Spring 2023.
Based on feedback from principals and teachers, the cash “wellness awards” were an important contributing factor to this improvement. The success of this project contributed to a decision by the West Virginia Bureau for Public Health and the West Virginia Department of Education to join forces to deliver the CMB curriculum to middle schools in all 55 counties of the state beginning in the Fall of 2023. 26
Students showed significant improvements in e-cigarette knowledge, use, and interest in future use after receiving CMB. However, on both pre-tests and post-tests, the majority of students maintained the mistaken belief that over 40% of high school students vape. The number has never been above 28% and fell to 10% on the 2023 National Youth Tobacco Survey (NYTS). 27 One of the goals of CMB is to demonstrate to students that not vaping is the norm.
This experience offers ideas for program improvement and future research. Future versions of the pre-test should include a question about “ever use” of e-cigarettes, which will allow for comparison to the NYTS and the YBRS estimates of population susceptibility. In addition to the school incentive, an incentive for teachers will be introduced in the next phase of the program. An important question is whether school districts or individual schools will add CMB to their regular health education curriculum when these incentives end.
Potential future directions in research include: investigating whether wide delivery of the course reduces e-cigarette disciplinary occurrences; investigating whether knowledge and behavioral outcomes continue to improve with repeated course deliveries; and determining whether better outcomes are achieved when teachers receive CMB’s 6-h interactive training versus the 1-h asynchronous training. Future research using this data set may also include subgroup analysis by grade level and gender to determine which students may benefit most from this intervention.
Limitations
The evaluation described within this study is limited by the real-world nature of the program implementation and training. That is to say, the data presented in this study were collected as part of a broader initiative to disseminate an evidence-based program 28 to youth in rural West Virginia and Kentucky. Therefore, schools were unable to be randomized or matched by treatment condition. As a result, this study may suffer from threats to internal validity, particularly selection bias, as it is plausible that the observed changes in e-cigarette knowledge and behavior may be due to characteristics within the sample that may not be generalizable to other populations; replication studies with control groups and randomization are needed.
As noted in the methods discussion, fidelity of course delivery was not observed or measured directly, and there was variation among schools in terms of how lessons were implemented. The variations reported would be expected to diminish not increase the positive changes that were observed. Future research applying methods of implementation science are needed to investigate these nuances.
While the measured impacts on e-cigarette knowledge, use, and interest in future use are impressive, these must be interpreted in light of known limitations of the single-group, pre- and post-test methodology used here. Marsden and Torgerson describe these limitations as history, maturation, test effects, and regression to the mean (RTM). 29 Because the time between pre-test, intervention, and post-test was usually less than 4 weeks, the history and maturation biases were minimized. Test effects and RTM likely contributed to the positive outcomes obtained, but the outcomes were in most cases so large and statistically strong that a positive effect of the CMB course is most probable.
Additionally, while this study evaluates changes in knowledge and use via pre-test/post-test study design, the program was not implemented to account for intra-class correlation (ICC), nesting of participants within schools, or other aspects of sample size parameters and power analysis. Future research designed and powered for extensive program evaluation, particularly via experimental study, are needed to further investigate the impact of CMB on adolescent e-cigarette knowledge and use.
A second, 6-month post-test could have provided information about the durability of the curriculum’s impacts on knowledge, use, and interest in future use.
The Family Educational Rights and Privacy Act of 1974 (FERPA) required student anonymity as part of this project, and this prevented a matched-pairs analysis of student responses. For further data collection, the surveys have been changed to include primarily multiple-choice answers and dropdown menus for demographic questions like school and county, rather than text entry, allowing for improved pairing of responses while maintaining student anonymity. In future projects, each student could be provided with a participant card that contains unique pre-survey and post-survey links.
Conclusion
This CMB implementation project was undertaken as a rapid response to a public health crisis. This report is a summative evaluation of that experience. This evaluation adds to a growing body of national data that the CMB curriculum is effective, well received by teachers and students, and easy to integrate into regular health curriculum offerings. 26 While CMB shows promise as a standard part of middle school health curricula, robust success in tobacco prevention and cessation efforts requires a comprehensive approach including legislation, media, and family involvement. 30
Footnotes
Acknowledgements
The authors wish to acknowledge important contributions to this project by the following individuals and organizations: project officer Laura Boone of the Pallottine Foundation of Huntington; teacher consultants Brianne Barker, Melissa Cyfers, Angela Harless, Holly Mitchell, and Ashley Stephens; PIHN staff Tom Kuhn and Hilary Payne; UpMetrics staff Briana Roberge, Luisa Sydlik-Smith, and Greg Woodburn; the Coalition for a Tobacco-Free West Virginia; Josh Grant of the WV Department of Education; and the principals, teachers, and students of participating schools in Kentucky and West Virginia. This study is a project of the West Virginia Alliance for Creative Health Solutions, an AHRQ-recognized Practice Based Research Network.
Declaration of Conflicting Interests
The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This project was funded by grants from the Pallottine Foundation of Huntington, WV and the CAMC Foundation of Charleston, WV.
Data Availability Statement
De-identified data from this manuscript are available by approved request from Adam Baus, PhD at
Ethics Approval
The study protocol was determined to be exempt from review on 9/1/2022 by the Charleston Area Medical Center Institutional Review Board. IRB Study #22-876.
