Abstract
The prevention science approach to emotional/behavioral disorders (EBD) focuses on early intervention targeting risk and resilience factors (e.g., early problem behaviors, teacher-child interactions, classroom climate). The current study investigates the effectiveness of BEST in CLASS, a classroom-based indicated preventive intervention targeting young children at risk for EBD, in terms of producing clinically meaningful reductions of problem behaviors and improvements in social skills. A total of 185 early childhood teachers (92 in the treatment condition) and 462 young children (230 in treatment) participated in the study. Teachers completed rating scale measures of problem behaviors and social skills. The clinical significance of the findings indicated that the BEST in CLASS condition had a lower percentage of children with clinical or borderline range scores at posttest when compared with children in the control classrooms. The BEST in CLASS prevention model, clinically significant intervention outcomes, and prevention of EBD are discussed.
Given the national focus on the importance of the early years of a child’s development as they relate to school readiness, it is increasingly critical that children be given high-quality early learning experiences to ensure that they can be successful when entering school. School readiness refers to children’s level of preparation for learning in the school setting, as determined by their physical, cognitive, academic, social, and emotional development (U.S. Department of Health and Human Services, 2015). With the expansion of federally and state-funded early childhood programs, the promotion of young children’s learning and development in early childhood school settings has received increased attention. Although high-quality learning experiences related to academic development, such as literacy and mathematics, are typically emphasized, social-emotional learning (SEL) is also critical to school readiness (Zins, Bloodworth, Weissberg, & Walberg, 2007). SEL includes skills that contribute to emotion regulation, positive interactions with teachers and peers, and appropriate behavior, which are important for school success (Collaborative for Academic, Social, and Emotional Learning, 2013).
Prevalence and Associated Factors
Unfortunately, the prevalence of young children who enter school with SEL difficulties and significant problem behavior seems to be increasing. An estimated 5% to 26% of children demonstrate serious social, emotional, and behavioral learning challenges (Brauner & Stephens, 2006). In addition, preschool children are suspended and expelled for demonstrating problem behaviors at three times the rate of K–12 students (Gilliam, 2005). The short- and long-term outcomes associated with problem behavior are poor. Early childhood problem behavior contributes to negative relationships with teachers and peers (Bulotsky-Shearer, Bell, & Dominguez, 2012; Bulotsky-Shearer, Dominguez, & Bell, 2012; Carter et al., 2010), learning and academic achievement difficulties (Hamre & Pianta, 2001), and later identification of emotional/behavioral disorders (EBD; Fanti & Henrich, 2010). According to research, problem behaviors develop early, and if they are not addressed when they are initially displayed, they worsen over time, require more services and resources, and increase the likelihood for long-term negative outcomes (Dunlap et al., 2006).
The prevalence and outcomes of problem behaviors are even more concerning among young children who experience a range of child, family, and environmental risk factors. For example, rates of externalizing problem behavior are especially elevated for children living in poverty (Barbarin, 2007; Feil et al., 2005; Qi & Kaiser, 2003). In addition, negative child temperament, child adjustment problems, poor family functioning, and maternal depression are predictive of problem behaviors (Nelson, Stage, Duppong-Hurley, Synhorst, & Epstein, 2007). Young children who experience adverse relationships with their teachers and a negative classroom climate are also at a higher risk for problem behavior (Curby, Rimm-Kaufman, & Ponitz, 2009; Myers & Pianta, 2008; Pianta & Stuhlman, 2004; Pianta et al., 2005). When young children are exposed to a greater number and combination of these risk factors, their likelihood for future development of EBD, such as conduct disorder and oppositional defiant disorder, increases (Qi & Kaiser, 2003; Webster-Stratton, 1997). Research indicating the origins and risk factors associated with problem behavior highlights the need to examine early intervention, aimed at preventing these deleterious behaviors and lessening the impact of risk factors on children’s school success.
While noted risk factors raise the likelihood of problem behavior, factors leading to resilience among young children play an important part in the prevention of problem behavior and the promotion of school readiness. Resilience (or protective) factors include child, family, and environmental characteristics that have a positive impact on a child’s life, despite level of risk (National Research Council and Institute of Medicine, 2009a). For example, positive teacher-child interactions serve as a resilience factor influencing future social-emotional, behavioral, and academic success in school (Baker, 2006; Bulotsky-Shearer, Manz, et al., 2012; McCabe & Altamura, 2011; O’Connor & McCartney, 2007; Vandell, Belsky, Burchinal, Steinberg, & Vandergrift, 2010). Even when controlling for child and family factors, the quality of teacher-student relationships has demonstrated an effect on students’ academic achievement (O’Connor & McCartney, 2007). In addition, the early development of social, emotional, and behavioral competence may prevent the onset of persistent problem behaviors (Bornstein, Chun-Shin, & Haynes, 2010). Therefore, addressing resilience factors, such as children’s SEL and behavioral needs, within the context of the early childhood classroom is essential for the prevention of problem behaviors and the promotion of school readiness.
Prevention Science
Despite research on the risk factors associated with problem behaviors and the interventions based on tiered systems of support, effective interventions are not typically well integrated into early childhood programs to prevent the development of risk among young children and to promote their resilience. Traditional approaches to social-emotional and behavioral interventions for children who need additional support are also not often preventively employed in practice; rather, they focus on the treatment of individual children once identified and diagnosed—specifically, to reduce indicators (National Research Council and Institute of Medicine, 2009b). In contrast, prevention science aims to prevent problem behavior in the early stages of development as services are offered to children who are at risk for EBD, to reduce their likelihood of developing the disorder and needing treatment (National Research Council and Institute of Medicine, 2009a). The prevention science approach to problem behavior connects research on risk and resilience factors.
To illustrate, risk and resilience factors associated with problem behavior have been targeted with varying levels of intervention practices and programs, including universal (Tier 1), selective (Tier 2), and indicated (Tier 3) preventive positive behavioral support interventions (Sugai et al., 2000). Given that the goal of prevention science is to address problem behavior prior to or in the early stages of development, early childhood programs represent an important context for implementing these preventive interventions and addressing risk and resilience factors. The Teaching Pyramid (Fox, Carta, Strain, Dunlap, & Hemmeter, 2010; Hemmeter, Fox, & Snyder, 2013; Hemmeter, Ostrosky, & Fox, 2006) is a comprehensive multitiered model of support aimed at the social and behavioral competence needs of all children in early childhood classrooms. Through professional development activities (i.e., training and practice-based coaching), the Teaching Pyramid supports early childhood teachers’ use of effective strategies to support children’s SEL and development (see Hemmeter, Snyder, Kinder, & Artman, 2011; Snyder, Hemmeter, & Fox, 2015). On a Tier 2 level, teachers provide focused social-emotional supports for children who are at risk for EBD and have been unresponsive to universal supports at the Tier 1 level. Finally, Tier 3 levels of support are provided to children who continue to engage in persistent and ongoing problem behaviors and who have not responded to a Tier 1 or 2 level of support. Research has documented the efficacy of the Teaching Pyramid in ameliorating the needs of young children who are at varying levels of risk (see Hemmeter, Snyder, Fox, & Algina, 2016).
In addition to a comprehensive tiered approach—which includes an integrated system of levels of support based on need—some stand-alone, classroom-based early childhood universal (Tier 1) preventive interventions targeting SEL outcomes have been examined. These universal preventive interventions are typically implemented with all children from a selected population, regardless of individual risk factors. For example, Incredible Years (Webster-Stratton, Reid, & Hammond, 2004) is a universal curriculum for preventing emotional and behavioral problems by enhancing social-emotional competence and reducing classroom risk factors among young children who come from socioeconomically disadvantaged backgrounds (Webster-Stratton, Reid, & Stoolmiller, 2008). Another example is Second Step (Committee for Children, 1991), a universal program aimed at preventing aggressive behavior among children and adolescents by increasing their social skills (Frey, Hirschstein, & Guzzo, 2000). Although universal preventive interventions aim to generally promote resilience and reduce risk associated with problem behavior and SEL difficulties, approximately 20% of children need additional supports at the Tier 2 or 3 level.
Unlike universal (Tier 1) preventive interventions, indicated (Tier 2) preventive interventions are aimed at high-risk children who have some signs or symptoms of a social, emotional, or behavioral disorder but do not currently meet diagnostic criteria (National Research Council and Institute of Medicine, 2009a). Indicated prevention of EBD commonly includes risk- and resilience-focused interventions for young children who have elevated aggression and behavioral difficulties. One such example is the FAST Track program (Conduct Problems Prevention Research Group, 1992), a prevention science approach targeting children at risk for conduct disorder and the developmental influences contributing to this elevated risk (Conduct Problems Research Prevention Group, 2002). The results of the efficacy trial showed that the FAST Track intervention reduced the lifetime prevalence of conduct disorder, attention-deficit/hyperactivity disorder, and oppositional defiant disorder among those at the highest initial risk (Conduct Problems Prevention Research Group, 2002; Dodge & Conduct Problems Prevention Research Group, 2007). In addition, First Steps to Success (Feil et al., 2014) is an intervention designed to reduce young children’s problem behaviors in early childhood settings. In a recent efficacy trial, children who received the Preschool First Step to Success intervention demonstrated higher social skills and significantly fewer problem behaviors. These results provide evidence for the success of indicated preventive interventions implemented in early childhood school settings to reduce problem behaviors and the prevalence of EBD.
Finally, indicated (Tier 3) interventions are needed for those children who continue to engage in persistent and significant problem behaviors that further place them at elevated risk for EBD. Researchers found that a systematic process of providing individualized positive behavior supports is often required for children who need Tier 3 interventions (Dunlap & Fox, 2009; Dunlap et al., 2006). In a recent article, Dunlap and colleagues (Dunlap, Lee, Joseph, & Strain, 2015) provided a description of the “prevent-teach-reinforce for young children” model, designed to employ functional assessment and individualized positive supports for those needing Tier 3 interventions.
BEST in CLASS: An Indicated Preventive Intervention
Similar to other indicated interventions, BEST in CLASS (i.e., Behavioral, Emotional, Social Training: Competent Learners Achieving School Success; Conroy & Sutherland, 2008) represents a prevention science model at the Tier 2 level in that it systematically identifies and targets young children who engage in elevated chronic problem behaviors that place them at risk for the future identification of EBD. Unlike other Tier 2 interventions (e.g., FAST Track), BEST in CLASS is implemented by teachers during ongoing classroom instruction, which provides them with the flexibility to use the practices when identified children most need them. Theoretically, BEST in CLASS leverages the impact of the social transactions between teachers and children (Sameroff, 2009) within the early childhood classroom ecology (Bronfenbrenner, 1979, 2005) in terms of their social, emotional, and behavioral development. In practice, BEST in CLASS is based on a combination of high-quality professional development training and practice-based coaching to support the use of effective teacher instructional practices in early childhood classrooms that promote positive teacher-child interactions, enhance child engagement, increase learning opportunities, and decrease the occurrence of problem behaviors among young children (Conroy & Sutherland, 2008). Given that a negative classroom climate raises the risk for problem behaviors (Curby et al., 2009; Myers & Pianta, 2008; Pianta & Stuhlman, 2004; Pianta et al., 2005), BEST in CLASS promotes a positive classroom climate by focusing on enhanced child academic and social engagement and high-quality learning opportunities. In addition, BEST in CLASS promotes a higher level of positive teacher-child interactions and relationships, which enhance SEL and school readiness (Baker, 2006; Bulotsky-Shearer, Manz, et al., 2012; McCabe & Altamura, 2011; O’Conner & McCartney, 2007; Vandell et al., 2010).
BEST in CLASS is designed to equip early childhood teachers with the skills necessary to address the needs of children who are considered at risk for the development of EBD. The goal of BEST in CLASS is to facilitate teachers’ fidelity of key instructional practices targeting specific children who are demonstrating problem behaviors across various contexts in the classroom. This is accomplished through the delivery of the following three core professional development components: a 1-day teacher training workshop, a teacher manual, and weekly practice-based coaching occurring over 14 weeks (see the appendix for a description of the BEST in CLASS professional development components).
Studies have been conducted on the BEST in CLASS intervention examining teacher outcomes (Conroy et al., 2017) and child outcomes (Conroy et al., 2015; Sutherland et al., 2018), as well as teacher implementation fidelity (Conroy et al., 2015) and coach implementation fidelity (Sutherland, Conroy, Vo, & Ladwig, 2015). Prior research studies conducted on the BEST in CLASS intervention showed that teachers are able to implement the practices with high levels of adherence and competence (Conroy et al., 2017; Sutherland et al., 2018). Furthermore, BEST in CLASS decreases child problem behaviors, increases child engagement and social skills, and improves teacher-child interactions and relationships, including reductions in negative teacher-child interactions and raise the level of positive teacher-child interactions (see Sutherland et al., 2018). These results suggest that BEST in CLASS is effective in decreasing problem behaviors. However, no research has been conducted that examines the clinical significance of BEST in CLASS and the potential impact that it has on the reduction of risk for EBD.
Clinical Efficacy of Preventive Interventions
As discussed, the efficacy of various social-emotional and behavioral preventive interventions, including BEST in CLASS, is often substantiated by an overall increase in social skills and a decrease in problem behavior from pre- to post-intervention. However, few efficacy studies move beyond the statistical significance of findings and examine the clinical significance of findings, or the practical benefit of the intervention to the life of the child (Kazdin, 2003). That is, few studies determine whether, from pre- to post-intervention, children in the intervention group moved toward a healthier range of functioning in social, emotional, and behavioral competence and well-being as compared with children in the comparison group.
Examining efficacy in terms of clinically meaningful reductions in externalizing problem behavior and improvements in social skills is especially important for indicated preventive interventions that target children who demonstrate such levels outside the typical range of functioning, placing them at risk for EBD. Additionally, examining clinically meaningful changes can help researchers interpret findings from efficacy studies to highlight what interventions might be more or less effective for individual children (i.e., what works for whom).
Several studies of early childhood preventive interventions for problem behavior examined the clinical significance of findings. One such study investigated the clinical significance of the Early Risers Program, implemented by neighborhood family resource centers with urban kindergarten and first-grade children screened for aggressive behavior (August, Lee, Bloomquist, Realmuto, & Hektner, 2003). Clinical significance was determined through examination of pre- and post-intervention mean scores within the normative and nonnormative ranges on the Behavior Assessment System for Children (i.e., a teacher report form; Reynolds & Kamphaus, 2004) between intervention and control groups. Although differences between the experimental and control groups were not found for children who were mildly aggressive, clinically important differences were found for children who demonstrated severe aggression. Children with severe aggression in the intervention and control groups improved during the first year of the program. However, during the second year, children who demonstrated severe aggression in the intervention group maintained their gains, and the children in the control group worsened. In another study, Morrison and Bratton (2010) investigated the clinical significance of findings following Child-Teacher Relationship Training, which focused on preschool children in Head Start with identified behavior problems. According to teacher report data from the Caregiver-Teacher Report Form (C-TRF; Achenbach & Rescorla, 2000), 9 of the 15 children in the experimental group who scored in the borderline or clinical range prior to the intervention had improved scores in the normal range following the intervention. Last, Walker et al. (1998) investigated the clinical significance of the First Step to Success intervention, aimed at kindergarteners displaying early signs of antisocial behavior patterns (e.g., aggression, defiance). Although a number of efficacy studies conducted on the First Step to Success intervention yielded statistically significant child outcomes (e.g., see Feil et al., 2014; Sumi et al., 2012; Walker et al., 2009), Walker et al. found that the average scores for both cohorts on the Child Behavior Checklist Aggression subscale (Achenbach & Rescorla, 2000) went from marginally at risk (pre-intervention) to average (post-intervention). In addition, children scoring in the clinical range before the intervention had less favorable profiles across the dependent measures following the intervention than those children scoring in the nonclinical range before the intervention. With a limited number of studies, there is a clear need for prevention science research to examine the clinical significance of findings to determine the practical impact of preventive interventions on the development of problem behavior and EBD.
Study Purpose
Despite the importance of earlier prevention efforts, the majority of school-based prevention programs have focused on children in first through 12th grade (Humphries & Keenan, 2006). The current study contributes to this prevention science research gap by investigating a social, emotional, and behavioral preventive intervention that teachers can deliver with fidelity in authentic early childhood settings. In addition, this study adds to the limited research investigating the clinical significance of early childhood preventive interventions for problem behaviors. BEST in CLASS is an indicated preventive intervention that targets risk and resilience factors for the problem behaviors of young children identified as being at risk for EBD. The goal of this study was to determine the efficacy of BEST in CLASS in producing clinically meaningful reductions in problem behaviors and improvements in social skills. The primary research question centered on determining how many children receiving BEST in CLASS improved from pretest clinical or borderline ranges to posttest normal ranges for social skills and problem behavior as compared with the control classrooms.
Method
The current work was conducted as part of a 4-year efficacy study of the BEST in CLASS intervention (see Sutherland et al., 2018). Randomization into treatment and comparison groups occurred at the teacher level via a multisite cluster randomized design (Spybrook et al., 2011). 1 Teachers were blocked at the school level and randomly assigned to a BEST in CLASS intervention group or a business-as-usual comparison group. Teachers in both groups continued to implement their program-approved early childhood curricula throughout the study. Institutional review boards at two universities independently approved all research activities.
Setting and Participants
Research was conducted in 78 early childhood program settings across five school districts and three Head Start programs in two southeastern states. All programs, regardless of location, were federally and/or state-funded early childhood programs serving children 3 to 5 years of age (e.g., Head Start, state-funded prekindergarten).
Teachers
Teachers were deemed eligible for the study if they were currently employed as a preschool teacher in an early childhood program, had at least one child who met criteria for inclusion in the study, and consented to participate. A total of 185 teachers participated in the study, with 92 teachers in the BEST in CLASS condition and 93 teachers in the comparison condition (see Table 1 for teacher demographics). The majority of teachers held an associate degree (30.3%), a bachelor degree (39.5%), or a master degree (25.9%).
Participant Characteristics
Values are presented as n (%) unless noted otherwise.
Children
Children who met the following four eligibility criteria were included in the study: (1) caregiver consent obtained, (2) classified as at risk based on teacher report on the Early Screening Project (Feil, Severson, & Walker, 1998), (3) no identified developmental delay as measured by the Battelle Developmental Inventory–Second Edition Screener (Newborg, 2005), and (4) fluent in English. A total of 462 children participated in the study, with 230 children in the BEST in CLASS condition and 232 children in the comparison condition. The sample was 64.7% male and 35.3% female. The mean age for children in the BEST in CLASS and comparison conditions was 4.28 and 4.36 years, respectively. The majority (66.2%) of children were African American. In addition, 17.1% of children were Caucasian, 4.5% Hispanic, and 12.1% other or unidentified.
Data Collection and Measures
Data were collected at pretest in the fall of the academic school year and again at posttest in the spring of the year upon completion of the BEST in CLASS intervention. Pretest data were collected following teacher consent, child caregiver consent, and eligibility determination. Throughout the study, implementation fidelity data were collected, including coaching and dosage, as well as the level and quality of the key practices constituting the BEST in CLASS intervention. For a detailed description of the BEST in CLASS data collection, including coaching and teacher fidelity procedures, refer to the appendix, and see Sutherland et al. (2018), Sutherland et al. (2015), and Sutherland, McLeod, Conroy, Abrams, and Smith (2014).
Child outcome measures
Teacher report measures were utilized to assess children’s social skills and levels of problem behavior. The Social Skills Improvement System–Rating Scales (SSIS-RS; Gresham & Elliott, 2008) and the C-TRF (Achenbach & Rescorla, 2000) were administered at pre- and posttest.
The SSIS-RS (Gresham & Elliott, 2008) is a teacher report measure designed to assess the social skills and problem behaviors of young children. The instrument includes 76 items rated on a 4-point scale from 0 (never) to 3 (almost always) that can be grouped into two subscales: Social Skills and Problem Behavior. On the Problem Behavior subscale, a standard score >115 is considered above average and >130, well above average. On the Social Skills subscale, a standard score <85 is considered below average and <70, well below average. Gresham and Elliott (2008) found strong levels of internal consistency and test-retest reliability. For the current study, internal consistency was calculated, resulting in Cronbach’s alphas of .94 (pretest) and .95 (posttest) for the Social Skills subscale and .90 (pretest) and .92 (posttest) for the Problem Behavior subscale.
The C-TRF (Achenbach & Rescorla, 2000) is a teacher report measure designed to assess the problem behaviors of young children aged 18 months through 5 years. The 100 items are scored on a 3-point Likert scale, with responses ranging from 0 (never true) to 2 (often true). The three subscales include Externalizing Problems, Internalizing Problems, and Total Problems. A standard score in the range of 60 to 63 is considered borderline and ≥64, clinical. For the current study, internal consistency was high, with Cronbach’s alphas across all subscales ranging from .86 (pretest) to .95 (posttest).
Fidelity measures
Throughout the BEST in CLASS intervention, data were obtained to assess the fidelity of implementation by participating teachers and BEST in CLASS coaching staff. These included measures of coaching as well as teachers’ use of the BEST in CLASS practices. Teacher implementation fidelity was evaluated in BEST in CLASS and comparison classrooms with the BEST in CLASS Adherence and Competence Scale (Sutherland et al., 2014). Staff in both states were trained in the use of the scale through a manual and collaborative training. A total of 756 observations were conducted at pre- and posttest across both classrooms: BEST in CLASS (n = 389) and comparison (n = 367). Interobserver agreement data were obtained on 27% (n = 204) of those observations. Intraclass correlations were calculated for each item on each subscale: .69 (SD = .08; range, .60–.82) for adherence and .44 (SD = .12; range, .27–.59) for competence.
The mean adherence for teacher implementation of the BEST in CLASS intervention practices in the business-as-usual comparison condition was 2.35 (SD = 1.50; range, 1.41–4.77) at pretest and 2.43 (SD = 1.61; range, 1.36–4.66) at posttest. The mean adherence for teacher implementation of intervention practices in the BEST in CLASS condition was 2.47 (SD = 1.42; range, 1.52–4.69) at pretest and 4.53 (SD = 0.98; range, 3.72–5.99) at posttest. The mean competence of teacher implementation of BEST in CLASS intervention practices in the business-as-usual comparison condition was 3.82 (SD = 0.45; range, 3.32–4.44) at pretest and 4.30 (SD = 0.42; range, 3.95–5.04) at posttest. The mean competence of teacher implementation of intervention practices in the BEST in CLASS condition was 3.78 (SD = 0.37; range, 3.31–4.37) at pretest and 5.51 (SD = 0.33; range, 5.13–5.97) at posttest.
Data Analysis
To address our research questions, we used a transitional model (Agresti, 2007, section 9.4). In a transitional model for two time points, the outcome variable is status, which is defined for the current study as falling into (a) the clinical or borderline range or (b) within the normal range at Time 2 (i.e., posttest). Status was measured separately for the three C-TRF subscales and the two SSIS-SR subscales. The independent variables for the current investigation were condition (BEST in CLASS vs. business-as-usual comparison), pretest status (clinical or borderline vs. normal range), and state (A vs. B). In addition, the two-way interactions among the independent variables and the three-way interaction were included in the model. Condition, pretest status, and state were coded –.5 or .5, with BEST in CLASS, clinical or borderline, and State B coded .5. We included state because the study was conducted in two southeastern states. However, in reporting the results, we focus on the effect of treatment and the relationship of pre- to posttest status.
To account for nesting of children in teachers and teachers in schools, an approach developed in survey sampling was used (Heeringa, West, & Berglund, 2010). As applied in the current study, a single-level logistic regression model was conducted, but the standard errors were adjusted for nesting. We used this approach rather than a multilevel analysis to estimate a marginal model rather than a conditional model. 2 The logistic regression results were obtained via the complex procedure in Mplus 7.4, which provides maximum likelihood estimates and cluster-robust standard errors (McNeish, Stapleton, & Silverman, 2016) for hypothesis testing. 3
Results
Table 2 provides frequencies (percentages) and logits for posttest status on the C-TRF subscales Externalizing Problems, Internalizing Problems, and Total Problems. The marginal model represents the percentages in the clinical or borderline range (or, equivalently, the logits for the clinical or borderline range) and therefore accounts for the relationships of the percentages to condition, pretest status, and state. Results for Externalizing Problems in Table 2 indicate that none of the children with a pretest score in the normal range who resided in State B and were in BEST in CLASS had a clinical or borderline status at posttest. Due to the zero frequency, it is not possible to calculate the logit for this group or to validly estimate the C × P × S interaction based on the model. Inspection of the percentages in the clinical or borderline range at posttest suggests (a) little or no treatment effect in either state for children in the normal range at pretest and (b) a treatment effect of similar magnitude in both states for children in the clinical or borderline range at pretest. Therefore, we removed the C × P × S interaction from the model for Externalizing Problems, which permitted estimating the logistic regression equation.
Frequencies (Percentages) and Logits for Posttest Status on C-TRF Subscales by State, Pretest Status, and Condition
Note. C-TRF, Caregiver-Teacher Report Form; Normal = normal range; C or B = clinical or borderline range; ND = not defined.
Table 3 presents the results of the logistic regression analysis for C-TRF. For all three C-TRF subscales, the effect of condition was significant and negative, indicating that the BEST in CLASS intervention reduced the percentage of children in the clinical or borderline range. We computed the success rate difference (SRD; Kraemer & Kupfer, 2006) as an effect size for condition for each of the four combinations of pretest status and state. SRD is the difference in the proportion of children in the posttest clinical or borderline range for BEST in CLASS and the comparison condition. To quantify the effect size for the main effect of condition, we report the SRD averaged over the four combinations. The effect size for Externalizing Problems was –.119, indicating that, as averaged over pretest status and states, the percentage of children in the clinical or borderline range was lower by 11.9 percentage points for the BEST in CLASS condition in comparison with the control condition. For Internalizing Problems and Total Problems, the effect sizes were –.194 and –.173, respectively.
Results of Logistic Regression Analysis for C-TRF Subscales
Note. C-TRF = Caregiver-Teacher Report Form; C = condition; P = pretest; S = state.
There was also a significant positive relationship of pre- and posttest status, indicating that, when compared with children who scored in the normal range at pretest, children who scored in the clinical or borderline range at pretest were more likely to be in the clinical or borderline range at posttest. We adapted SRD to quantify the strength of the relationship between pre- and posttest status by calculating the difference in the proportion of children in the posttest clinical or borderline range as compared with (a) children who were in the pretest clinical or borderline range and (b) those who were not. We refer to this effect size as the clinical or borderline rate differential (CBRD). The CBRDs for pretest status, as averaged across combinations of condition and states, were .493, .388, and .382 for Externalizing Problems, Internalizing Problems, and Total Problems, respectively. The relationship to pre- and posttest status was modified by a significant P × S interaction for both Externalizing Problems and Internalizing Problems. These interactions indicated that the pretest-posttest relationship effect was larger in State B. For Externalizing Problems, the CBRD was .563 in State B and .423 in State A. For Internalizing Problems, the CBRD was .521 for State B and .256 for State A.
Table 4 presents the frequencies (percentages) and logits for posttest status on the SSIS-RS Problem Behavior and Social Skills subscales. Table 5 presents the results of the logistic regression analysis on the relationships of the percentages of children in the clinical or borderline range at posttest to condition, state, and pretest status (i.e., classified in the normal vs. clinical or borderline range at pretest). The condition estimate was negative and significant for Problem Behavior, indicating that treatment reduced the percentage of children in the clinical or borderline range (SRD = −.186). There was a significant Condition × Pretest × State interaction for social skills. A simple interaction analysis resulted in a significant Condition × Pretest interaction for State A (z = −2.104, p = .035), with a significant condition effect for children in the clinical or borderline range at pretest (z = −3.112, p = .002, SRD = −.277). The simple Condition × Pretest interaction was not significant for State B. The condition effect for State B was significant (z = –2.361, p = .018, SRD = −.161). These analyses suggest that BEST in CLASS reduced in both states (A and B) the percentage of children who were in the clinical or borderline range at pretest. Additionally, pretest status was positively and significantly related to posttest status for both SSIS-RS subscales, indicating that children who scored in the clinical or borderline range at pretest had a tendency to be in the clinical or borderline range at posttest. CBRD effect sizes were .328 and .338 for Problem Behavior and Social Skills, respectively.
Frequency (Percentages) and Logits for Posttest Status on SSIS-RS Subscales by Research State, Pretest Status, and Treatment
Note. SSIS-RS, Social Skills Improvement System–Rating Scales; Normal = normal range; C or B = clinical or borderline range.
Results of Logistic Regression Analysis for SSIS-RS Subscales
Note. SSIS-RS, Social Skills Improvement System–Rating Scales; C = condition; P = pretest; S = state.
Discussion
With a focus on prevention science, the overall goal of the current study was to examine the influence of BEST in CLASS in producing clinically meaningful reductions in externalizing problem behaviors and improvements in social skills among young children identified as being at risk for EBD. The current investigation extends our previous research on the BEST in CLASS intervention and was conducted as part of a 4-year efficacy study (see, Sutherland et al., 2018), which found statistically significant differences between children in the BEST in CLASS and control conditions, with better outcomes for children who received the intervention. Overall findings from the current investigation also found positive outcomes for children who received the BEST in CLASS intervention as compared with the control group. Specifically, the percentage of children receiving the BEST in CLASS intervention who scored in the clinical or borderline range—as measured by the Externalizing Problems, Internalizing Problems, and Total Problems subscales of the C-TRF and the Problem Behavior and Social Skills subscales of the SSIS-RS—was lower at posttest as compared with children who did not receive the intervention. However, for social skills, inferential tests suggest that the smaller percentage was not realized for children in State A who were not in the clinical or borderline range at pretest.
Limited research has examined the clinical significance of preventive early childhood interventions aimed at young children who engage in significant problem behaviors that place them at risk for EBD. The current study adds to this important but overlooked area of research. Results of this study suggest that the BEST in CLASS intervention—an indicated preventive intervention that targets resilience factors related to problem behaviors among young children identified as being at risk for EBD—produces clinically significant outcomes. Researchers found that problem behaviors that interfere with young children’s learning and development emerge early and persist well into the school years (NICHD Early Child Care Research Network, 2002; Vandell et al., 2010). Kazdin and Weiz (2003) suggested that the successful prevention and intervention of young children’s problem behaviors depend on early treatment, when developmental trajectories are still malleable. Fortunately, when we intervene at a young age before these behaviors have become well established in children’s repertoires, we may change their trajectory and prevent many of the lifelong negative consequences manifested through their ongoing existence (Bornstein et al., 2010).
Findings from the current study are important for another reason. Unlike other Tier 2 interventions that found positive effects on clinical change (e.g., Early Risers), BEST in CLASS is delivered by teachers during naturally occurring classroom activities. The BEST in CLASS intervention focuses on increasing the use of key instructional practices during activities in which children are engaged throughout their school day, thus decreasing problem behaviors during these activities and improving interactions and relationships between teachers and young children as they naturally interact. Research suggested that improving teacher-child interactions (Early et al., 2006) and relationships (O’Connor & McCartney, 2007) can have positive long-term effects on child functioning. Results from the current study suggest that BEST in CLASS may have some influence on helping children move from a clinical or borderline category to the normal range of functioning. For example, as reported in Table 2, 42% to 78% of children in the BEST in CLASS condition with clinical or borderline scores on the C-TRF at pretest had normal scores at posttest, and as seen in Table 4, 49% to 61% of the children in the BEST in CLASS condition with clinical or borderline scores on the SSIS-RS at pretest had normal posttest scores. These findings highlight the potential importance of teacher-child interactions and relationships as a protective factor in promoting positive child outcomes. In addition, feasible teacher-delivered Tier 2 interventions that can help move children from a clinical or borderline range to a more normal developmental range, such as BEST in CLASS, can be particularly helpful in identifying nonresponders that may benefit from more intensive Tier 3 interventions.
Our results are similar to and support findings from previous research indicating that children who received intervention and scored in the clinical or borderline range at pretest raised their performance to within the normal range at posttest (e.g., Morrison & Bratton, 2010; Walker et al., 1998). However, previous research and our findings suggest that a number of additional factors may influence clinical outcomes. For example, some researchers found that the intensity of problem behaviors affects the clinical significance of the findings. August et al. (2003) examined the clinical significance of the Early Risers program and found differences between the experimental and treatment groups for children who demonstrated significant aggression but not for children who demonstrated mild aggression. Additionally, researchers found that some children’s behavior actually continues to worsen without treatment (e.g., August et al., 2003). Another factor that might influence the outcomes of the current investigation is the intensity or dosage of the intervention. The BEST in CLASS intervention targets children at risk for EBD and is delivered throughout the school day. In contrast, some universal, or Tier 1, interventions focus on all children (e.g., Frey et al., 2000) and are delivered during a series of short small group intervention sessions (e.g., Walker et al., 1998); therefore, they might not deliver a sufficient dosage of intervention to influence clinically significant outcomes for individual children at elevated risk. Unfortunately, these types of analyses were beyond the scope of the current investigation. However, there has been a recent emphasis on authentic change agents (i.e., teachers) implementing interventions within the context of the school day (e.g., Feil et al., 2014; Sutherland et al., 2018). Future research may be able to address these types of questions. Additionally, researchers will want to consider that, even though children may receive Tier 1 and 2 interventions, a small percentage of children continue to need additional supports (i.e., Tier 3). It may be that some children who require Tier 3 services are exposed to high-quality Tier 1 and 2 practices but require additional supports due to their individual developmental learning needs (e.g., Dunlap et al., 2015).
As noted, results differed across the two states involved in the study on some of the outcome variables. Differences in implementation of BEST in CLASS between States A and B are not a likely explanation for state differences in outcomes, because BEST in CLASS is a manualized intervention and no variation in implementation occurred between the states. A more likely explanation may be several demographic differences between participants in States A and B. For example, caregivers in State A were, on average, younger and had less education in comparison with caregivers in State B. Additionally, household income was lower in State A. All of these are risk factors that can contribute to EBD. Last, teachers in State B were more highly educated than teachers in State A.
As the literature suggested, the development of EBD is progressive and influenced by early exposure to a number of child risk factors (e.g., temperament, development delays; Benson & Aman, 1999; Reid, 1993), family risk factors (e.g., poverty; Kaiser, Cai, Hancock, & Foster, 2002), and school risk factors (e.g., coercive teacher-child relationships, negative classroom climate; Curby et al., 2009). BEST in CLASS is designed to primarily target risk factors that might be associated with classroom settings, such as teacher-child interactions and classroom climate; thus, the training of teachers might have played a role in teachers’ uptake of BEST in CLASS practices within their ongoing classroom instruction.
Limitations
The findings from the current study are promising, but several limitations should be considered when interpreting the findings. First, in this investigation, we report findings from teacher report measures rather than direct observations of children’s behavior in their classrooms. Although our findings are supported by two standardized measures and although the C-TRF and the SSIS-RS are both psychometrically sound and common instruments used for research and clinical purposes, direct observations of children’s problem behaviors and social skills would provide additional information and help to substantiate findings. Additionally, we did not include data from children’s families. Problem behaviors often occur across home and school settings. Obtaining further information from families to examine the pre- and posttest differences in home settings might be important when interpreting findings related to clinical significance. Finally, an area of utmost importance to our findings that we are unable to report is longitudinal data. Therefore, we are unsure whether the improvements that children in the BEST in CLASS intervention group made between pre- and posttest would continue over time and whether these improvements affected children’s future identification of EBD. Clearly, these limitations suggest that our findings be viewed with caution and are directions for future research.
Conclusion
In summary, these findings indicate that when children received the BEST in CLASS instructional practices intervention targeted at promoting engagement and positive teacher-child interactions, their SEL outcomes improved. As discussed earlier, a number of social-emotional and behavioral preventive interventions exist. However, few investigators have focused on the clinical and practical significance of the intervention outcomes (Kazdin, 2003). Findings from research often indicate that children’s behavior statistically improves following an intervention. Still, the field has yet to emphasize research that examines whether these gains are meaningful from a clinical perspective, meaning that children’s social, emotional, and behavioral competence skills are within the normal range of development following intervention, thereby potentially reducing their risk for developing EBD. Additional research should be conducted to examine the clinical significance of findings as well as various factors that might influence outcomes, such as the level and intensity of behaviors and dosage. Through the implementation of effective early intervention programs, we may be able to reduce the occurrence of young children entering school at risk for or later identified with EBD. As suggested by Hawkins and colleagues (2015), “prevention is the best investment we can make, and the time to make it is now” (p. 1).
Footnotes
Appendix
Authors’ Note
The opinions expressed in this article are those of the authors and may or may not represent views of the U.S. Department of Education, Institute of Education Sciences, U.S. Department of Health and Human Services, and the U.S. National Institutes of Health.
Funding
The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R324A110173 to the University of Florida.