Abstract
Executive function (EF) challenges, including difficulties with cognitive flexibility, planning/ organization, and emotional control, are common in neurodivergent children. We developed the Behavioral Observation of Classroom Executive Functioning (BOCEF) tool to examine observable EF-related behaviors of elementary students and EF-supporting strategies of their teachers. The BOCEF student score correlated significantly with teacher-completed ratings of similar behaviors. Autistic students (n = 48) and students with ADHD (n = 98) were rated as having similar rates of several EF-related behaviors, while autistic students were rated as significantly more likely to get “stuck.” Teachers were found to be more likely to display strategies of planning, clear instructions, and visual supports, and less likely to utilize favorable praise to correction ratios, transition priming, flexibility, rule referencing, or behavioral reward systems. Praise was the teacher strategy found to have the greatest association with student behavior. Implications for student accommodation/support and teacher training are discussed.
Lay abstract
Autistic children and children with ADHD often have difficulties with thinking flexibility, planning, and managing frustration. We developed a tool called the Behavioral Observation of Classroom Executive Functioning (BOCEF) to evaluate how children struggle with these executive functioning behaviors in the classroom, and how often teachers are using strategies known to support these behaviors. Our ratings of students on the BOCEF were similar to how teachers rated these same types of behaviors in these students. Autistic students (n = 48) and students with ADHD (n = 98) were rated as having similar rates of difficulties in the classroom, with the exception that autistic students got “stuck” more often, meaning they had more difficulty moving on from specific ideas or topics. Teachers were found to be more likely to display strategies of planning, clear instructions, and visual supports, and less likely to utilize adequate praise, warnings prior to transitions, flexibility, rule referencing, or behavioral reward systems. Praise was the teacher strategy found to be most related to student behavior. Findings from this study indicate the BOCEF could be helpful for schools to determine how much these types of challenges are impacting neurodivergent students in the classroom. Additional teacher training on incorporating praise and other ways to support neurodivergent students in the classroom is also likely to be important moving forward.
Executive Functioning (EF) refers to a set of top-down processes that help regulate or manage thoughts, behaviors, and emotions (Stuss & Alexander, 2000). The broad constellation of EF-related skills includes cognitive flexibility needed to adapt to changes in environment or routine, working memory to keep necessary tasks in mind and complete them, planning and organization to accomplish larger goals, and emotional control to manage frustrating or disappointing situations (Miyake et al., 2000). Underlying EF challenges have been found to be associated with poor outcomes in critical domains of adaptive functioning, including particularly academic performance, for both children diagnosed with attention-deficit/hyperactivity disorder (ADHD) and autism (Kenworthy et al., 2008; Toplak et al., 2008). Therefore, it is very important to be able to accurately identify and measure how these challenges manifest in the classroom.
EF challenges associated with autism and ADHD
EF difficulties for autistic children are well documented (Kenworthy et al., 2008), are relatively stable across development (Demetriou 2018), and affect multiple domains, including flexibility, planning, inhibition, and attention (Hill, 2004; Landry & Al-Taie, 2016; O’Hearn et al., 2008; Guerts et al., 2014). Similarly, EF challenges are considered a primary deficit for children with ADHD (Willcutt et al., 2005). EF difficulties in this population also manifest across domains, including activation or getting started on tasks, shifting and sustaining focus, managing emotions, and monitoring impulses (Brown, 2008). A literature review of 26 studies found that EF profiles shared several similarities across children with autism and/or ADHD, including deficits in working memory, attention, and fluency compared to typically developing children (Craig et al., 2016). Children with autism or comorbid autism and ADHD diagnoses had more difficulties with cognitive flexibility and planning than did children with just ADHD, while the latter group demonstrated more difficulties with response inhibition (Craig et al., 2016).
EF challenges in the classroom
EF difficulties are highly related to classroom performance, as the abilities to adapt to change, complete tasks, and organize work and materials are critical to navigating day-to-day school activities (Pugliese et al., 2020). Sparapani and colleagues (2016) found that directly observed classroom challenges of autistic children could be categorized into five factors: Emotional Regulation, Classroom Participation, Social Connectedness, Initiating Communication, and Flexibility. Steinbrenner and Watson (2015) similarly found that autistic students struggled to utilize EF-related skills to interact with education-related stimuli and people at the same time. They determined that such joint engagement was related to instructional group size, use of student-directed practices, number of autistic characteristics, and expressive communication skills (Steinbrenner & Watson, 2015). Children with ADHD have also been shown via classroom observations to display lower engagement and higher inattention than peers, specifically during teacher-led instruction (Steiner et al., 2014). A review study by Daley and Birchwood (2010) found that across the literature, the underlying EF deficits that accompany ADHD, rather than co-morbid conduct problems, are at the root of the academic difficulties these students experience. Within subsamples of students with ADHD, EF impairments have been shown to be uniquely associated with poor academic outcomes (Biederman et al., 2004; Tamm et al., 2021).
EF processes also involve seeing less immediate consequences for actions and following predetermined steps to achieve something, such as studying to get good grades. Studies have found EF task performance and caregiver ratings of child EF to be accurate predictors of school performance, including grades (e.g., Visu-Petra et al., 2011; Samuels et al., 2016). Furthermore, teachers may misunderstand or misinterpret EF-related challenges in school, attributing these difficulties to willful student ‘misbehavior’ or underachievement (Ashburner et al., 2010). This is likely to influence how teachers view these students and may limit students from receiving supports that could benefit them in managing EF difficulties to maximize their potential.
EF best practices to support learning
Given the EF challenges of neurodivergent children, there is great need to provide necessary accommodations to help them succeed. A survey of over 100 autistic students (aged 11–18 years) in mainstream classrooms in Australia reported EF demands and frequent transitions between educational tasks/settings as two of their top five challenges to school success (Saggers, 2016). Among solutions these students generated were greater assistance in the organizational and planning aspects of school, and more explicit reminders prior to transitions. Similarly, a qualitative study of teachers on inclusive practices for autistic students revealed the techniques they found most beneficial included having structured/consistent routines, utilizing creative planning and flexibility, and incorporating goal setting/rewards (Lindsay et al., 2014). Other evidence-based teacher practices include antecedent strategies, including schedules, priming for transitions, and visuals; as well as consequence strategies, such as reinforcement of appropriate behavior (Crosland & Dunlap, 2012). Increased rates of teacher praise have been specifically shown to increase appropriate behavior in the classroom (Kranak et al., 2017). Overall, the U.S. Department of Education suggests utilizing strategies across the domains of academic intervention, behavioral intervention, and classroom accommodation to support students with ADHD. Examples include providing behavior-specific praise and reinforcement, setting learning expectations, simplifying instructions, being predictable, dividing work into smaller units, highlighting key points, and using assistive technology (Jackson, 2004).
Despite existing literature on best practices for inclusion of these students in general education classrooms, knowledge and utilization of EF strategies might not always be commonplace among teachers and other school staff. Keenan et al. (2019) found that elementary school teachers had limited prior knowledge and comprehensive training around EF, but when provided with the definition of EF did feel it was an important set of skills that greatly impacted learning. Additionally, although EF challenges are frequently identified in student Individualized Education Programs (IEPs), specific strategies for targeting these skills are rarely included (Duncan et al., 2022).
Measurement of EF skills
Given the importance of EF skills in the classroom, it is critical to accurately measure EF. Presently, there are two main ways to assess EF in children: behavioral rating scales and lab-based tasks. EF rating scales are questionnaires typically completed by an informant familiar with the child, such as a caregiver or teacher, that ask about the frequency of occurrence of behaviors associated with EFs. While rating scale measures of EF do tend to have good reliability and validity, they are limited as any rating scales are by relying on the perspective and potential biases of raters (De Los Reyes et al., 2015). Lab-based tasks remove some of the biases associated with informant rating scales by having children complete tasks purported to measure EF domains. A common critique of lab-based EF measures is that they are testing EF in a contrived environment and may provide limited information about a child's ability to incorporate EF skills in real-world settings, such as school or home (Kenworthy et al., 2008).
In terms of measurement of EF skills in the classroom, there are several existing direct observation rating scales that target general on-task/ academic engagement behaviors in the classroom, and have been used specifically to identify ADHD. A recent review identified seven such published measures that required less than 25 h of training to use (Volpe et al., 2023). These are: the Behavioral Observation of Students in Schools (BOSS; Shapiro, 2011), which tracks specifically on-task behavior and engagement vs. off-task behavior; the Classroom Observation of Engagement, Disrespectful and Disruptive Behavior (COEDD; Volpe & Briesch, 2018); which focuses on engagement and disruptive behaviors; the Direct Observation Form (DOF; McConaughy & Achenbach, 2009), which rates 97 problem behavior items across different domains; the Revised Edition of the School Observation Coding System (REDSOCS; Jacobs et al., 2000), which focuses on compliance, disruptive, and off-task behaviors; the Responses to Physically Provoking Situations (RIPPS; Houghton et al., 2003), which consists of event recording a narrative of any inappropriate behaviors and their possible triggers; the State-Event Classroom Observation System (SECOS; Saudargas, 1997), which measures academic behaviors such as time working vs. out of seat; and the Student Observation System (SOS; Reynolds & Kamphaus, 2004), which rates adaptive and problem behaviors across several domains. Finally, the more recently created Regulation Related Skills Measure (RRSM (McCoy et al., 2022) focuses on skills more closely related to EF specifically, but for preschool and kindergarten children. While each of these scales have a number of strengths, they either were not designed for our age range or had limited focus on common presentations of EF-related behaviors in autistic students, including difficulties with flexibility, feelings of overwhelm, insistence on sameness, transitions between activities, etc.
Literature gap and current study
It is important to have a better idea of the EF strengths and weaknesses of students to best match instructional practices to their needs. Prior studies examining EF in neurodivergent children have relied predominantly on parent and teacher rating scales, lab-based tasks, or ADHD-specific direct observations. No prior study has created a profile of general EF strengths and weaknesses of these children based on direct observations. Prior research has also not examined the frequency with which teachers use EF-supporting best practices which could facilitate the inclusion and maximize the success of these students in general education. As education systems continue to support neurodiversity and inclusion of students in the classroom, these strategies only become more critical.
The present study aims to address these gaps in the literature, as better understanding these patterns of student behavior and teacher support can help meet the needs of these students. Specifically, we sought to address research questions in three main areas:
(1) What is the construct validity of a direct observation measure of classroom EF behaviors? (2) What are EF strengths and weaknesses in the classroom of students with primary diagnoses of autism as compared to students with primary diagnoses of ADHD? Are there any overall EF differences between these students? (3) How often are classroom teachers using practices known to support EF-related behaviors? Do these teacher practices relate to student behaviors during observations?
Method
The data for the present study were collected as part of the pre-intervention procedures for a larger randomized cluster comparative effectiveness trial examining school-based EF interventions. Findings about the sensitivity to treatment change can be found in Kenworthy et al. (2014).
Participants
Participants were third to fifth grade students (age: M = 9.68 years, SD = 0.87) at Title I elementary schools (i.e., those with a high percentage of students experiencing poverty) in the U.S. Students needed to meet research criteria for a diagnosis of autism (n = 50) or ADHD (n = 98). Students were nominated for participation by school staff based on behaviors associated with these conditions and cognitive/behavioral inflexibility (Troxel et al., 2024), and diagnoses were confirmed using gold-standard diagnostic measures, i.e., the Autism Diagnostic Observation Schedule, 2nd Edition (ADOS-2; Lord et al., 2012) and the Mini International Neuropsychiatric Interview for Children and Adolescents: ADHD Module (MINI-Kid; Sheehan et al., 2010). Students were typically administered only one diagnostic instrument, assessing for the most primary or apparent behaviors related to ADHD or autism, based on initial screening. Further diagnostic evaluations to ascertain the number of children meeting criteria for co-occurring autism and ADHD, therefore, were not conducted due to procedural feasibility limitations (Anthony et al., 2020). If a child had a previous or co-occurring diagnosis of ADHD and also met criteria for autism, that child was included with a primary diagnosis of autism. Additional study criteria included participants spending at least part of their day in the general education classroom, having a Full Scale Intelligence Quotient (FSIQ) score above 70, and a verbal age equivalence above six years of age. Finally, caregivers were required to speak and understand either English or Spanish in order to provide consent and utilize home components of the interventions.
Participants were from socioeconomically and ethnically diverse backgrounds (only 31% non-Hispanic White). While the mean household income of participants in the study appears high ($88,193), this was positively skewed by a few very high-income participants. The median income in the sample was $64,800, well below the median for the greater Washington D.C. area in which these data were collected. See Table 1 for more details. On average, children with autism diagnoses were older, had higher household incomes, and were more likely to be male, White, and observed outside of general education than children with ADHD diagnoses.
Descriptive statistics for students with diagnoses of ASD vs. ADHD.
Note. WASI = Wechsler Abbreviated Scale of Intelligence, BRIEF GEC = Behavior Rating Inventory of Executive Function Global Executive Composite, SKAMP = Swanson, Kotkin, Agler, M-Flynn, and Pelham Scale, BOCEF = Behavioral Observation of Classroom Executive Functioning.
Measures
Demographics
During baseline appointments, participants’ caregivers completed a demographic intake form with information such as child age, gender, racial/ethnic background, and household income.
BOCEF
The primary outcome measure in the present study was a direct observation of each student and their classroom teacher. Observers utilized the Behavioral Observation of Classroom EF (BOCEF; Anthony et al., 2023) measure to rate both the student and their primary classroom teacher at the time of observation on EF-related behaviors and supports respectively. The BOCEF coding form was designed by researchers needing an easy-to-use measurement tool that could track student EF strengths and weaknesses over time. For this reason, the BOCEF aims to quickly determine the presence or absence of six discrete behaviors across the domains of EF. The items on the form for both students and teachers were developed by a team of school staff and clinical researchers, who were experienced with EF supports, interventions and measurement. These authors were informed by discussions with neurodivergent students and their teachers about what EF-related behaviors and supports are most important for classroom success. The authors piloted items from a long list of these EF-related behaviors and pared it down to those behaviors that related to different aspects of EF in the fewest number of items. These items were selected based on being most observable, likely to change with treatment, and feasible for team members to reliably rate. The six EF-related student behaviors were: Social Reciprocity, Rule Following, Appropriate Transitioning, Getting Stuck, Negativity/ Overwhelm, and Classroom Participation. Teacher behaviors were derived from a school's research-informed manual on critical elements for a successful classroom for neurodivergent students. The eight teacher supporting practices were: Praise to Correction Ratio, Priming for Transitions, Flexibility, Planning/ Organization, Clear Instructions, Active Use of Visual Supports, References Rules/ Procedures, and Behavior Reward System. See Table 2 for more information on the definition and a few examples for each behavior and see Appendix A for the full measure.
Behavioral observation of classroom executive functioning (BOCEF).
Each student was observed for 15 min by a research team member unaware of the participant's diagnosis. Prior research has shown a 15-min classroom observation to be adequately reliable for low-stakes decisions (Ferguson et al., 2012). Observation periods were chosen by the school based on when participating students were available and were unlikely to be doing purely individual seat work (i.e., taking a test). Each observation period needed to include at least one transition, either from one activity to another or one setting to another and if a transition had not occurred, the observation period was extended by up to five minutes until it did.
During the observations, each student behavior and teacher practice was rated as present or absent. For student items with multiple opportunities for occurrence (Reciprocity and Participation), each opportunity was tallied as present or absent, and the predominant response was ultimately scored for that behavior. For example, if a student had three opportunities to demonstrate participation and did so twice, that student got credit for participation during that observation period. Student behaviors of Getting Stuck and Negativity/Overwhelm were reverse scored and summed to create a count of observed EF-related behaviors, in which a count of zero represented no observed EF-related behaviors and a count of six represented the highest number of observed appropriate EF behaviors. Of a total of 148 observations, there were three in which at least one behavior was not rated or could not be rated. These were instances in which students were engaged in exclusively independent work and therefore no rating of reciprocity was possible. We accounted for this by averaging behavior counts as number occurred/number possible multiplied by six to create an EF-related behavior count.
Ratings of teachers were based on their use of strategies at any point during the observation, regardless of whether they used strategies with the target student, other individual students, or the class as a whole. Teachers needed to actively utilize the strategy during the observation to be credited for it. For example, if a teacher had rules or a behavior reward system posted in the classroom but was not observed to reference them during the observation, they would be scored a zero for that practice. For teacher practices with multiple opportunities for occurrence and/or contrasting behaviors (Praise vs correction, Clear vs vague instruction), tallies were kept and the predominant outcome was scored. For example, if three instances of praise and four instances of correction occurred during an observation period, praise to correction ratio was scored a zero. For teachers, a total count of zero meant that no EF supporting behaviors were demonstrated, while a count of eight meant that all were demonstrated.
After receiving training on how to use the measure and how to interpret each item, classroom observers needed to achieve 80% inter-rater reliability on at least two students before conducting observations independently. New raters observed the same child as a reliable research staff member and ratings were compared. A total of eight observers conducted at least one independent observation, and there were no significant differences between observers on student BOCEF count.
SKAMP
The Swanson, Kotkin, Agler, M-Flynn, and Pelham Scale (SKAMP; Swanson, 1992) is a validated teacher-report rating scale of classroom-observed ADHD/ EF-related behaviors critical to success in the school setting, i.e., getting started on classroom assignments, following directions, and transitioning appropriately. We utilized the SKAMP in this study rather than more comprehensive teacher-report EF rating scales because it is a much shorter measure of similar behaviors and we wanted to maximize teacher response rates. The SKAMP consists of 13 items rated on a 7-point Likert scale, with higher scores indicating greater impairment (0 = “not a problem,” 3 = “moderate problems,” 6 = “extreme problem”). We also added two items (accepting criticism or feedback; having meltdowns or tantrums) to assess flexibility more specifically, as allowed in the published instructions for the SKAMP (Bhatara et al., 2006). Therefore, the range of possible SKAMP scores for this sample was zero (no ADHD/EF challenges) to 90 (greatest ADHD/EF challenges). There were three SKAMPs in which at least one behavior was not rated. Therefore, all SKAMP overall scores were calculated as an average by the number of completed items and then multiplied by 15 to create the total score. The original SKAMP has previously demonstrated good reliability (α > .90) and validity when rated by teachers in the community (Murray et al., 2009; Dieckhaus et al., 2021).
BRIEF-2
The Behavior Rating Inventory of Executive Function (BRIEF-2; Gioia et al., 2015) is an informant reported rating scale of EF problems in children. Caregivers of children participating in the study completed the parent rater version. Scores are generated in nine clinical domains (Initiate, Inhibit, Self-Monitor, Shift, Plan/Organize, Organization of Materials, Self-Monitor, Emotional Control, and Working Memory), which are aggregated into 3 index scores (Behavior-, Emotion and Cognitive-Regulation Indices) and a Global Executive Composite (GEC) total score. Higher scores indicate greater EF challenges, with T-scores greater than or equal to 70 considered clinically elevated. The BRIEF has previously demonstrated high internal consistency (.80-.98) across samples (Gioia et al., 2015).
WASI-II
The Wechsler Abbreviated Scale of Intelligence-Second Edition (WASI-II; Wechsler, 2011) is a brief cognitive measure that provides a full-scale standard IQ score based on four subtests of verbal comprehension (Vocabulary, Similarities) and perceptual reasoning (Block Design, Matrix Reasoning), and can be utilized with individuals between the ages of 6 and 90. The WASI-II has been demonstrated to have good psychometric properties, including high reliability and concurrent validity with longer IQ measures (McCrimmon & Smith, 2013).
Procedures
IRB approval was obtained from Children's National Hospital and from participating school districts. Consent and assent, demographics, diagnostic assessments, and parent-report rating scales were all collected at an initial baseline appointment. After participants were enrolled in the trial and randomized, but prior to beginning an intervention, treatment-masked research team members conducted the classroom observations. One student and one teacher were observed at a time for a 15-min period during the school day. The vast majority of the observations (91.1%) took place during an academic subject (i.e., reading, math, etc.). The rest occurred during specials (i.e., art, music, etc.). Observers positioned themselves in the classroom where they could see and hear the student but without making it obvious to the student that they were being observed. Some of the 148 participating students in this study were in the same classroom or had the same teacher. When students were in the same class, each observation was concluded prior to beginning the next one, and behaviors were only coded for that student and teacher during their observation window. We have data from observations of 114 teachers but some were observed in multiple observation windows with their different students.
Community involvement statement
Autism and ADHD self-advocates and parents of individuals with these conditions were involved in the larger project on which this study was based as principal investigators, co-investigators and leaders/members of a community advisory board. As described above, the BOCEF measure specifically was designed in consultation with neurodivergent students and their teachers about the EF-related behaviors and supports they found to be most critical.
Data analyses
Descriptive statistics and Pearson correlations were run on all predictor and outcome variables to better understand the data (see Tables 1 and 3). BOCEF student EF-related behavior counts, BOCEF teacher practices, BRIEF-2 GEC, and SKAMP were approximately normally distributed without severe skew (skewness between −1 and +1) or extreme outliers (scores beyond 2.5 SDs of the mean). To test the first research question, two separate multiple linear regressions evaluated whether the SKAMP (teacher EF measure) and BRIEF (parent EF measure) predicted BOCEF student EF-related behavior count, after controlling for student diagnosis, IQ, age, and household income. For the second research question, chi square analyses compared whether autistic students differed from those with ADHD in prevalence of each student behavior. A multiple linear regression compared autistic students and those with ADHD on BOCEF count, after controlling for demographics. Finally, for research question three, relationships were explored between teacher practices and student behaviors for the sample as a whole (see Table 4). Student BOCEF EF-related behavior counts were compared with independent t-tests with groups comprised of presence or absence each teacher strategy, and a multiple linear regression evaluated whether the number of teacher strategies utilized predicted student BOCEF counts, after controlling for demographics.
Pearson correlation matrix between predictor and outcome variables for the full sample (n = 148).
Note. *p < .05, **p < .01. WASI = Wechsler Abbreviated Scale of Intelligence, GEC = Global Executive Composite, SKAMP = Swanson, Kotkin, Agler, M-Flynn, and Pelham Scale, BOCEF = Behavioral Observations of Classroom Executive Functioning.
Pearson correlation (phi coefficient) matrix between student and teacher behaviors for the full sample (n = 148).
Note. *p < .05, **p < .01. Variable names correspond to BOCEF items (student behaviors 1–6, teacher practices 7–14). See Table 2 for full names.
Results
Research question 1: construct validity
Internal consistency on the BOCEF was low (Cronbach's α = .58), indicating a measure of six discrete behaviors rather than one unitary dimension. Teacher ratings of their students’ ADHD/EF-related behaviors on the SKAMP significantly predicted observed student BOCEF EF-related behavior count, after controlling for student diagnosis, IQ, age, and household income. This model accounted for 18.6% of the total variance in student BOCEF EF-related behavior count, R2 = .19, F(5, 108) = 4.93, p < .001; SKAMP score was the only significant predictor (β = -0.4, p < .001). Parent EF ratings on the BRIEF did not predict student BOCEF EF-related behavior count, after controlling for student diagnosis, IQ, age, and household income, R2 = .05, F(5, 119) = 1.31, p = .265.
Research question 2: student EF-related behaviors
Autistic students and those with ADHD demonstrated somewhat similar patterns of strengths and weaknesses, with no statistically significant differences in five of six observed behaviors (see Figure 1). Autistic students were significantly more likely than their ADHD peers to be rated as having gotten “stuck” on something during their observation (χ2 (4) = 4.36, p = .037). Students across diagnostic categories demonstrated relative strengths (> 67% of students observed to display) in reciprocity and rule following, and relative weaknesses (≥ 50% of students observed to display) with getting stuck and displaying negativity or overwhelm. Student diagnosis did not predict student BOCEF EF-related behavior count, after controlling for IQ, age, and household income, R2 = .03, F(4, 129) = 0.90, p = .464.
Graphs depicting frequency of student behaviors (1a) and teacher strategies (1b) during observations for the full sample (N = 148).
Research question 2: teacher EF-supporting practices
Teachers displayed variable use of strategies known to support EF in the classroom. Over 75% percent of teachers were observed to display strategies of planning, utilizing clear instructions, and actively using visual supports. In contrast, fewer than 30% of teachers were utilizing praise more than corrections/reprimands, priming for transitions, demonstrating flexibility, referencing rules, or utilizing a behavioral reward system. This discrepancy led to low inter-item correlations within teacher practices and low internal reliability (Cronbach's α < .50), indicating no unitary dimension was being measured. Correlations between teacher practices and student behaviors demonstrated that some practices were more likely to associate with student behavior than others. Student BOCEF EF-related behavior count did not differ based on whether teachers demonstrated clear instructions, priming, flexibility, planning, visuals, or reward systems. BOCEF counts were significantly higher during observations in which their teacher was rated as using praise (t = 3.61, p < .001) and significantly lower in observations in which their teacher referenced classroom rules (t = -2.05, p = .044). Total ratings of teacher practices did not predict student EF-related behavior count, after controlling for diagnosis, IQ, age, and household income, R2 = .03, F(5, 128) = 0.76, p = .581.
Discussion
We report on a novel classroom observational measure of EF-supporting teacher practices and EF behaviors of their students with primary research diagnoses of autism and ADHD. Regarding research question one, independently completed teacher ratings of student ADHD/ EF-related behaviors in the classroom on a standardized measure were significantly related to these direct observation behavior counts, indicating initial construct validity for the tool. Parent ratings of EF were not related to direct observation counts. However, this is not unexpected because parent ratings are based on an entirely different setting/context and rarely correlate strongly with teacher behavior ratings (De Los Reyes et al., 2015; Munzer et al., 2018).
Results from the direct observations of student behaviors (research question two) indicate that the classroom EF profiles of the autistic children and those with ADHD in this study were not drastically different. The overall EF-related behavior counts were very similar. Only one of the six EF behaviors significantly differed between the diagnostic groups; autistic students were more likely to get “stuck” than were students with ADHD, a categorization that included such behaviors as getting fixated, refusing to change topics, asking repetitive questions. These types of challenges are related to restricted/repetitive behaviors and insistence on sameness that are core characteristics of autism. These findings on getting “stuck” align with previous meta-analytic findings that cognitive flexibility might be the EF deficit most specifically related to autism as opposed to ADHD (Craig et al., 2016). Slightly higher percentages of children with ADHD in our study had challenges following classroom rules and participating appropriately in class. These types of difficulties for students with ADHD are consistent with prior EF literature on response inhibition that can be considered disruptive or off-task in school setting (Steiner et al., 2014; Craig et al., 2016).
Students across diagnostic groups struggled to avoid negativity, frustration, or overwhelm, with over half being rated as displaying one of these behaviors even in such a brief observation. Challenges with regulating emotions are common in both groups (e.g., Lipsky, 2011; Brown, 2008), and the participants in this study were specifically nominated based on difficulties in these areas. These results may speak to the need for additional accommodations to help students deal with often very stimulating environments of school. Students may also benefit from IEP goals and intervention services specifically targeting instruction of emotional regulation skills, including coping strategies.
Finally, in addressing research question three, teachers in the current sample were much more likely to utilize certain EF-related strategies in their classrooms and were much less likely to use others. The strategies that were commonly used (i.e., planning, clear instructions, and visual supports) may be more easily built into curricula and lesson plans that teachers already employ. Strategies utilized less often (i.e., praise, transition priming, flexibility, and rules/ reward system referencing) may be beyond the scope of typical teaching practices, harder to implement consistently, or less recognized as important and therefore used less frequently. A comprehensive review by the U.S. National Council on Teacher Quality found that less than half of teacher preparation programs were adequately requiring teachers-in-training to demonstrate their ability in research-based classroom management strategies (Pomerance & Walsh, 2020).
Using a favorable praise-to-correction ratio was the teacher strategy that had the strongest relationship with student EF-related behavior. We defined this strategy as simply using more praise statements than corrective or command statements with any or all students in the classroom. Prior research has indicated that a ratio of four or five praise statements for every one correction is the ideal praise rate to improve student behavior, although maintaining such a high ratio may not always be feasible (Sabey et al., 2019; Kranak et al., 2017). That praise was associated with total EF-related behavior count in our target students, even while being coded based on its use with the whole class, speaks to the potential power of this strategy. Praise may be particularly important for efficiently reinforcing appropriate coping strategies and other prosocial behaviors in the classroom. It is also important to note that the correlations between teacher practice and student behavior do not provide any insight into the directionality of the effect. For example, it may be that students behave more appropriately when reinforced with praise, or the reverse could be true and teachers are more likely to praise when behavior is already desirable. The only other significant association between a teacher practice and student EF-related behavior count in our study was that in observations where a teacher referenced the rules, students were observed to demonstrate fewer appropriate EF-related behaviors. Teachers were perhaps more likely to reference rules in situations where rules were not being followed, in the style of more reactive behavior management (Clunies-Ross et al., 2008). Teachers may benefit from additional training in explicit strategies known to support behavior in general, and related to EF (Keenan et al., 2019). Perhaps due in part to the low variance in teacher strategies and brief nature of the observations, there was no association between total teacher practice and student behavior counts in this sample. This relationship may be more difficult to parse out in such a brief single timepoint observation and warrants additional study.
Limitations
In order to optimize feasibility for school staff and others, and because brief observations have been shown to be adequately reliable for such purposes (Ferguson et al., 2012), we limited the observation period to 15 min. This timeframe allowed for observation of all behaviors in all but three of our 148 participating students. Additionally, we did not utilize a normed observation tool, but rather created an EF coding form as a research team for use in clinical trials examining EF interventions in school. This was due to the lack of an existing classroom behavior measurement specific to EF behaviors. This limitation is lessened by our finding of a strong correlation between our ratings and teacher-reported student EF. This novel tool also has the potential benefit of filling a gap in available assessment tools without requiring much intrusion in the classroom. Furthermore, we did not assess students for secondary or comorbid ADHD if they met criteria for autism as their ‘primary’ study diagnosis. This may limit the generalizability of these findings, given the high rates of comorbidity across these conditions. Finally, we did not utilize a normative comparison group for this study. Conducting observations of typically developing students or those not believed to have EF difficulties would be relevant for establishing the construct and discriminant validity of this measure, i.e., the ability of the BOCEF to distinguish those with EF challenges from those without. Finally, having multiple related items for each EF area would likely have improved internal consistency, but would have simultaneously detracted from real-world usability. This limitation has similarly been noted in other shorter observation-based autism scales, including the Modified Checklist for Autism in Toddlers Revised with Follow-Up (Robins et al., 2014) and Restrictive and Repetitive Behavior domain of the ADOS-2 (Bal et al., 2020), which are designed to rate one observable behavior per area of interest. If a short scale is designed to measure a broad construct, interitem correlations will always be low, resulting in low internal consistency (Ziegler et al., 2014). This is particularly likely to be true in the case of a construct like EF, which is characterized by the diversity of its subdomains (Miyake et al., 2000).
Future directions
The present study provides preliminary evidence for the utility of a very quick and easy-to-use measure of EF-related behaviors. Future research should explore the BOCEF in a larger sample that includes age-matched typically developing peers of neurodivergent children to provide more context to their EF strengths and weaknesses. This process could also include iterative testing of additional student or teacher items to improve internal reliability. The lack of major differences between autistic and ADHD students in this study also may support transdiagnostic support services for EF behaviors. This aligns with current calls in the field to move towards more transdiagnostic intervention approaches both for scientific (Astle et al., 2022) and neurodiversity-affirming (Fletcher-Watson, 2022) reasons. Additionally, teachers may benefit from additional training related to EF behaviors in the classroom. Teacher training programs, and particularly those for general education teachers, have a relatively limited focus on classroom behavior management. Teachers may benefit from additional information on what EF skills entail, and how to best support them in students who struggle in these areas. Strategies found to be utilized less frequently during our observations, including praise, priming for transitions, and active use of visual supports and reward systems, would be excellent targets for additional preparation.
Supplemental Material
sj-pdf-1-ndy-10.1177_27546330241258817 - Supplemental material for Observing executive functioning of neurodivergent students and supporting practices of their teachers in the classroom
Supplemental material, sj-pdf-1-ndy-10.1177_27546330241258817 for Observing executive functioning of neurodivergent students and supporting practices of their teachers in the classroom by Jonathan Safer-Lichtenstein, Lauren Kenworthy, Alyssa Verbalis, Caroline Ba, Susan Mikulich-Gilbertson, Bruno J. Anthony and Laura G. Anthony in Neurodiversity
Footnotes
Acknowledgements
The authors are grateful to the children and families who participated in this study. The authors also gratefully acknowledge the efforts of research staff and volunteers who made this study possible.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by a grant from the Patient-Centered Outcomes Research Institute (AD-1304-737) and an NIMH training grant for the first author (T32 MH015442).
Supplemental material
Supplemental material for this article is available online.
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
