Identifying Implementation Usability Issues of an Evidence-Based Behavioral Intervention for Use in Schools

Abstract

Background

Educators endorse challenging behavior as a concern for autistic students, which is compounded by the lack of adequate resources for behavioral intervention use at school. The RUBI program is an evidence-based intervention, initially developed for clinicians to implement with parents of autistic children ages 3–14 with co-occurring challenging behavior in outpatient settings. Using the Discover, Design/Build, Test framework, which combines human-centered design and implementation science, implementation usability issues of RUBI were identified for redesign to ensure intervention-setting fit when used in schools.

Method

RUBI content was collaboratively and iteratively redesigned with elementary school partners, including 41 staff members from 28 schools. During the Discover Phase, the research team conducted in-class behavioral observations (N = 8) and cognitive walkthroughs (N = 15) with educators to identify implementation usability issues in the original RUBI intervention. In the Design/Build Phase, collaborative redesign sessions (N = 6) and demonstration studies (N = 12) were conducted to develop potential solutions to these issues. Implementation usability issues were systematically rated for importance and feasibility prior to redesign to guide the adaptation process.

Results

Conventional content analysis was used to code qualitative data and identify implementation usability issues. Two implementation usability issues were identified: (1) integration with other school-based systems of support and (2) data collection in schools.

Conclusion

Identifying and addressing usability issues may promote greater utility and successful implementation of RUBI in schools. Using partner-engaged methods allowed for the identification of critical implementation usability issues prior to the implementation of the redesigned intervention, RUBI in Educational Settings, or RUBIES. Implications to implementation in school settings are discussed, including potential ways to integrate RUBIES within existing school frameworks (i.e., MTSS) and streamline data collection with the use of technology.

Plain Language Summary

Many teachers and school staff say they struggle with managing challenging behaviors exhibited by autistic students and feel they do not have enough training or resources to help these students. The RUBI program was first created for outpatient clinics, where parents learned ways to support their children with challenging behaviors. In this project, the research team worked with teachers and school staff to adapt RUBI, so it could be used by any educator in a classroom. Teachers and school staff reviewed the RUBI program, shared their thoughts on how it could work in schools, and suggested ways to overcome potential challenges to its use in schools. Two main challenges emerged: using RUBI alongside other school support programs that are required, and collecting data in the classroom. Addressing these issues is important to make sure the redesigned program—called RUBI in Educational Settings (RUBIES)—can be used successfully. Working closely with school staff helped identify these challenges, some of which are easy to fix, while others may take more time and creativity to address.

Keywords

autism schools behaviors intervention paraeducators usability implementation

Introduction

Autism spectrum disorder (ASD) is a lifelong neurodevelopmental disorder that affects 1 in 31 youth in the United States (American Psychiatric Association, 2022; Shaw et al., 2025). The rising prevalence of ASD significantly impacts the public education sector (US Department of Education, 2020) as schools are the primary setting in which autistic children receive intervention services and support (Brookman-Frazee et al., 2009, Brookman-Frazee & Stahmer, 2018). Autism-focused evidence-based practices (EBPs) can be challenging to implement in schools, particularly if implementation strategies are not tailored to the unique needs of this setting (Locke et al., 2015, 2017). The heterogeneity of ASD symptoms and the need for highly individualized programs to meet children's diverse needs are two of the challenges that educators face in the delivery of EBPs in public schools (Corkum et al., 2014).

Usual care to address challenging behaviors, such as meltdowns, difficulties with transitions, aggression, and self-injury, which commonly co-occur in autistic students, involves engagement of behavioral specialists to develop targeted behavioral intervention plans. While effective, this costly and time-intensive process (DeFilippis & Wagner, 2016) could be lessened if educators were trained to effectively use behavioral strategies to support autistic children. In particular, training for paraeducators (teaching assistants or instructional aides who support autistic students) is one of the most significant needs facing the autism field and public education system (Giangreco et al., 1999; Rispoli et al., 2011; Simpson et al., 2003; Swiezy et al., 2008). Although paraeducators are the individuals who spend the most time supporting children with disabilities in schools, most paraeducators receive limited training before beginning their positions and are not provided access to the same professional development as teachers (Giangreco et al., 2010). It is imperative that implementation efforts include training paraeducators in acquiring knowledge and skills to adequately support autistic students who exhibit challenging behaviors at school.

One of the most extensively studied behaviorally focused interventions for autistic children with co-occurring challenging behaviors is the Research Units in Behavioral Intervention (RUBI) parent-training program. RUBI is a low-intensity, manualized intervention designed to be delivered by a clinician to caregivers as part of outpatient care (Aman et al., 2009; Bearss et al., 2015; Handen et al., 2015). RUBI is based on the behavioral principle that challenging behaviors serve identifiable functions, such as obtaining tangible items, gaining attention, escaping or avoiding demands, or producing sensory stimulation (Iwata & Worsdell, 2005). Clinicians teach caregivers to assess the function of their child's behavior and to apply behavioral strategies that are specifically matched to that function. By aligning intervention strategies with behavioral function, caregivers can replace challenging behaviors with more appropriate alternatives, thereby promoting skill development and reducing challenging behavior (Horner et al., 2002). Grounded in behavioral skills training (Miltenberger, 2016), RUBI incorporates direct instruction, modeling, role-play, and guided practice with feedback to equip caregivers with the skills needed to address their child's behaviors. Each session concludes with a practice assignment in which caregivers track daily implementation of the targeted techniques. Multiple randomized controlled trials of RUBI have demonstrated clinically and statistically significant reductions in autistic youths’ challenging behaviors, such as meltdowns, aggression, and transition difficulties, as well as high acceptability and engagement among caregivers (Aman et al., 2009; Bearss et al., 2015; Handen et al., 2015; Postorino et al., 2017).

With RUBI established as an EBP, the logical next step is to expand its implementation across contexts, including evaluation of its usability and feasibility within school settings. To this end, RUBI intervention content was collaboratively and iteratively redesigned with public school partners (administrators, general and special education teachers, paraeducators, other school personnel) for use in elementary schools (RUBI in Educational Settings, RUBIES; Bearss et al., 2022) using the Discover, Design/Build, Test Framework (DDBT; Lyon et al., 2019a, 2019b), a three-phase framework that combines human-centered design and implementation science to enhance the usability, contextual fit, and uptake of an intervention. DDBT “is intended to gather the requisite information to drive iterative redesign of existing EBPs or implementation strategies to improve usability and implementation outcomes (e.g., contextual appropriateness, and adoption) while retaining an intervention's core components” (p. 5). The redesigned intervention, RUBI in Educational Settings (RUBIES), was developed using feedback from school partners regarding content relevance and applicability. RUBIES is now an 8-module program designed to equip educators—including general and special education teachers, paraeducators, and specialists—with the skills to interpret behavior as communication while considering autistic characteristics such as rigidities and sensory sensitivities. This approach addresses common misattributions of behavior as “oppositional” and instead guides educators to respond to the underlying causes of challenging behavior. Grounded in the premise that challenging behavior serves a communicative function—accessing tangibles, gaining attention, escaping demands, or obtaining sensory stimulation (Iwata & Worsdell, 2005)—RUBIES provides a framework for identifying behavioral function and applying targeted positive behavioral supports (e.g., visual supports, reinforcement) matched to that function (Conroy et al., 2005; Reichow & Barton, 2014). By improving educators’ ability to interpret behavior and implement appropriate strategies, RUBIES aims to promote self-management skills in children and reduce the occurrence of challenging behaviors. RUBIES builds educators’ interpretative skills and expands their behavioral management toolbox, supporting generalizability across emerging behaviors and new students each year.

Major changes to the RUBI manual that resulted from the redesign process include: the inclusion of school-specific examples throughout the manual, the addition of an “Autism 101” session to provide more background on the diagnosis to educators, the removal of teaching skills sessions and physical prompting strategies, and the collapsing of two reinforcement sessions into one session, among others. The aforementioned manuscript outlined intervention content usability issues that resulted in these changes, while this manuscript focuses on the implementation usability issues that were uncovered during the Discover and Design/Build phases of the RUBIES redesign. We are choosing to place a specific focus on implementation usability to ensure intervention-setting fit when RUBIES is used with educators in public schools.

Methods

As previously mentioned, the redesign study of RUBI used the DDBT Framework, and this manuscript focuses on outcomes from the first two phases in the DDBT Framework: (1) Discover and (2) Design/Build. In the Discover phase, we identified our school partners’ needs and contextual information to improve the usability and appropriateness of the existing RUBI program within the school setting through classroom behavioral observations and cognitive walkthroughs, a structured evaluation method used to assess how easy it is for a new user to learn and successfully use a behavioral management strategy like RUBI. We then conducted two demonstration studies, which aimed to test the feasibility, usability, and acceptability of RUBI in a real-world setting, with the purpose of showing how RUBI works in practice and generating evidence to support further refinement or broader implementation. In the Design and Build phases, we systematically and iteratively assessed, developed, and enhanced RUBI content for educational settings to create a more practical and usable intervention for schools via a series of collaborative redesign feedback sessions followed by two demonstration studies of the newly designed RUBIES intervention.

Participants

The university institutional review board approved the study. All participants provided informed consent prior to participation. The activities associated with the Discover and Design/Build phases occurred in a West Coast public school district and three Midwest public school districts in the United States. We used purposive sampling, which is widely used in qualitative research to select individuals who are especially knowledgeable about or experienced with a phenomenon of interest (Cresswell & Plano Clark, 2011), to identify participants. Following a human-centered design approach, each iteration involves new participants to enable fresh perspectives and reduce biases that could otherwise be carried over from prior contributions to the redesign process (Krueger & Casey, 2009). Elementary school principals from both locations were contacted via email, provided information about the study activities, and invited to participate and/or forward our email to their teachers and staff who work with an autistic student. Forty-one participants, including principals, general and special education teachers, paraeducators, and other school personnel who support autistic students from 28 elementary schools, were included (see Table 1). School partners varied in their roles to allow for input from unique perspectives (Krueger & Casey, 2009).

Table 1.

Demographics.

DemographicsN (%)	Study 1: InterviewsN = 8	Study 2: Demonstration Study of RUBIN = 15	Study 3: Collaborative RedesignN = 6	Study 4: Demonstration Study of RUBIESN = 12
Gender
Female	8 (100)	13 (86.7)	5 (83.3)	11 (91.7)
Male	0 (0)	1 (6.7)	1 (16.7)	1 (8.3)
Nonbinary	0 (0)	1 (6.7)	0 (0)	0 (0)
Race
White	7 (87.5)	14 (93.3)	5 (83.3)	9 (75)
Black	1 (12.5)	1 (6.7)	1 (16.7)	3 (25)
Ethnicity
Hispanic/Latino	1 (12.5)	1 (6.7)	1 (16.7)	1 (8.3)
Age M (SD)	40.86 (14.1)	44.8 (12.6)	43.5 (16.4)	40.58 (12.1)
Highest level of education
Technical, vocational, trade school	0 (0)	1 (6.70)	0 (0)	0 (0)
Some college, no degree	0 (0)	1 (6.70)	0 (0)	0 (0)
Bachelor's	1 (12.5)	3 (20)	0 (0)	3 (25)
Graduate	7 (87.5)	10 (66.7)	6 (100)	9 (75)
Years of ASD experience	9.75 (6.5)	12.53 (8.1)	10.83 (10.6)	7.75 (6.7)
Educational role
Mainstream classroom teacher	3 (37.5)	2 (12.3)	3 (50)	1 (8.33)
Special education classroom teacher	5 (62.5)	4 (26.7)	2 (33.3)	6 (50)
Principal/Vice Principal	0 (0)	3 (20)	0 (0)	0 (0)
Special education class aide/paraeducator	0 (0)	3 (20)	1 (16.7)	1 (8.3)
Other	0 (0)	3 (20)	0 (0)	4 (33.3)

The participating districts were socioeconomically and racially/ethnically diverse. In the West Coast district, approximately 22.0% of the student population qualify for the federal free and reduced meal program. The racial and ethnic breakdown of this district is: 45.4% White; 15.0% Black/African American; 13.1% Asian; 0.4% Native Hawaiian/Pacific Islander; 0.4% American Indian or Alaskan Native; 12.2% multiracial; and 13% Hispanic/Latinx. In the Midwest districts, the racial and ethnic breakdown is: 21.9% White; 61.7% Black/African American; 1.3% Asian; 0.1% Native Hawaiian/Pacific Islander; 0.1% American Indian or Alaskan Native; 6.3% multiracial; and 8.6% Hispanic/Latinx. Over 80.0% qualify for the federal free and reduced meal program.

Across iterative studies, eight participants did not show up (n = 2 for the demonstration study; n = 6 for the collaborative redesign) for the session. Our final sample included 9 (22.0%) general education teachers, 17 (41.5%) special education teachers, 3 (7.3%) principals/vice principals, 5 (12.2%) paraeducators, and 7 (17.1%) other support staff (i.e., occupational therapists, school counselor, school climate and culture specialist, speech language pathologist) from 28 elementary schools. The participants were predominantly female (90.0%) with an average age of 42.6 years and an average of 10.3 years working with autistic youth. The racial backgrounds were: 85.4% White and 14.6% Black/African American. In terms of ethnicity, 9.8% of participants identified as Hispanic/Latino. Seventy-eight percent of participants had a graduate degree, while 17.0% had a bachelor's degree (see Table 1).

Procedures

Qualitative data were gathered in three different phases of the study: (1) behavioral observation and interview, (2) demonstration study of RUBI/RUBIES, and (3) collaborative redesign session. The first activity during the Discover Phase included in-class behavioral observations followed by retrospective cognitive walkthroughs and interviews with n = 8 participants to identify the current standard practices for addressing off-task classroom behaviors, contextual constraints (e.g., classroom rules, school policies), barriers and facilitators, and targeted values and priorities of the school context and end users. The behavioral observation was conducted by the first and anchor authors—two female, PhD-level observers—who collected data on teachers’ usual care strategies to address off-task behaviors exhibited by autistic students in the classroom over the course of a 2-h observation (Newcomb & Hagopian, 2018). Off-task behaviors were defined as student engagement in any tasks other than the assigned classroom task or ongoing activity. Observers then completed an in-person retrospective cognitive walkthrough (approximately 30 min) using a “think-aloud” protocol (Kuusela & Paul, 2000), which involved having the teacher talk through their decision-making processes around responding to each documented episode of off-task behavior during the observation. Participants were offered a $50 gift card.

In the second research activity under the Discover Phase, after informed consent was obtained, participants (n = 15) received informational materials that described the purpose, structure, and content of RUBI and then subsequently engaged in one of two 4-h in-person demonstration studies designed to gather feedback on the original RUBI intervention using behavioral rehearsal, prospective think-aloud procedures, and structured assessments (Duong et al., 2020). Participants engaged in feedback discussion following the demonstration of four core segments of RUBI content (behavioral principles, prevention strategies, consequences, and teaching strategies), which included test scenarios and video vignettes to elicit RUBI's potential applicability and usability in schools. Summary feedback guided the development of the initial version of RUBIES (RUBI in Educational Settings), including potential adaptations to fit classroom and end-user identification. As an incentive, participants were offered a $200 gift card commensurate with their time.

For the first part of the Design/Build phase, participants (n = 6) attended one of three 90-min in-person collaborative redesign feedback sessions. Discussion at each session surrounded feedback around potential usability issues for each component of the original RUBI intervention (Lyon & Bruns, 2019; Nguyen et al., 2019; Xie et al., 2016), along with the generation of potential solutions, including: (1) input regarding the feasibility and appropriateness of the proposed RUBIES components; and (2) methods for supporting implementation. As an incentive, participants were offered a $75 gift card. In the second research activity under the Design/Build Phase, participants (n = 12) engaged in one of two 4-h demonstration studies (one session was held in person, one session was held via zoom) of the newly redesigned RUBIES intervention, which utilized behavioral rehearsal, prospective think-aloud procedures, and structured assessments (Duong et al., 2020) to refine intervention modifications as well as address remaining usability issues. As an incentive, participants were offered a $200 gift card commensurate with their time.

The aforementioned two female, PhD-level researchers collected all qualitative data. They had at least 5 years of qualitative research experience and expertise in the clinical care of autistic children. An additional PhD-level researcher with expertise in the clinical care of autistic children (the second author of this manuscript) was involved in data collection for the final demonstration study for the RUBIES intervention. Participants knew that one of the researchers was an original creator of RUBI, two researchers were licensed psychologists, and all three of the researchers valued the use of EBPs to support autistic youth in public schools. The remaining co-authors were not involved in data collection or initial qualitative data analysis. Researchers had no prior interactions with participants. At the start of each data collection activity, the interviewers provided the purpose of the research study. All interviews were audio recorded.

Measures

Our team developed systematic and comprehensive guides to stimulate discussion in the Discover and Design/Build phases.

Behavioral Observation and Cognitive Walkthrough

For the interviews following classroom behavioral observations, the interview guide prompted interviewers to first orient educators to the “think-aloud process” (i.e., educators were asked to walk the interviewer through what they were thinking about during a hypothetical situation). Interviewers then asked educators to think aloud through specific behavioral episodes that occurred during the observation. If no behavioral episodes occurred during the observation, educators were asked to think back to previous challenging behavioral episodes their students had previously exhibited and walk the interviewer through them. The interview guides then stipulated that the interviewer probe educators about: (1) strategies they used to support the child, (2) what may be difficult about using these strategies, (3) how their classroom team collaborates to manage behaviors, and (4) what else would be helpful to support behavior management. Lastly, educators were asked a series of questions regarding previous training on autism or behavior management and policies and procedures impacting their use of behavioral strategies in the classroom, including: (1) What might be difficult about using behavioral strategies to support children in the classroom?, (2) What would be helpful to you to support your approach to classroom behavior management?, and (3) Tell me about any policies or procedures that affect your classroom behavior management.

Demonstration Study

For the demonstration studies, interview guides provided queries for interviewers to use following the presentation of various components of the original RUBI program or the redesigned RUBIES program (e.g., How useful is information from this section? What modifications would you suggest to this section?). The interview guide then stipulated several workflow queries (i.e., What tools are currently in place to help with classroom behavior management? What additional tools would be helpful? What are active gaps in staff processes for managing behavior?) and end user queries (i.e., Who might be an appropriate end user for this program? What characteristics might the end user have?).

Collaborative Redesign

The interview guide for the collaborative redesign study contained questions about how educators thought specific portions of RUBI (i.e., Prevention Strategies, Consequences, and Teaching Skills) should be redesigned for the school context. Example questions to help redesign the Prevention strategies included: What other visual supports to prevent challenging behaviors would be helpful in schools? What format(s) of daily schedules would be best in the school environment? Who should “hold” the visual toolkit? Example questions to help redesign the Consequences strategies included: How do you currently utilize reinforcement, planned ignoring, and compliance training in schools? What are the barriers to implementing these skills in a school environment? Example questions to help redesign the Teaching Skills strategies included: How do you currently teach new skills to students at school? What are the barriers to implementing this skill in a school environment?

Data Analysis

Qualitative Data Analysis for all Aims

Responses from the interviews and feedback sessions following each of the phases were professionally transcribed. Conventional content analysis was used, in which meaning is derived from the content of verbal communications, across all qualitative data (Hsieh & Shannon, 2005). Led by the first author, the research team met as a group to develop a coding scheme using a rigorous, systematic, transparent, and iterative approach. Three members of the research team independently coded two initial transcripts to identify recurring codes. Then, they met as a group to discuss common codes to include in the final codebook, where operational definitions of each code were specified, and examples were provided of when to use and not use the code. Three coders coded all data using the agreed-upon codebook, and interrater reliability was calculated (MacPhail et al., 2015). They met together on a weekly basis to discuss, clarify, verify, and compare codes; disagreements were discussed to attain consensus. Consensus agreement between raters was excellent (percent agreement = 99% across all codes). Themes were derived and refined throughout the analytic process (Bradley et al., 2007), and data saturation was reached at the point at which no new insights were obtained, and no new themes were identified when the codebook was applied across the text segments (Guest et al., 2006; Saunders et al., 2018). Due to analyses occurring 2 years after data collection, member checking on identified themes was not performed.

Results

A total of 41 educators participated across the four study phases: 8 in behavioral observations and cognitive walkthrough, 15 in the RUBI demonstration studies, 6 in the collaborative redesign sessions, and 12 in the RUBIES demonstration studies. Systematic coding of transcripts from all phases yielded 14 codes describing both intervention usability issues (informing RUBI modifications; Bearss et al., 2022) and implementation usability issues. Six codes were specific to implementation usability, developed to ensure that the redesigned manual was appropriate for educators in public schools. These included: School Policies (policies or rules that may affect RUBIES use), Class Norms (expected behaviors in general or special education classrooms), Implementation Strategies (techniques supporting RUBIES use), Workflow (team processes that may impact implementation), Barriers and Facilitators (factors impeding or supporting implementation), and Other School Resources (existing supports, frameworks, or programs). These six codes were used to identify implementation-specific usability issues. See Table 2 for code descriptions and examples.

Table 2.

Definitions of Codes.

Theme	Description	Text Examples
School Policies	This code refers to any school policies or rules that are present and/or may affect teacher/staff use of RUBIES.	“So I actually felt strongly, that's why I wanted to say that I think these would be great for all of the students, not just a certain population of students. So, but it is sort of the system has it set up that you work with certain students that you work with.”“I was new last year, and we had to do a lot of changes to bring the interventions that had to with disruptive or violent behavior into alignment with district policies with CPI, which is a nonviolent intervention.”
Class Norms	This code refers to the rules of the special education or general education classroom, what is expected behavior in either of those settings, the structure, specific rules/rule setting, boundaries or other limits imposed on children. This code is child-focused and refers to what is expected of children to do/follow in a particular classroom.	“But it doesn't have to be taught as I think systematically to the majority of my students, it would be more for kids who do have an IEP or are struggling with some other things that way, it's not something I wouldn't say that like I'm going to use for my students on a regular basis. It might be the one or two kids who need a little extra help, but it wouldn't be something I'm going to use all the time.”“Because there are lots of kids that don't need this stuff. But for our kids in particular, we know that this can be the key to unlocking how we get them to move forward and be more successful.”
Implementation Strategies	This code refers to the use of a documented implementation strategy.	"I feel like having, it would be nice if someone could come in and like temporarily co-teach as part of this because I think it helps teachers in terms of when I'm trying to help with implementation of something…and that makes it seem more feasible for it to be applied. ”
Workflow	This code refers to the classroom team.	“I was able to get a colleague in, the special ed teacher, and she was able to observe and then track that data and then together we were able to work on interventions and that seemed more feasible. So I think using a team would be better.”
Barriers and Facilitators	This code refers to any barrier of or facilitator to implementation. Barriers may include lack of support, administrator unaware/understanding of special education. Facilitators are things that are conducive to implementation.	“I love when you have things already ready for teachers to use, sometimes that's all it takes is like, ‘Okay, it's already printed out for me. I don't have to reinvent the wheel. I just can slap a name on it and stick it on a board and try it out.’ Then they're much more willing to try things, which is great.”“And it was difficult because I am not able to focus on one child and really see what exactly was the antecedent and what exactly is a consequence. There's just too many [students] that I'm watching.”
Other School Resources	This code refers to the use of other resources such as frameworks and school-based programs used to support students.	“I think sometimes the biggest barrier is just the idea of learning something new or having to add another thing to your plate. I think always starting with like, “You're already doing most of this, just what you're going to call it or what it is just has a different…”“We are using RULER and also PBIS and how is this all connected? Then I see that why I am doing it instead of just throwing [in] another thing.”

Implementation Usability Issues

The analysis of these six codes revealed two key usability issues that educators reported might impact RUBIES fit with educators in an educational context: (1) RUBIES lack of clear integration with other school resources, and (2) Perception that data collection necessary for RUBIES is burdensome.

Usability Issue 1: RUBIES Lack of Clear Integration With Other School Resources

Throughout the Design/Build phase of the study, several educators expressed concern about how they might integrate RUBIES with other school resources they have been trained on, are already using, or are a part of their school policies and norms. Educators reported that an additive requirement creates a burden and leads to EBP fatigue, which makes it difficult to implement and integrate with other frameworks, such as the Multi-Tiered System of Supports (MTSS) used in schools. For example, one educator stated, “It's overwhelming to fit everything in. I just think some teachers might be like, this is just too much with everything going on.” Another educator emphasized, “We are using RULER and also PBIS [two school-based EBPs] and how is this all connected? Then I see that why I am doing it instead of just throwing [in] another thing.”

RUBIES is designed to be implemented in addition to other supports that are legally required through students’ individualized education programs (IEPs). Several educators confirmed this distinction and provided some suggestions and framing for how RUBIES could be conceptualized in an educational context and how it could fit in with other existing resources:

I think the differentiation [is] between PBIS being tier one and tier two and then [RUBIES] being tier three helps you parse out that we are using this for our kids that are on IEPs and we're targeting paraeducators who are working with these kids in special ed. [RUBIES] helps target special-ed students…I feel like that's a good distinction. I mean, in my own head, I was like, ‘Oh, okay. It's tier three.’

Similarly, another educator explained:

There are people who [are] like, ‘Oh my gosh, it's another program.’ I kind of think of it kind of like how I stream, what I watch online. I have Netflix, I have Hulu. I have all of those things. Yes. I could just stick with Netflix, but then I'm not going to get all the other things. And it's just acknowledging you do have to pick and choose. You have to combine different things together. So, I see [RUBIES] as… not just another program but, ‘Here's another… Here's Disney plus.’ And I can watch all my Disney shows too.

Additionally, many educators highlighted the need for the trainers or implementers of RUBIES to highlight the similarities between RUBIES and other school resources. For example, one educator states, “So, it's finding that universal language, especially for a new teacher and how to stick with it.”

Usability Issue 2: Perception That Data Collection Necessary for RUBIES is Burdensome

RUBIES is a function-based intervention, where the recommended strategies are based on the identified function of the observed behavior. Data collection around target behaviors is required to help generate hypotheses about potential functions of behaviors. Educators reported that data collection requires a lot of time and is perceived as burdensome, which often causes them to omit the practice altogether, potentially undercutting the entire RUBIES intervention. For example, one educator stated, “Also as a teacher, time, time, time. I do not have time to do pretty much anything. So that's the difficult part.” Similarly, another educator reported, “And, it was difficult because I am not able to focus on one child and really see what exactly was the antecedent and what exactly is a consequence. There are just too many [students] that I'm watching.”

Educators in the study also provided some suggestions on how to increase usability in terms of the data collection burden. The first suggestion given was to take more of a team approach to data collection in the classroom:

I was able to get a colleague in, the special ed teacher, and she was able to observe and then track that data and then together we were able to work on interventions and that seemed more feasible. So, I think using a team would be better.” Similarly, another educator suggested, “Maybe have a guest speaker at that moment, maybe the teacher can have a substitute or a peer come in for a chunk of time, just specifically 20–30 minutes of the most high frequency of behavior whatnot… would help [educators] feel more supported.

Another suggestion to address this usability issue was to make data collection more straightforward and streamlined:

Because classroom teachers are some of the busiest people on the planet. They don't have time to pee. They don't eat lunch. And then they have 20 [students], whatever it is, it has to be 100% take and bake. All the toppings have to be on the pizza and they have to be able to just put it in the oven and go.

Many educators suggested creating data collection forms that were easier to fill out, stating, “And, so in our ABC charts, instead of writing a lot out, it's a lot of just check boxes” and suggesting, “The specialized teacher could go in and try to see what some of the antecedents or consequences could be, and then write those out in a box so then the teacher just has to go and check it later on.”

An additional suggestion was to provide more clear training for educators on how to collect data more efficiently:

That's another thing I see a lot when collecting ABC data is they're writing everything out and they're exhausting themselves because they're just writing every single behavior that they see. And I'm like, no, we're only specifically looking for times that he bites, hits, or kicks, right? But maybe…in the beginning saying, ‘define clearly the only thing you're looking for’ and only take data on that at the moment.

Discussion

This study applied the DDBT framework to iteratively redesign RUBI in partnership with elementary school partners for use in school settings. Challenging behaviors in the classroom often are considered a barrier to meaningful engagement in academic and social tasks for autistic children in schools (Powell et al., 2007) and prevent autistic children from accessing their least-restricted environment (Bearss et al., 2022). While several existing school-based systems (e.g., Positive Behavioral Interventions and Supports, MTSS, and Class-Wide Function-related Intervention Teams) aim to prevent challenging behaviors and foster a supportive environment (Reinke et al., 2013; Wills et al., 2016), autistic children often have unique needs that require more individualized programs and interventions to set them up for success (Hartley et al., 2008). The current behavioral intervention standard of care for addressing challenging behaviors exhibited by autistic students involves enlisting highly trained personnel (e.g., special education teachers, behavioral consultants, and school psychologists), which might limit the availability and immediate use of interventions in classrooms. The effort to redesign the RUBI program to fit within the context of school settings and with educators as intervention providers may help to bridge the gap between service demand and availability of EBPs in the classroom and increase the likelihood of interventionist support for each autistic student. The current study highlights the value of engaging school partners in the redesign process to optimize its utility and usability in how RUBIES may be successfully implemented in schools. Content adaptations were made to what is included in the intervention materials, and this study posits the usability issues that pertain to implementation, which address how RUBIES may be delivered in school settings. Two implementation usability challenges were identified: (1) integration with other school-based systems of support and (2) data collection in schools. Implications of these findings and potential ways to ameliorate usability issues are discussed below. Potential usability solutions are discussed.

There are several EBPs that educators can use with autistic children in schools (Locke et al., 2022; Sam et al., 2021). General and special education educators often are tasked with using many EBPs, strategies, and interventions to support autistic students. Given implementation often does not occur in isolation, new innovations such as RUBIES will need to fit within existing programs and structures, and/or other supports will need to be de-implemented or pruned to make space for something new (Cook et al., 2019). Many U.S. public schools have adopted MTSS, a prevention framework that uses data to facilitate schools’ selection of appropriate EBPs aligned with student need, monitor students’ progress, and adjust service delivery based on that progress (Fuchs & Fuchs, 2006). The three tiers of prevention are: (1) Tier 1: Primary Prevention, where all members of a school receive universal supports to increase outcomes to enhance academic success; (2) Tier 2: Secondary Prevention, where some members of a school who have not been successful with Tier 1 support alone and are at an elevated risk for problems receive supplemental support to prevent more challenging behaviors; and (3) Tier 3: Tertiary Prevention, where a few members of a school at high risk or experiencing significant challenges receive individualized support to reduce severity (Nese et al., 2021). One way to address the first identified RUBIES implementation usability challenge is to leverage existing school-based implementation teams servicing students receiving more targeted and intensive supports. Given the lack of collaboration opportunities between general and special education teachers and teams to support autistic children (Al Jaffal, 2022), equipping school-based implementation teams to collaborate to understand what RUBIES entails (e.g., what behavior looks like, how training will unfold, what will be used in the classroom, etc.) may potentially address this usability challenge. Specifically, this could be done with a set of shared expectations of what can be implemented across general and special education settings, clear plans for communication, and alignment with autistic students’ IEPs. Effective implementation requires integration with existing school programs so that educators can manage multiple initiatives simultaneously.

The second potential implementation usability challenge that emerged was the use of data collection in schools. Because RUBIES uses data collection to determine the function of the autistic child's behavior to identify the most appropriate strategy to use to address that behavior, failure to engage in data collection will result in the improper execution of RUBIES. Data collection in many autism-focused EBPs often is a complex but critical component of the intervention (Pellecchia et al., 2016). Educators often report significant challenges related to data collection, including both the recording and completion of data sheets when a challenging behavior occurs and data analysis to support collaborative decision making (Marcu et al., 2013). Technology offers an opportunity to improve immediate data collection in the moment as well as data sharing and use across the general and special education teams (Marcu et al., 2013). The use of a behavioral health mobile application to enhance data collection, tracking, and sharing may potentially increase access and reach of RUBIES in public schools.

Limitations and Future Directions

The findings of this study are limited to the perspectives and experiences of educators within West Coast Public Schools and Midwest Public Schools. This geographic limitation may restrict the generalizability of the results to other regions or school contexts. Additionally, while efforts were made to include a diverse range of participants, the study predominantly focused on general education teachers, special education teachers, and paraeducators. Other valuable school partners, such as administrators, specialists, parents, and students, were not included in the study, potentially limiting the comprehensiveness of the insights obtained.

The study's scope was limited to exploring the adaptation of the RUBI intervention for school contexts. Other potentially relevant interventions or approaches within schools were not fully investigated, which might have offered additional insights into enhancing the integration of behavioral support strategies. Lastly, the study primarily focused on identifying usability issues related to the application of RUBI within schools. While these usability issues are crucial, the study did not extensively delve into potential solutions or alternative intervention designs to address the identified limitations. Future steps of this research should aim to address usability issues and test the real-world feasibility and effectiveness of RUBIES in the school context.

Conclusion

This qualitative study highlights the implementation usability challenges of RUBIES within school contexts, drawing on insights from general education teachers, special education teachers, and paraeducators. These perspectives provide valuable guidance for supporting RUBIES implementation in practice. The identified implementation usability issues emphasize the importance of tailoring interventions to the specific demands of schools while ensuring alignment with existing support systems to promote successful implementation. Future research should focus on developing strategies to streamline behavioral data collection, enhance integration with current school programs, and evaluate the feasibility of implementing the redesigned intervention on a broader scale.

Supplemental Material

sj-docx-1-irp-10.1177_26334895261448447 - Supplemental material for Identifying Implementation Usability Issues of an Evidence-Based Behavioral Intervention for Use in Schools

Supplemental material, sj-docx-1-irp-10.1177_26334895261448447 for Identifying Implementation Usability Issues of an Evidence-Based Behavioral Intervention for Use in Schools by Jill Locke, Daina Tagavi, Samantha Seaver, Yuanchen Kuo, Angel Fettig, Richard Torres and Karen Bearss in Implementation Research and Practice

Footnotes

ORCID iDs

Jill Locke

Daina Tagavi

Ethical Considerations

All procedures were approved by the Seattle Children's Hospital IRB (Study No. 00001890).

Informed Consent

This study was determined to be IRB exempt.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was funded by the National Institute of Mental Health (R34MH123598 and 2P50MH115837). The content is solely the responsibility of the authors and does not necessarily represent the view of the NIMH.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Supplemental Material

Supplemental material for this article is available online.

References

Al Jaffal

(2022). Barriers general education teachers face regarding the inclusion of students with autism. Frontiers in Psychology, 13, 873248. https://doi.org/10.3389/fpsyg.2022.873248

Aman

M. G.

McDougle

C. J.

Scahill

Handen

Arnold

L. E.

Johnson

Stigler

K. A.

Bearss

Butter

Swiezy

N. B.

Sukhodolsky

D. D.

Ramadan

Pozdol

S. L.

Nikolov

Lecavalier

Kohn

A. E.

Koenig

Hollway

J. A.

Korzekwa

Gavaletz

, …, Research Units on Pediatric Psychopharmacology Autism Network. (2009). Medication and parent training in children with pervasive developmental disorders and serious behavior problems: Results from a randomized clinical trial. Journal of the American Academy of Child and Adolescent Psychiatry, 48(12), 1143–1154. https://doi.org/10.1097/CHI.0b013e3181bfd669

American Psychiatric Association (2022). Diagnostic and statistical manual of mental disorders (5th ed., text rev.). American Psychiatric Association. https://doi.org/10.1176/appi.books.9780890425787 .

Bearss

Johnson

Smith

Lecavalier

Swiezy

Aman

McAdam

D. B.

Butter

Stillitano

Minshawi

Sukhodolsky

D. G.

Mruzek

D. W.

Turner

Neal

Hallett

Mulick

J. A.

Green

Handen

Deng

Scahill

(2015). Effect of parent training vs parent education on behavioral problems in children with autism spectrum disorder. JAMA, 313(15), 1524–1533. https://doi.org/10.1001/jama.2015.3150

Bearss

Tagavi

Lyon

A. R.

Locke

(2022). Iterative redesign of a caregiver-mediated intervention for use in educational settings. Autism: The International Journal of Research and Practice, 26(3), 666–677. https://doi.org/10.1177/13623613211066644

Bradley

E. H.

Curry

L. A.

Devers

K. J.

(2007). Qualitative data analysis for health services research: Developing taxonomy, themes, and theory. Health Services Research, 42, 1758–1772.

Brookman-Frazee

Baker-Ericzén

Stahmer

Mandell

Haine

R. A.

Hough

R. L.

(2009). Involvement of youths with autism spectrum disorders or intellectual disabilities in multiple public service systems. Journal of Mental Health Research in Intellectual Disabilities, 2(3), 201–219. https://doi.org/10.1080/19315860902741542

Brookman-Frazee

Stahmer

A. C.

(2018). Effectiveness of a multi-level implementation strategy for ASD interventions: Study protocol for two linked cluster randomized trials. Implementation Science, 13(1), 66. https://doi.org/10.1186/s13012-018-0757-2

Conroy

M. A.

Dunlap

Clarke

Alter

P. J.

(2005). A descriptive analysis of positive behavioral intervention research with young children with challenging behavior. Topics in Early Childhood Special Education, 25(3), 157–166. https://doi.org/10.1177/02711214050250030301

10.

Cook

C. R.

Lyon

A. R.

Locke

Waltz

Powell

B. J.

(2019). Adapting a compilation of implementation strategies to advance school-based implementation research and practice. Prevention Science: The Official Journal of the Society for Prevention Research, 20(6), 914–935. https://doi.org/10.1007/s11121-019-01017-1

11.

Corkum

Bryson

S. E.

Smith

I. M.

Giffin

Hume

Power

(2014). Professional development needs for educators working with children with autism spectrum disorders in inclusive school environments. Exceptionality Education International, 24(1), 1–15. https://doi.org/10.5206/eei.v24i1.7709

12.

Cresswell

J. W.

Plano Clark

V. L.

(2011). Designing and conducting mixed methods research. Sage Publications, Inc.

13.

DeFilippis

Wagner

K. D.

(2016). Treatment of autism spectrum disorder in children and adolescents. Psychopharmacology Bulletin, 46(2), 18–41. https://doi.org/10.64719/pb.4346

14.

Duong

M. T.

Cook

C. R.

Lee

Davis

C. J.

Vázquez-Colón

C. A.

Lyon

A. R.

(2020). User testing to drive the iterative development of a strategy to improve implementation of evidence-based practices in school mental health. Child and Adolescent Mental Health, 1–12.

15.

Fuchs

L. S.

(2006). Introduction to response to intervention: What, why, and how valid is it? Reading Research Quarterly, 41, 93–99. https://doi.org/10.1598/RRQ.41.1.4

16.

Giangreco

M. F.

Broer

S. M.

Edelman

S. W.

(1999). The tip of the iceberg: Determining whether paraeducator support is needed for students with disabilities in general education settings. Journal of the Association for Persons with Severe Handicaps, 24(4), 281–291. https://doi.org/10.2511/rpsd.24.4.281

17.

Giangreco

M. F.

Suter

J. C.

Doyle

M. B.

(2010). Paraeducators in inclusive schools: A review of recent research. Journal of Educational and Psychological Consultation, 20(1), 41–57. https://doi.org/10.1080/10474410903535356

18.

Guest

Bunce

Johnson

(2006). How many interviews are enough? An experiment with data saturation and variability. Field Methods, 18, 59–82.

19.

Handen

B. L.

Aman

M. G.

Arnold

L. E.

Hyman

S. L.

Tumuluru

R. V.

Lecavalier

Corbett-Dick

Pan

Hollway

J. A.

Buchan-Page

K. A.

Silverman

L. B.

Brown

N. V.

Rice

R. R.

Hellings

Mruzek

D. W.

McAuliffe-Bellin

Hurt

E. A.

Ryan

M. M.

Levato

Smith

(2015). Atomoxetine, parent training, and their combination in children with autism spectrum disorder and attention-deficit/hyperactivity disorder. Journal of the American Academy of Child & Adolescent Psychiatry, 54(11), 905–915. https://doi.org/10.1016/j.jaac.2015.08.013

20.

Hartley

S. L.

Sikora

D. M.

McCoy

(2008). Prevalence and risk factors of maladaptive behaviour in young children with autistic disorder. Journal of Intellectual Disability Research : JIDR, 52(10), 819–829. https://doi.org/10.1111/j.1365-2788.2008.01065.x

21.

Horner

Carr

Strain

Todd

Reed

(2002). Problem behavior interventions for young children. Journal of Autism and Developmental Disorders, 32, 423–446. https://doi.org/10.1023/A:1020593922901

22.

Hsieh

H.-F.

Shannon

S. E.

(2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15, 1277–1288. https://doi.org/10.1177/1049732305276687

23.

Iwata

B. A.

Worsdell

A. S.

(2005). Implications of functional analysis methodology for the design of intervention programs. Exceptionality, 13(1), 25–34. https://doi.org/10.1207/s15327035ex1301_4

24.

Krueger

R. A.

Casey

M. A.

(2009). Focus groups: A practical guide to applied science. Sage publications.

25.

Kuusela

Paul

(2000). A comparison of concurrent and retrospective verbal protocol analysis. American Journal of Psychology, 113, 387–404. https://doi.org/10.2307/1423365

26.

Locke

Hernandez

A. M.

Joshi

Hugh

M. L.

Bravo

Osuna

Pullmann

M. D.

(2022). Supporting the inclusion and retention of autistic students: Exploring teachers’ and paraeducators’ use of evidence-based practices in public elementary schools. Frontiers in Psychiatry, 13, 961219. https://doi.org/10.3389/fpsyt.2022.961219

27.

Locke

Olsen

Wideman

Downey

M. M.

Kretzmann

Kasari

Mandell

D. S.

(2015). A tangled web: The challenges of implementing an evidence-based social engagement intervention for children with autism in urban public school settings. Behavior Therapy, 46(1), 54–67. https://doi.org/10.1016/j.beth.2014.05.001

28.

Locke

Wolk

C. B.

Harker

Olsen

Shingledecker

Barg

Mandell

Beidas

(2017). Pebbles, rocks, and boulders: The implementation of a school-based social engagement intervention for children with autism. Autism: The International Journal of Research and Practice, 21(8), 985–994. https://doi.org/10.1177/1362361316664474

29.

Lyon

A. R.

Bruns

E. J.

(2019). User-centered redesign of evidence-based psychosocial interventions to enhance implementation-hospitable soil or better seeds?. JAMA Psychiatry, 76, 3–4. https://doi.org/10.1001/jamapsychiatry.2018.3060

30.

Lyon

A. R.

Cook

C. R.

Locke

Davis

Powell

B. J.

Waltz

T. J.

(2019a). Importance and feasibility of an adapted set of implementation strategies in schools. Journal of School Psychology, 76, 66–77. https://doi.org/10.1016/j.jsp.2019.07.014

31.

Lyon

A. R.

Munson

S. A.

Renn

B. N.

Atkins

D. C.

Pullmann

M. D.

Friedman

Areán

P. A.

(2019b). Use of human-centered design to improve implementation of evidence-based psychotherapies in low-resource communities: Protocol for studies applying a framework to assess usability. JMIR Research Protocols, 8(10), e14990. https://doi.org/10.2196/14990

32.

Macphail

Khoza

Abler

Ranganathan

(2015). Process guidelines for establishing intercoder reliability in qualitative studies. Qualitative Research, 16, 198–212.

33.

Marcu

Tassini

Carlson

Goodwyn

Rivkin

Schaefer

K. J.

Kiesler

(2013, April). Why do they still use paper? Understanding data collection and use in autism education. In Proceedings of the SIGCHI conference on human factors in computing systems (pp. 3177–3186).

34.

Miltenberger

(2016). Behavior modification: Principles and procedures, 6th Ed. (pp. 223–241). Cengage Learning:

35.

Nese

R. N. T.

Kittelman

Strickland-Cohen

M. K.

McIntosh

(2021). Examining teaming and tier 2 and 3 practices within a PBIS framework. Journal of Positive Behavior Interventions, 25(1), 16–27. https://doi.org/10.1177/10983007211051090

36.

Newcomb

E. T.

Hagopian

L. P.

(2018). Treatment of severe problem behaviour in children with autism spectrum disorder and intellectual disabilities. International Review of Psychiatry, 30(1), 96–109. https://doi.org/10.1080/09540261.2018.1435513

37.

Nguyen

M. H.

Van Weert

J. C. M.

Loos

E. F.

Tytgat

K. M. A. J.

Geijsen

Drenth

Janse

Smets

E. M. A.

(2019). Optimising eHealth tools for older patients: Collaborative redesign of a hospital website. European Journal of Cancer Care, 28, e12882. https://doi.org/10.1111/ecc.12882

38.

Pellecchia

Beidas

R. S.

Marcus

S. C.

Fishman

Kimberly

J. R.

Cannuscio

C. C.

Mandell

D. S.

(2016). Study protocol: Implementation of a computer-assisted intervention for autism in schools: A hybrid type II cluster randomized effectiveness-implementation trial. Implementation Science, 11, 1–11. https://doi.org/10.1186/s13012-016-0513-4

39.

Postorino

Sharp

W. G.

McCracken

C. E.

Bearss

Burrell

T. L.

Evans

A. N.

Scahill

(2017). A systematic review and meta-analysis of parent training for disruptive behavior in children with autism spectrum disorder. Clinical Child and Family Psychology Review, 20(4), 391–402. https://doi.org/10.1007/s10567-017-0237-2

40.

Powell

Fixsen

Dunlap

Smith

Fox

(2007). A synthesis of knowledge relevant to pathways of service delivery for young children with or at risk of challenging behavior. Journal of Early Intervention, 29(2), 81–106. https://doi.org/10.1177/105381510702900201

41.

Reichow

Barton

E. E.

(2014). Evidence-based psychosocial interventions for individuals with autism spectrum disorders. In Volkmar

F. R.

Rogers

S. J.

Paul

Pelphrey

K. A.

(Eds.), Handbook of autism and pervasive developmental disorders: Vol. 2. Assessment, interventions, and policy (4th ed., pp. 969–992). John Wiley & Sons.

42.

Reinke

W. M.

Herman

K. C.

Stormont

(2013). Classroom-level positive behavior supports in schools implementing SW-PBIS: Identifying areas for enhancement. Journal of Positive Behavior Interventions, 15(1), 39–50. https://doi.org/10.1177/1098300712459079

43.

Rispoli

Neely

Lang

Ganz

(2011). Training paraeducators to implement interventions for people autism spectrum disorders: A systematic review. Developmental Neurorehabilitation, 14(6), 378–388. https://doi.org/10.3109/17518423.2011.620577

44.

Sam

A. M.

Odom

S. L.

Tomaszewski

Perkins

Cox

A. W.

(2021). Employing evidence-based practices for children with autism in elementary schools. Journal of Autism and Developmental Disorders, 51(7), 2308–2323. https://doi.org/10.1007/s10803-020-04706-x

45.

Saunders

Sim

Kingstone

Baker

Waterfield

Bartlam

Burroughs

Jinks

(2018). Saturation in qualitative research: Exploring its conceptualization and operationalization. Quality & quantity, 52, 1893–1907.

46.

Shaw

K. A.

Williams

Patrick

M. E.

, & Autism and Developmental Disabilities Monitoring Network. (2025). Prevalence and early identification of autism spectrum disorder among children aged 4 and 8 years—autism and developmental disabilities monitoring network, 16 sites, United States, 2022. MMWR Surveillance Summaries, 74(2), 1–22. https://doi.org/10.15585/mmwr.ss7402a1

47.

Simpson

R. L.

de Boer-Ott

S. R.

Smith-Myles

(2003). Inclusion of learners with autism spectrum disorders in general education settings. Topics in Language Disorders, 23(2), 116–133. https://doi.org/10.1097/00011363-200304000-00005

48.

Swiezy

Stuart

Korzekwa

(2008). Bridging for success in autism: Training and collaboration across medical, educational, and community systems. Child and Adolescent Psychiatric Clinics of North America, 17(4), 907–922, xi. https://doi.org/10.1016/j.chc.2008.06.001

49.

US Department of Education (2020). 41st annual report to congress on the implementation of the individuals with disabilities education act, 2019. https://sites.ed.gov/idea/2019-annual-report-to-congress-IDEA/

50.

Wills

Kamps

Fleming

Hansen

(2016). Student and teacher outcomes of the class-wide function-related intervention team efficacy trial. Exceptional Children, 83(1), 58–76. https://doi.org/10.1177/0014402916658658

51.

Xie

Carayon

Cartmill

Cox

E. D.

Plotkin

J. A.

Kay

M. M.

(2016). Multi-stakeholder collaboration in the redesign of family-centered rounds process. Applied Ergonomics, 46, 115–123. https://doi.org/10.1016/j.apergo.2014.07.011

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.03 MB