Abstract
Implementation considerations should be integrated into the iterative development of educational innovations. The purpose of this design-based implementation research (DBIR) was to identify intervention components worthy of revision, examine the feasibility of a video manual and document barriers to the implementation of a multi-tiered dual-language instructional curriculum. Six teachers implemented the curriculum and provided feedback via structured interviews. Although four of the six teachers reported the intervention to be highly feasible, multi-tiered instruction did not align with the beliefs of two teachers. Teachers’ feedback combined with child-level narrative and vocabulary outcomes led to curriculum revisions and to the creation of additional implementation supports. Findings suggest system-level barriers influence the sustainability of multi-tiered instruction in early childhood settings.
Development of an intervention is not a quick or easy process. Almost never is the first draft of an intervention effective and feasible. Rather, it requires several informed iterations to construct an intervention that has the intended impact on child outcomes, is feasible to do within the parameters of the available resources and has a high probability of being sustained in the target setting (Diamond & Powell, 2011). Although there is a premium on intervention research that establishes evidence of efficacy, it is not wise to abbreviate the preparatory work requisite for successful implementation. Newly developed interventions should undergo careful feasibility testing before full-scale efficacy research is undertaken. Failure to plan for feedback from end users in early phases of development could result in a premature, wasteful and unsuccessful full-scale efficacy trial (Morgan et al., 2018) and an intervention that is undesirable to educators (Lieber et al., 2010).
Design-based implementation research (DBIR) is a systematic method for optimizing interventions for implementation early in the iterative process (Fishman et al., 2013; LeMahieu et al., 2017). DBIR draws from several traditions including evaluation, community-based participatory, design-based and implementation research to study an intervention while researchers are developing, testing and revising it (Fishman et al., 2013). It is especially useful for examining an intervention’s feasibility and for identifying implementation facilitators and barriers related to the capacity of the users’ system (Duda et al., 2013). Given that DBIR integrates community-based participatory research elements, it is perfectly suited to the development of culturally derived interventions for a specific setting and target population (Bernal et al., 2009).
In this paper, we describe a design-based implementation study of a multi-tiered dual-language curriculum for use in preschool classrooms. The purpose of such a curriculum is to hasten Spanish-speaking preschoolers’ acquisition of the English language while building upon the foundation of their home language. Our goal was to design the curriculum for maximum implementation potential and adequate contextual fit, not just for efficacy. Research has documented that close to 60% of reported implementation barriers are related to features of the intervention and that implementation supports aligned with specific barriers are more effective than generic supports (Long et al., 2016). Implementation supports are strategies that are deliberately designed to enhance the implementation of an intervention (Collier-Meek et al., 2019), and their creation occurs alongside the intervention’s development.
Motivation for the development of a multi-tiered dual-language intervention
In US preschools, Spanish-speaking dual-language learners (DLLs) with limited English oral academic language skills are at high risk for later reading difficulty (US Department of Education, National Center for Education Statistics, 2003). Several factors contribute to DLLs’ risk, including poverty, limited access to quality schools and cultural misalignment in US schools (Carlo et al., 2004; Castro et al., 2011). Regardless of the reasons for risk, English language skills (including vocabulary, story retell and listening comprehension) at kindergarten entry predict later English reading performance (Kieffer, 2012), and the need for early oral language promotion among young DLLs is well established (August et al., 2014; Carlo et al., 2004; Castro et al., 2011). Sustained and intensive early oral language support is recommended; otherwise, by the time DLLs are in fourth grade, the vast majority will read below grade level (McFarland et al., 2018).
It is also recommended that young Spanish-speaking DLLs receive at least some instruction in their home language (National Academies of Sciences, Engineering, and Medicine, 2017). A comprehensive bilingual education is currently not realistic for every DLL educated in the US, but supplemental Spanish language interventions have yielded positive results on the retention of children’s home language (Durán et al., 2016; Lugo-Neris et al., 2010; Méndez et al., 2015; Restrepo et al., 2010). Moreover, instructional procedures that work for most children work for DLLs (August et al., 2014), but those tailored to meet their specific needs may have a greater chance of being effective (Castro et al., 2011).
Effective practices for DLLs, including those that involve children’s home language, could potentially be embedded in multi-tiered systems of support (MTSS). In the US, and other countries, MTSS has become a popular education model in K-12 schools. The transition to this model for early childhood education is also in process (Wackerle-Hollman et al., 2021). To make this transition, however, preschool teachers need proper curriculum materials to support large-group instruction delivered to the entire class (Tier 1), small-group intervention to a subset of children showing mild/moderate risk or delays (Tier 2) and a smaller-group or individual intervention to a few children who exhibit the need for intensive instructional support (Tier 3). Importantly, interventions at Tier 2 and Tier 3 are not considered special education services, but depending on how schools, districts or programmes determine who receives supplemental or intensive intervention, children with disabilities can receive instruction and intervention at any of the conceptual tiers. Multilingual speakers may require supports beyond what is offered to other children in the classroom but not have disabilities. Therefore, they are a population of young children for whom a multi-tiered model of instruction could be particularly beneficial (Wackerle-Hollman et al., 2021).
As tiered models make their way into early childhood education (e.g., Gettinger & Stoiber, 2008; Goldstein et al., 2017; Spencer et al., 2017, 2020; Weddle et al., 2016), they promise to enhance children’s kindergarten readiness by ensuring all children receive a differentiated dose of instruction based on individual needs (Greenwood et al., 2011). While there are only descriptions of what an MTSS model for DLLs might look like (LaForett et al., 2012), research with monolingual English-speaking preschoolers shows that supplemental interventions (e.g., Tier 2) improve children’s phonemic awareness (Goldstein et al., 2017), alphabet knowledge (Olszewski et al., 2017), syntax and listening comprehension (Phillips, 2014), vocabulary (Pollard-Durodola et al., 2016) and narrative skills (Spencer et al., 2015). There is also preliminary evidence of interventions that incorporate Spanish to improve the vocabulary and narrative skills of Spanish-speaking DLLs in preschool (e.g., Méndez et al., 2015; Spencer et al., 2019). Despite the related research, Durán and Wackerle-Hollman (2019) maintain that there is an urgent need for more empirical work in this area because there are no studies examining multi-tiered interventions for DLLs specifically.
Description of iterative development
Before obtaining funding for this work, Head Start teachers and administrators identified supporting the oral language needs of the numerous Spanish-speaking children in their classrooms as one of their greatest challenges. Together, the programme administrator and researchers developed an initial blueprint for a Spanish/English curriculum (called Puente de Cuentos, or Bridge Made of Stories) for Tier 2 and Tier 3 interventions with an emphasis on vocabulary and storytelling. Upon conceptualization, Puente de Cuentos would honour children’s home-language resources and maximize their multilingual development in the spirit of broad academic success. Once funded, bilingual researchers conducted focus groups with caregivers of the target children (Spanish-speaking preschoolers) in Spanish to identify culturally relevant activities around which stories and lessons could be developed. Using caregivers’ suggestions, the research team developed an initial set of stories, 12 in Spanish and 12 in English, as well as small-group and individual lessons to go with them. Multiple phases including field tests, user feedback and lesson revisions occurred prior to the current study (Spencer et al., 2019). During these phases, Head Start teachers requested activities that could be delivered to the whole class; they reported that all their students could benefit from the content in the lessons. In this study we included a set of activities for the whole class; however, they were not the primary focus of the research because they were only in English.
The current study represents the fourth phase of the planned iterative development with a focus on implementation. We report on the refinement of Puente de Cuentos with three units. The initial set of 24 (12 in English and 12 in Spanish) stories/lessons became Unit A (see Spencer et al., 2019). Development of Unit B of the curriculum took place while teachers used the revised Unit A lessons, and Unit C was created as teachers implemented Unit B. For this study, teachers were interviewed after each unit so that we could respond to their feedback and implement revisions immediately.
Research questions
The purpose of this study was to systematically study the implementation of the multi-tiered dual-language curriculum. While we examined the impact of the intervention on child outcomes (see Supplemental Material), this was not to test its efficacy but to identify aspects of the intervention worthy of revision. The following research questions were addressed: (1) To what extent is the multi-tiered dual-language curriculum feasible in preschool classrooms? (2) To what extent do preschool teachers and teaching assistants report a video manual to be acceptable and use it as a just-in-time support? (3) To what extent do preschool teachers make modifications to the multi-tiered dual-language curriculum? (4) What aspects of the multi-tiered dual-language curriculum do preschool teachers like? (5) What barriers impede the implementation and sustainability of the multi-tiered dual-language curriculum? (6) What additional implementation supports are needed to overcome implementation barriers? (7) What aspects of the multi-tiered dual-language curriculum require revision?
Method
Setting and participants
This study was conducted in six Head Start preschool classrooms in the Southwest United States. Centre directors nominated teachers who typically had a large number of Spanish-speaking students. Teachers were informed of the study and consented.
Teachers and classrooms
When teachers consented to be in the study, they completed a demographic survey about themselves and their classrooms. The following information was captured by this survey and additional characteristics of teachers and classrooms are outlined in Table 1. Preschool-aged (3–5 years old, average = 48 months) children attended classes Monday through Thursday. Class size ranged from 14 to 20 students, and one to three children per class had a disability. Each classroom had one lead teacher and one teaching assistant (all female). All teachers reported using Creative Curriculum (Dodge et al., 2002), complemented by Teaching Strategies Gold (TSG; Heroman et al., 2010). None of the teachers reported being familiar with MTSS frameworks, and there was no evidence of such models in use. Independent and trained raters used the Classroom Assessment Scoring System (CLASS; Pianta et al., 2008) to observe each teacher between September and November of the study year.
Characteristics of treatment teachers/classrooms.
Children
The research team gathered parent permission to screen the children from Spanish-speaking homes. Children’s oral language skills in English and Spanish were assessed to determine whether supplemental (Tier 2) intervention would be necessary. The Narrative Language Measures (NLM) Listening in English and Spanish were developed and validated for the purpose of screening preschool children’s oral language skills within an MTSS framework (Spencer et al., 2023). To elicit a child’s narrative retell, examiners read a brief story and then asked children to retell that story. They audio-recorded the children’s stories and scored them at a later time for the inclusion and clarity of story grammar elements and complex sentences (see manual for specific scoring instructions). Children who displayed low English skills (scores below 8) and moderate to high Spanish skills (scores of 8 or above) on the NLM Listening qualified to receive the Tier 2 intervention (small group). Children who displayed low skills in both English and Spanish qualified to receive Tier 3 intervention (one on one). Screening yielded 23 children across the six treatment classrooms.
Research assistants
Six research assistants (RAs) were responsible for liaising between the researchers and the Head Start classrooms. This involved observing lessons and tracking fidelity of intervention, checking in with teachers at least once per week to discuss any concerns and collecting data. Three of the RAs identified as Latina and were fluent and literate in Spanish and English. RAs who spoke only English observed the English lessons and administered the English assessments during data collection.
The first author conducted rigorous training with the RAs prior to their contributions to the study. RAs read assessment manuals, watched videos of test administrations, practised administering and scoring and demonstrated fidelity of administration and reliability of scoring through a check-out process. They were also trained to use the fidelity checklists when they observed the teachers delivering the lessons. To learn the interventions, RAs practised delivering and coaching others to deliver the intervention in small groups and one on one with bilingual children who were not research participants. The first author observed each RA delivering the intervention in English to a small group of children to document that they could deliver the lessons with fidelity.
Overview and general procedures
This study was conducted across an entire school year. Before beginning the intervention, teachers and teaching assistants attended a two-hour training session and responded to a brief questionnaire about the training. Intervention began in October and finished in April. At the completion of each unit, teachers completed an intervention-usage questionnaire and provided feedback to the developers about the intervention via structured interviews. Finally, implementers were observed periodically across the entire year to document intervention fidelity.
Bilingual teacher E completed all the lessons herself. Teaching assistant turnover occurred in three of the six treatment classrooms, and as a result, two of the treatment classrooms (B and C) did not have consistent Spanish-speaking adults. Therefore, a Spanish-speaking research assistant (RA) completed the Spanish small-group and individual lessons in their classrooms. Another RA delivered the English lessons in Classroom D because the teacher was more comfortable teaching in Spanish and the classroom had only substitute teaching assistants.
Multi-tiered dual-language curriculum
Each unit of Puente de Cuentos included 12 large-group lessons in English, 12 small-group lessons in Spanish and 12 small-group lessons in English. Small-group lessons were used to deliver instruction individually (e.g., Tier 3). Each lesson featured a brief story that young children could relate to, which were selected from the list of culturally relevant themes generated by Spanish-speaking families of preschoolers. Spanish stories were original stories (not translations), yet the English and Spanish stories in corresponding lessons included conceptually similar vocabulary words (e.g., rough/áspero, leap/brincar). Two target vocabulary words were embedded in each story — one adjective (e.g., heavy/pesado, dangerous/peligroso, brave/valiente) and one verb (e.g., explore/explorar, chase/perseguir, drag/arrastrar).
Each story was depicted by a set of five illustrations that represented main parts of stories (i.e., story grammar). Illustrations were simple drawings with few colourful features. Some lessons required the use of materials (e.g., heavy and light objects) to demonstrate or reinforce target vocabulary words. Additional materials included colourful icons, each representing one of the major story grammar elements taught to the children (i.e., character, problem, feeling, action and ending). The five circular icons were printed on cardstock and trimmed to be 1.5 inches square. Interventionists used presentation books (i.e., semi-scripted lessons for teachers to follow) and a picture book to deliver the lessons. The picture book included photographic exemplars of the words taught in each lesson.
English stories were featured in the large, small and individual lessons where the goal of the lesson was to learn the target vocabulary words (adjectives and verbs) via explicit instruction and practice using the words and complex sentences during storytelling activities. Research suggests that because verbs and adjectives are less concrete and more difficult to picture, they require more explicit instruction (Hadley et al., 2016). Optional activities were included at the end of the small-group lessons for the purpose of extending the practice of target words. Spanish stories were featured in small-group and individual lessons, but not the large-group lessons.
All small-group and individual lessons consisted of the same teaching procedures and set of activities, which were scripted with instructions for the teacher and phrasing to use with the children. Importantly, teachers were not required to read the scripts word for word, and they were encouraged to deliver the lesson without relying on the scripts as soon as they felt comfortable. Instruction followed the same general steps regardless of arrangement: (1) teachers read a short story and children named the story grammar elements; (2) teachers identified the target words and provided definitions for each one; (3) children said and defined the target words; (4) children participated in a team retell (for large and small groups only), in which individual children retold a part of the story; (5) individual children took turns retelling the entire story and using the target words (small group and individual only); illustrations and icons were faded so that the last child retold the story without visual supports; (6) photos, unrelated to the story, provided multiple contexts for children as they practised using the target words in sentences.
Each lesson regardless of arrangement lasted 15 to 20 minutes, depending on the comfort and fluency of the implementer. Children who demonstrated delayed language in both Spanish and English received their intervention individually (one per classroom) and the other children participated in groups of three children (2–3 children per classroom). We encouraged teachers to deliver large-group lessons twice per week but considered them optional. Because they were the primary focus of this research, the research team ensured small-group and individual lessons were conducted four times per week, alternating Spanish and English.
Researchers provided take-home activities for teachers to give to caregivers. Each take-home activity corresponded to one of the lessons delivered at school. Arranged on a single piece of paper, each activity included a story (in English or Spanish), the illustrations and icons, and instructions for using the activities to facilitate storytelling and word learning at home.
Teacher training and video manual
Videos have been used in other MTSS research to support implementation (Sanetti & Collier-Meek, 2015), but it was unclear if video training could replace didactic training and lead to sufficiently high intervention fidelity. Nonetheless, we created a video manual to accompany the curriculum and tried it out in this study. The creation was driven by the concept of just-in-time supports, often enhanced by technology (Hardeman et al., 2019; Nahum-Shani et al., 2018), that can be accessed by the interventionist at the precise time they need them. The video manual consisted of 12 5-to-15-minute modules that covered the teaching procedures, including the following topics: (1) introduction; (2) materials; (3) behaviour rules; (4) large-group activities; (5) introduction to small groups; (6) teaching vocabulary in small groups; (7) teaching storytelling in small groups; (8) team retell in small groups; (9) individual retell in small groups; (10) prompting and correcting; (11) differentiation; and (12) one-on-one intervention.
The first author facilitated a group training meeting in which all the teachers and teaching assistants in the treatment classrooms watched the modules (only modules 1–11 due to time). In between viewing each module, teachers and teaching assistants asked clarifying questions. The total running time of the videos was 90 minutes and the total duration of the meeting was 120 minutes. All the teachers were given their curricular materials during this training meeting, which included a flash drive of the video manual. To be able to examine the impact of the video manual on fidelity, we did not provide additional training. Moreover, once intervention began, each teacher and teaching assistant (except Assistant B) received only one coaching session.
Measures and data collection
Fidelity checklists
The extent to which Head Start teachers and teaching assistants could deliver the Puente de Cuentos lessons with fidelity was documented. RAs used fidelity checklists consisting of 12 adherence items, three responsiveness items and nine quality items. Adherence items were specific to the procedures described above for delivering the lessons, responsiveness items addressed children’s engagement and success in the lesson, and quality items covered the teacher’s enthusiasm, pace, rate of praise and comfort as she delivered the lesson.
RAs observed 25% of the total number of small-group and individual intervention sessions to document the fidelity with which the lessons were delivered. The number of sessions observed varied for teachers and teaching assistants because the Ras’ schedules did not always align with the teachers’ schedules. Because large-group lessons were optional, our limited personnel resources were allocated to the small-group and individual lessons. As a result, fewer than 10% of the large-group lessons were observed for fidelity.
Video manual questionnaire
To address the acceptability of the video manual, teachers responded to a brief questionnaire immediately after viewing each module. Before the videos began, they were given a sheet of paper with two statements next to the title of each module: (1) ‘This module was understandable’; (2) ‘This module was helpful’. Teachers and teaching assistants were asked to rate each statement on a scale of 1 to 4, where 1 represented ‘strongly disagree’ and 4 represented ‘strongly agree’.
Structured interviews
Teachers were interviewed after completing each unit. Since the teachers completed the lessons according to centre schedules, it was not possible to conduct focus groups with all the teachers together. However, teachers from the same centre were interviewed together. Preplanned questions were designed to elicit specific information that helped answer research questions 3 to 6. Teachers reported on their use and modification of the lessons, what they liked about the curriculum, what barriers they experienced and what they thought would help them implement it more effectively. At the end of Unit C, the question addressing what the teachers liked about the curriculum was replaced with a question about their plan to continue using the curriculum in the coming school years; otherwise, the same questions were asked each time. Because the interviews consisted of specific questions, it was unnecessary to conduct thematic analyses.
Child-level measures
To identify which aspects of the multi-tiered dual-language instruction were less effective, we collected child-level outcomes related to retelling stories, understanding vocabulary and listening comprehension before and after each unit. Although the outcomes collected from children in the treatment classes were compared to children who did not receive the intervention, the study design was not intended to answer questions of efficacy. The purpose of collecting child-level data on proximal outcomes was purely to determine the potency of intervention components (e.g., narrative vs. vocabulary). For this reason, the embedded quasi-experimental design procedures, measures and results are only described in the Supplemental Material.
Results
Fidelity
To address research question 1, we report on the fidelity with which teachers and teaching assistants delivered the multi-tiered dual-language curriculum with fidelity. The ease of use, as documented by teachers’ adherence and quality of implementation, is a key indicator of whether it is feasible for them to do it in their classrooms given the resources available to them. Mean percent fidelity for small-group and individual lessons is displayed in Table 2, organized by teacher and teaching assistant. RAs’ lesson fidelity scores are also included in Table 2 to represent the fidelity of implementers with more training and practice than that which the teachers and teaching assistants received. The overall mean fidelity of the Head Start teachers and teaching assistants during the study was 91.6%, whereas the mean of the RAs’ fidelity was 94.9%, both of which are considered high. Not all teachers were observed during large-group lessons (n = 5) and RAs were only able to capture a few of them. The mean fidelity of large-group lessons was 94% for Teacher A, 76% for Teacher C and 91.5% for Teacher E.
Fidelity of lessons.
Note: RA = research assistant; RA 1 delivered Spanish small-group lessons in classes B and C; RA 2 delivered English small-group lessons in class D; fidelity of large-group activities was combined across five teachers.
Another aspect of feasibility is the extent to which teachers report plans for sustained use. Four teachers reported that they would continue to use the curriculum (A, D, E and F). Two teachers (B and C) reported that they would only teach the vocabulary words during child-directed activities, but their assistants expressed an interest in continuing with the large-group and small-group lessons. Teacher D reported that she would continue to use Puente de Cuentos, but she moved before we could follow up the next year.
Acceptability and use of the video manual
The second research question address the acceptability and use of the video manual, which was an intentional implementation support designed for Head Start contexts. Given that there is little time for reading manuals and that there is frequent turnover among the staff, video guidance was created as a versatile option for supporting ongoing implementation. Based on teachers’ and teaching assistants’ ratings on how understandable and helpful each module was, we calculated mean scores for each module with a range of 1 to 4. For all modules, teachers and teaching assistants reported that the modules were understandable/helpful to extremely understandable/helpful (see Table 3). Mean scores ranged from 3.43 to 4.00. Teachers and teaching assistants reported the module about establishing rules to be the most helpful and most understandable and the module about teaching retelling in small groups to be the least helpful and least understandable.
Teachers’ and teaching assistants’ feedback on video modules (means on a 1–4 scale).
We were also interested in the extent to which teachers used the video manual on their own as a just-in-time support. During Unit A structured interviews, two teachers reported that they and their teaching assistants reviewed all the modules again between the training session and when they began the lessons, which was about six weeks later. Two teachers reported that they reviewed some of the modules again. Two teachers (B and C) reported that they never reviewed the video modules on the flash drive. Only one teacher reported reviewing the video modules between Units B and C, but three teaching assistants who were hired after the training session watched all the modules before delivering lessons. During the Unit A interviews, the researcher asked if they thought the video manual was helpful now that they had put the content to use. Four teachers reported that it was extremely helpful while two teachers (B and C) reported that it was somewhat helpful.
Structured interviews
Research questions 3, 4, 5 and 6 were answered using data from the interviews. We were intentionally requesting information about how they modified the lessons to fit within their classroom routines, what they liked about the lessons, what they perceived as barriers to implementation and what type of improvements were needed to make the curriculum better or to make their implementation easier. Teachers’ answers to the preplanned interview questions are included in Table 3 of the Supplemental Material. Highlights of their responses are below.
Use of lessons
Two teachers were able to integrate the Puente de Cuentos curriculum into their routines easily (Teachers A and E). Teachers D and F encountered a number of barriers that limited their ability to implement the lessons as intended; however, they continued to enhance their implementation across the year and continued to be enthusiastic about the new curriculum and its impact on their students’ language. Teachers B and C did not deliver all the intended lessons and eventually transferred the responsibility of doing them to their assistants. Teachers were able to reduce their dependency on the scripts as they became more comfortable with the instructional formats. Likewise, most of the teachers made small adjustments to the lessons or their delivery that were appropriate for their specific circumstances. For example, Teacher E created two groups of two children instead of one group of three and having one child receive the lessons individually. Teacher A repeated lessons with one of her students because the student needed more practice with the words. A few teachers modified the lessons for children who were reticent to speak when directed. Teacher B reported simplifying the stories so that the language was no longer complex.
Lesson likes
Four (A, D, E and F) of the six teachers consistently reported that they liked the stories, lessons and the gains their children made. In particular, teachers liked that the stories were relatable, culturally and developmentally appropriate for Spanish-speaking preschoolers and that the presentation and picture books were structured in a way that made them easy to use. Teachers B and C liked the relatable stories, engaging pictures and the vocabulary words embedded in the stories, but stated philosophical objections to explicit instruction.
Barriers to implementation
Common barriers to implementing Puente de Cuentos as intended included time to plan and prepare, attempting to do lessons when the rest of the class was playing, behaviour management and finding a consistent time in the schedule for lessons. Some teachers wanted more involvement from caregivers but worried that the take-home activities were too difficult for them to use without more direction or simplified activities. Several classrooms experienced teaching assistant turnover that affected the consistency of lesson delivery.
Suggestions for improvement
Most teachers offered extremely helpful suggestions for how to improve the materials or lessons for their context. For example, several teachers requested something to simplify their planning. They suggested that because it was not part of their mandated schedules, it was difficult to find time to complete the lessons. Some teachers asked for ways to make it easier for them to integrate the target vocabulary into their routines because they were not using the optional activities in the small-group lessons. A summary of teachers’ suggestions is included in Table 3 of the Supplementary Material.
Effect of multi-tiered dual-language instruction on child outcomes
The final question addressed aspects of the curriculum, meaning narrative and/or vocabulary components, as well as Spanish and/or English components. Descriptive statistics and results of the embedded quasi-experimental study are reported in the Supplemental Materials (Tables 4 and 5). We used those results to identify the weakest components of the intervention. By exploring the effect on each outcome — narrative and vocabulary, English and Spanish — we linked the results to aspects of the curriculum according to their statistical significance and whether the effect is educationally meaningful (i.e., >.25; US Department of Education & Institute of Education Sciences, 2017). The effect on narrative retelling was significant for both English and Spanish measures, with meaningful effect sizes. This revealed that the narrative aspects of the Puente de Cuentos lessons were sufficiently potent to achieve the desired narrative outcome. For receptive picture vocabulary, there was only one statistically significant result (Spanish Unit B), suggesting that the vocabulary instruction and practice within the lessons were insufficient for children to master the target words. Nonetheless, effect sizes were meaningful for two English units and one Spanish unit. Although the effect on the listening comprehension outcome was not statistically significant, the effect size was meaningful.
Discussion
The purpose of this DBIR study was to examine the implementation of a newly developed, multi-tiered dual-language curriculum in Head Start classrooms serving Spanish-speaking preschoolers. The most important findings are related to the identification of areas of the intervention worthy of revision, implementation barriers that require tailored strategies to overcome and early childhood MTSS systemic drivers necessary for supporting the implementation of a multi-tiered curriculum.
Refinement of Puente de Cuentos
Revisions to lessons
A summary of all the changes made to the lessons is in Table 4. The most substantial change to the Puente de Cuentos curriculum took place between Units A and B. Most of the teachers reported not using the optional activities, which were designed to provide additional opportunities for children to practise the vocabulary words. Additionally, child outcomes indicated that the vocabulary components of the intervention were less effective than the narrative components. Because vocabulary researchers recommend that children should receive several opportunities to practise new words across a variety of contexts (Beck et al., 2013) and it was not happening, it was worth revising immediately. We altered the format of the large-group lessons to include four brief extension activities and removed the optional activities from the small-group lessons. Teachers reported that they were more likely to use the extensions if they could do them with all the students in their class. Including the extension activities in the large-group presentation book seemed to help teachers to do this. Moreover, we created posters of all the vocabulary words. These were large photos depicting the target words that teachers could use during circle time to review and practise the vocabulary.
Summary of changes made after Units A and C.
Although the inclusion of a stronger set of classroom extension activities could possibly increase learning of English vocabulary, a different strategy was needed for Spanish. Based on feedback teachers gave us about how caregivers were using or not using the take-home activities, we decided to make several revisions to them. This involved making them all in Spanish and putting the activities together in a booklet. These changes occurred at the end of Unit C. We completed a number of other revisions after Unit C, which included replacing a difficult word with a more relatable one, providing a box of materials so that teachers did not need to hunt for items to use, creating a bookmark to help teachers to keep their place and providing a weekly lesson plan to reduce their preparation.
Implementation supports
Training and coaching are among the most popular implementation strategies reported in the literature (Powell et al., 2015). However, given the complexity of early childhood environments and lack of resources for extensive professional development, it behooves intervention developers to be creative about what can work in under-resourced classrooms (Diamond & Powell, 2011). Our results indicate that the video manual was understandable and helpful and led to acceptable fidelity with only one coaching session per implementer. Teachers’ mean fidelity of small-group and individual lessons was comparable to that of the RAs who received more intensive and individualized training. Teachers A, E and F were observed delivering large-group lessons eight months following the end of the study. They were all able to maintain fidelity of lesson delivery above 80%. Plus, most of the teachers used the video manual as a just-in-time support when they determined support was needed or when a new assistant was hired. The implications of such high fidelity are important because of how little training and coaching the teachers received compared to other language intervention research studies (e.g., Hindman & Wasik, 2012). This strong fidelity to training ratio is likely due to the scripted nature of the lessons and contextual fit of the video manual.
One of the most common barriers that teachers identified was time. The lack of time was reported in relation to planning and having a consistent time slot in the schedule. Administrators continued to require teachers to create lesson plans using templates available through their Teaching Strategies Gold online platform (called TSG templates or TSG lesson plans), even though the Puente de Cuentos lessons were prepared and scripted for them. Their TSG lesson plans did not correspond to a multi-tiered organization of instruction, so it was challenging for teachers to integrate the two approaches. They perceived the multi-tiered language instruction to be in addition to what they were already doing, not replacing or enhancing it. Additionally, teachers did not readily see how the Puente de Cuentos lessons addressed the TSG objectives. To overcome this barrier, which is relevant to early childhood MTSS research generally, we designed weekly planning forms that did the integration for them. The weekly plans outlined each component of the Puente de Cuentos lessons within the traditional daily schedules (e.g., circle time, learning centres, motor time, etc.) and included the relevant TSG objectives.
Implications for early childhood MTSS
Through the conduct of this important DBIR, we identified a number of barriers that cannot be resolved easily with the creation of curriculum-specific implementation supports. Duda et al. (2013) taught us that an effective intervention is insufficient for sustained implementation. An enabling context is also required. Our data suggest that the intervention could not be examined in the absence of MTSS as a systemic framework. In other words, the context in which we conducted this study was not sufficiently enabling for a multi-tiered instructional system. Next, we discuss two system-level barriers and their implications.
First, because they shared their philosophical objections during the interviews, it was clear that Teachers B and C held beliefs about teacher-directed instruction that differed from those of the rest of the teachers. They were also the only non-Hispanic teachers in the group and had the highest education achievement of the teachers. Both of these characteristics may have influenced their beliefs and perceptions of the intervention; however, there is insufficient data to draw conclusions. The researchers learned that the director of centre 2 requested their participation. She was eager to be included in the research partnership and viewed teacher-directed small-group instruction as a positive addition to those classrooms. Teacher B suggested during structured interviews that the direct teaching of skills should be reserved for children in elementary school, and Teacher C reported that she preferred to let the children play. Teacher B removed complexity from the stories and reported more behaviour problems than the other teachers. Teacher C reported similar behaviour challenges, but to a lesser extent. By Unit C, the teaching assistants in both classrooms had taken over the lessons. Our results indicate that we failed to create an intervention that fit within these teachers’ contexts and that aligned with their notion of developmentally appropriate practice. Future work might benefit from gathering teachers’ beliefs and philosophies during recruitment to identify potential areas of misalignment and ensuring teachers are true volunteers and not simply complying with their director’s wishes.
Second, when we followed up with teachers eight months after the study was completed, we learned that teachers were primarily using the large-group lessons. They preferred them over the small-group and individual lessons because they wanted all their students to benefit. During the study, the emphasis was on small-group and individual interventions, but when the research team’s support was faded, there were competing pressures to deliver instruction to all the children in the class equally. Their conventional paradigm aligned with the concept of equality, where everyone receives the same thing (Takeuchi et al., 2018). This is in direct conflict with multi-tiered models of instruction that embrace a paradigm of equity, in which everyone gets what they need (The Division for Early Childhood of the Council for Exceptional Children (DEC) et al., 2013).
Overall, these findings indicate that additional implementation work is needed to prepare these environments for a system of multi-tiered supports (Metz et al., 2019; Peterson, 2013). It is possible that if the Head Start sites had already made the shift to MTSS, a new multi-tiered curriculum would have been more easily sustained. DBIR exposed the systemic barriers, and their discovery came early enough in our iterative process that we could make appropriate adaptations to the curriculum and seek or prepare enabling contexts for further research and development (Duda et al., 2013).
The importance of contextual fit and assessing a system’s readiness for change cannot be overstated (Peterson, 2013). Intervention researchers must attend to organization drivers of their intended contexts such as policies, data decision systems and facilitative administration (Metz et al., 2019), which makes intervention development work extremely complex. Additionally, research funding is often not large enough or for adequate durations to complete the slow work of exploration and preparation before installing new innovations (Metz et al., 2019). Innovative designs, such as DBIR, can empower intervention developers to intentionally create and adapt their interventions for maximum match with educational contexts. Moreover, interventions are best developed within the context of strong, long-term researcher–community partnerships (Schindler et al., 2017).
Limitations
Several weaknesses reduce the confidence in our findings — the small sample size in particular. Only six teachers provided input on the intervention and six Head Start classrooms in the Southwest do not adequately represent all Head Start classrooms. The cost of conducting such an in-depth examination of feasibility and implementation of a long intervention was too high to include more than six teachers. As a result, the generalizability of findings is reduced.
It is also likely that repeated turnover affected the consistency of intervention delivery, which may have impacted teachers’ perceptions of the intervention’s feasibility and acceptability. Even so, it would not have been an authentic test of implementation without turnover. Teacher and teaching assistant turnover in early childhood education is estimated to be 25–50% (Burton et al., 2002; Miller & Bogatova, 2009); what we experienced was within this range. In other words, turnover is a typical part of the target setting. As a possible way to counteract this problem, the video manual, as a just-in-time implementation support, did exactly what it was supposed to do. As turnover occurred naturally within the Head Start classrooms, new teaching assistants were able to watch the video manual on the flash drive and deliver the interventions with reasonable fidelity. These limitations suggest that the investigation of just-in-time implementation supports may be a fruitful avenue of research.
Conclusion
We consider the study of implementation early in the iterative development process to be an indispensable part of intervention science (Hodgson & Gitlin, 2016). DBIR was an innovative approach that allowed us to benefit from community engagement (Fishman et al., 2013) and deepen our understanding of how a multi-tiered dual-language curriculum is implemented in Head Start classrooms. While less traditional, DBIR is suitable for exploring several dimensions of implementation and intervention feasibility.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
