Iterative Pilot Testing of the POWR Intervention: Online Training to Support Paraeducator Implementation of Augmentative and Alternative Communication Strategies

Abstract

The development of communication is a fundamental part of early childhood. Yet many students with disabilities require supports such as augmentative and alternative communication to develop communication skills. Teachers and paraeducators play key roles in supporting communication for these students, but often lack effective and accessible training options that include relevant content, active engagement, and application with individualized coaching. To meet this need, we engaged in iterative development, revision, and testing of the POWR Training System, an online training and coaching program for teachers and their paraeducators. As part of a multi-year, federally funded grant, this iterative process was informed by feedback from educators and experts as well as by outcomes for paraeducators, a teacher, and children in a series of nonconcurrent single case studies. A description of the iterative process, outcomes for paraeducators, a teacher, and children as well as feedback related to the program are described. Future research directions are discussed.

Keywords

single subject design augmentative/alternative communication pre-k

Introduction

The development of communication skills is a fundamental part of early childhood and essential for academic success (Erickson & Geist, 2016). Although most children communicate primarily through speech (Dodd, 2011), many children with disabilities experience significant delays (Beukelman & Light, 2020). Approximately half of all children in early childhood special education settings require speech and language services (Morgan et al., 2016), including 12% who have complex communication needs (CCN; Binger & Light, 2006), meaning they are unable to use speech to meet their daily communication needs (Light & McNaughton, 2014). Augmentative and alternative communication (AAC) systems (e.g., picture symbols, communication apps, speech generating devices) can be used to support the development of communication for children with CCN (Beukelman & Light, 2020). However, learning to make effective use of AAC is not intuitive for children or their communication partners. Children who require AAC need the ongoing assistance of trained communication partners who support communication development by providing opportunities for communication, waiting for communication, responding to communication, and modeling AAC (Kent-Walsh et al., 2015). High quality AAC intervention to support communication partners can help achieve this goal.

Quality Intervention Development

The need for high quality intervention development has been recognized through United States federal funding agencies such as the Institute for Education Science (IES). In fact, since its inception (Education Sciences Reform, 2002) IES has consistently provided funding opportunities to develop and test new interventions (see https://ies.ed.gov/funding/grantsearch). Unfortunately, the ways in which intervention development is carried out is rarely reported in the published literature. There is a need for increased transparency in the process of intervention development particularly when public funding is provided (Hoddinott, 2015). Intervention development utilizes a systematic and iterative process with numerous steps. The National Center for Special Education Research (Buckley et al., 2014) indicates the iterative process includes “implementing, gathering feedback, revising, and implementing again” and notes the value of single case research designs as an important tool in the process. Variation exists in the literature related to how intervention development is carried out, but generally includes the following components: initial intervention development, review, refinement, and preliminary evaluation (see Kern et al., 2011; Wight et al., 2016).

Paraeducators as Critical Communication Partners

In this article, the authors describe the development of an online intervention to support paraeducators. Within educational settings, paraeducators are the most critical communication partner for children with CCN (Cole-Lade & Bailey, 2020). Federal law defines paraeducators as school employees who provide instructional support to students under the direct supervision of a teacher (Every Student Succeeds Act, 2015). Paraeducators are most often assigned to support students with disabilities (Binger et al., 2010), many of whom struggle with communication (Fisher & Pleasants, 2012). Children with CCN spend the majority of their school day interacting with adults (89% of interactions), with paraeducators as the most common communication partner (Chung et al., 2012). Despite the important role paraeducators play, interactions between children with CCN and paraeducators tend to be brief with little reciprocity (Chung et al., 2012), indicating a need for intervention to train paraeducators better support child communication.

Paraeducator Training

Although federal law requires paraeducator training (Every Student Succeeds Act, 2015) and professional standards emphasize the importance of providing training in communication skills (Council for Exceptional Children, 2022), lack of training for paraeducators remains a persistent problem (Jones et al., 2012). Challenges related to training paraeducators include: limited effectiveness of traditional training models, lack of supervision (Brock & Carter, 2013; Douglas et al., 2016), high turnover which creates a need for ongoing training (Fisher & Pleasants, 2012; Douglas et al., 2022b), lack of administrative support including limited funding, and logistical challenges (e.g., release time, coordination of coaching and supervision; Douglas et al., 2022a; 2022b). There are also unique challenges related to training paraeducators to support children with CCN. Many educational organizations lack personnel with expertise specific to children with CCN and may have only a handful of paraeducators who have training needs related to supporting these children (Douglas et al., 2016). Thus, the development of training programs of this nature may be a low priority or an area in which support is needed.

Online Training

Online training is one delivery method that can help address many of the aforementioned challenges and meet the needs of paraeducators as adult learners. Online training offers a flexible learning environment that can actively engage learners and meet specific needs across wide geographic areas, so training is accessible to paraeducators who support children with CCN in various educational organizations. Online training may likely be the wave of the future because online programs can be standardized, accessible, and utilized at any time – which addresses the need to train new paraeducators throughout the year. As indicated in a recent review of online asynchronous trainings for paraeducators supporting children with disabilities, these programs became especially common during the COVID-19 pandemic (Knowles et al., 2022), yet few address content relevant to supporting children who use AAC. Although not an exhaustive look at all available online training programs (commercial training programs were not reviewed), the number of programs highlights growing interest in online training for paraeducators.

Current best practice recommendations suggest that paraeducator training be based on existing professional standards (Council for Exceptional Children, 2022), and developed with adult learning theory in mind (Douglas & Uitto, 2021), such as the use of self-reflection to help paraeducators identify new skills (Bagawan et al., 2023). To ensure best practice for online paraeducator training, programs should include: (a) relevant content, (b) active engagement by paraeducators, and (c) application of skills to the classroom, in addition to being easy to access and use (Knowles et al., 2022; Dunst et al., 2010; Martin & Alborz, 2014). Relevant content within training programs is not just critical for supporting targeted skills and knowledge, but also because paraeducators will be more motivated to engage in the training if the content is applicable to their immediate needs (Martin & Alborz, 2014). Active engagement is also essential within online training programs (Dunst et al., 2010) and can be supported through the use of embedded activities, discussion questions, assessments, or requiring clicks within the online training program to continue to engage with training materials (Douglas et al., 2013; Layden et al., 2018). Online training should also prepare the paraeducator to implement learned skills in the classroom setting (Knowles et al., 2022). This includes materials for teachers to support the coaching and supervision of paraeducators as they implement learned skills in the classroom (Brock & Carter, 2013). Online materials should also be accessible and easy for paraeducators to use (Knowles et al., 2022) because materials that are cumbersome and offer a poor user experience will result in limited completion and application.

Existing Research

Although literature has looked at paraeducator training to support child communication (see Gregori et al., 2022), limited research has explored the efficacy of intervention packages that provide training for paraeducators who support young children who use AAC. However, training packages that utilize a strategy instruction approach outlined in the communication partner instructional model (Kent-Walsh & McNaughton, 2005) appear to be most promising based on literature reviews specific to paraeducators (Douglas, 2012) and other communication partners (Kent-Walsh et al., 2015). Key components of this paraeducator training model are: (a) a descriptive overview of the strategy, (b) modeling of the strategy, (c) rehearsal of the strategy steps, (d) practice implementing the strategy, and (e) feedback on strategy implementation.

Since the aforementioned literature reviews, a number of research projects have subsequently been conducted where communication partner training was successfully carried out using the communication partner instructional model (Kent-Walsh & McNaughton, 2005) with paraeducators in person (Douglas et al., 2013) and online (Douglas et al., 2014). Past training for paraeducators has also included common content designed to support young children who use AAC. For example, content has included: the importance of providing communication opportunities (i.e., asking questions, providing choices, commenting), waiting for the child’s communication, modeling AAC, and responding to the child’s communication (Binger et al., 2010; Douglas et al., 2013; 2014). Although results from previous studies have shown increased implementation of communication partner strategies by paraeducators, and improvements in child communication, variability in paraeducator and child performance was noted. This illuminates the need for further development and refinement of specific components (e.g., coaching) of the training to ensure implementation fidelity and maximize child outcomes.

The following primary research question guided the evaluation of the POWR Training System on paraeducator outcomes: Is there a functional relationship between the POWR Training System and the frequency of paraeducator communication behaviors (i.e., POWR: opportunities for communication, responses to communication; POWR+: AAC modeling)? Two secondary research questions were also posed to determine outcomes for child and teacher participants: Is there a functional relationship between paraeducator implementation of the POWR intervention on the frequency of child communication?; Does the POWR Training System support supervising teachers in providing high quality coaching (90% or above) to paraeducators related to their implementation of the POWR strategies? Paraeducator and teacher perceptions of the POWR Training System were also evaluated to determine the social validity of the intervention and inform adjustments to the intervention.

Method

This research was carried out as part of a large-scale, multi-year, federally funded grant project designed to refine and expand the original POWR intervention into a fully online training program with paraeducator and teacher components (Douglas et al., 2013; 2014). The iterative process was guided by the work of Wainer et al. (2018) and included three steps: (a) initial intervention development, (b) expert/educator panel review and revision of materials, and (c) evaluation and adjustment of the POWR Training System; the focus of this article is on the final step. All research was approved through the federal funder and university institutional review board. Following the (a) initial intervention development and (b) expert/educator panel review and revision of materials, iterative testing of the POWR Training System occurred using a series of nonconcurrent A-B-C single case design studies (i.e., baseline, intervention which consisted of training and post training, maintenance; Slocum et al., 2022) to measure how paraeducators implemented trained skills. This design was purposefully selected so the research team could engage in iterative revisions after each paraeducator completed training, and make adjustments to teacher supervision and coaching materials based on feedback and participant performance. Participation was staggered so changes could be made to training before the next participant began. Feedback about training was obtained from paraeducators and teachers using qualitative interviews.

The iterative testing of the online POWR Training System was staggered and took place over the course of three school years with 7 separate paraeducators who supported preschool children who used AAC. All participants who engaged in the evaluation process completed consent (paraeducators, teachers, parental consent for children). Data were collected during natural classroom activities with different levels of structure, as such, some variability in paraeducator and child behaviors was expected.

Participants

Seven paraeducators, one teacher, and seven children participated in the study. Five paraeducators and the children they supported (Dyads A-E) engaged in testing to refine paraeducator training materials. Two paraeducators, their supervising teacher, and the children they supported (Dyads F & G) took part in refinement of materials for teacher training and supervision to paraeducators. Full details about participants are provided in Tables 1 and 2.

Table 1.

Paraeducator Participant Demographics.

Paraeducator	Gender	Age	Race/Ethnicity	Education Level	Years of Experience
Paraeducator	Gender	Age	Race/Ethnicity	Education Level	Supporting children with Disabilities	In Current Setting
Paraeducator A	F	52	White	Bachelor’s degree	4 months	4 months
Paraeducator B	F	24	White	Bachelor’s degree	2 years	2 years
Paraeducator C	M	36	Black	Bachelor’s degree	7 years	5 years
Paraeducator D	F	59	White	High school diploma	14 years	14 years
Paraeducator E	F	48	White	Some college	10 years	1 year
Paraeducator F	F	62	White	Associate’s degree	15 years	7 years
Paraeducator G	F	35	White	High school diploma	3 years	3 years

Table 2.

Child Participant Demographics.

Child	Gender	Age (Year; Month)	Disability	Race/Ethnicity	Description of AAC system(s)
Child A	M	3; 6	DiGeorge syndrome, speech delay	White	Paper-based board with boardmaker symbols and 42 core words; LAMP words for life unity symbols on iPad with access to 7-8 activity pages (4-15 symbols per page)
Child B	M	4; 0	ASD	Hispanic	Paper-based picture exchanges up to 12 boardmaker symbols specific to the activity; paper-based board with 12 boardmaker symbols specific to the activity
Child C	F	3; 3	ASD, developmental delay	Biracial (black, white)	Voice recorded SGD with 8 boardmaker symbols available per activity
Child D	M	4; 2	ASD, developmental delay	White	Paper-based board with boardmaker symbols – 42 core words and 15 pages of fringe vocabulary
Child E	M	4; 2	ASD	Hispanic/Asian	Paper-based board with boardmaker symbols – 42 core words and 15 pages of fringe vocabulary
Child F	M	3:4	ASD	White	Paper-based picture symbols - approximately 20 symbols available each session, usually 2-4 presented at a time
Child G	F	5; 0	ASD	White	Paper-based picture symbols - approximately 20 symbols available each session, usually 2-4 presented at a time

Notes. *AAC systems are listed as defined at the start of the study; M = male; F = female; ASD = autism spectrum disorders; SGD = speech generating device; LAMP = Language Acquisition through Motor Planning; ELL = English language learner.

In line with iterative testing, data collection for the first five paraeducators (A-E), was staggered and took place for 4–5 months for each paraeducator depending on school closures, illness, paraeducator performance, and maintenance data collection. Data collection for the last two paraeducators (F & G), took place over 3–5 months. Because the COVID-19 pandemic occurred during parts of the project, some data collection was interrupted due to school closures or student/staff quarantine. Such interruptions, if applicable, are illustrated within the figures.

Materials

The POWR Training System consisted of online training materials for both paraeducators and supervising teachers. The POWR strategy steps included: Prepare the activity and AAC, Offer opportunities for communication, Wait for communication, Respond to communication, and ± Model AAC (see Figure 1 for details about the strategy and its components). The online training for paraeducators included two training sections, Paraeducator POWR (approximately 1.75 hr to complete; POWR steps) and Paraeducator POWR+ (approximately 1 hr to complete; + step). The online teacher training, completed by the teacher for Paraeducator F and G, included two separate sections: Teacher POWR Part 1 which focused on the POWR/POWR + strategies for paraeducators (approximately 1 hr to complete) and Teacher POWR Part 2 which focused on coaching paraeducator implementation of the POWR/POWR + strategies (approximately .75 hr to complete) and was completed prior to coaching the paraeducator.

Figure 1.

POWR/POWR + strategy components.

The online training materials for both paraeducators and teachers were developed using a strategy instruction approach (Kent-Walsh & McNaughton, 2005). This included: (a) a descriptive overview of the strategy (i.e., POWR/POWR+), (b) modeling of the strategy (e.g., video models), (c) rehearsal of the strategy steps (e.g., naming and describing strategy steps), (d) practice implementing the strategy (e.g., during naturally occurring opportunities in the classroom), and (e) feedback on strategy implementation (e.g., provided by the teacher during coaching sessions).

Trainings were divided into small sections to allow flexibility in training completion and included interactive materials such as engaging animations describing training steps, video examples of communication partners implementing each step, as well as learning, reflection, and application activities. A wide variety of AAC systems were illustrated in video models to ensure relevance for paraeducator and teacher participants who support children with various needs and AAC systems (e.g., sign, mobile communication apps, core boards). Paraeducators and teachers were provided with a packet of printed materials to support online training completion. These materials were also available in the online training system for participants. The packet included login information for the online training, images illustrating the login process and navigation of the training website, completion time for each section of training, and copies of materials for instructional and application activities.

Measures

The primary dependent variables were related to paraeducator behavior. Specifically, the number paraeducator communication opportunities, paraeducator responses to child communication, and AAC models provided by paraeducators were measured. Changes in the number of communication opportunities and responses were expected after POWR training, and changes in AAC models were expected after POWR + training which emphasized modeling. Although the primary focus of the study was to measure changes in paraeducator performance after completion of the POWR and POWR + training programs, the team also wanted to understand how implementation impacted child communication. As such, child communication turns were coded as a secondary measure. Although immediate change for children was hoped for, it was anticipated that it might take children time to adjust to the new interaction style of the paraeducators. Evaluation of the training materials also included qualitative interviews to gain feedback from paraeducators and teachers. Interviews occurred at the end of studies and were recorded for later transcription and analysis. Participants were asked to note the benefits, challenges, impact, and suggested changes to the training.

Coding for all dependent variables (i.e., communication opportunities provided by paraeducators, responses to child communication, AAC models provided by paraeducators, and child communication) was done using behavioral definitions provided to coders (a copy of the coding manual is available from the first author). Coding was completed using the Datavyu software program (Datavyu Team, 2014) which provides timestamps marking all coded behaviors. Prior to coding, all research assistants were trained until they demonstrated 90% interobserver agreement (IOA) for all dependent variables. IOA was calculated point by point for each behavior (start time within 1 s) across each dyad and was calculated by taking the number of agreements divided by the number of agreements plus disagreements and multiplying by 100. A minimum of 30% of sessions were randomly selected across dyads and conditions for IOA coding. Paraeducator communication opportunity IOA averaged 96% across all dyads (range = 84–100%). Paraeducator responses to child communication IOA averaged 95% across all dyads (range = 81–100%). Paraeducator AAC model IOA averaged 96% across all dyads (range = 60–100%). Child communication turn IOA averaged 93% across all dyads (range = 57–100%). Any coding disagreements were discussed until resolved.

Procedural fidelity was measured across all conditions and sessions of the study and training fidelity was measured within the online training system for all online trainings that were completed. In baseline and maintenance procedural fidelity included eight steps, for training and post training this included nine steps. Procedural fidelity across sessions was 100% for baseline, 99% for training (range = 88–100%), 100% for post training, and 100% for maintenance. The lower procedural fidelity a single missing step in two separate training sessions (i.e., researcher forgot to ask the paraeducator if they had questions, forgot to provide a prompt to use POWR at the beginning of the session). Fidelity of training completion was 100% for all participants.

Data Analysis

Data were graphed and visually analyzed for level, trend, variability, immediacy of effect, and overlap (Kratochwill et al., 2010). To compliment visual inspection and serve as a measure of intervention impact on paraeducator and child behaviors Tau-U, an effect size index that estimates nonoverlap between conditions and can control for baseline trend (Parker et al., 2011), was also calculated using www.singlecaseresearch.org. Baseline trend was controlled for when baseline slope was over 0.40 (Parker et al., 2011). Values were calculated for each participant, and then for the entire intervention to provide an overall aggregate of intervention effect. Tau-U values ranged from −1 to 1 with positive scores indicating desired improvement. Values below 0.19 indicated a small effect, 0.2 to 0.59 indicated a moderate effect, 0.60 to 0.79 indicated a large effect, and above 0.80 indicated a very large effect (Vannest & Ninci, 2015). For Teams F and G teacher coaching fidelity was also analyzed to determine if high fidelity was met (i.e., 90% fidelity). Interviews of paraeducators and teachers were transcribed and data were analyzed using a content analysis approach (Patton, 2015). Themes that were derived from this process informed material revision and provided social validity evidence.

Procedures

Each dyad completed three study conditions: baseline, training, and post training. When possible, a maintenance condition (i.e., Dyads A, B, C, and F) was also included. Maintenance data could not be collected for some dyads due to the end of the school year, school closures, or paraeducator position changes (see Figures 1, 2, and 3). Data collection of paraeducator-child interactions were staggered and collected during 10-minute sessions via video across 3–5 months to measure the impact of the POWR Training System on paraeducator and child behaviors. The interactions took place in a nearby room or quiet area of the classroom. Activities were pre-planned by paraeducators and designed to be motivating to the child (e.g., puzzles, games, toys, books). The child’s AAC system was provided during each session. A total of 13–20 sessions were observed and analyzed for each dyad.

Figure 2.

Paraeducator and child behaviors without teacher training and coaching.

Figure 3.

Paraeducator and child behaviors with teacher training and coaching provided.

Baseline

During baseline paraeducators and children engaged in normal interactions in the classroom. The only guidance provided was that the activity should be motivating to the child. No instruction, feedback, or coaching was provided to paraeducators during baseline.

Training

Prior to the training condition paraeducators completed the online Paraeducator POWR training modules. For Paraeducator F and G, the supervising teacher also completed the Teacher POWR training modules. During the training condition of the study, paraeducators engaged in paraeducator-child interactions with an activity planned that was motivating to the child. No instruction, feedback, or coaching was provided to Paraeducators A-E during this condition of the study. Paraeducators F and G were provided with teacher coaching after each paraeducator-child session. Teacher coaching focused only on components of the POWR strategy.

Post Training

Prior to the post training condition of the study paraeducators completed the online Paraeducator POWR + training modules. During the post training condition, they engaged in paraeducator-child interactions with an activity planned that was motivating to the child. No instruction, feedback, or coaching was provided to paraeducators A-E during this part of the study. Paraeducators F and G were provided with teacher coaching after each paraeducator-child session during this condition, which focused on components of the POWR + strategy.

Maintenance

During maintenance paraeducators and children engaged in normal interactions. No guidance, instruction, feedback, or coaching was provided to paraeducators during maintenance. Maintenance data were collected every 2 weeks after the completion of the post training condition. Up to five maintenance data points were collected (2, 4, 6, 8, 10 weeks).

Results

Results from the iterative testing of the POWR Training System are provided next including findings from nonconcurrent single case studies (Slocum et al., 2022) and interviews with paraeducators and teacher that informed social validity and intervention adjustment.

Nonconcurrent Single Case Design Studies

In each evaluation of the training materials, benefits were noted from completion and application of the POWR training system for paraeducators, children, and teachers (see Figures 2 and 3; Tables 3–6). For paraeducators, the frequency and types of communication opportunities, responses to child communication, and AAC models were measured. For children the frequency of child communication turns was measured. When teacher coaching was provided teacher coaching fidelity was measured.

Table 3.

Frequency of Paraeducator Behaviors.

Paraeducator	Behavior	Baseline		Training		Post Training		Maintenance
Paraeducator	Behavior	M	SD	M	SD	M	SD	M	SD
Communication opportunities
Paraeducator A (No teacher coaching)		85	17.1	86	16.9	102	13.8	89	14.0
Paraeducator B (No teacher coaching)		68	11.5	64	4.8	69	12.1	79	6.3
Paraeducator C (No teacher coaching)		59	5.5	67	6.6	61	3.2	64	3.5
Paraeducator D (No teacher coaching)		66	13.8	63	8.4	54	4.9	*
Paraeducator E (No teacher coaching)		51	18.9	74	13.7	*		*
Paraeducator F (Teacher coaching)		52	5.8	48	7.4	47	2.6	45	5.5
Paraeducator G (Teacher coaching)		55	3.7	60	3.4	59	4.9	*
Responses
Paraeducator A (No teacher coaching)		48	13.9	49	16.6	82	25.1	63	21.0
Paraeducator B (No teacher coaching)		11	5.9	9	3.0	28	15.9	45	8.6
Paraeducator C (No teacher coaching)		8	6.2	13	5.7	17	7.0	15	6.4
Paraeducator D (No teacher coaching)		47	12.0	53	10.3	40	16.7	*
Paraeducator E (No teacher coaching)		43	17.3	71	19.8	*		*
Paraeducator F (Teacher coaching)		6	6.0	25	8.8	8	4.1	7	4.7
Paraeducator G (Teacher coaching)		11	4.9	16	1.3	14	3.1	*
AAC models
Paraeducator A (No teacher coaching)		1	1.4	5	6.5	19	9.7	12	4.2
Paraeducator B (No teacher coaching)		1	1.3	1	1.0	15	8.0	16	7.8
Paraeducator C (No teacher coaching)		1	1.8	2	2.9	2	2.0	2	0.7
Paraeducator D (No teacher coaching)		9	8.5	29	8.8	57	32.0	*
Paraeducator E (No teacher coaching)		19	10.6	28	8.0	*		*
Paraeducator F (Teacher coaching)		12	9.7	30	20.3	59	5.7	48	7.0
Paraeducator G (Teacher coaching)		11	9.4	31	5.1	41	4.9	*

Note. *Missing data in the table is due to school COVID-19 closures which shortened the originally scheduled data collection period. These closures varied across districts.

Table 4.

Frequency of Child Communication Turns.

Child	Baseline		Training		Post Training		Maintenance
Child	M	SD	M	SD	M	SD	M	SD
Child A	56	18.7	54	18.7	90	26.3	71	17.4
Child B	13	7.0	9	3.0	29	16.7	46	8.7
Child C	9	6.0	13	6.4	18	7.8	16	7.8
Child D	83	13.7	70	7.3	50	4.8	*
Child E	50	21.3	78	20.9	*		*
Child F	10	5.4	28	8.4	14	7.7	10	3.5
Child G	18	2.6	25	4.1	23	2.1	*

Note. *Missing data in the table is due to school COVID closures which shortened the originally scheduled data collection period. These closures varied across districts.

Table 6.

Tau-U Results.

BehaviorParticipant	Tau-U	p-Value	90% CI	Degree of Effect
Paraeducator Communication Opportunities (comparison between baseline and training)
A	−.04	.92	−.67 to .59	None
B	−.30	.46	−.97 to .37	None
C	.96	.01	.33 to 1.0	Large
D	−.24	.53	−.87 to .39	None
E	.07	.83	−.45 to .59	None
F	.93	.10	.33 to 1	Large
G	.81	.06	.1 to 1	Moderate
All paraeducators	.30	.03	.07 to .54	Small
Paraeducators with teacher coaching	.88	<.01	.42 to 1	Moderate
Paraeducator Response to Child Communication (comparison between baseline and training)
A	−.08	.83	−.71 to .55	None
B	−.30	.46	−.97 to .37	None
C	.72	.06	.09 to 1	Moderate
D	.36	.35	−.27 to .99	Small
E	0	1	−.36 to .36	None
F	−.47	.20	−1 to .13	None
G	.56	.19	−.15 to 1	Small
All paraeducators	.76	.56	−.14 to .29	None
Paraeducators with teacher coaching	−.04	.89	−.50 to .42	None
Paraeducator AAC Models* (comparison between baseline and post training)
A	1	.01	.37 to 1	Very large
B	1	.01	.40 to 1	Very large
C	.52	.17	−.11 to 1	Small
D	1	.01	.33 to 1	Very large
F	.80	.05	.13 to 1	Moderate
G	1	.03	.23 to 1	Very large
All paraeducators	.88	.00	.61 to 1	Moderate
Paraeducators with teacher coaching	.89	.00	.38 to 1	Large
Child Communication Turns* (comparison between baseline and post training)
A	.76	.47	.13 to 1	Large
B	.47	.20	−.13 to 1	Small
C	1.04	.01	.41 to 1	Very large
D	−1	.01	−1 to .33	None
F	.30	.46	−.37 to .97	Small
G	.92	.05	.14 to 1	Large
All paraeducators	.42	.01	.18 to .65	Small
Paraeducators with teacher coaching	.56	.07	.12 to 1	Small

Note. *Tau-U calculation missing for Dyad E due to school COVID closures.

Relationship Between POWR Training System and Paraeducator Behaviors

In baseline, the level of communication opportunities provided by paraeducators to children varied among dyads, but aligned with the child’s communication level. Trend was relatively stable or descending for most paraeducators with limited variability, except Paraeducators A, D, and E. Paraeducator E showed an ascending trend in baseline, but baseline extended much longer than other participants. After baseline, variability reduced for communication opportunities provided by most paraeducators. Levels (see Table 3) increased for five paraeducators (A, B, C, E, and G) and decreased for two (D and F). Overlap to baseline occurred for all paraeducators and immediacy of effect was only noted for two paraeducators (i.e., C and E) although effects did not remain at the same level. Tau-U results comparing paraeducator communication opportunities from baseline to training (when communication opportunities were introduced) indicated no effect for Paraeducators A, B, D, or E, a moderate effect for Paraeducator G, and a large effect for Paraeducator C and F (see Table 6 for Tau-U results). Paraeducators who received teacher coaching had better outcomes.

In baseline, the level of responses to child communication closely mirrored opportunities, as well as child communication (see Figures 2 and 3), showing that paraeducators were generally responsive to child communication even during baseline. Paraeducator responses also showed level increases from baseline for all paraeducators (see Table 3), although the condition in which this occurred varied and in some cases the effect did not last (Paraeducator D and F). Overlap to baseline occurred for all paraeducators and immediacy of effect was only noted for Paraeducator E although this effect did not maintain at the same level. Tau-U results comparing paraeducator responses to child communication from baseline to training (when responses were introduced) indicated no effect for Paraeducators A, B, E, or F, small effect for Paraeducators D and G, and moderate effect for Paraeducator C (see Table 6).

In baseline, AAC models provided by paraeducators to children were low and relatively stable for all paraeducators (see Figures 2 and 3). The level of AAC models increased in post training compared to baseline for all paraeducators who completed post training, although the effect was lowest for Paraeducator C. While some paraeducators showed an increase in AAC models after POWR training (perhaps because some video examples illustrated this technique even though it was not explained, discussed, or part of the POWR strategy), most did not show notable growth in AAC models, or stable use of AAC models until after POWR + training which is consistent with what was taught within POWR+. Overlap to baseline was noted for two paraeducators (C and E) and immediacy of effect from baseline was noted for all but one paraeducator (C). Tau-U results comparing paraeducator AAC models from baseline to post training (when AAC models were introduced) indicated a small effect for Paraeducator C, moderate effect for Paraeducator F, and very large effect for A, B, D, and G. Tau-U could not be calculated for Paraeducator E because post training data were not collected due to COVID-19.

Relationship Between POWR Training System and Child Behaviors

In baseline child communication varied. The trend for child communication was relatively stable or descending for most children with noted variability for three children (A, D, and E). Child E showed an ascending trend in baseline, but baseline extended much longer than other participants. In most cases clear increases in child communication occurred after observed changes in paraeducator behavior, especially when paraeducators modeled AAC more often (see Dyad B data in Figure 2). Variability reduced for most children after baseline. Level changes (see Table 3) were observed for all but Child D after baseline. Overlap to baseline occurred for all children. Immediacy of effect for children was not clearly demonstrated, but was also not expected as it often takes children time to adjust to new expectations from their partner (see Douglas et al., 2018). Tau-U results comparing child communication from baseline to post training (when AAC models were introduced) indicated no effect for Child D, a small effect for Child B and F, a large effect for Child A and G, and a very large effect for Child C. Tau-U could not be calculated for Child E because post training data were not collected due to COVID-19.

POWR Training System and Teacher Behaviors

The teacher who engaged in paraeducator coaching with two separate paraeducators implemented coaching with high fidelity. Table 5 provides details of average coaching fidelity which was above 90% overall. The most common coaching fidelity steps missed included starting the session with praise, writing down goals at the end of the session (they were discussed, but sometimes not written down), and providing corrective feedback for the Wait step. The most often missed step during POWR + coaching was completing the POWR + rating before the session. Frequently, the teacher did this during the coaching session instead.

Table 5.

Coaching Fidelity by Teachers to Paraeducators.

Paraeducator/Teacher Team	POWR		POWR+
Paraeducator/Teacher Team	Mean, %	Range	Mean, %	Range
Team F	95	88–100	92	88–96
Team G	93	85–96	92	84–96

Note. The most often missed step during POWR coaching was starting the session with praise, and recording goals at the end of the session (at times the team ran out of time in their schedule to record the goals on the sheet, but they were discussed). The most often missed step during POWR+ was completing the POWR + rating before the session, instead teachers often did this as part of the coaching session.

Social Validity of the POWR Training System

Within interviews, paraeducators and teachers noted benefits and challenges, provided suggestions for improving the training, and indicated that they would recommend the training to other educational teams. Overall, feedback was positive with benefits noted for the teachers, paraeducators, and children. Paraeducator E noted the use of AAC with other children in the classroom. Teacher D noted that after training, Paraeducator D was “more comfortable using the AAC book.” Benefits were also noted for children. Paraeducator A indicated: “[Child A] became more comfortable with the AAC device.”

Similarly, teachers and paraeducators who engaged in training and coaching highlighted the value of coaching sessions. Paraeducator F indicated: “Coaching is very important because [Teacher F] made me feel like I was getting there.” Teachers and paraeducators also highlighted that improved teamwork was a valuable outcome of coaching. Teacher G noted:

I can validate their efforts for trying something new! I get to see what they did and how the child responded to them. Coaching helps with bonding and comradery. I validate what they do and this builds our relationship and teamwork in the class.

Despite the benefits of the training and coaching, some paraeducators noted challenges. For example, some paraeducators supported children who engaged in challenging behavior and found it difficult to implement the strategy when they needed to address challenging behaviors. Paraeducator F worked with a child who frequently left his seat and attempted to elope from the classroom: “we tried to do the session in the class and other students wanted to participate because it is fun. Child F got distracted though so a quiet spot to do communication with him is important.” Paraeducator G also supported a student who engaged in challenging behavior, but still saw benefits from implementing the POWR strategies: “[Child G] is not getting as frustrated as she usually would because she can communicate herself now.” Difficulty learning to wait for the child’s communication was noted even though teachers and paraeducators agreed on its importance. Paraeducator B said: “the training helped improve wait skills.”

Participants also provided important suggestions for improving training. This included providing more video models of the strategy steps, getting feedback from teachers on their performance and implementation of the POWR/POWR + strategy (provided from paraeducators in the Paraeducator POWR Evaluation), and adding frequently asked questions with answers in the online training website. Benefits of these adjustments to training were noted by subsequent teachers and paraeducators who completed training with these improvements. Teachers also provided suggestions for improving the coaching process, which focused on streamlining the process and including only one coaching form for teachers to complete during coaching sessions.

All participants indicated they would recommend the training to other paraeducators and teachers. Paraeducator E said the training would help other paraeducators “use the AAC with their students more and make them become more familiar. It makes you use it.” Paraeducator C’s advice to other paraeducators was simple: “take the training.” Paraeducator G indicated: “I feel comfortable maybe even teaching families how to use the communication system and some of these strategies.” Participants who viewed final materials were most positive about the training and excited to implement what they learned within their classrooms. During the interview, one teacher exclaimed, “I am all fired up now!” and she listed the various ways she would be implementing the POWR/POWR + strategies in her classroom (e.g., lunch table, circle, playsets). Another teacher said, “I enjoyed it! We want to share POWR with all our parapros!”

Discussion

The POWR Training System provides a fully online training program for teachers and paraeducators to support children with CCN. The training is built off best practice knowledge of online training (Knowles et al., 2022), adult learning theory (Dunst et al., 2010), coaching (Brock & Carter, 2013) and communication partner instruction (Kent-Walsh & McNaughton, 2005; Kent-Walsh et al., 2015). Findings from the development of this training program indicate it is effective and socially valid.

Additions to and Connections with the Literature

Data from the development of this project build on existing research (Douglas et al., 2013; 2014) and show adequate implementation of POWR/POWR + strategies by paraeducators and ways in which the intervention should be refined. Although numerous adjustments were made during iterative testing, these adjustments focused on instructional design rather than the overall content. After training the most consistent increase in paraeducator behavior was for AAC models, a common intervention target in the literature (Kent-Walsh et al., 2015). The level increase for AAC models from baseline to post training was 22 additional AAC models (range 1–48). However, it is important to note that the increase in use of AAC models varied among paraeducators. This highlights the variance in AAC system type (lower tech AAC systems often corresponded with fewer AAC models), the need for teacher coaching (Paraeducator C appears to be one who would have benefited from teacher coaching, something which was noted in the follow up interview), and the benefits of the revisions that were made (e.g., additional video models, the addition of coaching). Although variable, an increase in frequency of paraeducator responses to child communication was noted. This is critical because responsive partners are key to children with CCN learning communication skills (see Fäldt et al., 2020).

Although not the primary focus of this study, an improvement in child communication was also noted when paraeducators implemented the POWR Training System. To further explore the impact of the types of communication opportunities provided by paraeducators the team conducted a follow up study that helped us learn that children communicated more quickly after a choice, question, or AAC model, and least likely after a comment (Sun et al., 2023). These findings are not surprising given that all child participants were emerging communicators and choices, questions, and AAC models have a clearer expectation for a child response.

Findings also show that teachers were able to coach paraeducators with fidelity, a finding noted in related research (Gregori et al., 2022). Coaching for Paraeducator F and G showed high rates of fidelity (POWR M = 94%, range 85–100%; POWR + M = 92%, range 84–96%). Based on interview feedback from teachers related to coaching, coaching materials were also streamlined (e.g., use of only one form during coaching). The team continues to explore the fidelity and impact of teacher coaching in future research using the POWR Training System.

Finally, initial evidence supports that the POWR Training System is socially valid. Feedback of the training content and format by educator and expert panels (Step 2 of the project) was also designed to assess social validity of the POWR Training System. Results from participating teachers and paraeducators confirm that they found the POWR Training System to be effective and acceptable. Participants said they would recommend the training to other teachers and paraeducators who support children with CCN (Douglas et al., 2022c). Additional social validity measures are planned in future work including measurement of the goals, procedures, and outcomes by various stakeholders (Wolf, 1978).

Limitations and Implications for Research and Practice

Although this study provided important insights into the development of the POWR Training System, there were some limitations. The research team tested and revised the POWR Training System; however, the POWR Training System itself was continually revised to improve paraeducator and child outcomes which makes it difficult to generalize the findings from this phase of the project. The need to make iterative revisions also required the use of a series of nonconcurrent A-B-C single case design studies, which are generally considered less rigorous than concurrent single case design because of the lack of demonstrated effect across participants (Slocum et al., 2022). Additionally, due to the ongoing impact of the COVID-19, limited testing occurred for teacher coaching materials. Follow up feedback was sought to account for this lack of data from 4 additional teachers and 1 paraeducator. These individuals completed the finalized version of training and an interview to gain feedback and explore social validity of the training. Their responses indicated the intervention was ready for final testing.

Research is underway to test the finalized version of the POWR Training System using concurrent single case studies across paraeducator/child dyads with teacher-paraeducator teams who have not yet engaged in the POWR Training System. Data gathered will provide additional information about teacher coaching and demonstrate the overall impact of the POWR Training System on teacher coaching to paraeducators, paraeducator implementation of the POWR/POWR + strategies, and child communication. At the end of the evaluation phase, the training will be broadly available for use by teacher-paraeducator teams and important implications about use in school settings will be available. Given the early stage of development of the intervention, we encourage other research teams to examine the effects of the POWR Training System in varied contexts using the What Works Clearinghouse Standards (Kratochwill et al., 2010) to provide further evidence of its effect outside of this team and project.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclose receipt of the following financial support for the research, authorship, and/or publication of this article: This work was supported by the United States Department of Education grant R324A180122.

ORCID iD

Sarah N. Douglas

Author Biography

Sarah N. Douglas, PhD is an Associate Professor at Michigan State University. Her research focuses on autism, developmental disabilities, paraeducators, and interventions and technologies to support social and communication skill development for children with disabilities. This work has included AAC interventions for parents, siblings, and paraeducators using in person and online formats.

Ryan Bowles, PhD is an Associate Professor at Michigan State University. As an applied quantitative psychologist Dr. Bowles has expertise in research design and statistical methods in measurement and longitudinal analyses. His research focuses on early childhood language and literacy development for young children with and without disabilities.

Joshua Plavnick, PhD BCBA-D is an Associate Professor at Michigan State University. His research focuses on the use of behavioral interventions to support young children with autism spectrum disorder.

Sarah M. Dunkel-Jackson, PhD BCBA-D is a project director at Michigan State University. Her research focuses on autism, developmental disabilities, staff training, and youth sports.

Tiantian Sun is a doctoral candidate at Michigan State University. Her reserch focuses on measuring communication skills for children with disabilities and culturally responsive communication interventions.

Atikah Bagawan is a doctoral candidate at Michigan State University. Her research focuses on supporting families of individuals with disabilities as well as training and development for professionals and culturally responsive interventions.

References

Bagawan

Douglas

S. N.

Gerde

H. K.

(2023). Components of effective supervision and training for paraeducators. Intervention in School and Clinic, 58(4), 264–272. https://doi.org/10.1177/10534512221093778

Beukelman

Light

(2020). Augmentative and alternative communication: Supporting children and adults with complex communication needs (5th ed.). Brookes.

Binger

Kent-Walsh

Ewing

Taylor

(2010). Teaching educational assistants to facilitate multi-symbol message productions of young students who require AAC. American Journal of Speech Language Pathology, 19(2), 108–120. https://doi.org/10.1044/1058-0360(2010/09-0014)

Binger

Light

J. C.

(2006). Demographics of preschoolers who require AAC. Language, Speech and Hearing Services in Schools, 37(3), 200–208. https://doi.org/10.1044/0161-1461(2006/022

Brock

M. E.

Carter

E. W.

(2013). A systematic review of paraprofessional-delivered educational practices to improve outcomes for students with intellectual and developmental disabilities. Research and Practice for Persons with Severe Disabilities, 38(4), 211–221. https://doi.org/10.1177/154079691303800401

Buckley

J. A.

Speece

D. L.

McLaughlin

J. E.

(2014). The role of single-case designs in supporting rigorous intervention development and evaluation at the Institute of Education Sciences. In Kratochwill

T. R.

Levin

J. R.

(Eds.), Single-case intervention research: Methodological and statistical advances (pp. 283–296). American Psychological Association. https://doi.org/10.1037/14376-010

Chung

Y. C.

Carter

E. W.

Sisco

L. G.

(2012). Social interactions of students with disabilities who use augmentative and alternative communication in inclusive classrooms. American Journal on Intellectual and Developmental Disabilities, 117(5), 349–367. https://doi.org/10.1352/1944-7558-117.5.349

Cole-Lade

G. M.

Bailey

L. E.

(2020). Examining the role of paraeducators when supporting children with complex communication needs: A multiple case study. Teacher Education and Special Education, 43(2), 144–161. https://doi.org/10.1177/0888406419852778

Council for Exceptional Children (2022). Core competencies for special education paraeducators. https://exceptionalchildren.org/paraeducators

10.

Datavyu Team (2014). Datavyu: A video coding tool. Databrary Project. http://datavyu.org

11.

Dodd

(2011). Differentiating speech delay from disorder: Does it matter? Topics in Language Disorders, 31(2), 96–111. https://doi.org/10.1097/TLD.0b013e318217b66a

12.

Douglas

S. N.

(2012). Teaching paraeducators to support the communication of individuals who use augmentative and alternative communication: A literature review. Current Issues in Education, 15(1). http://cie.asu.edu/ojs/index.php/cieatasu/article/view/906

13.

Douglas

S. N.

Bowles

Kammes

(2022a). Elementary principals’ views on the policies and practices of paraeducators in special education. Journal of the American Academy of Special Education Professionals. http://aasep.org/aasep-publications/journal-of-the-american-academy-of-special-education-professionals-jaasep/jaasep-winter-2022/index.html

14.

Douglas

S. N.

Bowles

Kammes

(2022b). Paraeducators: An important member of the educational team for students with disabilities. Journal of the American Academy of Special Education Professionals. http://aasep.org/aasep-publications/journal-of-the-american-academy-of-special-education-professionals-jaasep/jaasep-springsummer-2022/index.html

15.

Douglas

S. N.

Bowles

Plavnick

Sun

Bagawan

Dunkel-Jackson

(2022c, January). Development and testing of an online training for paraeducators and teachers to support young children who use augmentative and alternative communication. [Conference presentation]. In: 2022 International Conference on Autism, Intellectual Disability & Developmental Disabilities. Clearwater Beach, FL, United States.

16.

Douglas

Chapin

Nolan

(2016). Special education teachers’ experiences supporting and supervising paraeducators: Implications for special and general education settings. Teacher Education and Special Education, 39(1), 60–74. https://doi.org/10.1177/0888406415616443

17.

Douglas

S. N.

Kammes

Nordquist

(2018). Online communication training for parents of children with autism spectrum disorders. Communication Disorders Quarterly, 39(3), 415–425. https://doi.org/10.1177/1525740117727491

18.

Douglas

S. N.

Light

J. C.

McNaughton

D. B.

(2013). Teaching paraeducators to support the communication of young children with complex communication needs. Topics in Early Childhood Special Education, 33(2), 27–37. https://doi.org/10.1177/0271121412467074

19.

Douglas

S. N.

McNaughton

D. B.

Light

J. C.

(2014). Online training for paraeducators to support the communication of young children. Journal of Early Intervention, 35(3), 223–242. https://doi.org/10.1177/1053815114526782

20.

Douglas

S. N.

Uitto

(2021). A collaborative approach to paraeducator training. Beyond Behavior, 30(1), 4–13. https://doi.org/10.1177/1074295621997177

21.

Dunst

C. J.

Trivette

C. M.

Hamby

D. W.

(2010). Meta-analysis of the effectiveness of four adult learning methods and strategies. International Journal of Continuing Education and Lifelong Learning, 3(1), 91–112.

22.

Education Sciences Reform

Act of § 2002, 20 U.S.C. § 9501.

23.

Erickson

K. A.

Geist

L. A.

(2016). The profiles of students with significant cognitive disabilities and complex communication needs. Augmentative and Alternative Communication, 32(3), 187–197. https://doi.org/10.1080/07434618.2016.1213312

24.

Every Student Succeeds Act . 2015. 20 U.S.C. § 1177.

25.

Fäldt

Fabian

Thunberg

Lucas

(2020). All of a sudden we noticed a difference at home too”: Parents’ perception of a parent-focused early communication and AAC intervention for toddlers. Augmentative and Alternative Communication, 36(3), 143–154. https://doi.org/10.1080/07434618.2020.1811757

26.

Fisher

Pleasants

S. L.

(2012). Roles, responsibilities, and concerns of paraeducators: Findings from a statewide survey. Remedial and Special Education, 33(5), 287–297. https://doi.org/10.1177/0741932510397762

27.

Gregori

Rispoli

Lory

Kim

David

(2022). Effects of teachers as coaches for paraprofessionals implementing functional communication training. Journal of Positive Behavior Interventions, 24(2), 133–144. https://doi.org/10.1177/1098300720983538

28.

Hoddinott

(2015). A new era for intervention development studies. Pilot and Feasibility Studies, 1(1), 1–4. https://doi.org/10.1186/s40814-015-0032-0.

29.

Jones

C. R.

Ratcliff

N. J.

Sheehan

Hunt

G. H.

(2012). An analysis of teachers’ and paraeducators’ roles and responsibilities with implications for professional development. Early Childhood Education Journal, 40(1), 19–24. https://doi.org/10.1007/s10643-011-0487-4

30.

Kent-Walsh

McNaughton

(2005). Communication partner instruction in AAC: Present practice and future directions. Augmentative and Alternative Communication, 21(3), 195–204. https://doi.org/10.1080/07434610400006646

31.

Kent-Walsh

Murza

K. A.

Malani

M. D.

Binger

(2015). Effects of communication partner instruction on the communication of individuals using AAC: A meta-analysis. Augmentative and Alternative Communication, 31(4), 271–284. https://doi.org/10.3109/07434618.2015.1052153

32.

Kern

Evans

S. W.

Lewis

T. J.

(2011). Description of an iterative process for intervention development. Education and Treatment of Children, 34(4), 593–617. https://doi.org/10.1353/etc.2011.0037

33.

Knowles

D’Agostino

Kunze

Uitto

Douglas

S. N.

(2022). A systematic review of asynchronous online learning opportunities for paraeducators. Journal of Special Education, 56(3), 168–178. https://doi.org/10.1177/00224669221085306

34.

Kratochwill

Hitchcock

Horner

Levin

Odom

Rindskopf

Shadish

(2010). Single-case designs technical documentation. What Works Clearinghouse.

35.

Layden

Hendricks

Inge

Sima

Erickson

Avellone

Wehman

(2018). Providing online professional development for paraprofessionals serving those with ASD: Evaluating a statewide initiative. Journal of Vocational Rehabilitation, 48(2), 285–294. https://doi.org/10.3233/JVR-180932

36.

Light

McNaughton

(2014). Communicative competence for individuals who require augmentative and alternative communication: A new definition for a new era of communication? Augmentative and Alternative Communication, 30(1), 1–18. https://doi.org/10.3109/07434618.2014.885080

37.

Martin

Alborz

(2014). Supporting the education of pupils with profound intellectual and multiple disabilities: The views of teaching assistants regarding their own learning and development needs. British Journal of Special Education, 41(3), 309–327. https://doi.org/10.1111/1467-8578.12070

38.

Morgan

P. L.

Hammer

C. S.

Farkas

Hillemeier

M. M.

Maczuga

Cook

Morano

(2016). Who receives speech/language services by 5 years of age in the United States? American Journal of Speech-Language Pathology, 25(2), 183–199. https://doi.org/10.1044/2015_AJSLP-14-0201

39.

Parker

R. I.

Vannest

K. J.

Davis

J. L.

Sauber

S. B.

(2011). Combining nonoverlap and trend for single-case research: Tau-U. Behavior Therapy, 42(2), 284–299. https://doi.org/10.1016/j.beth.2010.08.006

40.

Patton

M. Q.

(2015). Qualitative evaluation and research methods. Sage.

41.

Slocum

Pinkelman

Joslyn

P. R.

Nichols

(2022). Revisiting an analysis of threats to internal validity in multiple baseline designs. Perspective on Behavior Science, 45(3), 619–638. https://doi.org/10.1007/s40614-022-00326-1

42.

Sun

Bowles

Douglas

S. N.

Plavnick

(2023). Response time of young children with complex communication needs following a communication opportunity. Exceptional Children, 001440292211465. https://doi.org/10.1177/00144029221146574

43.

Vannest

K. J.

Ninci

(2015). Evaluating intervention effects in single-case research designs. Journal of Counseling & Development, 93(4), 403–411. https://doi.org/10.1002/jcad.12038

44.

Wainer

A. L.

Dvortcsak

Ingersoll

(2018). Designing for dissemination: The utility of the deployment-focused model of intervention development and testing for parent- mediated intervention. In Handbook of parent-implemented interventions for very young children with autism (pp. 425–440). Springer.

45.

Wight

Wimbush

Jepson

Doi

(2016). Six steps in quality intervention development (6SQuID). Journal of Epidemiology Community Health, 70(5), 520–525. https://doi.org/10.1136/jech-2015-205952

46.

Wolf

M. M.

(1978). Social validity: The case for subjective measurement or how applied behavior analysis is finding its heart. Journal of Applied Behavior Analysis, 11(2), 203–214. https://doi.org/10.1901/jaba.1978.11-203