Sage Journals: Discover world-class research

Abstract

Rationale/Background: The Alert Program® (AP) is widely used to support children with self-regulation challenges, yet inconsistent implementation may affect outcomes. A standardized fidelity measure is needed to ensure consistent delivery across research and practice settings. Objectives: To develop and validate a fidelity measure for assessing AP implementation for children with self-regulation needs. Methodology: Using an implementation framework, we conducted a narrative literature review following SANRA guidelines to identify structural, process, and content elements essential to AP delivery. A preliminary fidelity measure underwent expert review, revision, and inter-rater reliability assessment. Results: The literature review identified 14 studies with varying coverage of fidelity elements. Expert review (n = 4) demonstrated 93.4% agreement on core elements. Inter-rater reliability assessment showed 91% agreement between independent raters. Conclusion: The developed fidelity measure provides a foundation for research using the AP and offers practitioners a method to ensure implementation consistency in supporting children with self-regulation challenges.

Plain Language Summary

Ensuring Consistent Implementation of the Alert Program® Through Research

We developed a measure to verify that the Alert Program® for self-regulation is being delivered as intended by its developers. Through reviewing existing research, analyzing program materials, and consulting experts, we identified the essential components that should be present when using the program. Expert review showed strong agreement on the core elements, and testing demonstrated that different evaluators could use the measure consistently. This fidelity measure will help ensure that children reliably receive the program across different settings.

Keywords

self-regulation fidelity

Introduction

The occupational therapy profession emphasizes developing evidence-based practices that are effective, collaborative, and accessible for clients (American Occupational Therapy Association, 2017). To achieve this objective, implementing fidelity measures is essential as they assess whether program delivery adheres to the developers’ intended design. Such measures are vital for producing dependable evidence regarding the intervention’s efficacy (Song et al., 2010). While evidence-based practice has received significant attention in autism research (Steinbrenner et al., 2020), fidelity measures for interventions addressing core occupational therapy needs remain scarce. This gap is particularly problematic for occupational therapy interventions commonly sought by caregivers of autistic children; differences in sensory reactivity and self-regulation fall into this category (Pfeiffer et al., 2017).

Self-regulation is a complex concept encompassing the ability to regulate behaviors, arousal or attention levels, and emotions (Blackwell et al., 2014). Self-regulation challenges can significantly impact quality of life and impose severe limitations, as individuals lack effective strategies to manage their emotional and arousal responses (Martini et al., 2016). The inability to self-regulate has far-reaching consequences on a child’s engagement in various occupations, including academic pursuits and social interactions (Gill et al., 2018; Mac Cobb et al., 2014a). In the long term, these difficulties may lead to social isolation, as the child may struggle with underdeveloped social-emotional behaviors and experience challenges with self-esteem (Wells et al., 2012). Thus, addressing self-regulation challenges is a critical area of focus for occupational therapists.

Children with self-regulation challenges, including autistic children, often exhibit behavioral difficulties with self-control, frustration, socialization, and academic tasks (Barnes et al., 2008). Research indicates that 80%–90% of autistic people experience some degree of sensory reactivity differences (Ben-Sasson et al., 2009; Philpott-Robinson et al., 2016; Schaaf et al., 2012). These sensory processing differences involve challenges regulating behavioral responses to sensory inputs, manifesting as sensory over-responsivity, sensory under-responsivity, and/or sensory seeking (Miller et al., 2007). Such sensory differences are closely linked to the behavioral challenges of poor self-regulation (Miller et al., 2007), thereby exacerbating day-to-day challenges in learning environments where social interactions become increasingly complex (Wells et al., 2012). Occupational engagement in learning is greatly impacted by sensory differences, which are also linked with emotional dysregulation and anxiety (Lidstone et al., 2014).

Occupational therapists employ diverse interventions to enhance self-regulation skills, acknowledging each individual’s unique needs and strengths. The Alert Program® (AP) is one approach commonly used in clinical and research contexts (Williams & Shellenberger, 1996). The AP has several distinctive features that differentiate it from other self-regulation interventions. It utilizes a car engine metaphor that many children find relatable. The program features a sequential structure with progressive stages (further described in Methods). The AP is commonly implemented in clinical settings, potentially indicating practical utility. In addition, its emphasis on sensorimotor strategies aligns with occupational therapy’s holistic approach to intervention, which addresses physical, cognitive, and emotional aspects of functioning (Nash et al., 2015). The AP was specifically developed to assist children in developing social strategies and increasing their awareness of arousal through sensory-based coping interventions. Through program participation, children learn to identify and modulate their arousal levels, optimize their daily performance, and adapt to environmental requirements (Nash et al., 2015).

Despite the AP’s widespread use in clinical practice, a significant gap exists in the literature regarding standardized fidelity measures for this intervention. The AP has been utilized in various research contexts, with studies examining its effectiveness across different populations and settings. Research has investigated the program’s impact on children with autism spectrum disorder (Barnes et al., 2008), learning disabilities (Wells et al., 2012), and emotional regulation challenges (Nash et al., 2015). These studies have generally reported positive outcomes in improving self-regulation skills, though methodological approaches and implementation methods have varied considerably across investigations. This absence impacts both research and clinical practice in occupational therapy in several ways. First, without a fidelity measure, researchers cannot verify if the AP is being implemented as intended, potentially compromising the validity of outcome studies. Second, clinicians lack a structured framework to guide their implementation, which may lead to drift from the program’s core principles. Third, the absence of a standardized measure impedes comparison across studies, limiting the development of a robust evidence base for the AP.

Developing a fidelity instrument is vital for improving the validity and rigor of intervention research (Borrelli et al., 2005). A fidelity instrument for the AP would guide implementation and ensure consistent use of the program. Moncher and Prinz (1991) provided comprehensive recommendations for promoting and verifying fidelity within intervention studies, addressing critical elements such as treatment definitions, implementor training and characteristics, supervision, verification methods, and data reporting (see Supplemental Table 1). These recommendations extend beyond developing a fidelity measure by including guidelines for implementing and monitoring interventions effectively. By incorporating these established principles, researchers can ensure that interventions are implemented consistently and faithfully, allowing for better comparison and replication of results across studies.

Therefore, the primary objective of this study was to develop a fidelity instrument that can effectively assess the fidelity and adherence of occupational therapists when delivering the AP to children.

Method

Before beginning our fidelity measure development, it was important to understand the core structure of the AP. The AP consists of three progressive stages (Williams & Shellenberger, 1996). The first stage focuses on helping children learn the language and labeling associated with their engine levels. Using the metaphor of a car engine, the AP teaches children how to recognize, modify, and regulate their arousal states (Blackwell et al., 2014). This metaphorical approach helps children understand and conceptualize their internal arousal states in a relatable way. The second stage encourages children to employ various sensorimotor strategies to modulate their engine levels effectively. Finally, in the third stage, children are supported in developing personalized strategies to facilitate engagement in meaningful occupations and reflecting on their progress (Gill et al., 2018). This stage empowers children to recognize their emotions and attention state and develop the ability to adapt their arousal states to suit different situations (Mac Cobb et al., 2014a).

Study Design: We utilized a three-step implementation framework for establishing fidelity criteria and measurement as outlined by Mowbray et al. (2003):

Identification and specification of fidelity criteria,

Development of methods to quantify the fidelity criteria, and

Assessment of the reliability and validity of the fidelity criteria.

A final fourth step was implemented to establish inter-rater reliability. While inter-rater reliability could conceptually fit within Step 3, it was separated because Step 3 focused on establishing content validity through expert consensus, whereas Step 4 specifically tested the measure’s reliability in practical application with multiple raters.

Step 1: Identifying the Fidelity Criteria

Two concurrent activities comprised Step 1: conducting a narrative literature review and analyzing the AP intervention manual. While the AP manual contains comprehensive implementation details, the literature review was essential to understand how researchers and clinicians adapted the program across different contexts and populations. This dual approach ensured the fidelity measure would capture both the developer’s intentions and real-world implementation practices.

The narrative review was conducted following the Scale for the Assessment of Narrative Review Articles (SANRA) guidelines (Baethge et al., 2019) to ensure methodological quality. These guidelines address six key criteria: justification of the review’s importance, clear articulation of aims, comprehensive description of literature search methods, proper article referencing, scientific reasoning in interpretation, and appropriate presentation of relevant data.

For the narrative review, key search terms (self regul*, Alert program, and children) were used in major databases: CINAHL, Medline, PsychInfo, ERIC, and in hand-searching selected occupational therapy journals. Literature was searched through February 2018 with inclusion criteria: (a) published in English, (b) peer-reviewed, and (c) pertaining to the implementation of the AP intervention. All study designs that contained implementation details were included. Duplicate studies identified across different databases were screened and included only once in the final analysis.

Studies were reviewed independently by two reviewers, with the lead author reviewing all studies and each co-reviewed by another author (AEL/SJL/KPR). In cases of disagreement, a fourth author was consulted. Data extraction initially focused on structure and process elements. However, early in the review, a third element, content, was added (Figure 1). This content element was part of the theoretical framework recommended by Mowbray et al. (2003) but not explicitly included in the initial extraction plan. Upon reviewing the literature and AP manual, it became apparent that content elements (specific AP activities, language, and techniques) were essential for comprehensive fidelity assessment. This addition ensured alignment with established fidelity frameworks.

Figure 1.

Breakdown of Core Elements for Narrative Review of Literature.

All authors reviewed the AP intervention manual, and three authors completed the online AP training modules to verify core elements as intended by its developers.

Step 2: Development and Expert Review of the Fidelity Measure

Based on Step 1 outcomes, a preliminary fidelity measure was constructed using a systematic process. The identified elements were organized into three categories (structural, process, and content). For each element, specific criteria with operational definitions were created based on both the AP manual and literature. These were then formatted into a structured checklist with a rating system to assess implementation quality. This preliminary measure incorporated both essential program elements (e.g., use of engine terminology) and implementation variables (e.g., session length, environmental setup) identified in research studies.

Four experts from Australia and the United States reviewed the measure, focusing on three key areas: assessment of each item’s essentiality, evaluation of terminology alignment, and identification of missing elements. Experts were specifically asked to review:

The comprehensiveness of structural elements (session format, length, therapist qualifications)

The appropriateness of process elements (therapeutic alliance, participant engagement)

The accuracy of content elements (AP-specific components and activities)

The clarity and usability of the measure for research and clinical applications.

Step 3: Revision of the Fidelity Measure

In Step 3, the Mowbray framework (Mowbray et al., 2003) was adapted to prioritize content validity through expert consensus before formal reliability testing. This approach ensured the measure’s content accurately reflected AP core components before assessing implementation consistency.

Step 4: Inter-Rater Reliability

Step 4 specifically tested the practical reliability of the refined measure when used by different evaluators. Establishing strong inter-rater reliability is critical for any fidelity measure that will be used across different research teams or clinical settings to ensure consistent evaluation of program implementation.

Two independent raters (HB and EM) with AP training assessed 15 randomly selected intervention videos from a larger study (one session per child). Each rater independently evaluated whether core structural, process, and content elements were present in each session. Percent agreement was calculated to assess inter-rater reliability, which provides a direct measure of consistency between raters.

Results

Step 1: Identifying the Fidelity Criteria

The narrative literature review identified 59 studies, with the screening process depicted in Figure 2. Following duplicate removal and full-text review against inclusion criteria by the first author, 14 studies were included: Barnes et al. (2008); Blackwell et al. (2014); Gill et al. (2018); Mac Cobb et al. (2014a, 2014b); Mah and Doherty (2021); Nash et al. (2015); Pfeiffer et al. (2011); Soh et al. (2015); Soler et al. (2019); Wagner et al. (2017, 2020a, 2020b); and Wells et al. (2012). Data extraction was guided by a standardized form developed to capture the three key elements of fidelity (structural, process, and content) and their subcomponents.

Figure 2.

Reporting Items for Narrative Review (Adapted from Page et al., 2021).

Analysis of the included studies against the elements identified in Step 1 revealed varying levels of fidelity element coverage. Table 1 presents a comprehensive breakdown of how frequently each fidelity element was reported across the reviewed studies. Of the 14 studies reviewed, one study was a critical review, and one study was a systematic review and therefore are not included in the data presented in Table 1.

Table 1.

Fidelity Elements Reported in Included Studies.

Reported elements	Number of studies reported
Structural elements
Number of recipients	12
Number of interveners	11
Length of sessions	11
Professional training of therapist	10
Specific environmental features	11
Process elements
Therapeutic alliance	6
Participant responsiveness	8
Treatment dosage	12
Content elements
Essence of program included, i.e., engine/arousal levels	11
Sensory activities included (play elements)	10
Evidence of therapist’s observation of sensorimotor strategies and child’s preference	7

Note. Of the 14 studies reviewed, one study was a critical review, and one study was a systematic review and therefore is not included in the above data.

A detailed analysis of these elements revealed that two studies (Blackwell et al., 2014; Wagner et al., 2017) included all structural, process, and content elements as outlined earlier. Four studies included two of three complete fidelity elements, with Soler et al. (2019) omitting structural elements, Nash et al. (2015) and Wells et al. (2012) omitting process elements, and Wagner et al. (2020a) omitting content elements. All other remaining studies only included one complete fidelity element.

Structural Elements

Seven studies included all structural elements (Blackwell et al., 2014; Mac Cobb et al., 2014a; Nash et al., 2015; Soh et al., 2015; Wagner et al., 2017, 2020a; Wells et al., 2012). Two studies omitted the therapists’ professional training (Mac Cobb et al., 2014a; Soler et al., 2019), one did not report on specific environmental features (Mah & Doherty, 2021), and one did not discuss the length of intervention sessions (Barnes et al., 2008). Across all remaining studies, intervention sessions ranged from 10 to 90 minutes.

Process Elements

Five studies included all process elements (Blackwell et al., 2014; Soler et al., 2019; Wagner et al., 2017, 2020a, 2020b). Six studies did not report on the therapeutic alliance (Barnes et al., 2008; Mac Cobb et al., 2014a, 2014b; Mah & Doherty, 2021; Soh et al., 2015; Wells et al., 2012) and five did not state participant responsiveness (Mac Cobb et al., 2014b; Nash et al., 2015; Pfeiffer et al., 2011; Soh et al., 2015; Wells et al., 2012). However, all reported treatment dosages, which ranged from 3 to 16 sessions.

Content Elements

Six studies included all content elements (Blackwell et al., 2014; Mah & Doherty, 2021; Nash et al., 2015; Soler et al., 2019; Wagner et al., 2017; Wells et al., 2012). Two studies had no information regarding sensory activities (Barnes et al., 2008; Mac Cobb et al., 2014b), and five studies did not report evidence of therapeutic observations of sensorimotor strategies and child preferences (Barnes et al., 2008; Mac Cobb et al., 2014a, 2014b; Soh et al., 2015; Wagner et al., 2020a).

Step 2: Development and Expert Review of the Fidelity Measure

The preliminary fidelity measure was reviewed by all experts, with an agreement rate of 93.4% that the measure contained the core elements of the AP. Table 2 summarizes the agreement rate for the individualized AP stages, with Stage 3 having the least agreement (90.8%). One expert indicated that various additional elements were required within Stage 3. Overall, 15 narrative comments were received focusing on terminology alignment and implementation specifics.

Table 2.

Percentages of Expert Panel Agreement for Core Elements of AP.

Stage 1		93.5%
	Structural	93.8%
	Content	95%
	Process	96.4%
Stage 2		94.0%
	Structural	93.8%
	Content	95%
	Process	93.8%
Stage 3		90.8%
	Structural	87.5%
	Content	85%
	Process	95%
Overall		93.4%

Step 3: Revision of the Fidelity Measure

Expert panel feedback resulted in refinements to the measure (See Appendix A for the updated measure). As detailed in Table 3, these modifications included eight specific language and terminology adjustments to align with established AP standards, incorporation of previously omitted mile markers (a critical AP component), refinement of scoring criteria specifications, and adaptations to enhance applicability across diverse clinical settings. Six additional expert recommendations were documented but required no further actions as they were already addressed in the measure design.

Table 3.

Expert Panel Comments, Recommendations, and Resolutions.

Element	Comments/recommendations	Resolution
Content	Is summarizing session a core element? It’s an important part of group process, but perhaps not core	Summarizing is listed as a recommendation for each session within the AP intervention manual. No change required.
Content	I would leave this as “Discusses” and take out “Introduces” for this section.	Wording changed to reflect the suggestion.
General	Are planning on using the songs and games that go along with the Alert Program®?	This relates more to activities specified in the intervention manual; no change required.
General	Should language of the measure be more layman than OT focused?	No, as it is to be used in research, no change is required.
Process	Clarification around wording of labeling own & child’s arousal levels?	Additional clarification added.
Process	Preferences of the child?	Wording changed to clarify.
Process	Providing feedback to child seems similar to identify engine level of child?	Some similarities, but this is more about explaining therapists’ observations. Changes to wording to assist in clarification.
Process	Strategies offered to child. At least two?	Wording changed to clarify.
Process	Reflecting on interviewing process—not sure what this one means?	Listed as per Mile marker 11. No change required.
Process	Discussion of interview—not sure what this refers to?	Listed as per Mile marker 11. No change required.
Set up	Clarification of set up: preparation materials to be listed as examples instead.	Wording changed to reflect the suggestion.
Structural	Should the length of session be more a 30–60 min due to differing settings.	Change to length of sessions from 45–60 mins to 30–60 mins.
Structural	I think the focus here is often what the child can do when they don’t have access to the space/equipment they have during therapy, so space may not be necessary.	Space for activities is listed as a recommendation. No change required.
Structural	Without the 12 Mile Markers and listing them on the assessment, if I were using it to decide if sessions were adhering to the Alert Program® core principles, in its present form, I wonder if I would be able to differentiate those that were or were not adhering to the fidelity measures.	Mile markers added to stages.
Structural	“at least 1 trained Alert Program® Therapist” can this be changed to “At least one therapist is experienced in using the Alert Program®, read the Alert Program® books and/or taken the Alert Program® Online Course and is familiar with sensory processing information.”	Changes made to wording.

Step 4: Inter-Rater Reliability

The observed agreement between raters (HB & EM) indicated 91% agreement on the preliminary fidelity measure.

Discussion

Our preliminary fidelity measure was developed following established recommendations for core elements of fidelity (Century et al., 2010; Di Rezze et al., 2012; Mowbray et al., 2003). While existing literature around fidelity elements typically focuses on structural and process elements, we determined that content elements were essential to accurately represent the AP. The addition of content elements creates a more comprehensive approach to fidelity assessment that captures key aspects of the AP.

In revising our preliminary fidelity measure, expert advice led us to incorporate the specific language used in the AP, emphasizing the importance of identifying, labeling, and using child-appropriate language, the use of sensory measures and equipment, feedback mechanisms to child and carers, home-based labeling, and provision of strategies to assist children with self-regulation. The high level of expert agreement (93.4%) indicates substantial consensus on the measure’s core elements, supporting its content validity as recommended by Zamanzadeh and colleagues (2015), who emphasize the importance of expert panels in establishing content validity for newly developed measurement instruments.

The inter-rater reliability assessment showed excellent percent agreement (91%) between two independent clinicians, supporting the measure’s potential reliability for assessing AP fidelity. This high agreement suggests the measure offers clear, observable criteria that can be consistently identified by different raters—a critical characteristic for any fidelity measure intended for widespread use (Gisev et al., 2013; McHugh, 2012). Further testing with larger samples and diverse raters would strengthen confidence in the measure’s reliability across varied clinical and research contexts.

The developed fidelity measure addresses a gap in occupational therapy literature (see Appendix A). Prior to this study, standardized methods to assess AP implementation consistency were limited, with only preliminary work by Barnes et al. (2008) addressing aspects of implementation fidelity. This measure provides several potential benefits.

First, it offers a framework addressing essential components of the AP, supporting more consistent implementation. Second, it provides a tool for researchers to document fidelity in intervention studies, potentially enhancing research rigor. Third, it may serve as a reference for clinicians implementing the AP. Fourth, the measure’s format makes it applicable for both research documentation and clinical supervision contexts.

The fidelity measure provides a foundation for establishing consistency in AP implementation across research and practice settings. By incorporating Moncher and Prinz’s (1991) recommendations for fidelity assessment (see Supplemental Table 1), we developed a comprehensive measure that addresses not only the structural aspects of program delivery but also the essential process and content elements that define the AP. This approach aligns with current understanding of intervention fidelity as a multidimensional construct that requires attention to both the technical and theoretical aspects of program delivery.

Together, these benefits position the fidelity measure as a valuable addition to the occupational therapy toolkit for those implementing and researching the Alert Program®.

Limitations

Despite the study’s strengths, such as a comprehensive literature review and expert input, there were limitations. The study’s reliance on electronic journals and databases might have missed relevant literature from other sources. In addition, the small group of expert opinions may introduce unintended bias. To minimize this bias, experts were selected from multiple nations, Australia and the United States (Smith & Noble, 2014). However, while a formal Delphi study would have broadened our consensus building, this approach was not used in this study. Future research could benefit from a more extensive expert panel and a more structured consensus-building process. Regarding reliability, this study established percent agreement between two raters, representing initial evidence only. Additional testing using more raters, diverse clinical scenarios, and statistical methods such as intraclass correlation coefficients would further evaluate the measure’s reliability.

Implications for Occupational Therapy Practice

Assessing fidelity in intervention delivery is crucial for achieving consistency in how well a program is executed according to the established protocol. While fidelity does not determine intervention effectiveness, ensuring fidelity enables clinicians and researchers to more accurately measure the true impact of an intervention (Murphy & Gutman, 2012). Equipping researchers with intervention fidelity measures makes it possible to conduct more rigorous research studies that are replicable and comparable (Gill et al., 2018).

The fidelity measure was based on the core elements of the AP, ensuring that interventions remained faithful to the AP’s underlying principles (Murphy & Gutman, 2012). Higher-quality intervention research that examines and discloses fidelity can enhance practice by ensuring practitioners provide consistent support for children with self-regulation needs that are aligned with the AP’s underlying principles.

The fidelity measure may also assist those undertaking AP training to ensure they understand and can implement the AP as intended by the developers in research, education, and practice contexts (Parham et al., 2011). The AP interveners will also be able to test their adherence to the AP’s fidelity and assess whether their implementation of the program achieves the same results as found in clinical trials (Bellg et al., 2004).

Conclusion

Knowing that established interventions are true to their intent is key to best practice. The absence of intervention fidelity measures makes this challenging, particularly for widely used approaches like the Alert Program^®. Our fidelity tool for the AP provides a foundation for more consistent implementation assessment, potentially supporting both research and clinical applications. This measure offers a standardized approach to evaluating how closely the program delivery aligns with its intended design. Future research might examine how adherence to program components relates to outcomes for children with self-regulation challenges, furthering our understanding of this specific intervention approach.

Supplemental Material

sj-docx-1-otj-10.1177_15394492261422638 – Supplemental material for Development of a Fidelity Instrument for Delivering the Alert Program® for Self-Regulation

Supplemental material, sj-docx-1-otj-10.1177_15394492261422638 for Development of a Fidelity Instrument for Delivering the Alert Program® for Self-Regulation by Dianne Blackwell, Alison E. Lane, Kelsey Philpott-Robinson and Shelly J. Lane in OTJR: Occupational Therapy Journal of Research

Footnotes

Appendix

STAGE 3: Self-monitoring and independent strategy identification	Yes	No
Set up:
Examples include:• Prepare with ‘tune-up’ activity cards, individualized for child preferences and sensitivities as appropriate;• One larger picture of backpack or fanny pack (whatever seems appropriate) for use in identifying strategies for alternative environments;• Have “Changing how alert you feel’ handout and forms ready for discussion;• Have “Can you change your engine speed” form prepared for home use;• Have “What works, what bothers me” chart prepared for use;• Prepare “Problem solving” cards for play;• Prepare “Sensorimotor Preferences” checklist.	☐	☐
Structural elements	Yes	No
Mile Markers: ☐ 6 ☐ 7 ☐ 8 ☐ 9 ☐ 10 ☐ 11 ☐ 12	☐	☐
Length of session: 30–60 mins	☐	☐
At least one therapist is experienced in using the Alert Program®, read the Alert Program® books and/or taken the Alert Program® Online Course and is familiar with sensory processing information # present . . . . . . . . . . . . . . . . . . . .	☐	☐
Sufficient space for activities for sensory strategies	☐	☐
Sufficient space for activities for vestibular strategies	☐	☐
Content elements	Yes	No
Discusses terminology of “engine”	☐	☐
Discusses arousal levels	☐	☐
Presents opportunities for engagement in vestibular, proprioceptive and tactile activities	☐	☐
Discuss homework with parent and child	☐	☐
Summarizes session at close, including accomplishments and activities	☐	☐
Process elements	Yes	No
Discusses and reflects on preferences and sensitivities; shares own and listens/reflects on child comments	☐	☐
Includes strategies through session as needed to help child maintain optimal arousal for tasks	☐	☐
Provides feedback to child re preferences and sensitivities	☐	☐
Provides feedback, supports success, offers alternative strategy ideas	☐	☐
Provides opportunity to explore alternative strategies as appropriate	☐	☐
Encourages child to use and expand on strategies	☐	☐
Shares approaches used in alternative environments where all strategies may not work; encourages and prompts sharing from child	☐	☐
Reflects on strategies for alternative environments; helps child trouble shoot and problem solve	☐	☐
Reflects on interviewing process	☐	☐
Provides insight and feedback to child as the discussion of interviews unfolds	☐	☐

Comments:

Acknowledgements

We acknowledge the support of the Hunter Medical Research Institute and are grateful for the contributions made by the University of Newcastle occupational therapy clinical educators Ellen Mason and Jackson Brent, Hannah K Burke, PhD, our expert panel members Mary-Sue Williams and Sherry Shellenberger, Kelly Tanner, PhD, and Emily Saunderson.

Author’s Note

The research described in this study has been presented as an e-poster at the Australian Occupational Therapy Annual Conference in July 2019 (Sydney, NSW).

ORCID iDs

Dianne Blackwell

Alison E. Lane

Kelsey Philpott-Robinson

Shelly J. Lane

Ethical Considerations

This study was approved by the University of Newcastle Ethics Committee, H-2018-0036, as part of a more extensive study on the impact of sensory subtypes on outcomes of The Alert Program® for autistic children.

Funding

The authors disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The Hunter Medical Research Institute provided support for this research.

Declaration of Conflicting Interests

The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Data Availability Statement

The video data used in this study is not available due to privacy and ethical restrictions. However, the fidelity measure, inter-rater reliability sheets, and other supporting materials from this study are available from the corresponding author upon reasonable request

Supplemental Material

Supplemental material for this article is available online.

References

American Occupational Therapy Association. (2017). Vision 2025. American Journal of Occupational Therapy, 71(3), 1–1.

Baethge

Goldbeck-Wood

Mertens

(2019). SANRA — A scale for the quality assessment of narrative review articles. Research Integrity and Peer Review, 4(1), 5. https://doi.org/10.1186/s41073-019-0064-8

Barnes

K. J.

Vogel

K. A.

Beck

A. J.

Schoenfeld

H. B.

Owen

S. V.

(2008). Self-regulation strategies of children with emotional disturbance. Physical & Occupational Therapy in Pediatrics, 28(4), 369–387. https://doi.org/10.1080/01942630802307127

Bellg

A. J.

Borrelli

Resnick

Hecht

Minicucci

D. S.

Ory

, Treatment Fidelity Workgroup of the, N. I. H., & Behavior Change Consortium, . (2004). Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations from the NIH Behavior Change Consortium. Health Psychology, 23(5), 443–451. https://doi.org/10.1037/0278-6133.23.5.443

Ben-Sasson

Hen

Fluss

Cermak

S. A.

Engel-Yeger

Gal

(2009). A meta-analysis of sensory modulation symptoms in individuals with autism spectrum disorders. Journal of Autism and Developmental Disorders, 39(1), 1–11. https://doi.org/10.1007/s10803-008-0593-3

Blackwell

A. L.

Yeager

D. C.

Mische-Lawson

Bird

R. J.

Cook

D. M.

(2014). Teaching children self-regulation skills within the early childhood education environment: A feasibility study. Journal of Occupational Therapy, Schools, & Early Intervention, 7(3–4), 204–224. https://doi.org/10.1080/19411243.2014.966013

Borrelli

Sepinwall

Ernst

Bellg

A. J.

Czajkowski

Breger

Orwig

(2005). A new tool to assess treatment fidelity and evaluation of treatment fidelity across 10 years of health behavior research. Journal of Consulting and Clinical Psychology, 73(5), 852–860. https://doi.org/10.1037/0022-006X.73.5.852

Century

Rudnick

Freeman

(2010). A framework for measuring fidelity of implementation: A foundation for shared language and accumulation of knowledge. American Journal of Evaluation, 31(2), 199–218. https://doi.org/10.1177/1098214010366173

Di Rezze

Law

Gorter

J. W.

Eva

Pollock

. (2012). A narrative review of generic intervention fidelity measures. Physical & Occupational Therapy in Pediatrics, 32(4), 430–446. https://doi.org/10.3109/01942638.2012.713454

10.

Gill

Thompson-Hodgetts

Rasmussen

(2018). A critical review of research on the Alert Program^®. Journal of Occupational Therapy, Schools, & Early Intervention, 11(2), 1–17. https://doi.org/10.1080/19411243.2018.1432445

11.

Gisev

Bell

J. S.

Chen

T. F.

(2013). Interrater agreement and interrater reliability: Key concepts, approaches, and applications. Research in Social and Administrative Pharmacy, 9(3), 330–338. https://doi.org/10.1016/j.sapharm.2012.04.004

12.

Lidstone

Uljarević

Sullivan

Rodgers

McConachie

Freeston

Le Couteur

Prior

Leekam

(2014). Relations among restricted and repetitive behaviors, anxiety and sensory features in children with autism spectrum disorders. Research in Autism Spectrum Disorders, 8(2), 82–92. https://doi.org/10.1016/j.rasd.2013.10.001

13.

Mac Cobb

Fitzgerald

Lanigan-O’Keeffe

. (2014a). The Alert Program for self-management of behaviour in second level schools: Results of phase 1 of a pilot study. Emotional & Behavioural Difficulties, 19(4), 410–425. https://doi.org/10.1080/13632752.2014.903593

14.

Mac Cobb

Fitzgerald

Lanigan-O’Keeffe

Irwin

Mellerick

. (2014b). Students with social, emotional, and behavioral difficulties: The Alert Program trial in post-primary schools. Journal of Occupational Therapy, Schools, & Early Intervention, 7(2), 106–119. https://doi.org/10.1080/19411243.2014.930606

15.

Mah

J. W. T.

Doherty

(2021). Sensorimotor intervention group for children with ADHD and sensory processing difficulties: A feasibility study. Journal of Child and Family Studies, 30, 447–459. https://doi-org/10.1007/s10826-021-01900-y

16.

Martini

Cramm

Egan

Sikora

(2016). Scoping review of self-regulation: What are Occupational Therapists talking about? The American Journal of Occupational Therapy, 70(6), 1–15. https://doi.org/10.5014/ajot.2016.020362

17.

McHugh

M. L.

(2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276–282.

18.

Miller

Anzalone

Lane

Cermak

Osten

(2007). Concept evolution in sensory integration: A proposed nosology for diagnosis. American Journal of Occupational Therapy, 61(2), 135–140.

19.

Moncher

F. J.

Prinz

R. J.

(1991). Treatment fidelity in outcome studies. Clinical Psychology Review, 11(3), 247–266. https://doi.org/10.1016/0272-7358(91)90103-2

20.

Mowbray

C. T.

Holter

M. C.

Teague

G. B.

Bybee

(2003). Fidelity criteria: Development, measurement, and validation. The American Journal of Evaluation, 24(3), 315–340. https://doi.org/10.1016/S1098-2140(03)00057-2

21.

Murphy

S. L.

Gutman

S. A.

(2012). Intervention fidelity: A necessary aspect of intervention effectiveness studies. The American Journal of Occupational Therapy, 66(4), 387–388. https://doi.org/10.5014/ajot.2010.005405

22.

Nash

Stevens

Greenbaum

Weiner

Koren

Rovet

(2015). Improving executive functioning in children with fetal alcohol spectrum disorders. Child Neuropsychology, 21(2), 191–209. https://doi.org/10.1080/09297049.2014.889110

23.

Page

M. J.

McKenzie

J. E.

Bossuyt

P. M.

, et al. (2021). The PRISMA 2020 statement: An updated guideline for reporting systematic reviews. PLOS Medicine, 18(3), e1003583.

24.

Parham

L. D.

Roley

S. S.

May-Benson

T. A.

Koomar

Brett-Green

Burke

J. P.

Schaaf

R. C.

(2011). Development of a fidelity measure for research on the effectiveness of the Ayres sensory integration® intervention. American Journal of Occupational Therapy, 65(2), 133–142. https://doi.org/10.5014/ajot.2011.000745

25.

Pfeiffer

Coster

Snethen

Derstine

Piller

Tucker

(2017). Caregivers’ perspectives on the sensory environment and participation in daily activities of children with autism spectrum disorder. The American Journal of Occupational Therapy, 71(4), 7104220020p1-7104220028p9. https://doi.org/10.5014/ajot.2017.021360

26.

Pfeiffer

B. A.

Koenig

Kinnealey

Sheppard

Henderson

(2011). Effectiveness of sensory integration interventions in children with autism spectrum disorders: A pilot study. The American Journal of Occupational Therapy, 65(1), 76–85.

27.

Philpott-Robinson

Lane

A. E.

Harpster

(2016). Sensory features of toddlers at risk for autism spectrum disorder. American Journal of Occupational Therapy, 70(4), 1–8. https://doi.org/10.5014/ajot.2016.019497

28.

Schaaf

R. C.

Benevides

T. W.

Kelly

Mailloux-Maggio

(2012). Occupational therapy and sensory integration for children with autism: A feasibility, safety, acceptability and fidelity study. Autism, 16(3), 321–327. https://doi.org/10.1177/1362361311435157

29.

Smith

Noble

(2014). Bias in research. Evidence Based Nursing, 17(4), 100–101. https://doi.org/10.1136/eb-2014-101946

30.

Soh

D. W.

Skocic

Nasststh

Stevens

Turner

G. R.

Rovet

(2015). Self-regulation therapy increases frontal gray matter in children with fetal alcohol spectrum disorder: Evaluation by voxel-based morphometry. Frontiers in Human Neuroscience, 9, Article e00108. https://doi.org/10.3389/fnhum.2015.00108

31.

Soler

Hardwick

Perkes

I. E.

Dossetor

Bray

Dale

R. C.

(2019). An exploratory study into an adapted use of the Alert Program for tic disorder in children. Australasian Psychiatry, 27(2), 144–151. https://doi.org/10.1177/1039856218815750

32.

Song

M.-K.

Happ

M. B.

Sandelowski

(2010). Development of a tool to assess fidelity to a psycho-educational intervention. Journal of Advanced Nursing, 66(3), 673–682. https://doi.org/10.1111/j.1365-2648.2009.05216.x

33.

Steinbrenner

J. R.

Hume

Odom

S. L.

Morin

K. L.

Nowell

S. W.

Tomaszewski

Szendrey

McIntyre

N. S.

Yücesoy-Özkan

Savage

M. N.

(2020). Evidence-based practices for children, youth, and young adults with autism spectrum disorder. National Clearinghouse on Autism Evidence and Practice Review Team, FPG Child Development Center, University of North Carolina at Chapel Hill.

34.

Wagner

Cross

Adams

Symons

Mazzucchelli

T. G.

Watkins

Wright

Latimer

Carapetis

Boulton

(2020a). RE-AIM evaluation of a teacher-delivered programme to improve the self-regulation of children attending Australian Aboriginal community primary schools. Emotional and Behavioural Difficulties, 25, 42–58. https://doi.org/10.1080/13632752.2019.1672991

35.

Wagner

Fitzpatrick

Symons

Jirikowic

Cross

Latimer

(2017). The development of a culturally appropriate school-based intervention for Australian Aboriginal children living in remote communities: A formative evaluation of the Alert Program® intervention. Australian Occupational Therapy Journal, 64(3), 243–252. https://doi.org/10.1111/1440-1630.12352

36.

Wagner

Latimer

Adams

Carmichael Olson

Symons

Mazzucchelli

T. G.

Jirikowic

Watkins

Cross

Carapetis

(2020b). School-based intervention to address self-regulation and executive functioning in children attending primary schools in remote Australian Aboriginal communities. PLOS ONE, 15(6), Article e0234895. https://doi.org/10.1371/journal.pone.0234895

37.

Wells

A. M.

Chasnoff

I. J.

Schmidt

C. A.

Telford

Schwartz

L. D.

(2012). Neurocognitive habilitation therapy for children with fetal alcohol spectrum disorders: An adaptation of the Alert Program^®. American Journal of Occupational Therapy, 66(1), 24–34. https://doi.org/10.5014/ajot.2012.002691

38.

Williams

M.-S.

Shellenberger

(1996). ‘How does your engine run?’^®: A leader’s guide to the Alert Program^® for self-regulation. TherapyWorks.

39.

Zamanzadeh

Ghahramanian

Rassouli

Abbaszadeh

Alavi-Majd

Nikanfar

A.-R.

(2015). Design and implementation content validity study: Development of an instrument for measuring patient-centered communication. Journal of Caring Sciences, 4(2), 165–178. https://doi.org/10.15171/jcs.2015.017

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.02 MB

Development of a Fidelity Instrument for Delivering the Alert Program ® for Self-Regulation

Abstract

Plain Language Summary

Keywords

Introduction

Method

Step 1: Identifying the Fidelity Criteria

Step 2: Development and Expert Review of the Fidelity Measure

Step 3: Revision of the Fidelity Measure

Step 4: Inter-Rater Reliability

Results

Step 1: Identifying the Fidelity Criteria

Structural Elements

Process Elements

Content Elements

Step 2: Development and Expert Review of the Fidelity Measure

Step 3: Revision of the Fidelity Measure

Step 4: Inter-Rater Reliability

Discussion

Limitations

Implications for Occupational Therapy Practice

Conclusion

Supplemental Material

sj-docx-1-otj-10.1177_15394492261422638 – Supplemental material for Development of a Fidelity Instrument for Delivering the Alert Program® for Self-Regulation

Footnotes

Appendix

Acknowledgements

Author’s Note

ORCID iDs

Ethical Considerations

Funding

Declaration of Conflicting Interests

Data Availability Statement

Supplemental Material

References

Supplementary Material