Professional Development Specialist Inter-rater Reliability of Coaching Fidelity Scoring in a Statewide Professional Development Program

Abstract

Date Presented 04/20/2023

Intervention integrity is considered central to fidelity and the monitoring of intervention fidelity is important as a demonstrated mediator of outcomes. Documenting reliability among raters is critical to these functions. We describe development and field testing of a sensitive, research-based coaching fidelity measure to present study finding investigating whether professional development specialists maintain acceptable interrater reliability when using the measure. Implications are outlined.

Primary Author and Speaker: Scott Tomchek

Additional Authors and Speakers: Winnie Dunn, Lauren Little

Operational definitions of coaching have common themes, though often are not explicitly stated and lack consistency for comparison in intervention studies (Kemp & Turnbull, 2014). Intervention integrity is considered central to fidelity (Gearing et al., 2011) and the monitoring of intervention fidelity is important as a demonstrated mediator of outcomes (Mars et al., 2013). Documenting reliability among raters is critical to these functions. The purpose of the study was to determine if professional development (PD) specialists (PDS) were reliable when using a research-based coaching fidelity measure.

DESIGN: At present, 227 early intervention (EI) providers completed a statewide evidence-informed PD program to enhance coaching quality. Fidelity ratings at baseline and following PD from 6 PDS implementing the training program were analyzed in this correlational study.

METHOD: We developed and field tested a research-based fidelity tool to measure adherence to 7 coaching quality indicators (i.e., Fostering Trusting, Relationships, Beginning Joint Plan, Observation, Action/Practice, Feedback, Reflection, Ending Joint Plan) during video recorded EI visits. We established descriptors and ratings along a 5-point Likert scale (i.e., not yet, knowledge, awareness, application, mastery). We randomly selected 20% of all program video submissions and randomly assigned second raters for reliability analysis. Intraclass Correlation Coefficients were conducted for total and individual quality indicator scores.

RESULTS: To date 187 reliability checks have been conducted across the 6 PDS raters with ICC of .97 for the total score and ranging from .85 to .94 at the individual quality indicator level.

CONCLUSION: Findings note excellent interrater reliability among 6 PDS and contribute to program evaluation factors relevant to consumers of the PD, policy makers and researchers in the field.

IMPACT STATEMENT: Available measures of coaching typically rate characteristics as present or not present. We demonstrate the ability to maintain reliability using a more robust measure with 5 quality ratings of scale. Reliably measuring coaching fidelity using a measure sensitive to change is critical as at least 28 state EI programs are implementing coaching PD to address noted research to practice gaps between expressed value and implementation of coaching (Romano & Schnurr, 2020).

References

Kemp, P., & Turnbull, A. P. (2014). Coaching with parents in early intervention: An interdisciplinary research synthesis. Infants and Young Children, 27(4), 305–324. http://dx.doi.org/10.1097/IYC.0000000000000018

Gearing, R. E., El-Bassel, N., Ghesquiere, A., Baldwin, S., Gillies, J., & Ngeow, E. (2011). Major ingredients of fidelity: a review and scientific guide to improving quality of intervention research implementation. Clinical psychology review, 31(1), 79-88. https://doi.org/10.1016/j.cpr.2010.09.007

Mars, T., Ellard, D., Carnes, D., Homer, K., Underwood, M., & Taylor, S. J. (2013). Fidelity in complex behaviour change interventions: a standardised approach to evaluate intervention integrity. BMJ open, 3(11), e003555. http://dx.doi.org/10.1136/bmjopen-2013-003555

Romano, M., & Schnurr, M. (2020). Mind the gap: Strategies to bridge the research-to-practice divide in early intervention caregiver coaching practices. Topics in Early Childhood Special Education. https://doi.org/10.1177/0271121419899163