Date Presented 04/13/21
The purpose of this study was to establish interrater reliability between two blinded evaluators for a novel occupation-based coaching intervention. A high level of initial interrater agreement for the Occupation-Based Coaching Fidelity Measure (Shin et al., 2020) was established through extensive structured evaluator training, including open discussion, video analysis, and a clearly defined rubric. This poster discusses methods to support researchers in establishing interrater reliability.
Primary Author and Speaker: Kathryn Carlson
Additional Authors and Speakers: Paula Jarrard
PURPOSE: Clinical trials are considered the gold standard for testing the effectiveness of novel interventions. To ensure that interventions are delivered as intended, fidelity measures are frequently developed and utilized to measure intervention reliability. The Occupation-Based Coaching Fidelity Measure was developed for the purposes of ensuring that study outcomes from a clinical trial could be attributed to the intervention (Shin et al., 2020). The purpose of this study was to establish inter-rater reliability between two blinded evaluators for a novel occupation-based coaching intervention delivered via telehealth to families with a young child diagnosed with type 1 diabetes.
DESIGN: This quantitative study utilized psychometric testing to establish initial inter-rater reliability between two blinded evaluators for the Occupation-Based Coaching Fidelity Measure prior to a randomized clinical trial (Portney & Watkins, 2015).
METHOD: In phase one of the study, the blinded evaluators received comprehensive training on occupation-based coaching and on the Occupation-Based Coaching Fidelity Measure. Application of this well-defined rubric, video analysis, and team discussions facilitated consistent training of evaluators on the measure. The trainer and the evaluators watched three sample videos during initial training on the measure. Each independently completed the Occupation-Based Coaching Fidelity Measure and results were compared and discussed for clarification. In phase two, the evaluators independently completed the Occupation-Based Coaching Fidelity Measure after reviewing 10 occupation-based coaching sample videos. The videos were developed by team members for the purpose of team training and establishing inter-rater reliability and agreement. The videos were specifically developed to include participants who had been trained or had lived experience with the type 1 diabetes population for the pilot study, or who were healthy persons with type 1 diabetes. Videos in the final set represented full occupation-based coaching sessions, established a continuum of fidelity ratings, and included multiple interventionists. Data was analyzed for statistical significance for Cohen’s kappa (κ), and Cronbach’s alpha (α) for each set of data acquired for Phase One and Phase Two.
RESULTS: Inter-rater reliability and agreement were established for phase two to be κ = .881 (p < .001), indicating ‘almost perfect’ reliability and α = .999 establishing ‘excellent’ internal consistency between the evaluators (p < .001) (Landis & Koch, 1997). Cronbach's alpha for the final section of the Occupation-Based Coaching Fidelity Measure was .867, demonstrating ‘almost perfect’ reliability.
CONCLUSION: This study established strong inter-rater reliability between two blinded evaluators for the Occupation-Based Coaching Fidelity Measure, which strives to assess and enhance fidelity to intervention principles. Creating a thorough plan for evaluator training can support achievement of high interrater reliability and agreement. As demonstrated in this study, researchers looking to establish inter-rater reliability can utilize extensive evaluator training with open discussion, video review, clear criteria, and a detailed rubric throughout evaluator training. Establishing inter-rater agreement for tools used in research to assess fidelity to interventions is essential to supporting results in clinical research in occupational therapy and support occupational therapy as an evidence-based profession. With a thorough plan for evaluator training, inter-rater reliability can be established between raters even for interventions that are more fluid and individualized to patient care.
References
Shin, J., Russell, M., Jewell, V.D., Abbott, A.A., Carlson, K., Gordon, M. (2020). Development of the Occupation-Based Coaching Fidelity Measure. Manuscript submitted for publication.
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–74.
Portney, L. G., & Watkins, M. (2015). Foundations of clinical research: applications to practice: Upper Saddle River, N.J: Pearson/Prentice Hall.
