Date Presented 04/13/21
As part of a larger study to determine the effectiveness of an OT driving intervention to improve returning combat veterans' driver fitness, researchers measured interrater reliability among three driver rehabilitation specialists. We present the training process, results, and strategy to achieve strong rater reliability for the assessment of driving errors on a DriveSafety 250 high-fidelity simulator.
Primary Author and Speaker: Mary Jeghers
Contributing Authors: Amber L. Stober, Elisabeth Popoviciu, Gabriella Nieves, and Magaret Norton
PURPOSE: Motor vehicle crashes are the second leading cause of death among returning combat Veterans (henceforth referred to as Veterans). Medical conditions, battle training techniques, and deployment related exposure, such as the battle mindset, contribute to driving difficulty in Veterans. Researchers are studying the effectiveness of an Occupational Therapy Driving Intervention, compared to a Traffic Safety Intervention in a randomized controlled trial, to improve post-deployment driver fitness among Veterans. Multiple driver rehabilitation specialists evaluated Veterans’ driving errors using a high-fidelity driving simulator and an established rating protocol. Therefore, high inter-rater reliability among driver rehabilitation specialists was established prior to administering the Occupational Therapy Driving Intervention. The aim of this research poster is to explain the procedure, results, strategies, and value of establishing inter-rater reliability, among three driver rehabilitation specialists.
METHOD: The inter-rater reliability testing occurred over two phases. In Phase 1 we determined the inter-rater reliability among all three raters; and in Phase 2 we developed strategies for ubiquitous use to increase reliability. Three trained driver rehabilitation specialists assessed five volunteer drivers’ errors. One rater was a certified driver rehabilitation specialist with over twenty years of experience, and the others were driver rehabilitation specialists with about one-year experience. Each driver rehabilitation specialists rated a total of five sets of drives completed by different volunteer drivers: three sets of drives in Phase 1 and two sets of drives in Phase 2. Each set of drives included an acclimation drive to become familiar with the simulator, followed by a six-minute drive in a suburban/residential area and a 10-minute drive in an urban/highway area. All drives were video recorded for raters to play-back during review and rating. Raters independently evaluated the driving errors (i.e., gap acceptance, signaling, adjustment-to-stimuli, vehicle positioning, speed regulation, lane maintenance, and visual scanning) in each video. After independent review, raters jointly reviewed the drivers’ errors and calculated the intra-class correlation coefficient and percent agreement among each rater pair. A cut point of 90% was considered acceptable for both the intra-class correlation coefficients and rater percent agreements. At the end of each phase, the study team discussed discrepancies among raters, and devised strategies for consensus.
RESULTS: Phase 1: The intra-class correlation coefficients for five completed drives = 96-99% and the rater agreement = 48-100%. One driver did not complete the residential/suburban drive due to simulator sickness. Phase 2: The intra-class correlation coefficients for four completed drives = 95-99% and rater agreement = 48-67%. Although the intra-class correlation coefficients were high, the rater agreement was unacceptable. Therefore, raters reviewed and discussed each drive until 100% consensus was obtained. The team developed a guide that provided specific grading criteria for each driving scenario and driving error to increase rater agreement and improve accuracy while rating driving errors.
CONCLUSION: Results indicated that raters with different levels of experience can achieve adequate inter-rater reliability, identify inconsistencies, and develop strategies for rating driving errors. Consistency between raters increased the internal validity of the randomized control trial to assess the effectiveness of a driving intervention among Veterans. Occupational therapists performing driver assessment must consider and ensure rater reliability in clinical practice.
References
Classen, S., Winter, S., Monahan, M., Yarney, A., Link Lutz, A., Platek, K., & Levy, C. (2017). Driving intervention for returning combat veterans: interim analysis of a randomized controlled trial. OTJR: Occupation, Participation and Health, 37(2), 62-71.
Classen, S., Yarney, A. K. A., Monahan, M., Winter, S. M., Platek, K., & Lutz, A. L. (2015). Rater reliability to assess driving errors in a driving simulator. Advances in Transportation Studies, (36).
Classen, S., Cormack, N. L., Winter, S. M., Monahan, M., Yarney, A., Lutz, A. L., & Platek, K. (2014). Efficacy of an occupational therapy driving intervention for returning combat veterans. OTJR: occupation, participation and health, 34(4), 176-182.
Bédard, M., Parkkari, M., Weaver, B., Riendeau, J., & Dahlquist, M. (2010). Assessment of driving performance using a simulator protocol: Validity and reproducibility. American Journal of Occupational Therapy, 64(2), 336-340.
