Date Presented 03/26/20
This project identifies the importance of having advanced training for administration of assessments to have accurate and reliable results. With the identified gap in training for the ARAT assessment, a training program was created and tested pre- and posttraining with 15 participants. The interrater reliability results indicated that posttraining scores had greater accuracy in scoring. These results indicate a positive effect on accuracy of assessments with the application of training.
Primary Author and Speaker: Allison Drobish
Additional Authors and Speakers: Mary Hildebrand
PURPOSE/BACKGROUND: The objective of this study was to create and test the interrater reliability of an Action Research Arm Test (ARAT) training program. The ARAT is a standardized assessment of upper extremity function used to measure occupational therapy (OT) outcomes for people who have experienced a stroke1,2. Despite the use of the ARAT as an outcome measure, there is no training program for administration and scoring of the assessment, and few ARAT research studies report the use of training for the ARAT and most do not assess their interrater reliability. To obtain accurate results, the ARAT must have high interrater reliability, described as demonstrating that “with adequate training, practice, and recalibration, two different raters will consistently arrive at essentially the same score”3. This calls into question the accuracy of the results of these studies. This study addresses this deficit by creating, training, testing and analyzing the ARAT training program.
RESEARCH QUESTION: Do OT graduate students perform the ARAT, with high interrater reliability, after receiving a standardized training?
DESIGN: A quantitative, pre-experimental, single group, pretest-posttest research design with a convenience sample of 15 OT graduate students with no prior knowledge of the ARAT.
METHOD: Participants were provided with standardized written instructions of the ARAT, one week later they scored 3 pre-recorded client videos completing the ARAT. Two in-person training sessions followed including pictorial examples, detailed charts, a reviewed quiz, client video analysis, interaction with assessment tool, and Q&A of researchers. The final session participants scored 3 pre-recorded client videos (different clients from pre-testing videos) completing the ARAT. Data scoring sheets from each of the participants were entered into a secure database. Intraclass Correlation Coefficient (ICC) model 2 was used on IBM® SPSS for single item measures of each rater and for average rater scores and correct answers (determined by experienced researchers).
RESULTS: The total interrater reliability using ICC (2,1) for the pre-training was 0.868 (95% confidence interval: .817-.910, F=117.468, P <.001) and post-training was 0.808 (95% confidence interval: .746-.865, F=65.17, P<.001). The total mean and correct scores ICC (2, k) pre-training: 0.929 (95% confidence interval: .880-.958, F=14.066, P<.001) and post-training: 0.959 (95% confidence interval: .931-.976, F=24.779, P<.001).
CONCLUSION: The interrater reliability of the overall pre-training scores may have been higher because the participants scored the clients similarly pre-training but did not score accurately. Alternatively, the interrater reliability between the mean rater scores and correct scores improved from pre-to-post training. The participants scored much closer to the correct scores and therefore, were more accurate. This indicates that the ARAT training increased the interrater reliability and scoring accuracy. An ARAT training program can influence the accuracy of scoring for OT graduate students.
IMPACT STATEMENT: This project contributes to the OT profession by showing the importance of advanced training for the accuracy of scoring and administering of assessments. OT practice may benefit from this evidence with the addition of an ARAT training protocol promoting the use of the ARAT. It may encourage the creation of similar trainings for other clinical assessments for increased accuracy and trustworthiness.
HIGH INTERRATER RELIABILITY: ICC of 0.90 and above is considered reasonable reliability (4).
INTERRATER RELIABILITY: The degree to which two or more raters can obtain the same ratings for a given variable (4).
References
1. Hsieh, C., Hsueh, I., Chiang, F., & Lin, P. (1998). Inter-rater reliability and validity of the Action Research Arm Test in stroke patients. Journal of Age and Aging, 27, 107-113. doi:10.1093/ageing/27.2.107
2. Yozbatiran, N., Der-Yeghiaian, L., & Cramer, S. C. (2007). A standardized approach to performing the Action Research Arm Test. Journal of Neurorehabilitation and Neural Repair, 22, 78-90. doi:10.1177/1545968307305353
3. Hinojosa, J., & Kramer, P. (2014). Evaluation in occupational therapy: Obtaining and interpreting data, Fourth edition. Bethesda, MD: AOTA Press
4. Portney, L. G., & Watkins, M. P. (2015). Foundations of clinical research: Applications to practice (3rd ed.). Philadelphia, PA: F. A. Davis Company.
