Date Presented 04/06/19
The focus of the Home for Life Design (HFLD) app-based home assessment is to determine a person’s ability to safely complete an occupation in context, and if and how the environment limits safety. An inter-rater reliability study with OT students (n=32) used online training v.5; ICC between individual raters and the author was high (.92; 95% CI). These findings contribute to the evidence that the HFLD home assessment is a reliable tool for OTs making home-safety recommendations.
Primary Author and Speaker: Debra Lindstrom
Additional Authors and Speakers: Carolyn Sithong, Katherine Sullivan
PURPOSE: The purpose of this research was to establish the Home for Life Design (HFLD) Home Assessment as a reliable, standardized assessment. An inter-rater reliability study was completed with upper level OT graduate students. Many OT home safety assessments are paper-based, extensive or measure the environment exclusively. This HFLD Home Assessment is technology and performance-based. It measures personal safety, identifies the environmental barriers that limit occupational performance and shows examples of environmental modifications to increase safety and accessibility.
DESIGN: This psychometric inter-rater reliability study recruited a cohort of students from an occupational therapy program who were within one year of starting their Level II fieldwork. Students completed on-line training and a test assessing home safety. The current version of the training was the result of over 100 students viewing the on-line training and scoring 10 test scenarios.
METHOD: As part of a required classroom activity, the students viewed the on-line HFLD Home Assessment Training and scored 10 scenarios embedded in an on-line quiz to rate the person’s safety using the HFLD Home Assessment. After receiving the university’s ethical board approval, students were invited to allow the researchers to use their quiz scores anonymously (completed previously as required class activity) to compare individual students’ scores with the assessment author’s scores. The scores were analyzed with SPSS Statistics Version 24 to determine Intra-class correlation coefficients (ICC) between each student and the author for absolute agreement. Percent of agreement for each item was calculated by the on-line learning management tool.
RESULTS: ICC agreement was .92 (95 % CI) with individuals’ ratings ranging from .74-1. Agreement between the ratings of the test questions and the author was greater than or equal to 90% for eight of the ten scenarios. One test scenario (transferring off the toilet while using the bathtub and counter for support) was rated incorrectly by 40% of the raters. It was determined the rating error was attributed to a judgement error of the raters versus a problem with the training instructions or the test scenario.
CONCLUSION: The HFLD Home Assessment Training, v5 has strong inter-rater reliability ratings between OT graduate students and the author of the app. These findings contribute to establishing the HFLD home assessment as a standardized assessment, used reliably to assess the change between a pre and post home safety assessment. Impact statement: This study’s findings are very relevant for the profession as we strive to advance practice in a digital age. Utilizing technology to measure important outcomes is substantial only if the technology and user are reliable in capturing that data. This study advances the practice of home modifications in occupational therapy, as practitioners are proven, reliable users of the (HFLD) Home Assessment technology. Technology-based OT assessments can facilitate the collection of important meta-data and generate population-based outcomes of safety and accessibility during occupational performance.
References
Barras, S. (2005). A systematic and crucial review of the literature: The effectiveness of occupational therapy home assessment on a range of outcome measures. Australian Occupational Therapy Journal, 52, 326-336. https://doi.org/10.1111/j.1440-1630.2005.00496.x
Harris, S., James, E., Snow, P. (2007). Predischarge occupational therapy home assessment visits: Towards an evidence base. Australian Occupational Therapy Journal, 55, 85-95. https://doi.org/10.1111/j.1440-1630.2007.00684.x
Shrout, P. & Fleiss, J. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin 86 (2) 420-428. https://www.aliquote.org/cours/2012_biomed/biblio/Shrout1979.pdf
Siebert, C., Smallfield, S. & Stark, S. (2014). Occupational therapy practice guidelines for home modifications. Bethesda, MD: AOTA Press.
