Abstract
Comprehensive accessibility assessments are extensive and time consuming. The implementation of a branching question system is necessary to make them more efficient. However, such a system could negatively affect the reliability of the assessment.
Primary Author and Speaker: Qussai M. Obiedat
Additional Authors and Speakers: Roger O. Smith
Measuring the accessibility of public buildings is complex and multifaceted. Comprehensive accessibility assessments are extensive and assess each element of the built environment against a wide range of users’ needs. The implementation of branching question system is necessary to make such assessments more efficient. However, such a system could have a tremendous impact on the reliability of assessment. The purpose of this study is to study the effect of using the Trichotomous Tailored, Sub-branching Scoring (TTSS) system (Smith, 1992) on the AccessTools assessment (R2D2 Center, 2023). A simulation study design was implemented on the AccessTools assessment. The assessment contains 2,624 questions in total, with 6 branching levels (Obiedat et al., 2021). Multiple simulations were performed to observe the effect on leaf level agreement based on incorrectly answering 1-4 questions or 10% of the questions within each branching level, thereby finding the highest and lowest number of impacted leaf level questions that would be affected from each simulation scenario. The results show that incorrectly answering 10% of the questions on the first branching level could result in a high impact of dropping the AC1 coefficient to 0.743. Incorrectly answering 10% of the questions on the second and third branching levels could result in lowering the corresponding AC1 coefficients to below chance level. The implementation of the TTSS methodology is of great importance to minimize the time demand to perform such a comprehensive assessment. Particularly when expert raters are performing the assessment. However, novice raters should be careful when answering root level questions, as these questions will have a tremendous impact on the reliability of their assessment. The implications of this study include limiting the answering of root level branches by forcing raters to branch to lower levels or updating the question wording to increase the reliability of the answers.
Smith, R. O. (1992). Comparative reliability of dynamic trichotomous tailored sub-branching scoring (TTSS) and static functional assessment methods [Abstract]. Archives of Physical Medicine and Rehabilitation, 73.
R2D2 Center. (2023). Access Ratings for Buildings. Access Ratings for Buildings Project (ARB). https://uwm.edu/r2d2/projects/arb/
Obiedat, Q. M., Schwartz, J., Mendonca, R., Burns, S., Drake, M., O’Donnell, L., & Smith, R. O. (2021). Using video recordings as a COVID-19 adaptation to study the inter-rater reliability of the AccessTools app. RESNA Annual Conference. https://www.resna.org/sites/default/files/conference/2021/ServiceDeliveryandOutcomes/112_Obiedat.html