Assessing Readability,Understandability,and Actionability of ACPA Educational Resources and Utility of LLMs to Improve Accessibility for Caregivers

Abstract

Objective

It is important for parents and caregivers of children with cleft lip and/or palate to easily understand educational resources provided by the American Cleft Palate-Craniofacial Association (ACPA). This study aimed to assess the readability, understandability, and actionability of ACPA resources, and the utility of large language models (LLMs) in their improvement.

Design

ACPA resources were collected and assessed using a readability calculator and 2 readers assessed using the Patient Educational Material Assessment Tool (PEMAT). Each resource was modified by 3 LLMs (GPT-4o, Gemini, and Copilot) and their outputs reassessed for improvement.

Main Outcome Measures

Average reading level across Flesch-Kincaid Grade Level, Gunning Fog Index, SMOG Index, and Coleman-Liau Index; PEMAT understandability and actionability.

Results

ACPA educational materials are written at an average grade level of 10.7 ± 1.8 with a PEMAT understandability of 82.4 ± 7.0% and actionability of 42.3 ± 17.4%. Modification by LLMs decreased average reading level to 8.0 ± 1.2 (P = .0002) with a PEMAT understandability of 80.5 ± 7.9% (P = .5371) and actionability of 40.8 ± 18.2% (P = .6709). And 38.5% of ACPA resources included visual aids or illustrations and 42.3% provided explicit, actionable steps that parents and caregivers could take.

Conclusions

Although ACPA educational resources are high quality, they are written at reading levels nearly meeting NIH guidelines but not meeting AMA guidelines. LLMs prove valuable to improve readability without diminishing understandability or actionability.