“ChatBot” is a Two Syllable Word...Or Is It?: Using Generative AI for Survey Question Readability Assessments

Abstract

Market and survey researchers aim to write survey questions so that the target population can understand them. A common recommendation for general population studies is to write survey questions at an eighth-grade reading level. To evaluate whether questions meet this threshold, survey researchers turn to readability measures, such as the Flesch-Kincaid Reading Grade Level. Researchers may be able to streamline the calculation of question reading levels using artificial intelligence (AI) tools, such as ChatGPT, in situ to draft survey questions. One risk of using artificial intelligence tools is that they may incorrectly calculate readability measures. To our knowledge, whether AI tools calculate readability correctly has not been evaluated. In this paper, we examine readability calculations for 60 survey questions performed by commonly available AI tools, including both ChatGPT and Claude large language models (LLMs), at three time points (Summer 2024, November 2024, April 2025). We compare these to a “gold standard” online readability assessment tool (Readable.com), and calculations from packages in two open-source software programs, R and Python. We examine the Flesch-Kincaid Grade Level and four other readability measures and the inputs to their calculations (e.g., number of words, sentences). Although there is almost perfect alignment between these metrics as calculated by Readable and R, each LLM varies in its calculations across models and over time. We also examine the calculations of the inputs of the readability calculations for each tool, including implications for the reported overall readability score. Our results suggest that open-source tools are reliable and accurate. We also find that LLMs are evolving in their ability to accurately calculate readability of survey questions, with large variation over models and over time. Some, but not all, LLMs are being trained to use the same resources as open-source tools to calculate readability of passages of text.