A Preliminary Investigation of LLM Capability for Cognitive Interviewing

Abstract

Designing and operationalizing valid surveys during public health emergencies and similarly fast changing situations can be a challenge. In this research note, we present an exploration of whether an approach to cognitive pre-test interviewing through an online survey platform augmented by the use of AI prompts could help achieve robust pre-testing at pace, in a sensitive and effective manner. We integrated an LLM tool into an online survey system and configured it to generate qualitative prompts to better understand how a sample of 46 respondents understood and approached answering a series of questions about mpox. The results indicate using an LLM to conduct pre-test cognitive interviewing did not fully meet the need to conduct these activities to the standard needed by those working in quickly evolving situations such as health security incidents. However, this investigation of an untrained LLM with limited capability should be understood as an early step towards discovering the potential to cognitive pre-test survey questions using LLMs.