Web surveys have become the dominant mode of survey data collection. They offer advantages in terms of cost and time over other modes and they are the new normal in survey research for many populations and in many fields of study. Nevertheless, web surveys, like other survey modes, are affected by the consequences of satisficing behavior, which is attributed, among other things, to low motivation among respondents. We assume that the motivation of respondents can be increased by using a respondent-friendly approach in the design of the questionnaire (
Dillman, 2000), which was achieved by implementing a chatbot-like questionnaire interface. In a randomized field-experiment conducted among university students we employed a between-subject design comparing a chatbot-like interface and a traditional web survey design administering the same questionnaire. We assessed respondent evaluation, as well as data quality indicators like response time, non-differentiation, item missing rates, and the length of answers to narrative open-ended questions. Results indicate that respondents perceived the chatbot-like design as more original and entertaining than the traditional web survey design. By contrast, participants rated the chatbot-like interface as more difficult to navigate. The analyses of response time and character count of answers to open-ended questions showed no significant differences between the two designs. The proportion of respondents with at least one item-missing was marginally lower in the chatbot-like design, while the degree of differentiation of one of the multi-item scales was higher in the web survey design.
AdhikariDMHartlandAWeberI, et al. (2025) Ex-ploring LLMs for automated generation and adaptation of questionnaires. In: CUI 2025. Waterloo (ON), Canada.
2.
Al BaghalTLynnP (2015) Using motivational statements in web-instrument design to reduce item-missing rates in a mixed-mode context. Public Opinion Quarterly79(2): 568–79.
3.
AntounCCouperMPConradFG (2017) Effects of mobile versus PC web on survey response quality. A crossover experiment in a probability web panel. Public Opinion Quarterly81(Special Issue): 280–306.
4.
AraujoT (2018) Living up to the chatbot hype: The influence of anthropomorphic design cues and communicative agency framing on conversational agent and company perceptions. Computers in Human Behavior85: 183–89.
5.
BaierTFuchsM (2020) Page switching in mixed-device web surveys: prevalence and data quality. Survey Methods: Insights from the Field, Special issue: ‘Advancements in Online and Mobile Survey Methods’. DOI: 10.13094/SMIF-2020-00021. 1–7.
6.
BaisFSchoutenBToepoelV (2020) Investigating response patterns across surveys: Do respondents show consistency in undesirable answer behaviour over multiple surveys?Bulletin de Méthodologie Sociologique147: 150–68.
7.
BakkerJHaanMSchoutenB, et al. (2022) Testing the effects of automated navigation in a general population web survey. Methods, Data, Analyses16(1): 107–28.
8.
BisbeeJClintonJDorffC, et al. (2023) Synthetic replacements of human survey data? The perils of large language models. Political Analysis32(4): 401–16.
9.
BoschOJRevillaMPauraE (2019) Do Millennials differ in terms of survey participation?International Journal of Market Research61(4): 359–65.
10.
BurtonJLynnPBenzevalM (2020) How understanding society: The UK household longitudinal study adapted to the COVID-19 pandemic. Survey Research Methods14(2): 235–39.
11.
CaldariniGJafSMcGarryK (2022) A literature survey of recent advances in chatbots. Information13(41): 1–22.
12.
CallegaroMLozar ManfredaKVehovarV (2015) Web Survey Methodology. Los Angeles: SAGE.
13.
ČehovinGBosnjakMLozar ManfredaK (2022) Item nonresponse in web versus other survey modes: A systematic review and meta-analysis. Social Science Computer Review41(3): 926–45.
14.
CelinoIRe CalegariG (2020) Submitting surveys via a conversational interface: An evaluation of user acceptance and approach effectiveness. International Journal of Human-Computer Studies139: 1–16.
15.
CernatASakshaugJW (2021) Understanding the patterns of mode switching in longitudinal studies. Survey Research Methods15(3): 281–98.
16.
ChapraFHaalandI (2023) Conducting qualitative interviews with AI. CEBI Working Paper Series, No. 06/23, University of Copenhagen, Department of Economics, Center for Economic Behavior and Inequality (CEBI), Copenhagen. https://hdl.handle.net/10419/298540.
17.
ChuangCHLoJHWuYK (2023) Integrating Chatbot and augmented reality technology into biology learning during COVID-19. Electronics12(222): 1–29.
18.
CoffeySMaslovskayaOMcPheeC (2024) Recent innovations and advances in mixed-mode surveys. Journal of Survey Statistics and Methodology12: 507–31.
19.
ConradFGTourangeauRCouperMP, et al. (2017) Reducing speeding in web surveys by providing immediate feedback. Survey Research Methods11(1): 45–61.
20.
CuevasABrownEMScurrellJV, et al. (2023) Automated interviewer or augmented survey? Collecting social data with large language models. arXiv preprint arXiv:2309.10187.
21.
DaleTWalsoeH (2020) Optimizing grid questions for smartphones: A comparison of optimized and non-optimized designs and effects on data quality on different devices. In: BeattyPCCollinsDKayeL, et al. (eds) Advances in Questionnaire Design, Development, Evaluation, and Testing. Hoboken: Wiley, pp. 375–402.
22.
DillmanDA (2000) Mail and internet surverys. The tailored design method. New York: Wiley.
23.
EbadiSAminiA (2024) Examining the roles of social presence and human-likeness on Iranian EFL learners’motivation using artificial intelligence technology: A case of CSIEC chatbotInteractive Learning Environments32(2): 655–73.
24.
ErensBCollinsDManacordaT, et al. (2019) Comparing data quality from personal computers and mobile devices in an online survey among professionals. Social Research Practice Winter2019(7): 15–26.
25.
FeldererBKirchnerAKreuterF (2019) The effect of survey mode on data quality: Disentangling nonresponse and measurement error bias. Journal of Official Statistics35(1): 93–115.
26.
Jacinto RFigueroaArndtS (2018) Prevalence of the use of mobile instant messaging applications: The need to assess their usage in human factors investigations. Human Factors and Ergonomics Society Annual Meeting.406–10.
27.
GummerTChristmannPVerhoevenS, et al. (2022) Using a responsive survey design to innovate self-administered mixed-mode surveys. Journal of the Royal Statistical Society, Series A185: 916–32.
28.
GummerTHöhneJKRettigT, et al. (2023) Is there a growing use of mobile devices in web surveys? Evidence from 128 web surveys in Germany. Quality & Quantity57: 5333–53.
29.
HeerweghDLoosveldtG (2008) Face-to-face versus web surveying in a high-internet-coverage population. Differences in response quality. Public Opinion Quarterly72(5): 836–46.
30.
HuangWJiangJKingRB, et al. (2025) Chatbots and student motivation: a scoping review. International Journal of Educational Technology in Higher Education22(26): 1–27.
31.
IsraelGD (2010) Effects of answer space size on responses to open-ended questions in mail surveys. Journal of Official Statistics26(2): 271–85.
32.
JenkinsCRDillmanDA (1997) Towards a theory of self-administered questionnaire. In: LybergLEBiemerPPCollingsM, et al. (eds) Survey Measurement and Process Quality. New York: Wiley, pp. 165–96.
33.
KimSLeeJGweonG (2019) Comparing data from chatbot and web surveys: Effects of platform and conversational style on survey response quality. In: CHI, Glasgow.
34.
KnothNHahnelCEbersbachM (2025) Promoting online evaluation skills through educational chatbots. Computers in Human Behavior: Artificial Humans4: 100160.
35.
KrosnickJA (1991) Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology5(3): 213–36.
36.
LenznerTKaczmirekLLenznerA (2010) Cognitive burden of survey questions and response times: A psycholinguistic experiment. Applied Cognitive Psychology24(7): 1003–20.
37.
Manfreda KLozarBosnjakMBerzelakJ, et al. (2008) Web surveys versus other survey modes: A meta-analysis comparing response rates. International Journal of Market Research50(1): 79–104.
38.
LüdtkeDSchuppJ (2017) Wechsel von persönlichen Interviews zu web-basierten Interviews in einem laufenden Haushaltspanel. Befunde vom SOEP. In: EiflerSFaulbaumF (eds) Methodische Probleme von Mixed-Mode-Ansätzen in der Umfrageforschung. Wiesbaden: Springer VS, pp. 141–60.
39.
LugtigPToepoelVHaanM, et al. (2019) Recruiting young and urban groups into a probability-based online panel by promoting smartphone use. Methods, Data, Analyses13(2): 291–306.
40.
MavletovaA (2013) Data quality in PC and mobile web surveys. Social Science Computer Review31(6): 725–43.
41.
McCartyJAShrumLJ (2000) The measurement of personal values in survey research. A test of alternative rating procedures. Public Opinion Quarterly64(3): 271–98.
42.
McGonagleKASastryN (2023) Transitioning to a mixed-mode study design in a national household panel study: Effects on fieldwork outcomes, sample composition and costs. Survey Research Methods17(4): 411–27.
43.
OlsonKSmythJDHorwitzR, et al. (2021) Transitions from telephone surveys to self-administered and mixed-mode surveys: AAPOR Task Force Report. Journal of Survey Statistics and Methodology9: 381–411.
44.
OlsonKSmythJDPhillipsA, et al. (2019) Four questionnaire experiments in mixed-mode, mixed-device surveys: Answer boxes, response option order, check-all versus forced-choice, and ordinal scale versus number box items. In: 74th Annual Conference of the American Association for Public Opinion Research (AAPOR), Toronto, Canada, pp. 1-45.
45.
OudejansMChristianLM (2010) Using interactive features to motivate and probe responses to open-ended questions. In: DasMEsterPKaczmirekL (eds) Social and behavioral research and the internet. New York: Routledge, pp. 215–44.
RevillaMA (2015) Comparison of the quality estimates in a mixed-mode and a unimode design: an experiment from the European Social Survey. Quality & Quantity49: 1219–38.
48.
RevillaMACouperMP (2017) Comparing grids with vertical and horizontal item-by-item formats for PCs and smartphones. Social Science Computer Review. DOI: 10.1177/0894439317715626. 1–20.
49.
RevillaMAOchoaC (2016) Open narrative questions in PC and smartphones: is the device playing a role?Quality & Quantity50: 2495–2513.
50.
RevillaMAToninelliDOchoaC (2017) An experiment comparing grids and item-by-item formats in web surveys completed through PCs and smartphones. Telematics and Informatics34(1): 30–42.
51.
RhimJKwakMGongY, et al. (2022) Application of humanization to survey chatbots: Change in chatbot perception, interaction experience, and survey data quality. Computers in Human Behavior126: 1–12.
52.
RobertsCGilbertEAllumN, et al. (2019) Satisficing in surveys: A systematic review of the literature. Public Opinion Quarterly83(3): 598–626.
53.
SakshaugJWBesteJCobanM, et al. (2020) Impacts of the COVID-19 Pandemic on labor market surveys at the German Institute for Employment Research. Survey Research Methods14(2): 229–33.
54.
SchreppMHinderksAThomaschewskiJ (2017) Construction of a benchmark for the user experience questionnaire (ueq). International Journal of Interactive Multimedia and Artificial Intelligence4(4): 40–44.
55.
SelkäläACallegaroMCouperMP (2020) Automatic versus manual forwarding in web surveys - A cognitive load perspective on satisficing responding. In: MeiselwitzG (ed) Social Computing and Social Media. Design, Ethics, User Behavior, and Social Network Analysis. Springer Nature Switzerland, 130–55.
56.
SheehanKB (2002) Online research methodology: Reflections and speculations. Journal of Interactive Advertising3(1): 56–61.
57.
SöderströmAShatteAFuller-TyszkiewiczM (2021) Can intelligent agents improve data quality in online questiosnnaires? A pilot study. Behavior Research Methods53(5): 2238–51.
58.
ToepoelVLugtigPStruminskayaB, et al. (2020) Adapting surveys to the modern world: Comparing a research messenger design to a regular responsive design for online surveys. Survey Practice13(1): 1–10.
59.
ToepoelVMathonKTussenbroekP, et al. (2021) Probing in online mixed-device surveys: Is a research messenger layout more effective than a traditional online layout, especially on mobile devices?Bulletin de Méthodologie Sociologique151: 74–95.
60.
TourangeauRMaitlandARiveroG, et al. (2017) Web surveys by smartphone and tablets. Effects on survey responses. Public Opinion Quarterly81(4): 896–929.
61.
VehovarVCouperMPCehovinG (2022) Alternative layouts for grid questions in PC and mobile web surveys: An experimental evaluation using response quality indicators and survey estimates. Social Science Computer Review0(0): 1–23.
62.
WolfCChristmannPGummerT, et al. (2021) Conducting general social surveys as self-adminstered mixed-mode surveys. Public Opinion Quarterly85(2): 623–48.
63.
WooYKimSCouperMP (2015) Comparing a cell phone survey and a web survey of university students. Social Science Computer Review33(3): 399–410.
64.
WuCHLamHYKongA, et al. (2023) The design and evaluation of a digital learning-based English chatbot as an online self-learning method. International Journal of Engineering Business Management15(Feb): 1–22.
65.
WuttkeAAßenmacherMKlammC, et al. (2025) AI conversational interviewing: transforming surveys with LLMs as adaptive interviewers. In: Annual Conference of the American Association of Public Opinion Research, St. Louis (MO), United States.
66.
XiaoZZhouMXLiaoQV, et al. (2020) Tell me about yourself: Using an AI-powered chatbot to conduct conversational surveys with open-ended questions. ACM Transactions on Computer-Human Interaction1(1): 1–39.
67.
YunHSArjmandMSherlockP, et al. (2024) Keeping Users Engaged During Repeated Administration of the Same Questionnaire: Using Large Language Models to Reliably Diversify Questions. ACM International Conference on Intelligent Virtual Agents (IVA '24). DOI: 10.1145/3652988.3673929.
68.
ZaroualiBAraujoTOhmeJ, et al. (2023) Comparing chatbots and online surveys for (longitudinal) data collection: An investigation of response characteristics, data quality, and user evaluation. Communication Methods and Measures. DOI: 10.1080/19312458.2022.2156489. 1–20.
