Effects of the Number of Open-Ended Probing Questions on Response Quality in Cognitive Online Pretests

Abstract

Cognitive online pretests have, in recent years, become recognized as a promising tool for evaluating questions prior to their use in actual surveys. While existing research has shown that cognitive online pretests produce similar results to face-to-face cognitive interviews with regard to the problems detected and the item revisions suggested, little is known about the ideal design of a cognitive online pretest. This study examines whether the number of open-ended probing questions asked during a cognitive online pretest has an effect on the quality and depth of respondents’ answers as well as on respondents’ satisfaction with the survey. We conducted an experiment in which we varied the number of open-ended probing questions that respondents received during a cognitive online pretest. The questionnaire consisted of 26 survey questions, and respondents received either 13 probing questions (n = 120, short version) or 21 probing questions (n = 120, long version). The findings suggest that asking a greater number of open-ended probes in a cognitive online pretest does not undermine the quality of respondents’ answers represented by the response quality indicators: (1) amount of probe nonresponse, (2) number of uninterpretable answers, (3) number of dropouts, (4) number of words, (5) response times, and (6) number and type of themes covered by the probes. Furthermore, the respondents’ satisfaction with the survey is not affected by the number of probes being asked.

Keywords

open-ended questions probes online pretest web probing questionnaire design web survey

Get full access to this article

View all access options for this article.

References

Behr

Bandilla

Kaczmirek

Braun

(2014). Cognitive probes in web surveys: On the effect of different text box size and probing exposure on response quality. Social Science Computer Review, 32, 524–533.

Behr

Kaczmirek

Bandilla

Braun

(2012). Asking probing questions in web surveys: Which factors have an impact on the quality of responses? Social Science Computer Review, 30, 487–498.

Borg

Zuell

(2012). Write-in comments in employee surveys. International Journal of Manpower, 33, 206–220.

Denscombe

(2008). The length of responses to open-ended questions. A comparison of online and paper questionnaires in terms of a mode effect. Social Science Computer Review, 26, 359–368.

Dillman

D. A.

Smyth

J. D.

Christian

L. M.

(2009). Internet, mail, and mixed-mode surveys: The tailored design method (3rd ed.). Hoboken, NJ: John Wiley.

Kaczmirek

(2005). A framework for the collection of universal client side paradata (UCSP). Retrieved March 11, 2019, from http://kaczmirek.de/ucsp/ucsp.html

Kwak

Radler

(2002). A comparison between mail and web surveys: Response pattern, respondent profile, and data quality. Journal of Official Statistics, 18, 257–273.

Lenzner

Disch

Gebhardt

Menold

(2015). AUDITS—Methodological tools for the definition of local security policies. (Cognitive online pretest. GESIS Project Report [Version 1.0]). doi:10.17173/pretest12

Lenzner

Kaczmirek

Lenzner

(2010). Cognitive burden of survey questions and response times: A psycholinguistic experiment. Applied Cognitive Psychology, 24, 1003–1020.

10.

Lenzner

Neuert

C. E.

(2017). Pretesting survey questions via web probing—Does it produce similar results to face-to-face cognitive interviewing? Survey Practice, 10, 1–11.

11.

Meitinger

Behr

(2016). Comparing cognitive interviewing and online probing: Do they find similar results? Field Methods, 28, 363–380.

12.

Oudejans

Christian

L. M.

(2010). Using interactive features to motivate and probe responses to open-ended questions. In Das

Ester

Kaczmirek

(Eds.), Social and behavioral research and the internet (pp. 215–244). New York, NY: Routledge.

13.

Ratcliff

(1993). Methods for dealing with reaction time outliers. Psychological Bulletin, 114, 510–532.

14.

Reja

Manfreda

K. L.

Hlebec

Vehovar

(2003). Open-ended vs. close-ended questions in web questionnaires. Metodoloski zvezki, 19, 159–177

15.

Scholz

Zuell

(2012). Item non-response in open-ended questions: Who does not answer on the meaning of left and right? Social Science Research, 41, 1415–1428.

16.

Schuman

(1966). The random probe: A technique for evaluating the validity of closed questions. American Sociological Review, 31, 218–222.

17.

Willis

G. B.

(2004). Cognitive interviewing: A tool for improving questionnaire design. Sage Publications.

18.

Züll

. (2016). Open-ended questions. GESIS survey guidelines. Mannheim, Germany: GESIS—Leibniz Institute for the Social Sciences. doi:10.15465/gesis-sg_en_002

19.

Züll

Menold

Körber

(2014). The influence of answer box size on item nonresponse to open-ended questions in a web survey. Social Science Computer Review, 33, 115–122.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.33 MB