Accurate collection of response times is one of the main advantages of web-administered stated choice experiments and it can be thought of as a behavioral indicator of cognitive effort. We use data from a food choice experiment administered across three countries and estimate a panel Mixed Multinomial Logit Model to obtain individual-specific utility weights. These are used to construct two utility-based measures of contextual choice complexity, which are combined with subjective measures of cognitive resources as well as indicators of opt-out selection. We first develop and then test hypothesized effects of complexity at the level of single choice task and choice sequence on response times. By using a log-linear random effects model with choice task response-time as dependent variable, we isolate these effects from other background variables. Results suggest that as our measures of complexity increase, so do response times and such effects are robust across the three countries. We argue that these results broadly support the validity of web-based choice surveys to measure food preference. We suggest that computers can help improve survey design by implementing algorithms to improve the overall efficiency of choice tasks design, for example, by using adaptive design algorithms that control cognitive challenges in accordance with the respondent’s predicted ability to tackle cognitive effort.
ArentzeT.BorgersA.TimmermansH.Del MistroR. (2003). Transport stated choice responses: Effects of task complexity, presentation format and literacy, Transportation Research Part E: Logistics and Transportation Review, 39(3), 229–244. https://doi.org/10.1016/S1366-5545(02)00047-9
2.
BechM.KjaerT.LauridsenJ. (2011). Does the number of choice sets matter? Results from a web survey applying a discrete choice experiment, Health Economics, 20(3), 273–286. https://doi.org/10.1002/hec.1587
3.
Ben-AkivaM.LermanS. R. (1985). Discrete choice analysis. MIT Press.
4.
BlatterH.CajochenC. (2007). Circadian rhythms in cognitive performance: Methodological constraints, protocols, theoretical underpinning. Physiology and Behaviour, 90(2–3), 196–208. https://doi.org/10.1016/j.physbeh.2006.09.009
5.
BonsallP.LythgoeB. (2009). Factors affecting the amount of effort expended in responding to questions in behavioural choice experiments. Journal of Choice Modelling, 2(2), 216–236. https://doi.org/10.1016/S1755-5345(13)70011-3
6.
BörgerT. (2016). Are fast responses more random? Testing the effect of response time on scale in an online choice experiment, Environmental and Resource Economics, 65(2), 389–413. https://doi.org/10.1007/s10640-015-9905-1
7.
BrownT. C.KingsleyD.PetersonG. L.FloresN. E.ClarkeA.BirjulinA. (2008). Reliability of individual valuations of public and private goods: Choice consistency, response time, and preference refinement. Journal of Public Economics, 92(7), 1595–1606. https://doi.org/10.1016/j.jpubeco.2008.01.004
8.
CallegaroM.VillarA.KrosnickJ.YeagerD. (2014). A critical review of studies investigating the quality of data obtained with online panels. In CallegaroM.BakerR.BethlehemJ.GoritzA.KrosnickJ.LavrakasP. (Eds.), Online panel research: A data quality perspective (pp. 23–53). John Wiley & Sons.
9.
CameronT. A.DeShazoJ. R. (2010). Differential attention to attributes in utility-theoretic choice models. Journal of Choice Modelling, 3(3), 73-115. https://doi.org/10.1016/S1755-5345(13)70015-0
10.
CampbellD.ErdemS. (2019). Including opt-out options in discrete choice experiments: Issues to consider. The Patient-Patient-Centered Outcomes Research, 12(1), 1–14. https://doi.org/10.1007/s40271-018-0324-6
11.
CampbellD.MørkbakM. R.OlsenS. B. (2017). Response time in online stated choice experiments: the non-triviality of identifying fast and slow respondents. Journal of Environmental Economics and Policy, 6, 17-35. https://doi.org/10.1080/21606544.2016.1167632
12.
CampbellD.MørkbakM. R.OlsenS. B. (2018). The link between response time and preference, variance and processing heterogeneity in stated choice experiments. Journal of Environmental Economics and Management, 88(C), 18–34. https://doi.org/10.1016/j.jeem.2017.10.003
13.
CasparC. G.KroesenM. (2014). On the (im-)possibility of deriving transport policy implications from hybrid choice models. Transport Policy, 36, 217–222. https://doi.org/10.1016/j.tranpol.2014.09.001
14.
CaussadeS.OrtúzarJ. d. D.RizziL. I.HensherD. A. (2005). Assessing the influence of design dimensions on stated choice experiment estimates. Transportation Research Part B: Methodological, 39(7), 621–640. https://doi.org/10.1016/j.trb.2004.07.006
15.
ConradF. G.CouperM. P.TourangeauR.ZhangC. (2017). Reducing speeding in web surveys by providing immediate feedback. Survey Research Methods, 11(1), 45–61. https://doi.org/10.18148/srm/2017.v11i1.6304
16.
CookJ.JeulandM.MaskeryB.WhittingtonD. (2012). Giving stated preference respondents “time to think”: Results from four countries. Environmental and Resource Economics, 51(4), 473–496. https://doi.org/10.1007/s10640-011-9508-4
17.
CookJ.WhittingtonD.CanhD. G.JohnsonF. R.NyameteA. (2007). Reliability of stated preferences for cholera and typhoid vaccines with time to think in Hue, Vietnam. Economic Inquiry, 45(1), 100–114. https://doi.org/10.1111/j.1465-7295.2006.00038.x
18.
CouperM. (2000). Usability evaluation of computer-assisted survey instruments. Social Science Computer Review, 18(4), 384–396. https://doi.org/10.1177/089443930001800402
19.
CouperM.MillerP. V. (2008). Special issue: Web survey methods. Public Opinion Quarterly, 72(5), 831–835. https://doi.org/10.1093/poq/nfn066
20.
CouperM. P.PetersonG. J. (2016). Why do web surveys take longer on smartphones?Social Science Computer Review, 35(3), 357–377. https://doi.org/10.1177/0894439316629932
21.
CrawfordS. D.CouperM. P.LamiasM. J. (2001). Web surveys: Perceptions of burden. Social Science Computer Review, 19(2), 146–162. https://doi.org/10.1177/089443930101900202
22.
DayB.BatemanI.CarsonR.DupontD.LouviereJ.MorimotoS.ScarpaR.WangP. (2012). Ordering effects and choice set awareness in repeat-response stated preference studies. Journal of Environmental Economics and Management, 63(1), 73–91. https://doi.org/10.1016/j.jeem.2011.09.001
23.
DeShazoJ. RFermoF. (2002). Designing choice sets for stated preference methods: The effects of complexity on choice consistency. Journal of Environmental Economics and Management, 44(1), 123–143. https://doi.org/10.1006/jeem.2001.1199
24.
DharR. (1997). Consumer preference for a no‐choice option. Journal of Consumer Research, 24(2), 215–231. https://doi.org/10.1086/209506
25.
DuquetteE. N. (2010). Choice difficulty and risk perceptions in environmental economics. [Ph.D. Thesis, Department of Economics, Graduate School - University of Oregon].
26.
Espinoza-VarasB.WatsonC. S. (1994). Effects of decision criterion on response latencies of binary decisions. Perception & Psychophysics, 55(2), 190–203. https://doi.org/10.3758/BF03211666
27.
FanW.YanZ. (2010). Factors affecting response rates of the web survey: A systematic review. Computers in Human Behavior, 26(2), 132–139. https://doi.org/10.1016/j.chb.2009.10.015
28.
FazioR. H. (1990). A practical guide to the use of response latency in social psychological research. In HendrickC.ClarkM. S. (Eds.), Research methods in personality and social psychology (pp. 74–97). Sage.
29.
FischhoffB. (2005). Cognitive processes in stated preference methods. In MalerK. G.VincentJ. (Eds.), Handbook of environmental economics (pp. 937–968). Elsevier. https://doi.org/10.1016/S1574-0099(05)02018-8
30.
GalesicM.BosnjakM. (2009). Effects of questionnaire length on participation and indicators of response quality in a web survey. Public Opinion Quarterly, 73(2), 349–360. https://doi.org/10.1093/poq/nfp031
31.
GreszkiR.MeyerM.SchoenH. (2015). Exploring the effects of removing “too fast” responses and respondents from web surveys. The Public Opinion Quarterly, 79(2), 471–503. https://doi.org/10.1093/poq/nfu058
32.
GummerT.RossmanJ. (2015). Explaining interview duration in web surveys: A multilevel approach, Social Science Computer Review, 33(2), 217–234https://doi.org/10.1177/0894439314533479
33.
HaaijerR.KamakuraW.WedelM. (2000). Response latencies in the analysis of conjoint choice experiments. Journal of Market Research, 37(3), 376–382. https://doi.org/10.1509/jmkr.37.3.376.18775
34.
HeerweghD. (2003). Explaining response latencies and changing answers using client-side paradata from a web survey. Social Science Computer Review, 21(3), 360–373. https://doi.org/10.1177/0894439303253985
35.
HenningerM.PlieningerH. (2020). Different styles, different times: how response times can inform our knowledge about the response process in rating scale measurement. Assessment, https://doi.org/10.1177/1073191119900003
36.
HessS.RoseJ. M.PolakJ. (2010). Non-trading, lexicographic and inconsistent behaviour in stated choice data. Transportation Research Part D: Transport and Environment, 15(7), 405–417. https://doi.org/10.1016/j.trd.2010.04.008
37.
HessS.StathopoulosA. (2013). Linking response quality to survey engagement: A combined random scale and latent variable approach. Journal of Choice Modelling, 7, 1–12. https://doi.org/10.1016/j.jocm.2013.03.005
38.
HillygusD. S.JacksonN.YoungM. (2014). Professional respondents in non-probability online panels. In CallegaroM.BakerR.BethlehemJ.GoritzA.KrosnickJ.LavrakasP. (Eds.), Online panel research: A data quality perspective (pp. 219–237). John Wiley & Sons. https://doi.org/10.1002/9781118763520.ch10
39.
HöhneJ. K.SchlosserS.CouperM. P.BlomA. G. (2020). Switching away: Exploring on-device media multitasking in web surveys. Computers in Human Behavior, 111, 106417. https://doi.org/10.1016/j.chb.2020.106417
40.
HöhneJ. K.SchlosserS.KrebsD. (2017). Investigating cognitive effort and response quality of question formats in web surveys using paradata. Field Methods, 29(4), 365–382. https://doi.org/10.1177/1525822X17710640
41.
HolmesT.AlgerK.ZinkhanC.MercerE. (1998). The effect of response time on conjoint analysis estimates of rainforest protection values. Journal of Forest Economics, 4(1), 7–28.
42.
JohnsonR. M.OrmeB. K. (1996). How many questions should you ask in choice-based conjoint studies? Technical Paper, Sawtooth Software.
43.
JohnstonR. J.BoyleK. J.AdamowiczW.BennettJ.BrouwerR.CameronT. A.HanemannW. M.HanleyN.RyanM.ScarpaR.TourangeauR.VosslerC. A., (2017). Contemporary guidance for stated preference studies. Journal of the Association of Environmental and Resource Economics, 4(2), 319–405. https://doi.org/10.1086/691697
44.
Kaye-BlakeW. H.AbellW. L.ZellmanE. (2009). Respondents’ ignoring of attribute information in a choice modelling survey. Australian Journal of Agricultural and Resource Economics, 53(4), 547–564. https://doi.org/10.1111/j.1467-8489.2009.00467.x
45.
KreuterF.CouperM. P.LybergL. E. (2010). The use of paradata to monitor and manage survey data collection. In JSM Proceedings, Survery Research Methods Section. American Statistical Association. July 31 - August 5, 2010. Vancouver, British Columbia, Canada.
46.
KunzT.GummerT. (2019). Understanding respondents’ attitudes toward web paradata use, Social Science Computer Review, 38(6), 739–753https://doi.org/10.1177/0894439319826904
47.
LenznerT.KaczmirekL.LenznerA. (2010). Cognitive burden of survey questions and response times: A psycholinguistic experiment. Applied Cognitive Psychology, 24(7), 1003–1020. https://doi.org/10.1002/acp.1602
48.
LiebeU.GlenkK.OehlmannM.MeyerhoffJ. (2015). Does the use of mobile devices (tablets and smartphones) affect survey quality and choice behaviour in web surveys?Journal of Choice Modelling, 14, 17–31. https://doi.org/10.1016/j.jocm.2015.02.002
49.
LindhjemH.NavrudS. (2011). Are internet surveys an alternative to face-to-face interviews in contingent valuation?Ecological Economics, 70(9), 1628–1637. https://doi.org/10.1016/j.ecolecon.2011.04.002
LuceR. D. (1986). Response times: Their role in inferring elementary mental organization. Oxford University Press.
52.
LuceR. D. (1957). A theory of individual choice behavior. Columbia University, Departments of Mathematical Statistics and Sociology and the Bureau of Applied Social Research.
53.
MacMillanD. C.PhilipL.HanleyN.Alvarez-FarizoB. (2002). Valuing the nonmarket benefits of wild goose conservation: A comparison of interview and group-based approaches. Ecological Economics, 43(1), 49–59. https://doi.org/10.1016/S0921-8009(02)00182-9
54.
MatjašičM.VehovarV.ManfredaK.L. (2018). Web survey paradata on response time outliers: A systematic literature review. Metodološki zvezki, 15(1), 23–41. https://doi.org/10.51936/yoqn3590
55.
MayerlJ. (2013). Response latency measurement in surveys. Detecting strong attitudes and response effects. Survey Methods: Insights from the Field. https://doi.org/10.13094/SMIF-2013-00005
56.
McClainC. A.CouperM. P.HuppA. L.KeuschF.PetersonG.PiskorowskiA. D.WestB. T. (2019). A typology of web survey paradata for assessing total survey error. Social Science Computer Review, 37(2), 196–213. https://doi.org/10.1177/0894439318759670
57.
McFaddenD. (1974). Conditional logit analysis of qualitative choice behavior. In ZarembkaP. (Ed.), Frontiers in econometrics (pp. 105–142). Academic Press.
58.
McFaddenD. (1980). Econometric models for probabilistic choice among products. Journal of Business, 53(3), 13–29.
MertenT.RuchW. (1996). A comparison of computerised and conventional administration of the German version of the Eysenck personality questionnaire and the Carnell rating scale for depression. Personality and Individual Differences, 20(3), 281–291. https://doi.org/10.1016/0191-8869(95)00185-9
62.
ReveltD.TrainK. (1998). Mixed logit with repeated choices: Households’ choices of appliance efficiency level. Review of Economics and Statistics, 80(4), 647–657. https://doi.org/10.1162/003465398557735
63.
RoseJ.BlackI. (2006). Means matter, but variance matter too: Decomposing response latency influences on variance heterogeneity in stated preference experiments. Marketing Letters, 17(4), 295–310. https://doi.org/10.1007/s11002-006-8632-3
64.
SafirA.BlackT.SteinbachR. (2001). Using paradata to examine the effects of interviewer characteristics on survey response and data quality. In: Proceedings of the Annual Meeting of the American Statistical Association. August 5-9,2001. Atlanta, GA, United States.
65.
ScarpaR.GilbrideT. J.CampbellD.HensherD. A. (2009). Modelling attribute non-attendance in choice experiments for rural landscape valuation. European Review of Agricultural Economics, 36(2), 151–174. https://doi.org/10.1093/erae/jbp012
66.
ScarpaR.NotaroS.LouviereJ. J.RaffaelliR. (2011). Exploring scale effects of best/worst rank ordered choice data to estimate benefits of tourism in alpine grazing commons. American Journal of Agricultural Economics, 3(3), 813–828. https://doi.org/10.1093/ajae/aaq174
67.
ScarpaR.RoseJ. M. (2008). Design efficiency for nonmarket valuation with choice modelling: How to measure it, what to report and why. Australian Journal of Agricultural and Resource Economics, 52(3), 253–282. https://doi.org/10.1111/j.1467-8489.2007.00436.x
68.
ScottA.JeonS. H.JoyceC. M.HumphreysJ. S.KalbG.WittJ.LeahyA. (2011). A randomised trial and economic evaluation of the effect of response mode on response rate, response bias, and item non-response in a survey of doctors. BMC Medical Research Methodology, 11, 1–12. https://doi.org/10.1186/1471-2288-11-126
69.
SládekM.Kudrnáčová RöschováM.AdámkováV.HamplováD.SumováA. (2020). Chronotype assessment via a large scale socio-demographic survey favours yearlong standard time over daylight saving time in central Europe. Scientific Reports (Nature Research), 10(1), 1419. https://doi.org/10.1038/s41598-020-58413-9
70.
SternM. J. (2008). The use of client-side paradata in analyzing the effects of visual layout on changing responses in web surveys. Field Methods, 20(4), 377–398. https://doi.org/10.1177/1525822X08320421
71.
SvedsäterH. (2007). Ambivalent statements in contingent valuation studies: Inclusive response formats and giving respondents time to think. Australian Journal of Agricultural and Resource Economics, 51(1), 91–107. https://doi.org/10.1111/j.1467-8489.2007.00346.x
72.
SwaitJ. (2001). A non-compensatory choice model incorporating attribute cutoffs. Transportation Research Part B: Methodological, 35(10), 903–928. https://doi.org/10.1016/S0191-2615(00)00030-8
73.
SwaitJ.AdamowiczW. (2001a). Choice environment, market complexity, and consumer behavior: A theoretical and empirical approach for incorporating decision complexity into models of consumer choice. Organizational Behavior and Human Decision Processes, 86(2), 141–167. https://doi.org/10.1006/obhd.2000.2941
74.
SwaitJ.AdamowiczW. (2001b). The influence of task complexity on consumer choice: A latent class model of decision strategy switching. Journal of Consumer Research, 28(1), 135–148. https://doi.org/10.1086/321952
75.
TrainK. E. (2009). Discrete choice methods with simulation. (2nd ed.). Cambridge University Press.
76.
VassC. M.BoeriM. (2021). Mobilising the next generation of stated-preference studies: The association of access device with choice behaviour and data quality. The Patient-Patient-Centered Outcomes Research, 14(1), 55–63. https://doi.org/10.1007/s40271-020-00484-x
77.
VijA.WalkerJ. L. (2016). How, when and why integrated choice and latent variable models are latently useful. Transportation Research Part B: Methodological, 90, 192–217. https://doi.org/10.1016/j.trb.2016.04.021
78.
VistaA. B.RosenbergerR. S.CollinsA. R. (2009). If you provide it, will they read it? Response time effects in a choice experiment. Canadian Journal of Agricultural Economics, 57(3), 365–377. https://doi.org/10.1111/j.1744-7976.2009.01156.x
79.
VörösZ.RouetJ.-F. (2016). Laypersons’ digital problem solving: Relationships between strategy and performance in a large-scale international survey. Computers in Human Behavior, 64, 108–116. https://doi.org/10.1016/j.chb.2016.06.018
80.
WellsT.BaileyJ. T.LinkM. W. (2013). Comparison of smartphone and online computer survey administration. Social Science Computer Review, 32(2), 238–255. https://doi.org/10.1177/0894439313505829
81.
WestbrookA.BraverT. S. (2015). Cognitive effort: A neuroeconomic approach. Cognitive, Affective, & Behavioral Neuroscience15(2), 395–415. https://doi.org/10.3758/s13415-015-0334-y
82.
WhittingtonD.SmithV. K.OkoraforA.OkoreA.LiuJ. L.McPhailA. (1992). Giving respondents time to think in contingent valuation studies: A developing country application. Journal of Environmental Economics and Management, 22(3), 205–225. https://doi.org/10.1016/0095-0696(92)90029-V
83.
YanT.TourangeauR. (2008). Fast times and easy questions: The effects of age, experience and question complexity on web survey response times. Applied Cognitive Psychology, 22(1), 51–68. https://doi.org/10.1002/acp.1331
84.
YaoR.T.ScarpaR.RoseJ.M.TurnerJ.A. (2015). Experimental design criteria and their behavioural efficiency: An evaluation in the field. Environmental and Resource Economics, 62(3), 433–455. https://doi.org/10.1007/s10640-014-9823-7
85.
ZhangC.ConradF. (2014). Speeding in web surveys: The tendency to answer very fast and its association with straightlining. Survey Research Methods, 8(2), 127–135. https://doi.org/10.18148/srm/2014.v8i2.5453
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
