This paper describes an experiment in which a single questionnaire was fielded in four different styles of presentation: Text Only, Decoratively Visual, Functionally Visual and Gamified. Respondents were randomly assigned to only one presentation version. To understand the effect of presentation style on survey experience and data quality, we compared response distributions, respondent behaviour (such as time to complete), and self-reports regarding the survey experience and level of engagement across the four experimental presentations. While the functionally visual and gamified treatments produced higher satisfaction scores from respondents, we found no real differences in respondent engagement measures. We also found few differences in response patterns.
BradburnN. (1977) Respondent burden. Health Survey Research Methods: Research Proceedings from the 2nd Biennial Conference. PHS Publication No 79-3207, pp. 49–53.
3.
CannellC.F., & KahnR.L. (1968) Interviewing, in LindzeyG., & AronsonE. (eds) Handbook of Social Psychology, Vol. 2. Reading, MA: Addison Wesley.
4.
CouperM.P., SingerE., TourangeauR., & ConradF.G. (2006) Evaluating the effectiveness of visual analog scales: a web experiment.Social Science Computer Review, 24, 2, pp. 227–245.
5.
Downes-Le GuinT., MechlingJ., & BakerR. (2006) Great results from ambiguous sources: cleaning internet panel data. ESOMAR World Research Conference: Panel Research 2006. Amsterdam: ESOMAR.
6.
FisherS. (2007) Practices that minimize online panelist satisficing behavior.Quirk's Marketing Research Review, March, p. 44.
7.
GalešićM., & BošnjakM. (2009) Effects of questionnaire length on participation and indicators of response quality in a web survey.Public Opinion Quarterly, 73, 2, pp. 349–360.
8.
GMIInteractive Whitepaper, (2011) Redesigning Mintel's Online Brand Research Survey to Re-Engage Respondents and Improve Data Quality.Bellvue, WA: Global Market Insight, Inc.
9.
HeerweghD., & LoosveldtG. (2008) Face-to-face versus web surveying in a high-internet-coverage population: differences in response quality.Public Opinion Quarterly, 72, 5, pp. 836–846.
10.
HerzogA.R., & BachmanJ.G. (1981) Effects of questionnaire length on response quality.Public Opinion Quarterly, 45, pp. 549–559.
11.
JohnsonL.D., SievekingN.A., & ClantonE.S.III (1974) Effects of alternative positioning of open-ended questions in multiple-choice questionnaires.Journal of Applied Psychology, 59, pp. 776–778.
12.
KrautA.I., WolfsonA.D., & RothenburgA. (1975) Some effects of position on opinion survey items.Journal of Applied Psychology, 60, pp. 774–776.
13.
KrosnickJ. (1991) Response strategies for coping with the cognitive demands of attitude measures in surveys.Applied Cognitive Psychology, 5, pp. 213–236.
14.
LugtigheidA., & RathodS. (2005) Questionnaire Length and Response Quality: Myth or Reality?Stamford, CT: Survey Sampling International.
15.
MalhotraN. (2008) Completion time and response order effects in web surveys.Public Opinion Quarterly, 72, 5, pp. 914–934.
McGonigalJ. (2011) Reality is Broken: Why Games Make Us Better and How They Can Change the World.New York, NY: Penguin Press HC.
18.
MillerC. (2009) Respondent technology preferences. CASRO Technology Conference, New York.
19.
MillerJ. (2007) Burke Panel Quality R&D Summary: ARF Online Research Quality Council.Cincinnati: Burke, Inc.
20.
MillerJ. (2008) Burke Panel Quality R and D.Cincinnati: Burke, Inc.
21.
PulestonJ. (2011a) Market Research Game Theory.Bellevue, WA: Global Market Insight, Inc.
22.
PulestonJ., & SleepD. (2011) The game experiments: researching how gaming techniques can be used to improve the quality of feedback from online research. ESOMAR Congress 2011, Amsterdam.
23.
ReidJ., MordenM., & ReidA. (2007) Maximizing respondent engagement: the use of rich media. ESOMAR Congress 2007, Berlin.
24.
SchellJ. (2008) The Art of Game Design: A Book of Lenses.Burlington, MA: Elsevier.
25.
SleepD., & PulestonJ. (2011) Measuring the Value of Respondent Engagement: Summary of Research Findings.Bellevue, WA: Global Market Insight.
