Studies examining the similarity of online self-report survey responses using different item formats have yielded inconclusive results. Additionally, no studies have used appropriate methods for thoroughly and correctly examining equivalence across conditions. We examined the comparability of survey responses across four item formats—horizontal radio button, text box, drop-down menu, and vertical radio button—in two studies. The second study added two response categories: optional responding and forced responding. Participants were college students at two institutions of higher education who were randomly assigned to conditions. They completed measures of computer self-efficacy, personality, and social desirability. Results of both studies indicated quantitative (mean scores) and qualitative (internal consistency estimates and scale intercorrelations) equivalence. However, there were notable differences in auxiliary equivalence such that participants in the text box condition had lower amounts of missing data than those in the other conditions, those in the horizontal radio button condition completed the study in the shortest amount of time, and participants across conditions generally preferred to use drop-down menus compared to other item formats.
ArnauR. C.ThompsonR. L.CookC. (2001). Do different response formats change the latent structure of responses? An empirical investigation using taxometric analysis. Educational and Psychological Measurement, 61, 23–44. https://doi.org/10.1177/00131640121971040
2.
AustinT. M.RichterR. R.ReinkingM. F. (2008). A primer on web surveys. Journal of Allied Health, 37, 180–186.
3.
BiJ.KuestenC. (2012). Intraclass correlation coefficient (ICC): A framework for monitoring and assessing performance of trained sensory panels and panelists. Journal of Sensory Studies, 27, 352–364. https://doi.org/10.1111/j.1745-459X.2012.00399.x
4.
BrehmJ. W. (1989). Psychological reactance: Theory and applications. In SrullT. K. (Ed.), Advances in consumer research: Volume16 (pp. 72–75). Association for Consumer Research.
5.
BrionesE. M.BenhamG. (2017). An examination of the equivalency of self-report measures obtained from crowdsources versus undergraduate student samples. Behavior Research Methods, 49, 320–334. https://doi.org/10.3758/s13428-016-0710-8
6.
BuchananT.JohnsonJ. A.GoldbergL. R. (2005). Implementing a five-factor personality inventory for use on the Internet. European Journal of Psychological Assessment, 21, 115–127. https://doi.org/10.1027/1015-5759.18.1.115
7.
BuskirkT. D.SaundersT.MichaudJ. (2015). Are sliders too slick for surveys? An experiment comparing slider and radio button scales for smartphone, tablet and computer based surveys. Methods, Data, Analyses, 9, 229–260. https://doi.org/10.12758/mda.2015.013
8.
ChristianL. M.ParsonsN. L.DillmanD. A. (2009). Designing scalar questions for web surveys. Sociological Methods & Research, 37, 39–425. https://doi.org/10.1177/0049124108330004
9.
CouperM. P.TourangeauR.ConradF. G.CrawfordS. D. (2004). What they see is what we get: Response options for web surveys. Social Science Computer Review, 22, 111–127. https://doi.org/10.1177/0894439303256555
10.
CrowneD. P.MarloweD. (1960). A new scale of social desirability independent of psychopathology. Journal of Consulting Psychology, 24, 349–354. https://doi.org/10.1037/h0047358
11.
de BruijneM.WijnantA. (2014). Improving response rates and questionnaire design for mobile web surveys. Public Opinion Quarterly, 78, 951–962. https://doi.org/10.1093/poq/nfu046
12.
DeCastellarnauA. (2018). A classification of response scale characteristics that affect data quality: A literature review. Quality and Quantity, 52, 1523–1559. https://doi.org/10.1007/s11135-017-0533-4
13.
DécieuxJ. P.MergenerA.NeufangK. M.SischkaP. (2015). Implementation of the forced answering option within online surveys: Do higher item response rates come at the expense of participation and answer quality?Psihologija, 48, 311–326. https://doi.org/10.2298/PSI1504311D
14.
DenizG.DurduP. O. (2019). A comparison of mobile form controls for different tasks. Computer Standards & Interfaces, 61, 97–106. https://doi.org/10.1016/j.csi.2018.06.001
15.
DillmanD. A.BowkerD. K. (2001). The web questionnaire challenge to survey methodologists. In ReipsU. D.BosnjakM. (Eds.), Dimensions of Internet science (pp. 53–71). Pabst Science.
16.
EckfordR. D.BarnettD. L. (2016). Comparing paper-and-pencil and Internet survey methods conducted in a combat-deployed environment. Military Psychology, 28, 209–225. https://doi.org/10.1037/mil0000118
FeldtL. S.WoodruffD. J.SalihF. A. (1987). Statistical inference for coefficient alpha. Applied Psychological Measurement, 11, 93–103. https://doi.org/10.1177/014662168701100107
19.
FischerD. G.FickC. (1993). Measuring social desirability: Short forms of the Marlowe-Crowne Social Desirability Scale. Educational and Psychological Measurement, 53, 417–424. https://doi.org/10.1177/0013164493053002011
20.
GoldbergL. R.JohnsonJ. A.EberH. W.HoganR.AshtonM. C.CloningerC. R.GoughH. G. (2006). The international personality item pool and the future of public-domain personality measures. Journal of Research in Personality, 40, 84–96. https://doi.org/10.1016/j.jrp.2005.08.007
21.
GräfL. (2002). Assessing Internet questionnaires: The online pretest lab. In BatinicB.ReipsU. D.BosnjakM. (Eds.), Online social sciences (pp. 49–68). Hogrefe & Huber.
22.
HarrisonA. W.RainerR. K. (1992). An examination of the factor structures and concurrent validities for the computer attitude scale, the computer anxiety rating scale, and the computer self-efficacy scale. Educational and Psychological Measurement, 52, 735–745. https://doi.org/10.1177/0013164492052003024
23.
HeerweghD.LoosveldtG. (2002). An evaluation of the effect of response formats on data quality in web surveys. Social Science Computer Review, 20, 471–484. https://doi.org/10.1177/089443902237323
24.
International Personality Item Pool. (2019). A scientific collaboratory for the development of advanced measures of personality and other individual differences. http://ipip.ori.org/
25.
KleinJ. D.ThomasR. K.SutterE. J. (2007). Self-reported smoking in online surveys: Prevalence estimate validity and item format effects. Medical Care, 45, 691–695. https://doi.org/10.1097/MLR.0b013e3180326145
LiuM.ConradF. G. (2016). An experiment testing six formats of 101-point rating scales. Computers in Human Behavior, 55, 364–371. https://doi.org/10.1016/j.chb.2015.09.036
28.
MaloshonokN.TerentevE. (2016). The impact of visual design and response formats on data quality in a web survey of MOOC students. Computers in Human Behavior, 62, 506–515. https://doi.org/10.1016/j.chb.2016.04.025
29.
MeyersonP.TryonW. W. (2003). Validating Internet research: A test of the psychometric equivalence of Internet and in-person samples. Behavior Research Methods, Instruments, & Computers, 35, 614–620. https://doi.org/10.3758/BF03195541
30.
MurphyC. A.CooverD.OwenS. V. (1989). Development and validation of the computer self-efficacy scale. Educational and Psychological Measurement, 49, 893–899. https://doi.org/10.1177/001316448904900412
PreckelF.ThiemannH. (2003). Online- versus paper-pencil-version of a high potential intelligence test. Swiss Journal of Psychology, 62, 131–138. https://doi.org/10.1024//1421-0185.62.2.131
RogersJ. L.HowardK. I.VesseyJ. T. (1993). Using significance tests to evaluate equivalence between two experimental groups. Psychological Bulletin, 113, 553–565. https://doi.org/10.1037/0033-2909.113.3.553
35.
RusticusS. A.LovatoC. Y. (2011). Applying tests of equivalence for multiple group comparisons: Demonstration of the confidence interval approach. Practical Assessment, Research & Evaluation, 16(11), 1–6.
36.
RusticusS. A.LovatoC. Y. (2014). Impact of sample size and variability on the power and Type I error rates of equivalence test: A simulation study. Practical Assessment, Research & Evaluation, 19(11), 1–11.
37.
StiegerS.ReipsU. D.VoracekM. (2007). Forced-response in online surveys: Bias from reactance and an increase in sex-specific dropout. Journal of the American Society for Information Science and Technology, 58, 1653–1660. https://doi.org/10.1002/asi.20651
38.
TabachnickB. G.FidellL. S. (2001). Using multivariate statistics (4th ed.). Allyn & Bacon.
39.
TourangeauR. (2014, October). Measurement properties of web surveys. [Paper presentation] Statistics Canada Symposium, Gatineau, Québec, Canada.
40.
TryonW. W. (2001). Evaluating statistical difference, equivalence, and indeterminacy using inferential confidence intervals: An integrated alternative method of conducting null hypothesis statistical tests. Psychological Methods, 6, 371–386. https://doi.org/10.1037/1082-989X.6.4.371
41.
VicenteP.ReisE. (2010). Using questionnaire design to fight nonresponse bias in web surveys. Social Science Computer Review, 28, 251–267. https://doi.org/10.1177/0894439309340751
42.
WeigoldA.WeigoldI. K.DrakefordN. M.DykemaS. A.SmithC. A. (2016). Equivalence of paper-and-pencil and computerized self-report surveys in older adults. Computers in Human Behavior, 54, 407–413. https://doi.org/10.1016/j.chb.2015.08.033
43.
WeigoldA.WeigoldI. K.NateraS. N. (2018). Mean scores for self-report surveys completed using paper-and-pencil and computers: A meta-analytic test of equivalence. Computers in Human Behavior, 86, 153–164. https://doi.org/10.1016/j.chb.2018.04.038
44.
WeigoldA.WeigoldI. K.RussellE. J. (2013). Examination of the equivalence of self-report survey-based paper-and-pencil and Internet data collection methods. Psychological Methods, 18, 53–70. https://doi.org/10.1037/a0031607
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
