Little is known about the reliability and validity in web surveys, although this is crucial information to evaluate how accurate the results might be and/or to correct for measurement errors. In particular, there are few studies based on probability-based samples for web surveys, looking at web-specific response scales and considering the impact of having smartphone respondents. In this article, we start filling these gaps by estimating the measurement quality of sliders compared to radio button scales controlling for the device respondents used. We conducted therefore two multitrait–multimethod (MTMM) experiments in the Norwegian Citizen Panel (NCP), a probability-based online panel. Overall, we find that if smartphone respondents represent a nonnegligible part of the whole sample, offering the response options in form of a slider or a radio button scale leads to a quite similar measurement quality. This means that sliders could be used more often without harming the data quality. Besides, if there are no smartphone respondents, we find that sliders can also be used, but that the marker should be placed initially in the middle rather than on the left side. However, in practice, there is no need to shift from radio buttons to sliders since the quality is not highly improved by providing sliders.
AndrewsF. M. (1984). Construct validity and error of survey measures: A structural modeling approach. Public Opinion Quarterly, 48, 409–442.
3.
BuskirkT. D.AndrusC. H. (2014). Making mobile browser surveys smarter: Results from a randomized experiment comparing online surveys completed via computer or smartphone. Field Methods, 26, 322–342. Retrieved fromhttps://doi.org/10.1177/1525822X14526146
4.
BuskirkT. D.SaundersT.MichaudJ. (2015). Are sliders too slick for surveys? An experiment comparing slider and radio button scales for smartphone, tablet and computer based surveys. Methods, Data, Analyses: A Journal for Quantitative Methods and Survey Methodology, 9, 229–260.
5.
CampbellD. T.FiskeD. W. (1959). Convergent and discriminant validation by the multitrait–multimethod matrix. Psychological Bulletin, 56, 81–105. Retrieved fromhttps://doi.org/10.1037/h0046016
ChristianL. M.DillmanD. A.SmythJ. D. (2007). Helping respondents get it right the first time: The influence of words, symbols, and graphics in Web surveys. Public Opinion Quarterly, 71, 113–125. Retrieved fromhttps://doi.org/10.1093/poq/nfl039
8.
CobanogluC.WardeB.MoreoP. J. (2001). A comparison of mail, fax and web-based survey methods. International Journal of Market Research, 43, 405–410.
9.
CookC.HeathF.ThompsonR. L.ThompsonB. (2001). Score reliability in webor internet-based surveys: Unnumbered graphic rating scales versus likert-type scales. Educational and Psychological Measurement, 61, 697–706. Retrieved fromhttps://doi.org/10.1177/00131640121971356
10.
CortenI. W.SarisW. E.CoendersG.van der VeldW. M.AalbertsC. E.KornelisC. (2002). Fit of different models for multitrait–multimethod experiments. Structural Equation Modeling: A Multidisciplinary Journal, 9, 213–232. Retrieved fromhttps://doi.org/10.1207/S15328007SEM0902_4
11.
CouperM. P.MillerP. V. (2008). Web survey methods: Introduction. Public Opinion Quarterly, 72, 831–835. Retrieved fromhttps://doi.org/10.1093/poq/nfn066
12.
CouperM. P.TourangeauR.ConradF. G.CrawfordS. D. (2004). What they see is what we get: Response options for web surveys. Social Science Computer Review, 22, 111–127. Retrieved fromhttps://doi.org/10.1177/0894439303256555
13.
CouperM. P.TourangeauR.ConradF. G.SingerE. (2006). Evaluating the effectiveness of visual analog scales: A web experiment. Social Science Computer Review, 24, 227–245. Retrieved fromhttps://doi.org/10.1177/0894439305281503
14.
DeCastellarnauA.RevillaM. (2017). Two approaches to evaluate measurement quality in online surveys: An application using the Norwegian citizen panel. Survey Research Methods, 11(4), 415–433. https://doi.org/10.18148/srm/2017.v11i4.7226
de LeeuwE. D.HoxJ.DillmanD. (Eds.) (2008). International handbook of survey methodology. New York, NY: Taylor & Francis. Retrieved fromhttps://doi.org/10.4324/9780203843123
17.
DillmanD. A. (2000). Mail and internet surveys: The tailored design method. New York, NY: John Wiley.
18.
DillmanD. A.PhelpsG.TortoraR.SwiftK.KohrellJ.BerckJ.MesserB. L. (2008). Response rate and measurement in mixed mode surveys using mail, telephone, interactive voice response, and internet. Social Science Research, 38, 1–48.
19.
FunkeF. (2016). A web experiment showing negative effects of slider scales compared to visual analogue scales and radio button scales. Social Science Computer Review, 34, 244–254. Retrieved fromhttps://doi.org/10.1177/0894439315575477
20.
FunkeF.ReipsU.ThomasR. K. (2011). Sliders for the smart: Type of rating scale on the web interacts with educational level. Social Science Computer Review, 29, 221–231. Retrieved fromhttps://doi.org/10.1177/0894439310376896
21.
HeerweghD. (2009). Mode differences between face-o-face and web surveys: An experimental investigation of data quality and social desirability effects. International Journal of Public Opinion Research, 21, 111–121. Retrieved fromhttps://doi.org/doi:10.1093/ijpor/edn054
KreuterF.PresserS.TourangeauR. (2008). Social desireability bias in CATI, IVR, and web surveys: The effects of modeand question sensibility. Public Opinion Quarterly, 72, 847–865. Retrieved fromhttps://doi.org/10.1093/poq/nfn063
LiuM.ConradF. G. (2016). An experiment testing six formats of 101-point rating scales. Computers in Human Behavior, 55, 364–371. Retrieved fromhttps://doi.org/10.1016/j.chb.2015.09.036
28.
RevillaM. (2010). Quality in unimode and mixed-mode designs: A Multitrait-Multimethod approach. Survey Research Methods, 4(3), 151–164.
29.
RevillaM. (2015). Comparison of the quality estimates in a mixed-mode and a unimode design: an experiment from the European Social Survey. Quality and Quantity, 49(3), 1219–1238. https://doi.org/10.1007/s11135-014-0044-5
30.
RevillaM.OchoaC. (2015). Quality of different scales in an online survey in Mexico and Colombia. Journal of Politics in Latin America, 7, 157–177.
31.
RevillaM.OchoaC.LoeweG. (2013). How the form of asking respondents’ three preferred brands changes the results. Metodologia de Encuestas, 16, 67–82.
32.
RevillaM.SarisW. E. (2013a). A Comparison of the Quality of Questions in a Face-to-face and a Web Survey. International Journal of Public Opinion, 25(2), 419–432.
33.
RevillaM.SarisW. E. (2013b). The split-ballot multitrait–multimethod approach: Implementation and problems. Structural Equation Modeling: A Multidisciplinary Journal, 20, 27–46.
34.
RevillaM.SarisW. E. (2015). Estimating and comparing the quality of different scales of an online survey using an MTMM approach. In EngelU. (Ed.), Survey measurement: techniques, data quality and sources of error (pp. 53–74). Frankfurt, New York.
35.
RevillaM.SarisW. E.KrosnickJ. A. (2014). Choosing the number of categories in agree–disagree scales. Sociological Methods & Research, 43, 73–97. Retrieved fromhttps://doi.org/10.1177/0049124113509605
36.
RevillaM.Zavala-RojasD.SarisW. E. (2016). Creating a good question: How to use cumulative experience. In WolfC.JoyeD.SmithT. W.FuY. (Eds.), The SAGE-handbook of survey methodology (pp. 236–254). London, UK: Sage.
37.
RodgersW.AndrewsF.HerzogA. (1992). Quality of survey measures: A structural modeling approach. Jounal of Official Statistics, 8, 251–275.
38.
RosterC. A.LucianettiL.AlbaumG. (2015). Exploring slider vs. categorical response formats in web-based surveys. Journal of Research Practice, 11, 1.
39.
SarisW. E.AalbertsC. (2003). Different explanations for correlated disturbance terms in MTMM studies. Structural Equation Modeling: A Multidisciplinary Journal, 10, 193–213. Retrieved fromhttps://doi.org/10.1207/S15328007SEM1002_2
40.
SarisW. E.AndrewsF. M. (1991). Evaluation of measurement instruments using a structural modeling approach. In BiemerP. P.GrovesR. M.LybergL. E.MathiowetzN. A.SudmanS. (Eds.), Measurement errors in surveys (pp. 575–598). New York: John Wiley and Sons, Inc.
41.
SarisW. E.GallhoferI. N. (2007). Design, evaluation, and analysis of questionnaires for survey research. Hoboken, NJ: John Wiley.
SarisW. E.RevillaM. (2016). Correction for measurement errors in survey research: Necessary and possible. Social Indicators Research, 127, 1005–1020. Retrieved fromhttps://doi.org/10.1007/s11205-015-1002-x
44.
SarisW. E.RevillaM.KrosnickJ. A.ShaefferE. M. (2010). Comparing questions with agree/disagree response options to questions with item-specific response options. Survey Research Methods, 4, 61–79. Retrieved fromhttps://doi.org/10.18148/srm/2010.v4i1.2682
45.
SarisW. E.SatorraA.CoendersG. (2004). A new approach to evaluating the quality of measurement instruments: The split-ballot MTMM design. Sociological Methodology, 34, 311–347.
46.
SarisW. E.SatorraA.van der VeldW. M. (2009). Testing structural equation models or detection of misspecifications?Structural Equation Modeling: A Multidisciplinary Journal, 16, 561–582. Retrieved fromhttps://doi.org/10.1080/10705510903203433
47.
ScherpenzeelA. C.SarisW. E. (1997). The validity and reliability of survey questions: A meta-analysis of MTMM studies. Sociological Methods & Research, 25, 341–383. Retrieved fromhttps://doi.org/10.1177/0049124197025003004
SikkelD.SteenbergenR.GrasS. (2014). Clicking vs. dragging: Different uses of the mouse and their implications for online surveys. Public Opinion Quarterly, 78, 177–190. Retrieved fromhttps://doi.org/10.1093/poq/nft077
Van der VeldW. M.SarisW. E.SatorraA. (2009). Jrule 2.0: User manual (Unpublished manuscript, internal report). Radboud University Nijmegen, the Netherlands.
52.
van MeursA.SarisW. E. (1990). Memory effects in MTMM studies. In SarisW. E.van MeursA. (Eds.), Evaluation of measurement instruments by meta-analysis of multitrait–multimethod studies (pp. 134–146). Amsterdam, the Netherlands: North-Holland.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
