Scientific and commercial researchers around the world are relying increasingly on the responder insights obtained from online panels. However, from the researcher viewpoint, what distinguishes a good panel from a bad one is not clear and new metrics need to be developed to consider panel health. Given that online panels are composed of individuals, it is proposed that new individual-based metrics, taken across an individual's career, should also be developed. This paper proposes that each respondent's overall average relative response time (OARRT) across all surveys they have completed is one such measure, and examines how it is affected by a range of control factors. Three cohorts of panellists “ that is, ‘ongoing’, ‘unsubscribers’ and ‘three-strikers’ “ are examined. We found that the number of surveys completed and the respondent's age affected OARRT across the three cohorts, with participation in other panels affecting OARRT for unsubscribers. Gender, education and employment status did not impact OARRT for any group. The results suggest there may be a learning effect.
Get full access to this article
View all access options for this article.
References
1.
BakerR., BlumbergS.J., BrickJ.M., CouperM.P., CourtrightM., DennisJ.M., DillmanD., FrankelM.R., GarlandP., GrovesR.M., KennedyC., KrosnickJ., & LavrakasP.L. (2010) Research synthesis AAPOR report on online panels Public OpinionQuarterly, 74, 4, pp. 711–781
2.
BaldingerA.L. (2001) Online research and data quality: stepping towards the truth Council of American Survey ResearchOrganizations Journal, pp. 93–96Available online at: http://www.casro.org/?page=journal (accessed 20 March 2010)
3.
BosnjakM., TracyL.T., & WernerW.W. (2005) Unit (non)response in web-based access panel surveys: an extended planned-behavior approachPsychology and Marketing, 22, 6, pp. 489–505
4.
BosnjakM., HaasI., GalesicM., KaczmirekL., BandillaW., & CouperM.P. (2013) Sample composition discrepancies in different stages of a probability-based online panel.Field Methods, 25, 4, pp 338–360.
5.
BrüggenE., & DholakiaU.M. (2010) Determinants of participation and response effort in web panel surveysJournal of Interactive Marketing, 24, 3, pp 239–250.
CobanogluC., & CobanogluN. (2003) The effect of incentives in web surveys: application and ethical considerationsInternational Journal of Market Research, 45, 4, pp 475–488.
CouperM.P., TraugottM.W., & LamiasM.J. (2001) Web survey design and administrationPublic Opinion Quarterly, 65, 2, pp 230–253.
10.
CourtrightM., & BrienD. (2009) The devil is in the data: a respondent classification analysis comparing the best and worst survey respondentsQuirk's Marketing Research Review, 23, 4, pp 56–62.
DennisJ.M. (2001) Are internet panels creating professional respondents?Marketing Research, 13, 2, pp 34–39.
13.
DeutskensE., de RuyterK., & WetzelsM. (2006) An assessment of equivalence between online and mail surveys in service researchJournal of Service Research, 8, 4, pp 346–355.
14.
DuffyB., SmithK., TerhanianG., & BremerJ. (2005) Comparing data from online and face-to-face surveysInternational Journal of Market Research, 47, 6, pp 615–639.
FurrerO., & SudharshanD. (2001) Internet marketing research: opportunities and problems Qualitative Marketing Research:An Interdisciplinary Journal, 4, 3, pp 123–129
17.
GalesicM., & BosnjakM. (2009) Effects of questionnaire length on participation and indicators of quality in a web surveyPublic Opinion Quarterly, 73, 3, pp 349–360.
18.
GöritzA.S. (2004) The impact of material incentives on response quantity, response quality, sample composition, survey outcome, and cost in online access panelsInternational Journal of Market Research, 46, 3, pp 327–345.
19.
GöritzA.S. (2006) Cash lotteries as incentives in online panelsSocial Science Computer Review, 24, 4, pp 445–459.
20.
GöritzA.S. (2008) The long-term effect of material incentives on participation in online panelsField Methods, 20, 3, pp 211–225.
21.
HanV., AlbaumG., WileyJ.B., & ThirkellP. (2009) Applying theory to structure respondents' stated motivations for participating in web surveysQualitative Marketing Research: An Interdisciplinary Journal, 12, 4, pp 428–442.
22.
HaunbergerS. (2011) To participate or not to participate: decision processes related to survey non-response Bulletin of SociologicalMethodology/Bulletin de Méthodologie Sociologique, 109, 1, pp 39–55
23.
HeerweghD., VanhoveT., MatthijsK., & LoosveldtG. (2005) The effect of personalization on response rates and data quality in web surveysInternational Journal of Social Research Methodology, 8, 2, pp 85–99.
24.
HoogendoornA.W., & DaalmansJ. (2009) Non-response in recruitment of an internet panel based on probability samplingSurvey Research Methods, 3, 2, pp 50–72.
25.
IlievaJ., BaronS., & HealeyN.M. (2002) Online surveys in marketing research: pros and consInternational Journal of Marketing Research, 44, 3, pp 361–376.
26.
ISO (2009) Access Panels in Market, Opinion and Social Research - Vocabulary and Service Requirements (ISO 26362). Geneva, Switzerland: International Standards Organisation
27.
LarsonA.J., & SachauD.A. (2009) Effects of incentives and the big five personality dimensions on internet panellists' ratingsInternational Journal of Market Research, 51, 5, pp 687–706.
28.
LenznerT., KaczmirekL., & LenznerA. (2010) Cognitive burden of survey questions and response times: a psycholinguistic experimentApplied Cognitive Psychology, 24, 7, pp 1003–1020.
29.
LipnerW.E. (2007) The future of online market researchJournal of Advertising Research, 47, 2, pp 142–146.
30.
LoosveldtG., & CartonA. (2002) Utilitarian individualism and panel non-responseInternational Journal of Public Opinion Research, 14, 3, pp 428–438.
31.
MalhotraN. (2008) Completion time and response order effects in web surveysPublic Opinion Quarterly, 72, 5, pp 914–934.
32.
ManfredaK.L., BosnjakM., BerzelakJ., HaasI., & VehovarV. (2008) Web surveys versus other survey modes: a meta-analysis comparing response ratesInternational Journal of Market Research, 50, 1, pp 70–104.
NancarrowC., PallisterJ., & BraceI. (2001) A new research medium, new research populations and seven deadly sins for internet researchersQualitative Market Research: An International Journal, 4, 3, pp 136–149.
35.
NeslinS.A., NovakT.P., BakerK.R., & HoffmanD.L. (2009) An optimal contact model for maximizing online panel response ratesManagement Science, 55, 5, pp 727–737.
36.
PeytchevA., CouperM.O., McCabeS.E., & CrawfordS.D. (2006) Web survey design: paging versus scrollingPublic Opinion Quarterly, 70, 4, pp 596–607.
37.
PorterS.R., & WhitcombM.E. (2003) The impact of contact types of web survey response ratesPublic Opinion Quarterly, 67, 4, pp 579–588.
38.
PorterS.R., WhitcombM.E., & WeitzerW.H. (2004) Multiple surveys of students and survey fatigueNew Directions for Institutional Research, 121, pp 63–73.
39.
ScherpenzeelA., & ToepoelV. (2012) Recruiting a probability sample for an online panel effect of contact mode, incentives, and informationPublic Opinion Quarterly, 76, 3, pp 470–490.
40.
ShihT.-H., & FanX. (2008) Comparing response rates from web and mail surveys: a meta-analysisField Methods, 20, 3, pp 249–271.
41.
SparrowN. (2007) Quality issues in online researchJournal of Advertising Research, 47, 2, pp 179–182.
42.
TeislM.F., RoeB., & VaydaM. (2005) Incentive effects on response rates, data quality, and survey administration costsInternational Journal of Public Opinion Research, 18, 3, pp 364–373.
43.
ToepoelV., DasM., & van SoestA. (2008) Effects of design in web surveys: comparing trained and fresh respondentsPublic Opinion Quarterly, 72, 5, pp 985–1007.
44.
YanT., & TourangeauR. (2008) Fast times and easy questions: the effects of age, experience and question complexity on web survey response timesApplied Cognitive Psychology, 22, pp 51–68.
