We report sample composition discrepancies related to demographic and personality variables occurring in different stages of development of a probability-based online panel. The first stage—selecting eligible participants—produces differences between Internet users and nonusers in age, education, and gender distribution as well as in the personality traits of openness to experience, conscientiousness, and extraversion. The second and third stages of panel development—asking about willingness to participate in the panel and actual participation in online surveys—result in fewer and smaller discrepancies. The results suggest that among the three potential sources of sample composition bias considered, the largest impact comes from coverage differences with regard to Internet access.
BakerR. BlumbergS. J.BrickJ. M.CouperM. P.CourtrightM.DennisJ. M.DillmanD. A.. 2010. Research synthesis: AAPOR report on online panels. Public Opinion Quarterly74:711–81.
2.
BandillaW.BosnjakM.AltdorferP.2003. Survey administration effects? A comparison of web-based and traditional written self-administered surveys using the ISSP environment module. Social Science Computer Review21:235–43.
3.
BarrickM. R.StewartG. L.PiotrowskiM.2002. Personality on job performance: Test of the mediating effects of motivation among sales representatives. Journal of Applied Psychology87:43–51.
4.
BethlehemJ. G.2002. Weighting nonresponse adjustments based on auxiliary information. In Survey nonresponse, eds. GrovesR. M.DillmanD. A.EltingeJ. L.LittleR. J. A., 275–88. New York: Wiley.
5.
BosnjakM.GalesicM.TutenT.2007. Personality determinants of online shopping: Explaining online purchase intentions using a hierarchical approach. Journal of Business Research60:597–605.
6.
CapraraG. V.ZimbardoP. G.2004. Personalizing politics: A congruency model of political preference. American Psychologist59:581–94.
7.
CaspiA.RobertsB. W.ShinerR. L.2005. Personality development: Stability and change. Annual Review of Psychology56:453–84.
8.
ChangL.-C.KrosnickJ. A.2009. National surveys via RDD telephone versus the Internet: Comparing sample representativeness and response quality. Public Opinion Quarterly73:641–78.
9.
CohenJ.1992. A power primer. Psychological Bulletin112:155–59.
10.
CouperM. P.2000. Web surveys: A review of issues and approaches. Public Opinion Quarterly64:464–94.
11.
CouperM. P.2007. Issues of representation in eHealth research (with a focus on web surveys). American Journal of Preventive Medicine32: S83–89.
12.
CouperM. P.BosnjakM.2010. Internet surveys. In Handbook of survey research, eds. MarsdenP. V.WrightJ. D., 527–50. Bingley, UK: Emerald Group Publishing.
13.
CouperM. P.KapteynA.SchonlauM.WinterJ.2007. Noncoverage and nonresponse in an Internet survey. Social Science Research36:131–48.
14.
DeNeveK. M.CooperH.1998. The happy personality: A meta-analysis of 137 personality traits and subjective well-being. Psychological Bulletin124:197–229.
15.
De RaadB.2000. The Big Five personality factors. Toronto, Canada: Hogrefe & Huber Publishers.
16.
DienerE.LucasR. E.1999. Personality and subjective well-being. In Well-being: The foundations of hedonic psychology, eds. KahnemanD.DienerE.SchwarzN., 213–29. New York: Russell-Sage.
17.
DollingerS. J.LeongF. T. L.1993. Volunteer bias and the five-factor model. The Journal of Psychology127:29–36.
18.
DonnellanM. B.LucasR. E., 2008. Age differences in the Big Five across the life span: Evidence from two national samples. Psychology and Aging23:558–66.
19.
FergusonC. J.2009. An effect size primer: A guide for clinicians and researchers. Professional Psychology: Research and Practice40:532–38.
20.
FrickerS.GalesicM.TourangeauR.YanT.2005. An experimental comparison of web and telephone surveys. Public Opinion Quarterly69:370–92.
21.
GelmanA.CarlinJ. B.2002. Poststratification and weighting adjustments. In Survey nonrersponse, eds. GrovesR. M.DillmanD. A.EltingeJ. L.LittleR. J. A., 289–302. New York: Wiley.
22.
GoldbergL. R.1990. An alternative “description of personality”: The Big-Five factor structure. Journal of Personality and Social Personality59:1216–29.
23.
GöritzA.2010. Using lotteries, loyalty points, and other incentives to increase participant response and completion. In Advanced methods for conducting online behavioral research, eds. GoslingS.JohnsonA., 219–33. Washington, DC: American Psychological Association.
24.
HasselbladV.HedgesL. V.1995. Meta-analysis of screening and diagnostic test. Psychological Bulletin117:167–78.
25.
HoogendoornA. W.DaalmansJ.2009. Nonresponse in the recruitment of an Internet panel based on probability sampling. Survey Research Methods3:59–72.
26.
HugginsV. J.EyermanJ.2001. Probability based Internet surveys: A synopsis of early methods and survey research results. Paper presented at the Research Conference of the Federal Committee on Statistical Methodology, Arlington, VA, November 14–16, 2001. http://www.fcsm.gov/01papers/Huggins.pdf(accessed October 30, 2012).
27.
JohnO. P.SrivastavaS.1999. The Big-Five trait taxonomy: History, measurement, and theoretical perspectives. In Handbook of personality: Theory and research(2nd ed.), eds. PervinL. A.JohnO. P., 102–38. New York: Guilford Press.
28.
KanukL.BerensonC.1975. Mail surveys and response rates: A literature review. Journal of Marketing Research12:440–53.
29.
KnoefM.de VosK.2009. The representativeness of the LISS panel. Tilburg, the Netherlands: CentERdata.
30.
LeeS.2006. An evaluation of nonresponse and coverage errors in a prerecruited probability web panel survey. Social Science Computer Review24:460–75.
31.
LönnqvistJ.-E.PaunonenS.VerkasaloM.LeikasS.Tuulio-HenrikssonA.LönnqvistJ.2007. Personality characteristics of research volunteers. European Journal of Personality21:1017–30.
32.
McCraeR. R.ZondermanA. B.CostaP. T.BondM. H.PaunonenS. V.1996. Evaluating replicability of factors in the revised NEO Personality Inventory: Confirmatory factor analysis versus procrustes rotation. Journal of Personality and Social Personality70:552–66.
33.
MarcusB.SchützA.2005. Who are the people reluctant to participate in research? Personality correlates of four different types of nonresponse as inferred from self- and observer ratings. Journal of Personality73:959–84.
34.
MowenJ. C.2000. The 3M model of motivation and personality: Theory and empirical applications to consumer behavior. Boston: Kluwer Academic.
35.
NTIA (National Telecommunications and Information Administration). 2011. Digital nation: Expanding Internet usage. Washington, DC: U.S. Department of Commerce.
36.
OstendorfF.AngleitnerA.2004. Neo-Persönlichkeitsinventar (revidierte Form, NEO-PI-R) nach Costa und McCrae(The NEO personality inventory, revised form, NEO-PI-R according to Costa and McCrea). Göttingen, Germany: Hogrefe.
37.
ParksL.GuayR.2009. Personality, values, and motivation. Personality and Individual Differences47:675–84.
38.
RammstedtB.2007. The 10-item Big Five inventory—Norms values and investigation of sociodemographic effects based on a German population representative sample. European Journal of Psychological Assessment23:193–201.
39.
RammstedtB.GoldbergL. R.BorgI.2010. The measurement equivalence of Big-Five factor markers for persons with different levels of education. Journal of Research in Personality44:53–61.
40.
RammstedtB.JohnO. P.2007. Measuring personality in one minute or less: A 10-item short version of the Big Five inventory in English and German. Journal of Research in Personality41:203–12.
41.
RoccasS.SagivL.SchwartzS. H.KnafoA.2002. The Big Five personality factors and personal values. Personality and Social Psychology Bulletin28:789–801.
42.
RogelbergS. G.ConwayJ. M.SederburgM. E.SpitzmuellerC.AzizS.KnightW. E.2003. Profiling active and passive nonrespondents to an organizational survey. Journal of Applied Psychology88:1104–14.
43.
SaucierG.2000. Isms and the structure of social attitudes. Journal of Personality and Social Psychology74:366–85.
44.
ScherpenzeelA. C.BethlehemJ. G.2011. How representative are online panels? Problems of coverage and selection and possible solutions. In Social and behavioral research and the Internet: Advances in applied methods and research strategies, eds. DasM.EsterP.KaczmirekL., 105–32. New York: Routledge.
45.
ScherpenzeelA. C.DasM.2011. “True” longitudinal and probability-based Internet panels: Evidence from the Netherlands. In Social research and the Internet, eds. DasM.EsterP.KaczmirekL., 77–104. New York: Taylor and Francis.
46.
SmithT. W.2003. An experimental comparison of knowledge networks and the GSS. International Journal of Public Opinion Research15:167–79.
47.
The American Association for Public Opinion Research (AAPOR). 2011. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 7th edition. AAPOR.
48.
TrullT. J.DurrettC.2005. Categorical and dimensional models of personality disorder. Annual Review of Clinical Psychology1:355–80.
49.
van EimerenB.FreesB.2011. Drei von vier Deutschen im Netz – ein Ende des digitalen Grabens in Sicht? Ergebnisse der ARD/ZDF-Onlinestudie 2011 (Three out of four Germans are online – Is the end of the digital devide in sight? Results of the ARD/ZDF online study 2011). Media Perspektiven7/8:334–349.
50.
VisC. M.MarchandM.2011. Challenges in reaching hard-to-reach groups in Internet panel research. In Social and behavioral research and the Internet: Advances in applied methods and research strategies, eds. DasM.EsterP.KaczmirekL., 271–90. New York: Routledge.
