Study participants are vital to personality research. Thus, individuals not following study invitations (self-selection), not completing all administered measures (showing dropout) and/or not participating at subsequent measurement occasions (showing attrition) pose a significant challenge to personality research. Building on existing research documenting that non-response does not occur randomly, we herein test for potential predictors of non-responding at each of the three stages by combining official register (census) data from a randomly selected participant pool of 100,175 adults, representative for the target population (adults living in Denmark), with data from a personality panel (N = 14,071) recruited from this participant pool. We find that each socio-demographic characteristic considered (i.e., sex, age, marital status, migration background, income, highest completed education, and region of residence) as well as five of the six HEXACO personality dimensions (Honesty-Humility, Emotionality, Extraversion, Conscientiousness, and Openness to Experience; but not Agreeableness vs. Anger) were associated with self-selection, selective dropout, and/or selective attrition. We further compare the sample representativeness at each of the stages and illustrate how to assess potential biases (regarding the obtained HEXACO scores) due to self-selection.
Plain Language Summary
Study participants are important to personality research, so if invited individuals do not participate in a study or do not answer all questions this makes it challenging to obtain data with high quality for personality research. In this investigation, we combined official register (census) data on various variables (e.g., education, income) from a randomly selected potential participant pool of 100,175 adults, representative for the entire adult population of Denmark, with personality data from 14,071 individuals who followed the invitation to participate in a personality panel study. The results showed that each socio-demographic characteristic considered (i.e., sex, age, marital status, migration background, income, highest completed education, region of residence) as well as five of the six HEXACO basic personality dimensions (Honesty-Humility, Emotionality, Extraversion, Conscientiousness, and Openness to Experience; but not Agreeableness vs. Anger) were associated with who participated in the study or not, who filled out all measures of the survey or not (called dropout), and/or who participated at the second round of the panel study about one year later or not (called attrition). We also illustrate how the representativeness of a sample can be estimated and how estimates might be biased due to who participates in research and who not.
AlsonoA.de IralaJ.Martínez-GonzálezM. A. (2007). Representativeness, losses to follow-up and validity in cohort studies. European Journal of Epidemiology, 22(7), 481–482. https://doi.org/10.1007/s10654-007-9124-3
2.
ArslanR. C.WaltherM. P.TataC. S. (2020). formr: A study framework allowing for automated feedback generation and complex longitudinal experience-sampling studies using R. Behavior Research Methods, 52(1), 376–387. https://doi.org/10.3758/s13428-019-01236-y
3.
AshtonM. C.LeeK. (2007). Empirical, theoretical, and practical advantages of the HEXACO model of personality structure. Personality and Social Psychology Review, 11(2), 150–166. https://doi.org/10.1177/1088868306294907
4.
AshtonM. C.LeeK. (2009). The HEXACO-60: A short measure of the major dimensions of personality. Journal of Personality Assessment, 91(4), 340–345. https://doi.org/10.1080/00223890902935878
5.
BosnjakM.HaasI.GalesicM.KaczmirekL.BandillaW.CouperM. P. (2013). Sample composition discrepancies in different stages of a probability-based online panel. Field Methods, 25(4), 339–360. https://doi.org/10.1177/1525822X12472951
6.
BouchardK. N.StewartJ. G.BoyerS.HoldenR. R.PukallC. F. (2019). Sexuality and personality correlates of willingness to participate in sex research. The Canadian Journal of Human Sexuality, 28(1), 26–37. https://doi.org/10.3138/cjhs.2018-0028
7.
BowdenR. J. (1986). Self-selection biases in correlational studies based on questionnaires. Psychometrika, 51(2), 313–325. https://doi.org/10.1007/BF02293987
8.
BurgardT.BošnjakM.WedderhoffN. (2020). Response rates in online surveys with affective disorder participants: A meta-analysis of study design and time effects between 2008 and 2019. Zeitschrift für Psychologie, 228(1), 14–24. https://doi.org/10.1027/2151-2604/a000394
9.
CarnahanT.McFarlandS. (2007). Revisiting the Stanford prison experiment: Could participant self-selection have led to the cruelty?Personality and Social Psychology Bulletin, 33(5), 603–614. https://doi.org/10.1177/0146167206292689
10.
CoppockA.LeeperT. J.MullinixK. J. (2018). Generalizability of heterogeneous treatment effect estimates across samples. Proceedings of the National Academy of Sciences, 115(49), 12441–12446. https://doi.org/10.1073/pnas.1808083115
11.
DienerE.EmmonsR. A.LarsenR. J.GriffinS. (1985). The satisfaction with life scale. Journal of Personality Assessment, 49(1), 71–75. https://doi.org/10.1207/s15327752jpa4901_13
12.
EgglestonJ. (2024). Frequent survey requests and declining response rates: Evidence from the 2020 Census and Household Surveys. Journal of Survey Statistics and Methodology, 12(5), 1138–1156. (in press). https://doi.org/10.1093/jssam/smae022
13.
EvansC. R.WilliamsD. R.OnnelaJ.-P.SubramanianS. V. (2018). A multilevel approach to modeling health inequalities at the intersection of multiple social identities. Social Science and Medicine, 203, 64–73. https://doi.org/10.1016/j.socscimed.2017.11.011
14.
GrovesR.CialdiniR.CouperM. (1992). Understanding the decision to participate in a survey. Public Opinion Quarterly, 56(4), 475–495. https://doi.org/10.1086/269338
15.
HanssonI.BergA. I.ThorvaldssonV. (2018). Can personality predict longitudinal study attrition? Evidence from a population-based sample of older adults. Journal of Research in Personality, 77, 133–136. https://doi.org/10.1016/j.jrp.2018.10.002
16.
HaynesA.RobinsonE. (2019). Who are we testing? Self-Selection bias in laboratory-based eating behaviour studies. Appetite, 141, Article 104330. https://doi.org/10.1016/j.appet.2019.104330
17.
HoA. K.SidaniusJ.KteilyN.Sheehy-SkeffingtonJ.PrattoF.HenkelK. E.FoelsR.StewartA. L. (2015). The nature of social dominance orientation: Theorizing and measuring preferences for intergroup inequality using the new SDO- scale. Journal of Personality and Social Psychology, 109(6), 1003–1028. https://doi.org/10.1037/pspi0000033
18.
HoltD.ElliotD. (1991). Methods of weighting for unit non-response. Journal of the Royal Statistical Society - Series D: The Statistician, 40(3), 333–342. https://doi.org/10.2307/2348286
19.
HoltomB.BaruchY.AguinisH.A BallingerG. (2022). Survey response rates: Trends and a validity assessment framework. Human Relations, 75(8), 1560–1584. https://doi.org/10.1177/00187267211070769
20.
HorvitzD. G.ThompsonD. J. (1952). A generalization of sampling without replacement from a finite Universe. Journal of the American Statistical Association, 47(260), 663–685. https://doi.org/10.2307/2280784
21.
JensenH. A. R.LauC. J.DavidsenM.FeveileH. B.ChristensenA. I.EkholmO. (2022). The impact of non-response weighting in health surveys for estimates on primary health care utilization. The European Journal of Public Health, 32(3), 450–455. https://doi.org/10.1093/eurpub/ckac032
22.
KellerL.LüdtkeO.PreckelF.BrunnerM. (2023). Educational inequalities at the intersection of multiple social categories: An introduction and systematic review of the Multilevel Analysis of Individual Heterogeneity and Discriminatory Accuracy (MAIHDA) approach. Educational Psychology Review, 35(1), 31. https://doi.org/10.1007/s10648-023-09733-5
23.
LeeK.AshtonM. C. (2006). (2006). Further assessment of the HEXACO Personality Inventory: Two new facet scales and an observer report form. Psychological Assessment, 18(2), 182–191. https://doi.org/10.1037/1040-3590.18.2.182
24.
LeeperT. J. (2019). Where have the respondents gone? Perhaps we ate them all. Public Opinion Quarterly, 83(S1), 280–288. https://doi.org/10.1093/poq/nfz010
LevayK. E.FreeseJ.DruckmanJ. N. (2016). The demographic and political composition of Mechanical Turk Samples. Sage Open, 6(1), 1–7. https://doi.org/10.1177/2158244016636433
27.
LönnqvistJ.-E.PaunonenS.VerkasaloM.LeikasS.Tuulio-HenrikssonA.LönnqvistJ. (2007). Personality characteristics of research volunteers. European Journal of Personality, 21(8), 1017–1030. https://doi.org/10.1002/per.655
28.
MarcusB.SchützA. (2005). Who are the people reluctant to participate in research? Personality correlates of four different types of nonresponse as inferred from self- and observer ratings. Journal of Personality, 73(4), 959–984. https://doi.org/10.1111/j.1467-6494.2005.00335.x
29.
MasonM. J. (1999). A review of procedural and statistical methods for handling attrition and missing data in clinical research. Measurement and Evaluation in Counseling and Development, 32(2), 111–118. https://doi.org/10.1080/07481756.1999.12068976
30.
McAbeeS. T.ConnellyB. S. (2016). A multi-rater framework for studying personality: The trait-reputation-identity model. Psychological Review, 123(5), 569–591. https://doi.org/10.1037/rev0000035
31.
MellonJ.ProsserC. (2016). Correcting polling error in over-engaged samples using representative turnout weighting. SSRN. https://doi.org/10.2139/ssrn.2720206
32.
MerloJ.WagnerP.LeckieG. (2019). A simple multilevel approach for analysing geographical inequalities in public health reports: The case of municipality differences in obesity. Health and Place, 58, Article 102145. https://doi.org/10.1016/j.healthplace.2019.102145
33.
MooreJ. C.DurrantG. B.SmithP. W. F. (2018). Data set representativeness during data collection in three UK social surveys: Generalizability and the effects of auxiliary covariate choice. Journal of the Royal Statistical Society - Series A: Statistics in Society, 181(1), 229–248. https://doi.org/10.1111/rssa.12256
34.
MooreJ. C.DurrantG. B.SmithP. W. F. (2021). Do coefficients of variation of response propensities approximate non-response biases during survey data collection?Journal of the Royal Statistical Society - Series A: Statistics in Society, 184(1), 301–323. https://doi.org/10.1111/rssa.12624
35.
MoshagenM.HilbigB. E.ZettlerI. (2018). The dark core of personality. Psychological Review, 125(5), 656–688. https://doi.org/10.1037/rev0000111
36.
MoshagenM.ZettlerI.HilbigB. (2020). Measuring the dark core of personality. Psychological Assessment, 32(2), 182–196. https://doi.org/10.1037/pas0000778
37.
MurphyR. O.AckermannK. A.HandgraafM. J. J. (2011). Measuring social value orientation. Judgment and Decision Making, 6(8), 771–781. https://doi.org/10.1017/s1930297500004204
38.
NowokB.RaabG. M.DibbenC. (2016). synthpop: bespoke creation of synthetic data in R. Journal of Statistical Software, 74(11), 1–26. https://doi.org/10.18637/jss.v074.i11
39.
OishiS.ChoiH.ButtrickN.HeintzelmanS. J.KushlevK.WestgateE. C.TuckerJ.EbersoleC. R.AxtJ.GilbertE.NgB. W.BesserL. L. (2019). The psychologically rich life questionnaire. Journal of Research in Personality, 81, 257–270. https://doi.org/10.1016/j.jrp.2019.06.010
40.
OteroI.SalgadoJ. F.MoscosoS. (2022). Cognitive reflection, cognitive intelligence, and cognitive abilities: A meta-analysis. Intelligence, 90, Article 101614. https://doi.org/10.1016/j.intell.2021.101614
41.
PaganJ. L.EatonN. R.TurkheimerE.OltmannsT. F. (2006). Peer-reported personality problems of research nonparticipants: Are our samples biased?Personality and Individual Differences, 41(6), 1131–1142. https://doi.org/10.1016/j.paid.2006.04.017
PeerE.RothschildD.GordonA.EverndenZ.DamerE. (2022). Data quality of platforms and panels for online behavioral research. Behavior Research Methods, 54(4), 1643–1662. https://doi.org/10.3758/s13428-021-01694-3
44.
PikeG. R. (2008). Using weighting adjustments to compensate for survey nonresponse. Research in Higher Education, 49(2), 153–171. https://doi.org/10.1007/s11162-007-9069-0
45.
ProsserC.MellonJ. (2018). The twilight of the polls? A review of trends in polling accuracy and the causes of polling misses. Government and Opposition, 53(4), 757–790. https://doi.org/10.1017/gov.2018.7
46.
RichterD.KörtnerJ. L.SaßenrothD. (2014). Personality has minor effects on panel attrition. Journal of Research in Personality, 53, 31–35. https://doi.org/10.1016/j.jrp.2014.08.001
47.
RungeJ. M.LangJ. W. B. (2019). Can people recognize their implicit thoughts? The motive self-categorization test. Psychological Assessment, 31(7), 939–951. https://doi.org/10.1037/pas0000720
48.
SatherleyN.MilojevP.GreavesL. M.HuangY.OsborneD.BulbuliaJ.SibleyC. G. (2015). Demographic and psychological predictors of panel attrition: Evidence from the New Zealand attitudes and values study. PLoS One, 10(3), Article e0121950. https://doi.org/10.1371/journal.pone.0121950
49.
SchildC.HeckD. W.ŚcigałaK. A.ZettlerI. (2019). Revisiting REVISE: (Re)Testing unique and combined effects of REminding, VIsibility, and SElf-engagement manipulations on cheating behavior. Journal of Economic Psychology, 75, Article 102161. https://doi.org/10.1016/j.joep.2019.04.001
50.
SchildC.LilleholtL.ZettlerI. (2021). Behavior in cheating paradigms is linked to overall approval rates of crowdworkers. Journal of Behavioral Decision Making, 34(2), 157–166. https://doi.org/10.1002/bdm.2195
51.
SchoelerT.SpeedD.PorcuE.PirastuN.PingaultJ. B.KutalikZ. (2023). Participation bias in the UK Biobank distorts genetic associations and downstream analyses. Nature Human Behavior, 7(7), 1216–1227. https://doi.org/10.1038/s41562-023-01579-9
52.
SchoeniR. F.StaffordF.McgonagleK. A.AndreskiP. (2013). Response rates in national panel surveys. The Annals of the American Academy of Political and Social Science, 645(1), 60–87. https://doi.org/10.1177/0002716212456363
53.
SchoutenB.ShlomoN. (2017). Selecting adaptive survey design strata with Partial R-indicators. International Statistical Review, 85(1), 143–163. https://doi.org/10.1111/insr.12159
54.
ShihT.-H.FanX. (2008). Comparing response rates from web and mail surveys: A meta-analysis. Field Methods, 20(3), 249–271. https://doi.org/10.1177/1525822X08317085
55.
SimonsD. J.ShodaY.LindsayD. S. (2017). Constraints on Generality (COG): A proposed addition to all empirical papers. Perspectives on Psychological Science, 12(6), 1123–1128. https://doi.org/10.1177/1745691617708630
56.
SirotaM.JuanchichM. (2018). Effect of response format on cognitive reflection: Validating a two- and four-option multiple choice question version of the Cognitive Reflection Test. Behavior Research Methods, 50(6), 2511–2522. https://doi.org/10.3758/s13428-018-1029-4
57.
SøgaardA. J.SelmerR.BjertnessE.ThelleD. (2004). The Oslo Health Study: The impact of self-selection in a large, population-based survey. International Journal for Equity in Health, 3(1), 3. https://doi.org/10.1186/1475-9276-3-3
58.
StegerM. F.FrazierP.OishiS.KalerM. (2006). The meaning in life questionnaire: Assessing the presence of and search for meaning in life. Journal of Counseling Psychology, 53(1), 80–93. https://doi.org/10.1037/0022-0167.53.1.80
59.
StewartR. D.MõttusR.SeebothA.SotoC. J.JohnsonW. (2022). The finer details? The predictability of life outcomes from Big Five domains, facets, and nuances. Journal of Personality, 90(2), 167–182. https://doi.org/10.1111/jopy.12660
60.
TaylorL. K.TongX.MaxwellS. E. (2020). Evaluating supplemental samples in longitudinal research: Replacement and refreshment approaches. Multivariate Behavioral Research, 55(2), 277–299. https://doi.org/10.1080/00273171.2019.1628694
61.
TiffanyD. W.CowanJ. R.BlinnE. (1970). Sample and personality biases of volunteer subjects. Journal of Consulting and Clinical Psychology, 35(1), 38–43. https://doi.org/10.1037/h0029621
62.
Van AltenS.DomingueB. W.FaulJ.GalamaT.MareesA. T. (2024). Reweighting UK Biobank corrects for pervasive selection bias due to volunteering. International Journal of Epidemiology, 53(3), dyae054. https://doi.org/10.1093/ije/dyae054
63.
VansteelandtS.CarpenterJ.KenwardM. G. (2010). Analysis of incomplete data using inverse probability weighting and doubly robust estimators. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 6(1), 37–48. https://doi.org/10.1027/1614-2241/a000005
64.
VazireS. (2010). Who knows what about a person? The self–other knowledge asymmetry (SOKA) model. Journal of Personality and Social Psychology, 98(2), 281–300. https://doi.org/10.1037/a0017908
65.
WardM. K.MeadeA. W.AllredC. M.PappalardoG.StoughtonJ. W. (2017). Careless response and attrition as sources of bias in online survey assessments of personality traits and performance. Computers in Human Behavior, 76, 417–430. https://doi.org/10.1016/j.chb.2017.06.032
Winston-ProctorC. E. (2023). “It's something to own”: A psychobiographical exploration of the life story of Michelle LaVaughn Robinson Obama. Journal of Personality, 91(1), 150–164. https://doi.org/10.1111/jopy.12779
69.
WuM.-J.ZhaoK.Fils-AimeF. (2022). Response rates of online surveys in published research: A meta-analysis. Computers in Human Behavior, 7, Article 100206. https://doi.org/10.1016/j.chbr.2022.100206
70.
ZettlerI.LilleholtL.BöhmR.GondanM. (2021). Comparing responses in repeated cross-sectional and panel studies: Results across eight weeks during the first COVID-19 lockdown in Denmark. Psychological Assessment, 33(8), 691–704. https://doi.org/10.1037/pas0001048
71.
ZettlerI.ThielmannI.HilbigB. E.MoshagenM. (2020). The nomological net of the HEXACO model of personality: A large-scale meta-analytic investigation. Perspectives on Psychological Science, 15(3), 723–760. https://doi.org/10.1177/1745691619895036
72.
ZhangD. C.HighhouseS.NyeC. D. (2018). Development and validation of the General Risk Propensity Scale (GRiPS). Journal of Behavioral Decision Making, 32(2), 152–167. https://doi.org/10.1002/bdm.2102
73.
ZhouH.FishbachA. (2016). The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusions. Journal of Personality and Social Psychology, 111(4), 493–504. https://doi.org/10.1037/pspa0000056
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
