A burgeoning line of research examining the relation between personality traits and political variables relies extensively on convenience samples. However, our understanding of the extent to which using convenience samples challenges the generalizability of these findings to target populations remains limited. We address this question by testing whether associations between personality and political characteristics observed in representative samples diverged from those observed in the sub–populations most commonly studied in convenience samples, namely, students and Internet users. We leverage 10 high–quality representative datasets to compare the representative samples with the two subsamples. We did not find any systematic differences in the relationship between personality traits and a broad range of political variables. Instead, results from the subsamples generalized well to those observed in the broader and more diverse representative sample.
American National Election Studies (2014). User's guide and codebook for the ANES 2012 time series study. Ann Arbor: University of Michigan and Stanford University.
2.
American National Election Studies (2018). User's guide and codebook for the ANES 2016 time series study. Ann Arbor, Michigan, and Palo Alto, California: University of Michigan and Stanford University.
3.
BakkerB. N., & LelkesY. (2018). Selling ourselves short? How abbreviated measures of personality change the way we think about personality and politics. The Journal of Politics, 80, 1311–1325. https://doi.org/10.1086/698928.
4.
BerinskyA. J., HuberG. A., & LenzG. S. (2012). Evaluating online labor markets for experimental research: Amazon.com's Mechanical Turk. Political Analysis, 20, 351–368. https://doi.org/10.1093/pan/mpr057.
5.
BerrensR. P., BoharaA. K., Jenkins–SmithH., SilvaC., & WeimerD. L. (2003). The advent of Internet surveys for political research: A comparison of telephone and Internet samples. Political Analysis, 11, 1–22. https://doi.org/10.1093/pan/11.1.1.
6.
BestS. J., KruegerB., HubbardC., & SmithA. (2001). An assessment of the generalizability of Internet surveys. Social Science Computer Review, 19, 131–145. https://doi.org/10.1177/089443930101900201.
CausadiasJ. M., VitriolJ., & AtkinA. L. (2018). Do we overemphasize the role of culture in the behavior of racial/ethnic minorities? Evidence of a cultural (mis)attribution bias in American psychology. American Psychologist, 73, 243–255. https://doi.org/10.1037/amp0000099.
9.
ChenP. G., ApplebyJ., BorgidaE., CallaghanT. H., EkstromP., FarhartC. E., … WilliamsA. (2014). The Minnesota Multi–Investigator 2012 Presidential Election Panel Study. Analyses of Social Issues and Public Policy, 14, 78–104. https://doi.org/10.1111/asap.12041.
CliffordS., JewellR. M., & WaggonerP. D. (2015). Are samples drawn from Mechanical Turk valid for research on political ideology?Research & Politics, 2, 1–9. https://doi.org/10.1177/2053168015622072.
12.
CooperC. A., McCordD. M., & SochaA. (2011). Evaluating the college sophomore problem: The case of personality and politics. The Journal of Psychology: Interdisciplinary and Applied, 145, 23–37. https://doi.org/10.1080/00223980.2010.528074.
13.
CoppockA. (2019). Generalizing from survey experiments conducted on Mechanical Turk: A replication approach. Political Science Research and Methods, 7, 613–628. https://doi.org/10.1017/psrm.2018.10.
14.
CoppockA., & GreenD. P. (2015). Assessing the correspondence between experimental results obtained in the lab and field: A review of recent social science research. Political Science Research and Methods, 3, 113–131. https://doi.org/10.1017/psrm.2014.10.
15.
CoppockA., LeeperT.J., & MullinixK.J. (2018). The Generalizability of Heterogeneous Treatment Effect Estimates Across Samples. Proceedings of the National Academy of Science, 115(49) 12441–12446.
16.
DeBellM., WilsonC., SeguraG., JackmanS., & HutchingsV. (2011). Methodology report and user's guide for the ANES 2010–2012 Evaluations of Government and Society Study. Palo Alto, CA, and Ann Arbor, MI: Stanford University and University of Michigan.
17.
Delli CarpiniM. X., & KeeterS. (1996). What Americans know about politics and why it matters. New Haven, CT: Yale University Press.
18.
DruckmanJ. N., & KamC. D. (2011). Students as experimental participants: A defense of the ‘narrow data base’. In DruckmanJ. N., GreenD. P., KuklinskiJ. H., & LupiaA. (Eds.), Cambridge handbook of experimental political science (pp. 70–101). New York: Cambridge University Press.
19.
DuckittJ., & SibleyC. G. (2016). Personality, ideological attitudes, and group identity as predictors of political behavior in majority and minority ethnic groups. Political Psychology, 37, 109–124. https://doi.org/10.1111/pops.12222.
20.
FedericoC. M., & SchneiderM. C. (2007). Political expertise and the use of ideology: Moderating effects of evaluative motivation. Public Opinion Quarterly, 71, 221–252. https://doi.org/10.1093/poq/nfm010.
21.
FieldhouseE., GreenJ., EvansG., SchmittH., van der EijkC., MellonJ., & ProsserC. (2015). British Election Study Internet Panel Wave 10. https://doi.org/10.15127/1.293723
GerberA. S., HuberG. A., DohertyD., DowlingC. M., & HaS. E. (2010). Personality and political attitudes: Relationships across issue domains and political contexts. American Political Science Review, 104, 111–133. https://doi.org/10.1017/S0003055410000031.
24.
GerberA. S., HuberG. A., DohertyD., DowlingC. M., RasoC., & HaS. E. (2011). Personality traits and participation in political processes. Journal of Politics, 73, 692–706. https://doi.org/10.1017/S0022381611000399.
25.
GoldbergL. R., JohnsonJ. A., EberH. W., HoganR., AshtonM. C., CloningerC. R., & GoughH. G. (2006). The international personality item pool and the future of public–domain personality measures. Journal of Research in Personality, 40, 84–96. https://doi.org/10.1016/j.jrp.2005.08.007.
26.
GoslingS. D., RentfrowP. J., & SwannW. B. (2003). A very brief measure of the Big–Five personality domains. Journal of Research in Personality, 37, 504–528. https://doi.org/10.1016/S0092–6566(03)00046–1.
27.
GoslingS. D., VazireS., SrivastavaS., & JohnO. P. (2004). Should we trust web–based studies? A comparative analysis of six preconceptions about Internet questionnaires. American Psychologist, 59, 93–104. https://doi.org/10.1037/0003–066X.59.2.93.
28.
HahnE., GottschlingJ., & SpinathF. M. (2012). Short measurements of personality—Validity and reliability of the GSOEP Big Five Inventory (BFI–S). Journal of Research in Personality, 46, 355–359. https://doi.org/10.1016/j.jrp.2012.03.008.
HenrichJ., HeineS. J., & NorenzayanA. (2010). The weirdest people in the world?The Behavioral and Brain Sciences, 32, 61–83. https://doi.org/10.1017/S0140525X0999152X.
31.
HoogheM., StolleD., MahéoV. A., & VissersS. (2010). Why can't a student be more like an average person? Sampling and attrition effects in social science field and laboratory experiments. The Annals of the American Academy of Political and Social Science, 628, 85–96. https://doi.org/10.1177/0002716209351516.
KandlerC., BleidornW., & RiemannR. (2012). Left or right? Sources of political orientation: The roles of genetic factors, cultural transmission, assortative mating, and personality. Journal of Personality and Social Psychology, 102, 633–645. https://doi.org/10.1037/a0025560.
34.
KrupnikovY., & LevineA. S. (2014). Cross–sample comparisons and external validity. Journal of Experimental Political Science, 1, 59–80. https://doi.org/10.1017/xps.2014.7.
35.
LandersR. N., & BehrendT. S. (2015). An inconvenient truth: Arbitrary distinctions between organizational, Mechanical Turk, and other convenience samples. Industrial and Organizational Psychology, 8, 142–164. https://doi.org/10.1017/iop.2015.13.
LeeK., AshtonM. C., GriepY., & EdmondsM. (2018). Personality, religion, and politics: An investigation in 33 countries. European Journal of Personality, 32, 100–115. https://doi.org/10.1002/per.2142.
38.
LudekeS. G., & LarsenE. G. (2017). Problems with the Big Five assessment in the World Values Survey. Personality and Individual Differences, 112, 103–105. https://doi.org/10.1016/j.paid.2017.02.042.
39.
LüdtkeO., RobertsB. W., TrautweinU., & NagyG. (2011). A random walk down university avenue: Life paths, life events, and personality trait change at the transition to university life. Journal of Personality and Social Psychology, 101, 620–637. https://doi.org/10.1037/a0023743.
40.
MellonJ., & ProsserC. (2017). Twitter and Facebook are not representative of the general population: Political attitudes and demographics of British social media users. Research & Politics, 4, 1–9. https://doi.org/10.1177/2053168017720008.
41.
MondakJ. J., & HalperinK. D. (2008). A framework for the study of personality and political behaviour. British Journal of Political Science, 38, 335–362. https://doi.org/10.1017/S0007123408000173.
42.
MullinixK. J., LeeperT. J., DruckmanJ. N., & FreeseJ. (2015). The generalizability of survey experiments. Journal of Experimental Political Science, 2, 109–138. https://doi.org/10.1017/XPS.2015.19.
43.
PetersonR. A. (2001). On the use of college students in social science research: Insights from a second–order meta–analysis. Journal of Consumer Research, 28, 450–461. https://doi.org/10.1086/323732.
RammstedtB., & JohnO. P. (2007). Measuring personality in one minute or less: A 10–item short version of the Big Five Inventory in English and German. Journal of Research in Personality, 41, 203–212. https://doi.org/10.1016/j.jrp.2006.02.001.
47.
RentfrowP. J., JostJ. T., GoslingS. D., & PotterJ. (2004). Statewide differences in personality predict voting patterns in 1996–2004 US presidential elections. In JostJ. T., KayA. C., & ThorisdottirH. (Eds.), Social and psychological bases of ideology and system justification (pp. 314–349). Oxford, United Kingdom: Oxford University Press.
48.
RoetsA., CornelisI., & Van HielA. (2014). Openness as a predictor of political orientation and conventional and unconventional political activism in Western and Eastern Europe. Journal of Personality Assessment, 96, 53–63. https://doi.org/10.1080/00223891.2013.809354.
49.
SchalmR. L., & KellowayE. K. (2001). The relationship between response rate and effect size in occupational health psychology research. Journal of Occupational Health Psychology, 6, 160–163. https://doi.org/10.1037/1076–8998.6.2.160.
50.
ScherpenzeelA. C., & DasM. (2010). “True” longitudinal and probability–based Internet panels: Evidence from the Netherlands. In DasM., EsterP., & KaxzmirekL. (Eds.), Social and behavioral research and the Internet: Advances in applied methods and research strategies (pp. 77–104). New York: Taylor & Francis.
51.
SchönbrodtF. D., & PeruginiM. (2013). At what sample size do correlations stabilize?Journal of Research in Personality, 47, 609–612. https://doi.org/10.1016/j.jrp.2013.05.009.
52.
SearsD. O. (1986). College sophomores in the laboratory: Influences of a narrow data base on social psychology's view of human nature. Journal of Personality and Social Psychology, 51, 515. https://doi.org/10.1037/0022–3514.51.3.515.
53.
SibleyC. G., & DuckittJ. (2008). Personality and prejudice: A meta–analysis and theoretical review. Personality and Social Psychology Review, 12, 248–279. https://doi.org/10.1177/1088868308319226.
54.
SibleyC. G., HardingJ. F., PerryR., AsbrockF., & DuckittJ. (2010). Personality and prejudice: Extension to the HEXACO personality model. European Journal of Personality, 24, 515–534. https://doi.org/10.1002/per.
55.
SibleyC. G., OsborneD., & DuckittJ. (2012). Personality and political orientation: Meta–analysis and test of a Threat–Constraint Model. Journal of Research in Personality, 46, 664–677. https://doi.org/10.1016/j.jrp.2012.08.002.
56.
SimonsD. J., ShodaY., & LindsayD. S. (2017). Constraints on Generality (COG): A proposed addition to all empirical papers. Perspectives on Psychological Science, 12, 1123–1128. https://doi.org/10.1177/1745691617708630.
57.
Van DeursenA. J., & van DijkJ. A. (2015). Toward a multifaceted model of Internet access for understanding digital divides: An empirical investigation. The Information Society, 31, 379–391. https://doi.org/10.1080/01972243.2015.1069770.
58.
Van HielA., CornelisI., & RoetsA. (2007). The intervening role of social worldviews in the relationship between the Five–Factor Model of personality and social attitudes. European Journal of Personality, 148, 131–148. https://doi.org/10.1002/per.618.
59.
VecchioneM., & CapraraG. V. (2009). Personality determinants of political participation: The contribution of traits and self–efficacy beliefs. Personality and Individual Differences, 46, 487–492. https://doi.org/10.1016/j.paid.2008.11.021.
60.
VoorpostelM., TillmannR., KuhnU., LippsO., RyserV. A., & SchmidF. (2016). Swiss household panel user guide (1999–2016). Lausanne: FORS.
61.
VowlesJ., CoffeH., & CurtinJ. (2017). A bark but no bite: Inequality and the 2014 New Zealand general election. Canberra, Australia: ANU Press.
62.
WangW., RothschildD., GoelS., & GelmanA. (2015). Forecasting elections with non–representative polls. International Journal of Forecasting, 31, 980–991. https://doi.org/10.1016/j.ijforecast.2014.06.001.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
