We examine the appropriateness of response speed and response consistency as data quality indicators within online samples. Across several inventories, results show that response consistency decreases dramatically at response rates faster than 1 second per item. Our results suggest that careless responding may be fairly common in online samples and often functions to increase the expected correlation between items in a survey, which has implications for the likelihood of false positives and the analysis of factor structure. Given how careless responding can influence estimated associations between variables, we strongly recommend that researchers include response speed and consistency screens in their research and provide empirically informed cut points for data screens that should be useful across a wide range of instruments and settings.
AnusicI.SchimmackU.PinkusR. T.LockwoodP. (2009). The nature and structure of correlations among Big Five ratings: The halo-alpha-beta model. Journal of Personality and Social Psychology, 97, 1142–1156.
2.
BehrendT. S.SharekD. J.MeadeA. W.WiebeE. N. (2011). The viability of crowdsourcing for survey research. Behavior Research Methods, 43, 800–813.
3.
BohannonJ. (2016). Mechanical Turk upends social sciences. Science, 352, 1263–1264.
4.
BuhrmesterM.KwangT.GoslingS. D. (2011). Amazon’s Mechanical Turk a new source of inexpensive, yet high-quality, data?Perspectives on Psychological Science, 6, 3–5.
5.
CaspiA.HerbenerE. S. (1990). Continuity and change: Assortative marriage and the consistency of personality in adulthood. Journal of Personality and Social Psychology, 58, 250–258.
6.
ChandlerJ.MuellerP.PaolacciG. (2014). Nonnaïveté among Amazon Mechanical Turk workers: Consequences and solutions for behavioral researchers. Behavior Research Methods, 46, 112–130.
7.
ClevelandW. S.DevlinS. J. (1988). Locally weighted regression: An approach to regression analysis by local fitting. Journal of the American Statistical Association, 83, 596–610.
8.
CredéM. (2010). Random responding as a threat to the validity of effect size estimates in correlational research. Educational and Psychological Measurement, 70, 596–612.
9.
CredéM.HarmsP. D. (2015). 25 years of higher-order confirmatory factor analysis in the organizational sciences: A critical review and development of reporting recommendations. Journal of Organizational Behavior, 36, 845–872.
10.
CronbachL. J. (1947). Test “reliability”: Its meaning and determination. Psychometrika, 12, 1–16.
DeSimoneJ. A.HarmsP.DeSimoneA. J. (2015). Best practice recommendations for data screening. Journal of Organizational Behavior, 36, 171–181.
13.
FeitosaJ.JosephD. L.NewmanD. A. (2015). Crowdsourcing and personality measurement equivalence: A warning about countries whose primary language is not English. Personality and Individual Differences, 75, 47–52.
14.
GoslingS. D.VazireS.SrivastavaS.JohnO. P. (2004). Should we trust web-based studies? A comparative analysis of six preconceptions about internet questionnaires. American Psychologist, 59, 93–104.
15.
GuttmanL. (1945). A basis for analyzing test–retest reliability. Psychometrika, 10, 255–282.
16.
HarmsP.DeSimoneJ. A. (2015). Caution! MTurk workers ahead—Fines doubled. Industrial and Organizational Psychology, 8, 183–190.
17.
HuangJ. L.CurranP. G.KeeneyJ.PoposkiE. M.DeShonR. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27, 99–114.
18.
HuangJ. L.LiuM.BowlingN. A. (2015). Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology, 100, 828–845.
19.
JohnsonJ. A. (2005). Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality, 39, 103–129.
20.
JonasonP. K.WebsterG. D. (2010). The dirty dozen: A concise measure of the dark triad. Psychological Assessment, 22, 420–432.
21.
JonesD. N.PaulhusD. L. (2014). Introducing the short dark triad (SD3) a brief measure of dark personality traits. Assessment, 21, 28–41.
22.
LinW.GreenD. P. (2016). Standard operating procedures: A safety net for pre-analysis plans. Political Science & Politics, 49, 495–500.
23.
MasonW.SuriS. (2012). Conducting behavioral research on Amazon’s Mechanical Turk. Behavior Research Methods, 44, 1–23.
24.
McCraeR. R.KurtzJ. E.YamagataS.TerraccianoA. (2011). Internal consistency, retest reliability, and their implications for personality scale validity. Personality and Social Psychology Review, 15, 28–50.
25.
MeadeA. W.CraigS. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437–455.
26.
O’ReillyC. A.ChatmanJ.CaldwellD. F. (1991). People and organizational culture: A profile comparison approach to assessing person-organization fit. Academy of Management Journal, 34, 487–516.
27.
PaulhusD. L.WilliamsK. M. (2002). The dark triad of personality: Narcissism, Machiavellianism, and psychopathy. Journal of Research in Personality, 36, 556–563.
28.
SchmittN.StultsD. M. (1985). Factors defined by negatively keyed items: The result of careless respondents?Applied Psychological Measurement, 9, 367–373.
29.
SchonlauM.ToepoelV. (2015). Straightlining in Web survey panels over time. Survey Research Methods, 9, 125–137.
30.
SotoC. J.JohnO. P. (2016). The next Big Five Inventory (BFI-2): Developing and assessing a hierarchical model with 15 facets to enhance bandwidth, fidelity, and predictive power. Journal of Personality and Social Psychology. Advance online publication. doi:10.1037/pspp0000096.
31.
SotoC. J.JohnO. P.GoslingS. D.PotterJ. (2008). The developmental psychometrics of big five self-reports: Acquiescence, factor structure, coherence, and differentiation from ages 10 to 20. Journal of Personality and Social Psychology, 94, 718–737.
32.
WatsonD.ClarkL. A. (1999). The PANAS-X: Manual for the Positive and Negative Affect Schedule-Expanded Form. Iowa City, IA: University of Iowa.
33.
WoodD.FurrR. M. (2016). The correlates of similarity estimates are often misleadingly positive: The nature and scope of the problem, and some solutions. Personality and Social Psychology Review, 20, 79–99.
34.
WoodD.HarmsP. D.LowmanG.SotoC. J.QiuL.JohnO. P.JiahuiL. (2017). Rethinking reliability: Evaluating the viability and utility of within-session retest correlations as reliability estimates. Tuscaloosa: University of Alabama.
35.
WoodD.NyeC. D.SaucierG. (2010). Identification and measurement of a more comprehensive set of person-descriptive trait markers from the English lexicon. Journal of Research in Personality, 44, 258–272.
36.
Woods, (2006). Careless responding to reverse-worded items: Implications for confirmatory factor analysis. Journal of Psychopathology and Behavioral Assessment, 28, 189–194.
37.
ZhangC.ConradF. (2014). Speeding in web surveys: The tendency to answer very fast and its association with straightlining. Survey Research Methods, 8, 127–135.
38.
ZhouH.FishbachA. (2016). The pitfall of experimenting on the web: How unattended selective attrition leads to surprising (yet false) research conclusions. Journal of Personality and Social Psychology, 114, 493–504. doi:10.1037/pspa0000056
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
