Insufficient effort responding (IER) affects many forms of assessment in both educational and psychological contexts. Much research has examined different types of IER, IER’s impact on the psychometric properties of test scores, and preprocessing procedures used to detect IER. However, there is a gap in the literature in terms of practical advice for applied researchers and psychometricians when evaluating multiple sources of IER evidence, including the best strategy or combination of strategies when preprocessing data. In this study, we demonstrate how the use of different IER detection methods may affect psychometric properties such as predictive validity and reliability. Moreover, we evaluate how different data cleansing procedures can detect different types of IER. We provide evidence via simulation studies and applied analysis using the ACT’s Engage assessment as a motivating example. Based on the findings of the study, we provide recommendations and future research directions for those who suspect their data may contain responses reflecting careless, random, or biased responding.
American Educational Research Association, American Psychological Association, & National Council on Measurement Education. (2014). Standards for educational and psychological testing. Retrieved from https://www.apa.org/science/programs/testing/standards
3.
AustinE. J.DearyI. J.GibsonG. J.McGregorM. J.DentJ. B. (1998). Individual response spread in self-report scales: Personality correlations and consequences. Personality and Individual Differences, 24, 421-438.
4.
BerryD. T. R.WetterM. W.BaerR. A.LarsenL.ClarkC.MonroeK. (1992). MMPI-2 Random responding indices: Validation using a self-report methodology. Psychological Assessment, 4, 340-345.
5.
BreitsohlH.SteidelmüllerC. (2018). The impact of insufficient effort responding detection methods on substantive responses: Results from an experiment testing parameter invariance. Applied Psychology, 67, 284-308.
6.
ChengY.YuanK-H.LiuC. (2012). Comparison of reliability measures under factor analysis and item response theory. Educational and Psychological Measurement, 72, 52-67. doi:10.1177/0013164411407315
7.
ClarkM. E.GirondaR. J.YoungR. W. (2003). Detection of back random responding: Effectiveness of MMPI-2 and personality assessment inventory validity indices. Psychological Assessment, 15, 223-234.
8.
CurranP. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4-19.
9.
DrasgowF.LevineM. V.WilliamsE. A. (1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38, 67-86.
10.
DunnA. M.HeggestadE. D.ShanockL. R.TheilgardN. (2018). Intra-individual response variability as an indicator of insufficient effort responding: Comparison to other indicators and relationships with individual differences. Journal of Business and Psychology, 33, 105-121.
11.
FalkC. F.CaiL. (2016). A flexible full-information approach to the modeling of response styles. Psychological Methods, 21, 328-347.
12.
GoldbergL. R.KilkowskiJ. M. (1985). The prediction of semantic consistency in self-descriptions. Characteristics of persons and of terms that affect the consistency of responses to synonym and antonym pairs. Journal of Personality and Social Psychology, 48, 82-98.
13.
HauserD. J.SchwarzN. (2016). Attentive turkers: MTurk participants perform better on online attention checks than do subject pool participants. Behavior Research Methods, 48, 400-407.
14.
HongM.ChengY. (2019a). Clarifying the effect of test speededness. Applied Psychological Measurement. Advance online publication. doi:10.1177/0146621618817783
15.
HongM.ChengY. (2019b). Robust maximum marginal likelihood (RMML) estimation for item response theory models. Behavior Research Methods, 1, 573-588. doi:10.3758/s13428-018-1150-4
16.
HuangJ. L.BowlingN. A.LiuM.LiY. (2015). Detecting insufficient effort responding with an infrequency scale: Evaluating validity and participant reactions. Journal of Business and Psychology, 30, 299-311.
17.
HuangJ. L.CurranP. G.KeeneyJ.PoposkiE. M.DeShonR. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27, 99-114.
18.
HuangJ. L.LiuM.BowlingN. A. (2015). Insufficient effort responding: Examining an insidious confound in survey data. Journal of Applied Psychology, 100, 828-845. doi:10.1037/a0038510
19.
JohnsonJ. A. (2005). Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality, 39, 103-129.
20.
KarabatsosG. (2003). Comparing the aberrant response detection performance of thirty-six person-fit statistics. Applied Measurement in Education, 16, 277-298. doi:10.1207/S15324818AME1604
21.
KimD. S.ReiseS. P.BentlerP. M. (2018). Identifying aberrant data in structural equation models with IRLS-ADF. Structural Equation Modeling, 25, 343-358.
22.
LeH.CasillasA.RobbinsS. B.LangleyR. (2005). Motivational and skills, social, and self-management predictors of college outcomes: Constructing the student readiness inventory. Educational and Psychological Measurement, 65, 482-508.
23.
LokenE.GelmanA. (2017). Measurement error and the replication crisis. Science, 355, 584-585.
24.
MahalanobisP. (1936). On the generalized distance in statistics. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, 2(1), 49-55.
25.
MariantiS.FoxJ.-P.AvetisyanM.VeldkampB. P.TijmstraJ. (2014). Testing for aberrant behavior in response time modeling. Journal of Educational and Behavioral Statistics, 39, 426-451. doi:10.3102/1076998614559412
26.
MaxwellS. E.LauM. Y.HowardG. S. (2015). Is psychology suffering from a replication crisis? What does “failure to replicate” really mean?American Psychologist, 70, 487-498. doi:10.1037/a0039400
27.
MeadeA. W.CraigS. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17, 437-455. doi:10.1037/a0028085
28.
MeijerR. R.SijtsmaK. (2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25, 107-135. doi:10.1177/01466210122031957
29.
NiessenA. S. M.MeijerR. R.TendeiroJ. N. (2016). Detecting careless respondents in web-based questionnaires: Which method to use?Journal of Research in Personality, 63, 1-11. doi:10.1016/j.jrp.2016.04.010
30.
PauszekJ. R.SztybelP.GibsonB. S. (2017). Evaluating Amazon’s Mechanical Turk for psychological research on the symbolic control of attention. Behavior Research Methods, 49, 1969-1983. doi:10.3758/s13428-016-0847-5
31.
RhemtullaM.Brosseau-LiardP. É.SavaleiV. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17, 354-373. doi:10.1037/a0029315
32.
ShaoC.LiJ.ChengY. (2016). Detection of test speededness using change-point analysis. Psychometrika, 81, 1118-1141. doi:10.1007/s11336-015-9476-7
33.
SinharayS. (2015). The asymptotic distribution of ability estimates: Beyond dichotomous items and unidimensional IRT models. Journal of Educational and Behavioral Statistics, 40, 511-528. doi:10.3102/1076998615606115
34.
SnijdersT. A. B. (2001). Asymptotic null distribution of person fit statistics with estimated person parameter. Psychometrika, 66, 331-342. doi:10.1007/BF02294437
35.
SteedleJ.HongM.ChengY. (2019). The effects of inattentive responding on construct validity evidence when measuring social-emotional learning competencies. Educational Measurement Issues and Practice, 38, 101-111.
36.
SuhY.ChoS. J.WollackJ. A. (2012). A comparison of item calibration procedures in the presence of test speededness. Journal of Educational Measurement, 49, 285-311. doi:10.1111/j.1745-3984.2012.00176.x
37.
SwainS. D.WeathersD.NiedrichR. W. (2008). Assessing three sources of misresponse to reversed Likert items. Journal of Marketing Research, 45, 116-131. doi:10.1509/jmkr.45.1.116
38.
TendeiroJ. N. (2017). The lz(p)* Person-fit statistic in an unfolding model context. Applied Psychological Measurement, 41, 44-59. doi:10.1177/0146621616669336
39.
TendeiroJ. N.MeijerR. R.NiessenA. S. M. (2016). PerFit : an R package for person-fit analysis in IRT. Journal of Statistical Software, 74(5). doi:10.18637/jss.v074.i05
Van Krimpen-StoopE. M. L. A.MeijerR. R. (2002). Detection of person misfit in computerized adaptive tests with polytomous items. Applied Psychological Measurement, 26, 164-180. doi:10.1177/01421602026002004
42.
WangC.XuG.ShangZ. (2018). A two-stage approach to differentiating normal and aberrant behavior in computer based testing. Psychometrika, 83, 223-254. doi:10.1007/s11336-016-9525-x
WoodsC. M. (2006). Careless responding to reverse-worded items: Implications for confirmatory factor analysis. Journal of Psychopathology and Behavioral Assessment, 28, 189-194. doi:10.1007/s10862-005-9004-7
45.
YamamotoK.EversonH. (2003). Estimating the effects of test length and test time on parameter estimation using the hybrid model. ETS Research Report Series, 1995, 277-298. doi:10.1002/j.2333-8504.1995.tb01637.x
46.
YuX.ChengY. (2019). A change-point analysis procedure based on weighted residuals to detect back random responding. Psychological Methods. Advance online publication. doi:10.1037/met0000212
47.
YuanK. H.ZhongX. (2008). Outliers, leverage observations, and influential cases in factor analysis: Using robust procedures to minimize their effect. Sociological Methodology, 38, 329-368. doi:10.1111/j.1467-9531.2008.00198.x
48.
YuanK. H.ZhongX. (2013). Robustness of fit indices to outliers and leverage observations in structural equation modeling. Psychological Methods, 18, 121-136. doi:10.1037/a0031604
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
