Aberrant responding on tests and surveys has been shown to affect the psychometric properties of scales and the statistical analyses from the use of those scales in cumulative model contexts. This study extends prior research by comparing the effects of four types of aberrant responding on model fit in both cumulative and ideal point model contexts using graded partial credit (GPCM) and generalized graded unfolding (GGUM) models. When fitting models to data, model misfit can be both a function of misspecification and aberrant responding. Results demonstrate how varying levels of aberrant data can severely impact model fit for both cumulative and ideal point data. Specifically, longstring responses have a stronger impact on dimensionality for both ideal point and cumulative data, while random responding tends to have the most negative impact on data model fit according to information criteria (AIC, BIC). The results also indicate that ideal point data models such as GGUM may be able to fit cumulative data as well as the cumulative model itself (GPCM), whereas cumulative data models may not provide sufficient model fit for data simulated using an ideal point model.
AkaikeH. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19(6), 716–723. https://doi.org/10.1109/TAC.1974.1100705
2.
AndrichD. (1988). The application of an unfolding model of the PIRT type to the measurement of attitude. Applied Psychological Measurement, 12(1), 33–51. https://doi.org/10.1177/014662168801200105
3.
AndrichD. (1996). A hyperbolic cosine latent trait model for unfolding polytomous responses: Reconciling Thurstone and Likert methodologies. British Journal of Mathematical and Statistical Psychology, 49(2), 347–365. https://doi.org/10.1111/j.2044-8317.1996.tb01093.x
4.
AndrichD.GuanzhongL. (1993). A hyperbolic cosine latent trait model for unfolding dichotomous single-stimulus responses. Applied Psychological Measurement, 17(3), 253–276. https://doi.org/10.1177/014662169301700307
5.
BaumgartnerH.SteenkampJ.-B. E. M. (2001). Response styles in marketing research: A cross-national investigation. Journal of Marketing Research, 38(2), 143–156. https://doi.org/10.1509/jmkr.38.2.143.18840
6.
BerryD. T. R.WetterM. W.BaerR. A.LarsenL.ClarkC.MonroeK. (1992). MMPI-2 random responding indices: Validation using a self-report methodology. Psychological Assessment, 4(3), 340–345. https://doi.org/10.1037/1040-3590.4.3.340
7.
CarterN.DalalD. (2010). An ideal point account of the JDI work satisfaction scale. Personality and Individual Differences, 49(7), 743–748. https://doi.org/10.1016/j.paid.2010.06.019
8.
CarterN.DalalD.BoyceA.O’ConnellM.KungM-C.DelgadoK. (2014). Uncovering curvilinear relationships between conscientiousness and job performance: How theoretically appropriate measurement makes an empirical difference. Journal of Applied Psychology, 99(4), 564–586. https://doi.org/10.1037/a0034688
9.
ChalmersR. (2012). Mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(1), 1–29. https://doi.org/10.18637/jss.v048.i06
10.
ChernyshenkoO. S.StarkS.DrasgowF.RobertsB. W. (2007). Constructing personality scales under the assumptions of an ideal point response process: Toward increasing the flexibility of personality measures. Psychological Assessment, 19(1), 88–106. https://doi.org/10.1037/1040-3590.19.1.88
11.
DalalD. K.CarterN. T.LakeC. J. (2014). Middle response scale options are inappropriate for ideal point scales. Journal of Business and Psychology, 29(3), 463–478. https://doi.org/10.1007/s10869-013-9326-5
12.
DavisonM. (1977). On a metric, unidimensional unfolding model for attitudinal and developmental data. Psychometrika, 42(4), 523–548.
13.
DeSimoneJ. A.HarmsP. D. (2018). Dirty data: The effects of screening respondents who provide low-quality data in survey research. Journal of Business and Psychology, 33(5), 559–577. https://doi.org/10.1007/s10869-017-9514-9
DrasgowF.HulinC. L. (1990). Item response theory. In Handbook of industrial and organizational psychology, (Vol. 1, 2nd ed., pp. 577–636). Consulting Psychologists Press.
16.
DrasgowF.LevineM. V.TsienS.WilliamsB.MeadA. D. (1995). Fitting polytomous item response theory models to multiple-choice tests. Applied Psychological Measurement, 19(2), 143–166. https://doi.org/10.1177/014662169501900203
17.
FreundP. A.LohbeckA. (2021). Modeling self-determination theory motivation data by using unfolding IRT. European Journal of Psychological Assessment, 37(5), 388–396. https://doi.org/10.1027/1015-5759/a000629
18.
GessaroliM. E.De ChamplainA. F. (1996). Using an approximate chi-square statistic to test the number of dimensions underlying the responses to a set of items. Journal of Educational Measurement, 33(2), 157–179. https://doi.org/10.1111/j.1745-3984.1996.tb00487.x
19.
GreenleafE. A. (1992). Measuring extreme response style. Public Opinion Quarterly, 56(3), 328–351. https://doi.org/10.1086/269326
20.
HambletonR. K.SwaminathanH.RogersH. J. (1991). Fundamentals of item response theory. Sage.
21.
JinK.-Y.ChenH.-F.WangW.-C. (2018). Mixture item response models for inattentive responding behavior. Organizational Research Methods, 21(1), 197–225. https://doi.org/10.1177/1094428117725792
22.
JooS.-H.LeeP.ParkJ. Y.StarkS. (2023). Assessing dimensionality of the ideal point item response theory model using posterior predictive model checking. Organizational Research Methods, 26(2), 353–382. https://doi.org/10.1177/10944281211050609
23.
LikertR. (1932). A technique for the measurement of attitudes. Archives of Psychology, 22(140), 55–55.
24.
LiuC.-W.ChalmersR. P. (2018). Fitting item response unfolding models to Likert-scale data using mirt in R. PLoS One, 13(5), e0196292. https://doi.org/10.1371/journal.pone.0196292
25.
LiuC.-W.WangW.-C. (2019). A general unfolding IRT model for multiple response styles. Applied Psychological Measurement, 43(3), 195–210. https://doi.org/10.1177/0146621618762743
26.
LiuJ.ZhangJ. (2020). An item-level analysis for detecting faking on personality tests: Appropriateness of ideal point item response theory models. Frontiers in Psychology, 10, 3090. https://doi.org/10.3389/fpsyg.2019.03090
27.
LiuT.SunY.LiZ.XinT. (2019). The impact of aberrant response on reliability and validity. Measurement: Interdisciplinary Research and Perspectives, 17(3), 133–142. https://doi.org/10.1080/15366367.2019.1584848
28.
LiuY.DouglasJ. A.HensonR. A. (2009). Testing person fit in cognitive diagnosis. Applied Psychological Measurement, 33(8), 579–598. https://doi.org/10.1177/0146621609331960
29.
LordF. M. (1980). Applications of item response theory to practical testing problems. Lawrence Erlbaum.
30.
LordF. M.NovickM. R. (1968). Statistical theories of mental test scores (with contributions by A. Birnbaum). Addison-Wesley.
31.
LuoG. (1998a). A general formulation for unidimensional unfolding and pairwise preference models: Making explicit the latitude of acceptance. Journal of Mathematical Psychology, 42(4), 400–417. https://doi.org/10.1006/jmps.1998.1206
32.
LuoG. (1998b). A general formulation of multidimensional unfolding models involving the latitude of acceptance. In RizziA.VichiM.BockH.-H. (Eds.), Advances in data science and classification: Proceedings of the 6th conference of the international federation of classification societies (IFCS-98) università “La Sapienza”, Rome, 21–24 July, 1998 (pp. 483–488). Springer.
33.
LuoG. (2001). A class of probabilistic unfolding models for polytomous responses. Journal of Mathematical Psychology, 45(2), 224–248. https://doi.org/10.1006/jmps.2000.1310
34.
ManiaciM. R.RoggeR. D. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48, 61–83. https://doi.org/10.1016/j.jrp.2013.09.008
35.
Maydeu-OlivaresA.CaiL.HernandezA. (2011). Comparing the fit of item response theory and factor analysis models. Structural Equation Modeling: A Multidisciplinary Journal, 18(3), 333–356. https://doi.org/10.1080/10705511.2011.581993
36.
MeijerR. R. (1997). Person fit and criterion-related validity: An extension of the Schmitt, Cortina, and Whitney study. Applied Psychological Measurement, 21(2), 99–113. https://doi.org/10.1177/01466216970212001
MurphyM. K. (1996). Psychological aspects of survey methodology: Experiments on the response process. University of London.
39.
NyeC. D.JooS.-H.ZhangB.StarkS. (2020). Advancing and evaluating IRT model data fit indices in organizational research. Organizational Research Methods, 23(3), 457–486. https://doi.org/10.1177/1094428119833158
40.
R Core Team (2016). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URLhttps://www.R-project.org/.
41.
RobertsJ. S.DonoghueJ. R.LaughlinJ. E. (2000). A general item response theory model for unfolding unidimensional polytomous responses. Applied Psychological Measurement, 24(1), 3–32. https://doi.org/10.1177/01466216000241001
42.
RobertsJ.LaughlinJ. (1996). A unidimensional item response model for unfolding responses from a graded disagree-agree response scale. Applied Psychological Measurement, 20(3), 231–255. https://doi.org/10.1177/014662169602000305
43.
RossJ.CliffN. (1964). A generalization of the interpoint distance model. Psychometrika, 29(2), 167–176. https://doi.org/10.1007/bf02289698
44.
RuppA. (2013). A systematic review of the methodology for person fit research in Item Response Theory: Lessons about generalizability of inferences from the design of simulation studies. Psychological Test and Assessment Modeling, 55(1), 3–38.
StarkS.ChernyshenkoO. S.DrasgowF.WilliamsB. A. (2006). Examining assumptions about item responding in personality assessment: Should ideal point methods be considered for scale development and scoring?Journal of Applied Psychology, 91(1), 25–39. https://doi.org/10.1037/0021-9010.91.1.25
47.
TayL.AliU. S.DrasgowF.WilliamsB. (2011). Fitting IRT models to dichotomous and polytomous data: Assessing the relative model–data fit of ideal point and dominance models. Applied Psychological Measurement, 35(4), 280–295. https://doi.org/10.1177/0146621610390674
48.
TayL.DrasgowF. (2012). Theoretical, statistical, and substantive issues in the assessment of construct dimensionality: Accounting for the item response process. Organizational Research Methods, 15(3), 363–384. https://doi.org/10.1177/1094428112439709
49.
TendeiroJ. (2017). The lz(p)* person-fit statistic in an unfolding model context. Applied Psychological Measurement, 41(1), 44–59. https://doi.org/10.1177/0146621616669336
50.
TendeiroJ. N.Castro-AlvarezS. (2019). GGUM: An R package for fitting the generalized graded unfolding model. Applied Psychological Measurement, 43(2), 172–173. https://doi.org/10.1177/0146621618772290
51.
TendeiroJ. N.MeijerR. R. (2014). Detection of invalid test scores: The usefulness of simple nonparametric statistics. Journal of Educational Measurement, 51(3), 239–259. https://doi.org/10.1111/jedm.12046
TourangeauR.RasinskiK. A. (1988). Cognitive processes underlying context effects in attitude measurement. Psychological Bulletin, 103(3), 299–314. https://doi.org/10.1037/0033-2909.103.3.299
54.
WeekersA.MeijerR. (2008). Scaling response processes on personality items using unfolding and dominance models: An illustration with a Dutch dominance and unfolding personality inventory. European Journal of Psychological Assessment, 24(1), 65–77. https://doi.org/10.1027/1015-5759.24.1.65
55.
WilgusS.TravisJ. (2019). A comparison of ideal-point and dominance response processes with a trust in science Thurstone scale (pp. 415–428). https://doi.org/10.1007/978-3-030-01310-3_36
56.
WilliamsE. J.(2015). Dimensionality assessment of proximity-based data in unfolding model applications (Doctoral dissertation, Georgia Institute of Technology).
57.
XiaY.ZhengY. (2018). Asymptotically normally distributed person fit indices for detecting spuriously high scores on difficult items. Applied Psychological Measurement, 42(5), 343–358. https://doi.org/10.1177/0146621617730391
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
