Social desirability bias (SDB) has been a major concern in educational and psychological assessments when measuring latent variables because it has the potential to introduce measurement error and bias in assessments. Person-fit indices can detect bias in the form of misfitted response vectors. The objective of this study was to compare the performance of 14 person-fit indices to identify SDB in simulated responses. The area under the curve (AUC) of receiver operating characteristic (ROC) curve analysis was computed to evaluate the predictive power of these statistics. The findings showed that the agreement statistic outperformed all other person-fit indices, while the disagreement statistic , dependability statistic , and the number of Guttman errors also demonstrated high AUCs to detect SDB. Recommendations for practitioners to use these fit indices are provided.
ArtnerR. (2016). A simulation study of person-fit in the Rasch model. Psychological Test and Assessment Modeling, 58(3), 531–563. https://lirias.kuleuven.be/retrieve/523896
2.
BeckM. F.AlbanoA. D.SmithW. M. (2019). Person-fit as an index of inattentive responding: A comparison of methods using polytomous survey data. Applied Psychological Measurement, 43(5), 374–387. https://doi.org/10.1177%2F0146621618798666
3.
BernreuterR. G. (1933). Validity of the personality inventory. Personnel Journal, 11, 383–386.
4.
BirnbaumA. (1968). Some latent trait models and their use in inferring an examinee’s ability, contributed chapter. In LordF. M.NovicM. R. (Eds.), Statistical theories of mental test scores (pp. 397–479). Addison-Wesley.
CrockerL.AlginaJ. (1986). Introduction to classical and modern test theory. Holt, Rinehart and Winston.
9.
CrowneD. P.MarloweD. (1960). A new scale of social desirability independent of psychopathology. Journal of Consulting Psychology, 24(4), 349–354. https://psycnet.apa.org/doi/10.1037/h0047358
10.
CrowneD. P.MarloweD. (1964). The approval motive: Studies in evaluative dependence. Wiley.
11.
DawesS. E.PalmerB. W.AllisonM. A.GaniatsT. G.JesteD. V. (2011). Social desirability does not confound reports of wellbeing or of socio-demographic attributes by older women. Ageing & Society, 31(3), 438–454. https://doi.org/10.1017/S0144686X10001029
12.
DimitrovD.SmithR. M. (2006). Adjusted Rasch person-fit statistics. Journal of Applied Measurement, 7, 170–183.
13.
DonlonT. F.FischerF. E. (1968). An index of an individual’s agreement with group-determined item difficulties. Educational and Psychological Measurement, 28(1), 105–113. https://psycnet.apa.org/doi/10.1177/001316446802800110
14.
DrasgowF.LevineM. V.WilliamsE. A. (1985). Appropriateness measurement with polychotomous item response models and standardized indices. British Journal of Mathematical and Statistical Psychology, 38(1), 67–86. https://doi.org/10.1111/j.2044-8317.1985.tb00817.x
15.
EmbretsonS. E.ReiseS. P. (2013). Item response theory. Psychology Press.
16.
EmonsW. H. (2008). Nonparametric person-fit analysis of polytomous item scores. Applied Psychological Measurement, 32(3), 224–247. https://doi.org/10.1177/0146621607302479
17.
FalkC. F.CaiL. (2016). A flexible full-information approach to the modeling of response styles. Psychological Methods, 21(3), 328–347. https://doi.org/10.1037/met0000059
18.
FernandezE.WoldgabrealY.GuharajanD.DayA.KiageriV.RamtahalN. (2019). Social desirability bias against admitting anger: Bias in the test-taker or bias in the test?Journal of Personality Assessment, 101(6), 644–652. https://doi.org/10.1080/00223891.2018.1464017
19.
FerrandoP. J. (2005). Factor analytic procedures for assessing social desirability in binary items. Multivariate Behavioral Research, 40(3), 331–349. https://doi.org/10.1207/s15327906mbr4003_3
20.
FisherT. D.MooreZ. T.PittengerM. J. (2012). Sex on the brain? An examination of frequency of sexual cognitions as a function of gender, erotophilia, and social desirability. Journal of Sex Research, 49(1), 69–77. https://doi.org/10.1080/00224499.2011.565429
21.
GaoX.HeY.ZhangM.DiaoX.JingX.RenB.JiW. (2021). A multiclass classification using one-versus-all approach with the differential partition sampling ensemble. Engineering Applications of Artificial Intelligence, 97, 104034. https://doi.org/10.1016/j.engappai.2020.104034
22.
HallquistM. N.WileyJ. F. (2018). MplusAutomation: An R package for facilitating large-scale latent variable analyses in M plus. Structural Equation Modeling: A Multidisciplinary Journal, 25(4), 621–638. https://doi.org/10.1080/10705511.2017.1402334
23.
HanleyJ. A.McNeilB. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology, 143(1), 29–36. https://doi.org/10.1148/radiology.143.1.7063747
24.
HarnischD.LinnR. (1981). Analysis of item response patterns: Questionable test data and dissimilar curriculum practices. Journal of Educational Measurement, 18(3), 133–146. http://www.jstor.org/stable/1434737
25.
HeckD. W.HoffmannA.MoshagenM. (2018). Detecting nonadherence without loss in efficiency: A simple extension of the crosswise model. Behavior Research Methods, 50(5), 1895–1905. https://doi.org/10.3758/s13428-017-0957-8
26.
HolbrookA. L.KrosnickJ. A. (2010). Social desirability bias in voter turnout reports: Tests using the item count technique. Public Opinion Quarterly, 74(1), 37–67. https://doi.org/10.1093/poq/nfp065
27.
HongM.LinL.ChengY. (2021). Asymptotically corrected person fit statistics for multidimensional constructs with simple structure and mixed item types. Psychometrika, 86, 464–488. https://doi.org/10.1007/s11336-021-09756-3
28.
JohnsonR. E.RosenC. C.DjurdjevicE. (2011). Assessing the impact of common method variance on higher order multidimensional constructs. Journal of Applied Psychology, 96(4), 744–761. https://psycnet.apa.org/doi/10.1037/a0021504
29.
JonesE. E.SigallH. (1971). The bogus pipeline: A new paradigm for measuring affect and attitude. Psychological Bulletin, 76(5), 349–364. https://psycnet.apa.org/doi/10.1037/h0031617
30.
KamataA.BauerD. J. (2008). A note on the relation between factor analytic and item response theory models. Structural Equation Modeling: A Multidisciplinary Journal, 15(1), 136–153. https://doi.org/10.1080/10705510701758406
31.
KaminskaO.FoulshamT. (2016). Eye-tracking social desirability bias. Bulletin of Sociological Methodology/Bulletin de Méthodologie Sociologique, 130(1), 73–89. https://doi.org/10.1177%2F0759106315627591
32.
KaneM. T.BrennanR. L. (1980). Agreement coefficients as indices of dependability for domain-referenced tests. Applied Psychological Measurement, 4(1), 105–126. https://doi.org/10.1177%2F014662168000400111
KarabatsosG. (2003). Comparing the aberrant response detection performance of thirty-six person-fit statistics. Applied Measurement in Education, 16(4), 277–298. https://doi.org/10.1207/S15324818AME1604_2
35.
KimY.MuthénB. O. (2009). Two-part factor mixture modeling: Application to an aggressive behavior measurement instrument. Structural Equation Modeling, 16(4), 602–624. https://doi.org/10.1080%2F10705510903203516
36.
LeiteW. L.BeretvasS. N. (2005). Validation of scores on the Marlowe-Crowne social desirability scale and the balanced inventory of desirable responding. Educational and Psychological Measurement, 65(1), 140–154. https://doi.org/10.1177%2F0013164404267285
37.
LeiteW. L.CooperL. A. (2010). Detecting social desirability bias using factor mixture models. Multivariate Behavioral Research, 45(2), 271–293. https://doi.org/10.1080/00273171003680245
38.
LeiteW. L.NazariS. (2020). Marlowe-Crowne social desirability scale. In Zeigler-HillV.ShackelfordT. K. (Eds.), Encyclopedia of personality and individual differences. Springer (pp. 2751–2753). https://doi.org/10.1007/978-3-319-24612-3_45
39.
LengC. H.HuangH. Y.YaoG. (2020). A social desirability item response theory model: Retrieve–deceive–transfer. Psychometrika, 85(1), 56–74. https://doi.org/10.1007/s11336-019-09689-y
40.
LinacreJ. M. (2002). What do infit and outfit, mean-square and standardized mean. Rasch Measurement Transactions, 16(2), 878.
LubkeG. H.MuthénB. O. (2007). Performance of factor mixture models as a function of model size, covariate effects, and class-specific parameters. Structural Equation Modeling, 14(1), 26–47. https://doi.org/10.1080/10705510709336735
43.
MeijerR. R. (1994). The number of Guttman errors as a simple and powerful person-fit statistic. Applied Psychological Measurement, 18(4), 311–314. https://doi.org/10.1177%2F014662169401800402
NazariS.LeiteW. L.Huggins-ManleyA. C. (2021). Detecting Careless Responses to Assessment Items in a Virtual Learning Environment Using Person-fit Indices and Random Forest. In Proceedings of the 14th International Conference on Educational Data Mining, 635–640. https://educationaldatamining.org/EDM2021/virtual/static/pdf/EDM21_paper_155.pdf
49.
OlejnikS.AlginaJ. (2003). Generalized eta and omega squared statistics: Measures of effect size for some common research designs. Psychological Methods, 8(4), 434–447. https://psycnet.apa.org/doi/10.1037/1082-989X.8.4.434
PaulhusD. L. (1991). Measurement and control of response bias. In RobinsonJ. P.ShaverP. R.WrightsmanL. S. (Eds.), Measures of personality and social psychological attitudes (pp. 17–59). Academic Press. https://doi.org/10.1016/B978-0-12-590241-0.50006-X
53.
PodsakoffP. M.MacKenzieS. B.LeeJ. Y.PodsakoffN. P. (2003). Common method biases in behavioral research: A critical review of the literature and recommended remedies. Journal of Applied Psychology, 88(5), 879–903. https://doi.org/10.1037/0021-9010.88.5.879
54.
R Core Team. (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing. http://www.R-project.org/
RuppA. A. (2013). A systematic review of the methodology for person fit research in item response theory: Lessons about generalizability of inferences from the design of simulation studies. Psychological Test and Assessment Modeling, 55(1), 3–38.
SijtsmaK.MeijerR. R. (1992). A method for investigating the intersection of item response functions in Mokken's nonparametric IRT model. Applied Psychological Measurement, 16(2), 149–157. https://doi.org/10.1177/014662169201600204
61.
SinharayS. (2015). Assessment of person fit for mixed-format tests. Journal of Educational and Behavioral Statistics, 40(4), 343–365. https://doi.org/10.3102%2F1076998615589128
62.
SinharayS. (2017). How to compare parametric and nonparametric person-fit statistics using real data. Journal of Educational Measurement, 54(4), 420–439. https://doi.org/10.1111/jedm.12155
63.
SnijdersT. A. (2001). Asymptotic null distribution of person fit statistics with estimated person parameter. Psychometrika, 66(3), 331–342. https://doi.org/10.1007/BF02294437
64.
StarkT. H.van MaarenF. M.KrosnickJ. A.SoodG. (2019). The impact of social desirability pressures on whites’ endorsement of racial stereotypes: A comparison between oral and ACASI reports in a national survey. Sociological Methods & Research, 51(2), 605–631. https://doi.org/10.1177%2F0049124119875959
TatsuokaK. K.TatsuokaM. M. (1982). Detection of aberrant response patterns and their effect on dimensionality. Journal of Educational Statistics, 7(3), 215–231. https://doi.org/10.3102%2F10769986007003215
68.
TatsuokaK. K.TatsuokaM. M. (1983). Spotting erroneous rules of operation by the individual consistency index. Journal of Educational Measurement, 20(3), 221–230. http://www.jstor.org/stable/1434713
69.
TendeiroJ. N. (2017). The lz (p)* person-fit statistic in an unfolding model context. Applied Psychological Measurement, 41(1), 44–59. https://doi.org/10.1177%2F0146621616669336
70.
TendeiroJ. N.MeijerR. R. (2014). Detection of invalid test scores: The usefulness of simple nonparametric statistics. Journal of Educational Measurement, 51(3), 239–259. https://doi.org/10.1111/jedm.12046
71.
TendeiroJ. N.MeijerR. R.NiessenA. S. M. (2016). PerFit: An R package for person-fit analysis in IRT. Journal of Statistical Software, 74(5), 1–27. https://doi.org/10.18637/jss.v074.i05
72.
TourangeauR.RipsL. J.RasinskiK. (2000). The psychology of survey response. Cambridge University Press.
73.
Van der FlierH. (1977). Environmental factors and deviant response patterns. Basic problems in cross cultural psychology. Swets & Seitlinger.
74.
Van der FlierH. (1980). Vergelijkbaarheid van individuele testprestaties. Swets & Zeitlinger.
75.
Van der FlierH. (1982). Deviant response patterns and comparability of test scores. Journal of Cross-Cultural Psychology, 13(3), 267–298. https://doi.org/10.1177%2F0022002182013003001
76.
VésteinsdóttirV.JoinsonA.ReipsU. D.DanielsdottirH. B.ThorarinsdottirE. A.ThorsdottirF. (2019). Questions on honest responding. Behavior Research Methods, 51(2), 811–825. https://doi.org/10.3758/s13428-018-1121-9
77.
WarnerS. L. (1965). Randomized response: A survey technique for eliminating evasive answer bias. Journal of the American Statistical Association, 60(309), 63–69. https://doi.org/10.1080/01621459.1965.10480775
