When scales or tests are used to make decisions about individuals (e.g., to identify which adults should be assessed for psychiatric disorders), it is crucial that these decisions be accurate and consistent. However, it is not obvious how to assess accuracy and consistency when the scale was administered only once to a given sample and the true condition based on the latent variable is unknown. This article describes a method based on the linear factor model for evaluating the accuracy and consistency of scale-based decisions using data from a single administration of the scale. We illustrate the procedure and provide R code that investigators can use to apply the method in their own data. Finally, in a simulation study, we evaluate how the method performs when applied to discrete (vs. continuous) items, a practice that is common in published literature. The results suggest that the method is generally robust when applied to discrete items.
AchenbachT. M.RescorlaL. A. (2000). Manual for the ASEBA preschool forms and profiles: An integrated system of multi-informant assessment. University of Vermont Department of Psychiatry.
2.
AllisonC.AuyeungB.Baron-CohenS. (2012). Toward brief “red flags” for autism screening: The short autism spectrum quotient and the short quantitative checklist in 1,000 cases and 3,000 controls. Journal of the American Academy of Child & Adolescent Psychiatry, 51, 202–212.
3.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing.
4.
BeauducelA.HerzbergP. Y. (2006). On the performance of maximum likelihood versus means and variance adjusted weighted least squares estimation in CFA. Structural Equation Modeling, 13, 186–203.
5.
BessahaM. L. (2017). Factor structure of the Kessler psychological distress scale (K6) among emerging adults. Research on Social Work Practice, 27, 616–624.
6.
CarletonR. N.ThibodeauM. A.TealeM. J.WelchP. G.AbramsM. P.RobinsonT.AsmundsonG. J. (2013). The Center for Epidemiologic Studies–Depression scale: A review with a theoretical and empirical examination of item content and factor structure. PLOS ONE, 8(3), Article e58067.
7.
DavidovE.SchmidtP.SchwartzS. H. (2008). Bringing values back in: The adequacy of the European Social Survey to measure values in 20 countries. Public Opinion Quarterly, 72, 420–445.
8.
DengN. (2011). Evaluating IRT- and CTT- based method of estimating classification consistency and accuracy indices from single administrations [Unpublished doctoral dissertation]. University of Massachusetts, Amherst.
9.
DiStefanoC. (2002). The impact of categorization with confirmatory factor analysis. Structural Equation Modeling, 9, 327–346.
10.
DrapeauA.Beaulieu-PrévostD.MarchandA.BoyerR.PrévilleM.KairouzS. (2010). A life-course and time perspective on the construct validity of psychological distress in women and men. Measurement invariance of the K6 across gender. BMC Medical Research Methodology, 10, 1–16.
11.
EdwardsM. C.HoutsC. R.CaiL. (2018). A diagnostic procedure to detect departures from local independence in item response theory models. Psychological Methods, 23, 138–149.
12.
GonzalezO.GeorgesonA. R.PelhamW. E.IIIFouladiR. T. (2021). Estimating classification consistency of screening measures and quantifying the impact of measurement bias. Psychological Assessment, 37, 596–609.
13.
GonzalezO.PelhamW. E.III. (2021). When does differential item functioning matter for screening? A method for empirical evaluation. Assessment, 28, 446–456.
14.
HaertelE. H. (2006). Reliability. In BrennanR. L. (Ed.), Educational measurement (4th ed.). Greenwood.
15.
JorgensenT. D.JohnsonA. R. (2022). How to derive expected values of structural equation model parameters when treating discrete data as continuous. Structural Equation Modeling: A Multidisciplinary Journal, 29(4), 639–650.
16.
KesslerR. C.BarkerP. R.ColpeL. J.EpsteinJ. F.GfroererJ. C.HiripiE.HowesM. J.NormandS.-L. T.ManderscheidR. W.WaltersE. E.ZaslavskyA. M. (2003). Screening for serious mental illness in the general population. Archives of General Psychiatry, 60, 184–189.
17.
LaiM. H.KwokO. M.YoonM.HsiaoY. Y. (2017). Understanding the impact of partial factorial invariance on selection accuracy: An R script. Structural Equation Modeling: A Multidisciplinary Journal, 24, 783–799.
18.
LathropQ. N. (2015). Practical issues in estimating classification accuracy and consistency with R Package cacIRT. Practical Assessment, Research & Evaluation, 20, Article 18.
19.
LathropQ. N.ChengY. (2013). Two approaches to estimation of classification accuracy rate under item response theory. Applied Psychological Measurement, 37, 226–241.
20.
LeeW. C. (2010). Classification consistency and accuracy for complex assessments using item response theory. Journal of Educational Measurement, 47, 1–17.
21.
LiC.-H. (2016). Confirmatory factor analysis with ordinal data: Comparing robust maximum likelihood and diagonally weighted least squares. Behavior Research Methods, 48, 936–949.
22.
LivingstonS. A.LewisC. (1995). Estimating the consistency and accuracy of classifications based on test scores. Journal of Educational Measurement, 32, 179–197.
23.
MeadeA. W.LautenschlagerG. J. (2004). A comparison of item response theory and confirmatory factor analytic methodologies for establishing measurement equivalence/invariance. Organizational Research Methods, 7, 361–388.
24.
MellenberghJ. G. (2017). Models for continuous responses. In van der LindenW. J. (Ed.), Handbook of item response theory, volume one: Models (pp. 153–166). Chapman and Hall.
25.
MillsapR. E.KwokO. M. (2004). Evaluating the impact of partial factorial invariance on selection in two populations. Psychological Methods, 9, 93–115.
26.
MuthénB.KaplanD. (1985). A comparison of some methodologies for the factor analysis of non-normal Likert variables. British Journal of Mathematical and Statistical Psychology, 38(2), 171–189.
27.
NyeC. D.BradburnJ.OlenickJ.BialkoC.DrasgowF. (2019). How big are my effects? Examining the magnitude of effect sizes in studies of measurement equivalence. Organizational Research Methods, 22, 678–709.
28.
PengC. Y. J.SubkoviakM. J. (1980). A note on Huynh’s normal approximation procedure for estimating criterion-referenced reliability. Journal of Educational Measurement, 17, 359–368.
29.
PepeM. S. (2003). The statistical evaluation of medical tests for classification and prediction. Oxford University Press.
30.
ReiseS. P.WallerN. G. (2009). Item response theory and clinical measurement. Annual Review of Clinical Psychology, 5, 27–48.
31.
RhemtullaM.Brosseau-LiardP. É.SavaleiV. (2012). When can categorical variables be treated as continuous? A comparison of robust continuous and categorical SEM estimation methods under suboptimal conditions. Psychological Methods, 17, 354–373.
32.
SamejimaF. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph Supplement, 34, 1–97.
33.
SunderlandM.HobbsM. J.AndersonT. M.AndrewsG. (2012). Psychological distress across the lifespan: Examining age-related item bias in the Kessler 6 Psychological Distress Scale. International Psychogeriatrics, 24, 231–242.
34.
ThissenD. (2000). Reliability and measurement precision. In WainerH. (Ed.), Computerized adaptive testing: A primer (2nd ed., pp. 159–183). Lawrence Erlbaum.
35.
ThissenD. (2017, January). Similar DIFs: Differential item functioning and factorial invariance for scales with seven (“plus or minus two”) response alternatives [Paper presentation]. 81st International Meeting of the Psychometric Society, Asheville, NC, United States.
36.
ThissenD.SteinbergL.PyszczynskiT.GreenbergJ. (1983). An item response theory for personality and attitude scales: Item analysis using restricted factor analysis. Applied Psychological Measurement, 7, 211–226.
37.
WirthR. J.EdwardsM. C. (2007). Item factor analysis: Current approaches and future directions. Psychological Methods, 12, 58–79.
