The literature showing that subscores fail to add value is vast; yet despite their typical redundancy and the frequent presence of substantial statistical errors, many stakeholders remain convinced of their necessity. This article describes a method for identifying and reporting unexpectedly high or low subscores by comparing each examinee’s observed subscore with a discrete probability distribution of subscores conditional on the examinee’s overall ability. The proposed approach turns out to be somewhat conservative due to the nature of subscores as finite sums of item scores associated with a subdomain. Thus, the method may be a compromise that satisfies score users by reporting subscore information as well as psychometricians by limiting misinterpretation, at most, to the rates of Type I and Type II error.
AlbaneseM. A. (2014). The testing column: Differences in subject area subscores on the MBE and other illusions. The Bar Examiner, 83(2), 26–31.
3.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (2014). Standards for educational and psychological testing. American Educational Research Association.
4.
AndersenE. B. (1972). The numerical solution of a set of conditional estimation equations. The Journal of the Royal Statistical Society, 34(Series B), 42–54.
5.
BelovD. I. (2013). Detection of test collusion via Kullback–Leibler divergence. Journal of Educational Measurement, 50, 141–163.
6.
BiscarriW.ZhaoS. D.BrunnerR. J. (2018). A simple and fast method for computing the Poisson binomial distribution function. http://doi.org/10.1016/j.csda.2018.01.007
7.
BrennanR. L. (2012). Utility indexes for decisions about subscores (CASMA Research Report 33). Center for Advanced Studies in Measurement and Assessment.
8.
ClauserA. L.RickF. (2016). Evaluating score report prototypes for a licensure examination [Paper presentation]. Washington, DC: American Educational Research Association Annual Meeting.
9.
EvansD. L.LeemisL. M. (2004). Algorithms for computing the distributions of sums of discrete random variables. Mathematical and Computer Modelling, 40(13), 1429–1452.
10.
FeinbergR. A.JurichD. P. (2017). Guidelines for interpreting and reporting subscores. Educational Measurement: Issues and Practice, 36(1), 5–13.
11.
FeinbergR. A.RubrightJ. D. (2016). Conducting simulations in psychometrics. Educational Measurement: Issues and Practice, 35(2), 36–49.
12.
FeinbergR. A.WainerH. (2014). When can we improve subscores by making them shorter? The case against subscores with overlapping items. Educational Measurement: Issues and Practice, 33(3), 47–54.
13.
GonzalezJ.WibergM.von DavierA. A. (2016). A note on the Poisson binomial distribution in item response theory. Applied Psychological Measurement, 40(4), 302–310.
14.
GoodP. (2005). Permutation, parametric and bootstrap tests of hypotheses (3rd ed.). Springer.
GustafssonJ.-E. (1980). A solution of the conditional estimation problem for long tests in the Rasch model for dichotomous items. Educational and Psychological Measurement, 40, 327–385.
17.
HabermanS. J. (2008). When can subscores have value?Journal of Educational and Behavioral Statistics, 33(2), 204–229.
18.
HensonB. A. (1994). Note: Extension of Lord–Wingersky algorithm to computing test score distributions for polytomous items. Retrieved February 1, 1994, fromhttp://www.b-a-h.com/papers/note9401.html
19.
JiangZ.RaymondM. (2018). The use of multivariate generalizability theory to evaluate the quality of subscores. Applied Psychological Measurement, 42(8), 595–612.
20.
LordF. M.NovickM. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
21.
LordF. M.WingerskyM. (1984). Comparison of IRT true-score and equipercentile observed-score equatings. Applied Psychological Measurement, 8, 453–461.
22.
MarshH. W. (1994). Sport motivation orientations: Beware of jingle-jangle fallacies. Journal of Sport & Exercise Psychology, 16(4), 365–380.
23.
MislevyR. J.SteinbergL. S.AlmondR. G. (2003). On the structure of educational assessments. Measurement: Interdisciplinary Research and Perspectives, 1(1), 3–67.
24.
OrlandoM.ThissenD. (2003). Further investigation of the performance of S-X2: An item fit index for use with dichotomous item response theory models. Applied Psychological Measurement, 27(4), 289–298.
PuhanG.SinharayS.HabermanS. J.LarkinK. (2010). The utility of augmented subscores in a licensure exam: An evaluation of methods using empirical data. Applied Measurement in Education, 23, 266–285.
28.
RaschG. (1966). An individualistic approach to item analysis. In LazarsfeldP. F.HenryN. W. (Eds.), Readings in mathematical social science (pp. 89–107). The MIT Press.
29.
ReckaseM. D. (2007). Multidimensional item response theory. In RaoC. R.SinharayS. (Eds.), Handbook of statistics (Vol. 26, pp. 607–642). Amsterdam, The Netherlands: North-Holland.
30.
RickF.ClauserA. L. (2016). What score report features promote accurate remediation? Insights from cognitive interviews [Paper presentation]. Annual Meeting of the National Council on Measurement in Education (NCME), Washington, DC, United States.
31.
RijmenF.JeonM.von DavierM.Rabe-HeskethS. (2014). A third-order item response theory model for modeling the effects of domains and subdomains in large-scale educational assessment surveys. Journal of Educational and Behavioral Statistics, 38, 32–60.
32.
RizopoulosD. (2006). ltm: An R package for latent variable modelling and item response theory analyses. Journal of Statistical Software, 17(5), 1–25.
33.
SinharayS. (2010). How often do subscores have added value? Results from operational and simulated data. Journal of Educational Measurement, 47(2), 150–174.
34.
SinharayS. (2017). Detection of item preknowledge using likelihood ratio test and score test. Journal of Educational and Behavioral Statistics, 42, 46–68.
35.
SinharayS.HabermanS. J.WainerH. (2011). Do adjusted subscores lack validity? Don’t blame the messenger. Educational and Psychological Measurement, 71(5), 789–797.
36.
ThissenD.PommerichM.BilleaudK.WilliamsV. S. L. (1995). Item response theory for scores on tests including polytomous items with ordered responses. Applied Psychological Measurement, 19(1), 39–49.
37.
ThissenD.WainerH. (2001). An overview of test scoring. In ThissenD.WainerH. (Eds.), Test scoring (pp. 1–19). Lawrence Erlbaum.
38.
VerhelstN. D. (2012). Profile analysis: A closer look at the PISA 2000 reading data. Scandinavian Journal of Educational Research, 56(3), 315–332.
39.
von DavierM. (2010). Why sum scores may not tell us all about test takers. In Wang, Leighl. (Ed.), Special issue on Quantitative research methodology. Newborn and Infant Nursing Reviews, 10(1), 27–36.
40.
von DavierM. (2016). The Rasch model. In van der LindenW. (Ed.), Handbook of item response theory (Chapter 3, Vol. 1, 2nd ed., pp. 31–48). CRC Press.
41.
von DavierM.RostJ. (1995). Polytomous mixed Rasch models. In FischerG. H.MolenaarI. W. (Eds.), Rasch models—Foundations, recent developments and applications (pp. 371–379). Springer.
