How to compute and report subscores for a test that was originally designed for reporting scores on a unidimensional scale has been a topic of interest in recent years. In the research reported here, we describe an application of multidimensional item response theory to identify a subscore structure in a test designed for reporting results using a unidimensional scale. This research also dealt with the problem of planned missing data due to low levels of item overlap among multiple test forms. Furthermore, we provided evidence for the generalizability of the multidimensional structure using multiple forms of the same test. We also compared the subscores from multiple groups to show the usefulness of the subscores. The research provides evidence that subscores can be identified and produced to provide useful information about different constructs for multiple examinee groups even though the test data were well fit by a unidimensional model.
AckermanT. A.GierlM. J.WalkerC. M. (2003). Using multidimensional item response theory to evaluate educational and psychological tests. Educational Measurement: Issues and Practice, 22(3), 37-51.
2.
AckermanT.ShuZ. (2009, April). Using confirmatory MIRT modeling to provide diagnostic information in large scale assessment. Paper presented at the meeting of the National Council of Measurement in Education, San Diego, CA.
3.
American Educational Research Association, American Psychological Association, National Council on Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.
4.
CarrollJ. B. (1993). Human cognitive abilities: A survey of factor analytic studies. New York, NY: Cambridge University Press.
5.
HabermanS. J. (2008a). When can subscores have value?Journal of Educational and Behavioral Statistics, 33, 204-229.
6.
HabermanS. J.SinharayS. (2010). Reporting of subscores using multidimensional item response theory. Psychometrika, 75, 209-227.
7.
HarrisD. J.HansonB. A. (1991, March). Methods of examining the usefulness of subscores. Paper presented at the meeting of the National Council on Measurement in Education, Chicago, IL.
8.
KimJ. L. (2001). Proximity measures and cluster analyses in multidimensional item response theory (Unpublished doctoral dissertation). Michigan State University, East Lansing.
9.
LedesmaR. D.Valero-MoraP. (2007). Determining the number of factors to retain in EFA: An easy-to-use computer program for carrying out parallel analysis. Practical Assessment, Research & Evaluation, 12, 1-11.
10.
MessickS. (1993). Validity. In LinnR. L. (Ed.), Educational measurement (3rd ed., pp. 13-103). Phoenix, AZ: Oryx Press.
MuthénL. K.MuthénB. O. (2005). Mplus user’s guide (Version 3). Los Angeles, CA: Muthén & Muthén.
13.
National Research Council. (2001). Knowing what students know: The science and design of educational assessment. Washington, DC: National Academies Press.
14.
Pearson Longman. (2010). The official guide to PTE: Pearson Test of English Academic. Hong Kong SAR: Pearson Longman Asia ELT.
15.
ReckaseM. D. (1997). The past and future of multidimensional item response theory. Applied Psychological Measurement, 21, 25-36.
16.
ReckaseM. D. (2009). Multidimensional item response theory. New York, NY: Springer.
17.
SinharayS. (2010). How often do subscores have added value? Results from operational and simulated data. Journal of Educational Measurement, 47, 150-174.
18.
SinharayS.HabermanS. J.PuhanG. (2007). Subscores based on classical test theory: To report or not to report. Educational Measurement: Issues and Practice, 26(4), 21-28.
19.
SinharayS.PuhanG.HabermanS. J. (2011). An NCME instructional module on subscores. Educational Measurement: Issues and Practice, 30(3), 29-40.
20.
StoneC. A.YeF.ZhuX.LaneS. (2010). Providing subscale scores for diagnostic information: A case study when the test is essentially unidimensional. Applied Measurement in Education, 23, 63-86.
21.
StoutW. (1987). A nonparametric approach for assessing latent trait unidimensionality. Psychometrika, 52, 589-617.
22.
von DavierM. (2008). A general diagnostic model applied to language testing data. British Journal of Mathematical and Statistical Psychology, 61, 287-307.
23.
WainerH.VeveaJ. L.CamachoF.ReeveB. B.RosaK.NelsonL.. . . ThissenD. (2001). Augmented scores—“borrowing strength” to compute scores based on small numbers of items. In ThissenD.WainerH. (Eds.), Test scoring (pp. 343-387). Mahwah, NJ: Lawrence Erlbaum.
24.
WangM. (1985). Fitting a unidimensional model to multidimensional item response data: the effect of latent space misspecification on the application of IRT (Research Report MW: 6-24-85). Iowa City, IA, University of Iowa.
25.
WangM. (1986). Fitting a unidimensional model to multidimensional item response data. Paper presented at the Office of Naval Research Contractors Meeting, Gatlinburg, TN.
26.
WendlerC. L.WalkerM. E. (2006). Practical issues in designing and maintaining multiple test forms for large-scale programs. In DowningS. M.HaladynaT. M. (Eds.), Handbook of test development (pp. 445-467). Mahwah, NJ: Lawrence Erlbaum.
27.
YaoL. H.BoughtonK. A. (2007). A multidimensional item response modeling approach for improving subscale proficiency estimation and classification. Applied Psychological Measurement, 31, 83-105.
28.
ZhangJ.StoutW. (1999). The theoretical DETECT index of dimensionality and its application to approximate simple structure. Psychometrika, 64, 213-249.
