A Comparison of Content-Balancing Procedures for Estimating Multiple Clinical Domains in Computerized Adaptive Testing: Relative Precision,Validity,and Detection of Persons With Misfitting Responses
Restricted accessOtherFirst published online September, 2010
A Comparison of Content-Balancing Procedures for Estimating Multiple Clinical Domains in Computerized Adaptive Testing: Relative Precision,Validity,and Detection of Persons With Misfitting Responses
This simulation study sought to compare four different computerized adaptive testing (CAT) content-balancing procedures designed for use in a multidimensional assessment with respect to measurement precision, symptom severity classification, validity of clinical diagnostic recommendations, and sensitivity to atypical responding. The four content-balancing procedures were (a) no content balancing, (b) screener-based, (c) mixed (screener plus content balancing), and (d) full content balancing. In full content balancing and in mixed content balancing following administration of the screener items, item selection was based on (a) whether the target number of items for the item’s subscale was reached and (b) the item’s information function. Mixed and full content balancing provided the best representation of items from each of the main subscales of the Internal Mental Distress Scale. These procedures also resulted in higher CAT to full-scale correlations for the Trauma and Homicidal/Suicidal Thought subscales and improved detection of atypical responding.
American Psychiatric Association. (2000). Diagnostic and statistical manual of mental disorders (DSM-IV-TR) (4th ed., text rev.). Washington, DC: Author.
2.
Bond, T.G., & Fox, C.M. ( 2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd ed.). Mahwah, NJ: Lawrence Erlbaum.
3.
Briggs, D.C., & Wilson, M. ( 2003). An introduction to multidimensional measurement using Rasch models. Journal of Applied Measurement, 4, 87-100.
4.
Conrad, K.J., Bezruczko, N., Park, H., Chan, Y.F., Riley, B.B., Diamond, G., et al. (2008). Atypical suicide risk using Rasch person fit statistics to improve screening and understanding. Manuscript in preparation.
5.
Conrad, K.J., Conrad, K.M., Dennis, M.L., & Riley, B.B. ( 2008). GAIN working papers: Rasch analysis of the Internal Mental Distress Scale. Retrieved from http://www.chestnut.org/li/gain/psychometricreports/gainworkingpapersraschanalysisoftheinternalmentaldistressscale.pdf
6.
Dennis, M.L., Chan, Y.-F., & Funk, R. ( 2006). Development and validation of the GAIN Short Screener (GSS) for internalizing, externalizing, and substance use disorders and crime/violence problems among adolescents and adults. American Journal on Addictions, 15, 80-91.
7.
Dennis, M.L., Titus, J.C., White, M.K., Unsicker, J., & Hodgkins, D. ( 2003). Global appraisal of individual needs: Administration guide for the GAIN and related measures. Bloomington, IL : Chestnut Health Systems.
Dimitrov, D.M., & Smith, R.M. ( 2006). Adjusted Rasch person fit statistics. Journal of Applied Measurement, 7, 170-183.
10.
Fayers, P.M. ( 2007). Applying item response theory and computer adaptive testing: The challenges for health outcomes assessment. Quality of Life Research, 16(Suppl. 1), 187-194.
11.
Gardner, W., Kelleher, K.J., & Pajer, K.A. ( 2002). Multidimensional adaptive testing for mental health problems in primary care. Medical Care, 40, 812-823.
12.
Gardner, W., Shear, K., Kelleher, K.J., Pajer, K.A., Mammen, O., Buysse, D., et al. (2004). Computerized adaptive measurement of depression: A simulation study. BMC Psychiatry, 4, 13.
13.
Gibbons, R.D., Weiss, D.J., Kupfer, D.J., Frank, E., Fagiolini, A., Grochocinski, V.J., et al. (2008). Using computerized adaptive testing to reduce the burden of mental health assessment. Psychiatric Services, 59, 361-368.
14.
Haley, S.M., Fragala-Pinkham, M., & Ni, P. ( 2006). Sensitivity of a computer adaptive assessment for measuring functional mobility changes in children enrolled in a community fitness programme . Clinical Rehabilitation, 20, 616-622.
15.
Handel, R.W., Ben-Porath, Y.S., & Watt, M. ( 1999). Computerized adaptive assessment with the MMPI-2 in a clinical setting. Psychological Assessment, 11, 369-380.
16.
Kane, M.T., & Brennan, R.L. ( 1980). Agreement coefficients as indices of dependability for domain referenced tests. Applied Psychological Measurement , 4, 105-126.
17.
Karabatsos, G. ( 2003). Comparing the aberrant response detection performance of thirty-six person fit statistics. Applied Measurement in Education, 16, 277-298.
18.
King, D.W., King, L.A., Fairbank, J.A., Schlenger, W.E., & Surface, C.R. ( 1993). Enhancing the precision of the Mississippi Scale for Combat-Related Posttraumatic Stress Disorder: An application of item response theory. Psychological Assessment, 5, 457-471.
19.
Li, M.F., & Olejnik, S. ( 1997). The power of Rasch person-fit statistics in detecting unusual response patterns. Applied Psychological Measurement , 21, 215-231.
20.
Linacre, J.M. ( 2006). Winsteps Rasch measurement computer program (Version X.X) [Software]. Available from http://www.winsteps.com/
21.
Lipman, R.S., Covi, L., & Shapiro, A.K. ( 1979). The Hopkins Symptom Checklist (HSCL): Factors derived from the HSCL-90. Journal of Affective Disorders, 1, 9-24.
22.
Luecht, R.M. ( 1996). Multidimensional computerized adaptive testing in a certification or licensure context. Applied Psychological Measurement, 20, 389-404.
23.
Luecht, R.M., Champlain, A.D., & Nungester, R.J. (1998). Maintaining content validity in computerized adaptive testing. Advances in Health Sciences Education, 3, 29-41.
24.
Meijer, R.R., Molenaar, I.W., & Sijtsma, K. ( 1994). Item, test and group characteristics and their influence on nonparametric appropriateness measurement. Applied Psychological Measurement, 18, 111-120.
25.
Meijer, R.R., & Sijtsma, K. ( 2001). Methodology review: Evaluating person fit. Applied Psychological Measurement, 25, 107-135.
26.
Molenaar, I.W., & Hoijink, H. ( 1990). The many null distributions of person fit indices. Psychometrika, 55, 75-106.
27.
Molenaar, I.W., & Hoijink, H. ( 1996). Person fit and the Rasch model, with an application of knowledge of logical quantors. Applied Measurement in Education , 9, 27-45.
28.
Nering, M.L. ( 1995). The distribution of person fit using true and estimated person parameters. Applied Psychological Measurement, 19, 121-129.
29.
Petersen, M.A., Groenvold, M., Aaronson, N., Fayers, P., Sprangers, M., & Bjorner, J.B. ( 2006). Multidimensional computerized adaptive testing of the EORTC QLQ-C30: Basic developments and evaluations. Quality of Life Research, 15, 315-329.
30.
Reise, S.P., & Due, A.M. ( 1991). The influence of test characteristics on the detection of aberrant response patterns. Applied Psychological Measurement , 15, 217-226.
31.
Riley, B.B., Conrad, K.J., Bezruczko, N., & Dennis, M.L. ( 2007). Relative precision, efficiency and construct validity of different starting and stopping rules for a computerized adaptive test: The GAIN Substance Problem Scale. Journal of Applied Measurement , 8, 48-64.
32.
Sacks, S., Melnick, G., Coen, C., Banks, S., Friedmann, P.D., Grella, C., et al. (2007). CJDATS Co-occur ring Disorders Screening Instrument (CODSI) for mental disorders (MD): A validation study. Criminal Justice & Behavior, 34, 1198-1215.
Segall, D.O. ( 2001). General ability measurement: An application of multidimensional item response theory. Psychometrika, 66, 79-97.
35.
Smith, R.M., Schumacker, R.E., & Bush, M.J. ( 1998). Using item mean squares to evaluate fit to the Rasch model. Journal of Outcome Measurement, 2, 66-78.
36.
Tseng, F.-L. ( 2001). Multidimensional adaptive testing using the weighted likelihood estimation. Dissertation Abstracts International Section A: Humanities & Social Sciences, 61(12-A), 4746.
37.
van der Linden, W.J. ( 1999). Multidimensional adaptive testing with a minimum error-variance criterion. Journal of Educational and Behavioral Statistics , 24, 398-412.
38.
van Krimpen-Stoop, E.M., & Meijer, R.R. ( 1999). The null distribution of person-fit statistics for conventional and adaptive tests. Applied Psychological Measurement, 23, 327-345.
39.
van Krimpen-Stoop, E.M., & Meijer, R.R. ( 2000). Detecting person-misfit in adaptive testing using statistical process control techniques. In W. J. van der Linden & C. A. W. Glas (Eds.), Computer adaptive testing: Theory and practice (pp. 1-25). Boston : Kluwer Academic.
40.
van Krimpen-Stoop, E.M., & Meijer, R.R. ( 2001). CUSUM-based person-fit statistics for adaptive testing . Journal of Educational and Behavioral Statistics, 26, 199-218.
