Because of the requirement of reporting effect sizes and in the interest of measurement of change within the item response theory framework, their combination becomes a new issue. In the present study, repeated measures are decomposed as an initial ability and one or more modifiabilities (gain score) using a multidimensional Rasch model. The modifiability can be directly interpreted in terms of logit scale. The standardized mean modifiability is recommended for meta-analysis when test equating is not possible across studies. A simulation study was conducted to assess parameter recovery. It appeared that the point estimates were accurate whereas the error variances were underestimated. The bootstrap method was used and found appropriate for estimating the error variances. Implications and applications are illustrated through an empirical example.
Adams, R. J., & Wilson, M. R. (1996). Formulating the Rasch models as a mixed coefficients multinomial logit. In G. Engelhard & M. R. Wilson (Eds.), Objective measurement: Theory into practice (Vol. 3, pp. 143-166). Norwood, NJ: Ablex.
2.
Adams, R. J., Wilson, M. R., & Wang, W.-C. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21, 1-23.
3.
American Psychological Association. (1994). Publication manual of the American Psychological Association (4th ed.). Washington, DC: Author.
4.
American Psychological Association. (2001). Publication manual of the American Psychological Association (5th ed.). Washington, DC: Author.
5.
Andersen, E. B. (1985). Estimating latent correlations between repeated testings. Psychometrika, 50, 3-16.
6.
Andrich, D. (1978). A rating formulation for ordered response categories. Psychometrika, 43, 561-573.
7.
Andrich, D. (1988). Rasch models for measurement. Newbury Park, CA: Sage.
8.
Baugh, F. (2002). Correcting effect sizes for score reliability: A reminder that measurement and substantive issues are linked inextricably. Educational and Psychological Measurement, 62, 254-263.
9.
Béguin, A. A., & Glas, C. A. W. (2001). MCMC estimation and model-fit analysis of multidimensional IRT models. Psychometrika, 66, 541-562.
10.
Bereiter, C. (1963). Some persisting dilemmas in the measurement of change. In C. W. Harris (Ed.), Problems in measuring change. Madison: University of Wisconsin Press.
11.
Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinees ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
12.
Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46, 443-459.
13.
Bock, R. D., Gibbons, R., & Muraki, E. J. (1988). Full information item factor analysis. Applied Psychological Measurement, 12, 261-280.
14.
Bock, R. D., & Lieberman, M. (1970). Fitting a response model for n dichotomously scored items. Psychometrika, 35, 179-197.
15.
Bock, R. D., & Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6, 431-444.
16.
Bock, R. D., Muraki, E., & Pfeiffenberger, W. (1988). Item pool maintenance in the presence of item parameter drift. Journal of Educational Measurement, 25, 275-285.
17.
Bond, T. G., & Fox, C. M. (2001). Applying the Rasch model: Fundamental measurement in the human sciences. Mahwah, NJ: Lawrence Erlbaum.
18.
Capraro, R. M., & Capraro, M. M. (2002). Treatments of effect sizes and statistical significance tests in textbooks. Educational and Psychological Measurement, 62, 771-782.
19.
Cohen, J. (1969). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum.
20.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). New York: Academic Press.
21.
Collins, L. M., & Horn, J. L. (1991). Best methods for the analysis of change. Washington, DC: American Psychological Association.
22.
Cribbie, R. A., & Jamieson, J. (2000). Structural equation models and the regression bias for measuring correlates of change. Educational and Psychological Measurement, 60, 893-907.
23.
Cronbach, L. J., & Furby, L. (1970). How we should measure change or should we?Psychological Bulletin, 74, 68-80.
24.
DeMars, C. (2001). Group difference based on IRT scores: Does the model matter?Educational and Psychological Measurement, 61, 60-70.
25.
Dugard, P., & Todman, J. (1995). Analysis of pre-test post-test control group designs in educational research. Educational Psychology, 15, 181-198.
26.
Edwards, J. R. (1993). On the use of polynomial regression equations as an alternative to difference scores in organizational research. Academy of Management Journal, 36, 1577-1613.
27.
Edwards, J. R. (1995). Alternatives to difference scores as dependent variables in the study of congruence in organizational research. Organizational Behavior and Human Decision Processes, 64, 307-324.
28.
Efron, B. (1979). Bootstrap methods: Another look at the jack knife. Annals of Statistics, 7, 1-26.
29.
Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. New York: Chapman & Hall.
30.
Eliason, S. R. (1993). Maximum likelihood estimation: Logic and practice. Newbury Park, CA: Sage.
31.
Embretson, S. E. (1991). A multidimensional latent trait model for measuring learning and change. Psychometrika, 56, 495-515.
32.
Embretson, S. E. (1993). Psychometric models for learning and cognitive processes. In N. Frederiksen, R. J. Mislevy, & I. I. Bejar (Eds.), Test theory for a new generation of tests (pp. 125-150). Hillsdale, NJ: Lawrence Erlbaum.
33.
Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.
34.
Fischer, G. H. (1995). Derivation of the Rasch model. In G. H. Fischer & I. W. Molenaar (Eds.), Rasch models: Foundations, recent developments, and applications (pp. 15-38). New York: Springer-Verlag.
35.
Fischer, G. H. (2003). The precision of gain scores under an item response theory perspective: A comparison of asymptotic and exact conditional inference about change. Applied Psychological Measurement, 27, 3-26.
36.
Fischer, G. H., & Pazer, P. (1991). An extension of the rating scale model with an application to the measurement of change. Psychometrika, 56, 637-651.
37.
Fischer, G. H., & Ponocny, I. (1994). An extension of the partial credit model with an application to the measurement of change. Psychometrika, 59, 177-192.
38.
Fraser, C. (1988). NOHARM: A Fortran program for fitting unidimensional and multidimensional normal ogive models of latent trait theory [Computer software]. Armidale, New South Wales, Australia: University of New England, Centre for Behavioral Studies.
39.
Geenen, R., & van de Vijver, F. J. R. (1993). A simple test of the law of initial values. Psychophysiology, 30, 525-530.
40.
Hake, R. R. (1998). Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses. American Journal of Physics, 66, 64-74.
41.
Henson, R. K. (2001). Understanding internal consistency reliability estimates: A conceptual primer on coefficient alpha. Measurement and Evaluation in Counseling and Development, 34, 177-189.
42.
Holland, P. W., & Wainer, H. (Eds.). (1993). Differential item functioning. Hillsdale, NJ: Lawrence Erlbaum.
43.
Jamieson, J. (1994). Measurement of change and the law of initial values: A computer simulation study. Educational and Psychological Measurement, 55, 38-46.
44.
Lee, P. M. (1997). Bayesian statistics: An introduction (2nd ed.). New York: John Wiley.
45.
Lord, F. M. (1956). The measurement of growth. Educational and Psychological Measurement, 16, 421-437.
46.
Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.
47.
Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
48.
Malgady, R. G., & Colon-Malgady, G. (1991). Comparing the reliability of difference scores and residuals in analysis of covariance. Educational and Psychological Measurement, 51, 803-807.
49.
Masters, G. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149-174.
50.
Maxwell, S. E., & Delaney, H. D. (1990). Designing experiments and analyzing data: A model comparison perspective. Belmont, CA: Wadsworth.
51.
May, K., & Nicewander, W. A. (1998). Measuring change conventionally and adaptively. Educational and Psychological Measurement, 58, 882-897.
52.
McDonald, R. P. (1982). Linear versus nonlinear models in latent trait theory. Applied Psychological Measurement, 6, 379-396.
53.
McKinley, R. L., & Reckase, M. D. (1983). MAXLOG: A computer program for the estimation of the parameters of a multidimensional logistic model. Behavior Research Methods and Instrumentation, 15, 389-390.
54.
Mislevy, R. J., Beaton, A. E., Kaplan, B., & Sheehan, K. M. (1992). Estimating population characteristics from sparse matrix samples of item responses. Journal of Educational Measurement, 29, 133-161.
55.
Overall, J. E., & Tonidandel, S. (2002). Measuring change in controlled longitudinal studies. British Journal of Mathematical and Statistical Psychology, 55, 109-124.
56.
Perline, R., Wright, B. D., & Wainer, H. (1977). The Rasch model as additive conjoint measurement. Applied Psychological Measurement, 3, 237-255.
57.
Rasch, G. (1960). Probabilistic models for some intelligent and attainment tests. Copenhagen, Denmark: Institute of Educational Research.
58.
Reckase, M. D. (1985). The difficulty of test items that measure more than one dimension. Applied Psychological Measurement, 9, 401-412.
59.
Rogosa, D., & Willett, J. B. (1983). Demonstrating the reliability of the difference score. Journal of Educational Measurement, 20, 335-343.
60.
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph, No. 17.
61.
SAS Institute. (1999). SAS Online Doc (Version 8) [Computer software manual on CD-ROM]. Cary, NC: Author.
Shapiro, S. S., & Wilk, M. B. (1965). An analysis of variance test for normality (complete samples). Biometrika, 52, 591-611.
64.
Thompson, B., & Snyder, P. A. (1998). Statistical significance and reliability analyses in recent JCD research articles. Journal of Counseling and Development, 76, 431-436.
65.
Thompson, B., & Vacha-Haase, T. (2000). Psychometrics is data metrics: The test is not reliable. Educational and Psychological Measurement, 60, 174-195.
66.
Vacha-Haase, T., Ness, C., Nilsson, J., & Reetz, D. (1999). Practices regarding reporting of reliability coefficients: A review of three journals. Journal of Experimental Education, 67, 335-341.
67.
Wang, W.-C., & Chen, H.-C. (2004). The standardized mean difference within the framework of item response theory. Educational and Psychological Measurement, 64, 201-223.
68.
Wang, W.-C., Wilson, M. R., & Adams, R. J. (1998). Measuring individual differences in change with Rasch models. Journal of Outcome Measurement, 2, 240-265.
69.
Wilkinson, L., & the APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604.
70.
Williams, R. H., & Zimmerman, D. W. (1996). Are simple gain scores obsolete?Applied Psychological Measurement, 20, 59-69.
71.
Williams, R. H., & Zimmerman, D. W. (1999). Nonindependence of parameters of the validity and reliability of gain scores. Perceptual & Motor Skills, 88, 679-682.
72.
Wilson, D. T., Wood, R., & Gibbons, R. (1991). TESTFACT: Test scoring, item statistics, and item factor analysis [Computer software]. Mooresville, IN: Scientific Software.
73.
Wolfinger, R. D., & SAS Institute. (n.d.). Fitting nonlinear mixed models with the new NLMIXED procedure. Retrieved August 17, 2003, from http://support.sas.com/rnd/app/papers/nlmixedsugi.pdf.
74.
Wright, B. D., & Stone, M. H. (1979). Best test design. Chicago: MESA Press.
75.
Wu, C.-I. (1999). The etiology of adolescents substance abuse: A social learning model (Tech. Rep. No. DOH88-HR-621). Taipei, Taiwan: Academia Sinica.
76.
Wu, M., Adams, R. J., & Wilson, M. R. (1998). ACER ConQuest: Generalised item response modeling software [Computer software]. Camberwell, Victoria: Australian Council for Educational Research.
77.
Zimmerman, D. W., & Williams, R. H. (1982). Gain scores in research can be highly reliable. Journal of Educational Measurement, 19, 149-154.
78.
Zimmerman, D. W., & Williams, R. H. (1998). Reliability of gain scores under realistic assumptions about properties of pre-test and post-test scores. British Journal of Mathematical and Statistical Psychology, 51, 343-351.
