As item response theory (IRT) becomes popular in educational and psychological testing, there is a need of reporting IRT-based effect size measures. In this study, we show how the standardized mean difference can be generalized into such a measure. A disattenuation procedure based on the IRT test reliability is proposed to correct the attenuation on the sampling variance due to measurement error. Simulations were conducted to investigate whether the direct generalization and the disattenuation procedure were appropriate. The independent variables were (a) item response model, (b) test length, (c) sample size, (d) effect size, and (e) test-person match. The dependent variable was the empirical effect size measure. The results indicated that the direct generalization of the standardized mean difference into the IRT context is applicable and the disattenuation procedure is appropriate. An empirical example about English assents for Taiwan students is given.
Get full access to this article
View all access options for this article.
References
1.
Adams, R. J., Wilson, M. R., & Wang, W.-C. (1997). The multidimensional random coefficients multinomial logit model. Applied Psychological Measurement, 21, 1-23.
2.
Algina, J., & Moulder, B. C. (2001). Sample sizes for confidence intervals on the increase in the squared multiple correlation coefficient. Educational and Psychological Measurement, 61, 633-649
3.
American Psychological Association. (1994). Publication manual of the American Psychological Association (4th ed.). Washington, DC: Author.
4.
American Psychological Association. (2001). Publication manual of the American Psychological Association (5th ed.). Washington, DC: Author.
5.
Andrich, D. (1978). A rating formulation for ordered response categories. Psychometrika, 43, 561-573.
6.
Baugh, F. (2002). Correcting effect sizes for score reliability: A reminder that measurement and substantive issues are linked inextricably. Educational and Psychological Measurement, 62, 254-263.
7.
Bird, K. D. (2002). Confidence intervals for effect sizes in analysis of variance. Educational and Psychological Measurement, 62, 197-226.
8.
Birnbaum, A. (1968). Some latent trait models and their use in inferring an examinee’s ability. In F. M. Lord & M. R. Novick (Eds.), Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
9.
Bock, R. D., & Aitkin, M. (1981). Marginal maximum likelihood estimation of item parameters: An application of the EM algorithm. Psychometrika, 46, 443-459.
10.
Bock, R. D., & Lieberman, M. (1970). Fitting a response model for n dichotomously scored items. Psychometrika, 35, 179-197.
11.
Bock, R. D., & Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 6, 431-444.
12.
Capraro, R. M., & Capraro, M. M. (2002). Treatments of effect sizes and statistical significance tests in textbooks. Educational and Psychological Measurement, 62, 771-782.
13.
Chen, H.-C. (2002). The influence of different accents and text types on listening performance. Paper presented at the Proceedings of the Eleventh International Symposium on English Teaching, Taipei, Taiwan.
14.
Cohen, J. (1969). Statistical power analysis for the behavioral sciences (1st ed.). Hillsdale, NJ: Erlbaum.
15.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
16.
Cumming, G., & Finch, S. (2001). A primer on the understanding, use, and calculation of confidence intervals that are based on central and noncentral distributions. Educational and Psychological Measurement, 61, 532-574.
17.
Efron, B. (1979). Bootstrap methods: Another look at the jack knife. Annals of Statistics, 7, 1-26.
18.
Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. New York: Chapman & Hall.
19.
Eliason, S. R. (1993). Maximum likelihood estimation: Logic and practice. Newbury Park, CA: Sage.
20.
Embretson, S. E., & Reise, S. P. (2000). Item response theory for psychologists. Mahwah, NJ: Erlbaum.
21.
Fidler, F., & Thompson, B. (2001). Computing correct confidence intervals for ANOVA fixed- and random-effects effect sizes. Educational and Psychological Measurement, 61, 575-604.
22.
Fowler, R. J. (1985). Point estimates and confidence intervals in measure of association. Psychological Bulletin, 98, 160-165.
23.
Glass, G. V., & Hopkins, K. D. (1996). Statistical methods in education and psychology (3rded.). Boston: Allyn & Bacon.
24.
Hedges, L. V., & Olkin, L. (1985). Statistical methods for meta-analysis. Orlando, FL: Academic Press.
25.
Henson, R. K. (2001). Understanding internal consistency reliability estimates: A conceptual primer on coefficient alpha. Measurement and Evaluation in Counseling and Development, 34, 177-189.
26.
Huberty, C. J. (2002). A history of effect size indices. Educational and Psychological Measurement, 62, 227-240.
27.
Huberty, C. J., & Lowman, L. L. (2000). Group overlap as a basis for effect size. Educational and Psychological Measurement, 60, 543-563.
28.
Kirk, R. E. (1996). Practical significance: A concept whose time has come. Education and Psychological Measurement, 56, 746-759.
29.
Kirk, R. E. (2001). Promoting good statistical practices: Some suggestions. Educational and Psychological Measurement, 61, 213-218.
30.
Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Erlbaum.
31.
Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
32.
Maxwell, S. E., & Delaney, H. D. (1990). Designing experiments and analyzing data: A model comparison perspective. Belmont, CA: Wadsworth.
33.
Meehl, P. E. (1997). Theory testing in psychology and physics: A methodological paradox. Philosophy of Science, 34, 103-115.
34.
Mislevy, R. J., Beaton, A. E., Kaplan, B., & Sheehan, K. M. (1992). Estimating population characteristics from sparse matrix samples of item responses. Journal of Educational Measurement, 29, 133-161.
35.
Muraki, E. (1992). A generalized partial credit model: Applications of an EM algorithm. Applied Psychological Measurement, 17, 351-363.
36.
Murphy, K. R. (1997). Editorial. Journal of Applied Psychology, 82, 3-5.
37.
Nickerson, R. S. (2000). Null hypothesis significance testing: A review of an old but continuing controversy. Psychological Methods, 5, 241-301.
38.
Rasch, G. (1960). Probabilistic models for some intelligent and attainment tests. Copenhagen: Institute of Educational Research.
39.
Robertsm, J. K., & Henson, R. K. (2002). Correction for bias in estimating effect sizes. Educational and Psychological Measurement, 62, 241-253.
40.
Rosenthal, R. (1984). Meta-analytic procedures for social research. Beverly Hills, CA: Sage.
41.
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometric Monograph, No. 17.
42.
Sanchez, M. J., & Marin-Martinez, F. (2001). Meta-analysis of 2 × 2 tables: Estimating a common risk difference. Educational and Psychological Measurement, 61, 249-276.
43.
Schmidt, F. L. (1996). Statistical significance testing and cumulative knowledge in psychology: Implications for the training of researchers. Psychological Methods, 1, 115-129.
44.
Smithson, S. (2001). Correct confidence intervals for various regression effect sizes and parameters: The importance of noncentral distributions in computing intervals. Educational and Psychological Measurement, 61, 605-632.
45.
Snyder, P., & Lawson, S. (1993). Effect size estimates. The Journal of Experimental Education, 61, 334-349.
46.
Spapiro, S. S., & Wilk, M. B. (1965). An analysis of variance test for normality (complete samples). Biometrika, 52, 591-611.
Thompson, B. (1994). Guidelines for authors. Educational and Psychological Measurement, 54, 837-847.
49.
Thompson, B. (1996). AERA editorial policies regarding statistical significance testing: Three suggested reforms. Educational Researchers, 25, 26-30.
50.
Thompson, B., & Snyder, P. A. (1998). Statistical significance and reliability analyses in recent JCD research articles. Journal of Counseling and Development, 76, 431-436.
51.
Thompson, B., & Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement, 60, 174-195.
52.
Vacha-Haase, T. (2001). Statistical significance should not be considered one of life’s guarantees: Effect sizes are needed. Educational and Psychological Measurement, 61, 219-224.
53.
Vacha-Haase, T., Ness, C., Nilsson, J., & Reetz, D. (1999). Practices regarding reporting of reliability coefficients: A review of three journals. Journal of Experimental Education, 67, 335-341.
54.
Wang, W.-C. (1999). Direct estimation of correlations among latent traits within IRT framework. Methods of Psychological Research Online, 4(2), 47-68.
55.
Wilkinson, L., & the APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604.
56.
Wright, B. D., & Masters, G. N. (1982). Rating scale analysis. Chicago: MESA Press.
57.
Wu, M., Adams, R. J., & Wilson. M. R. (1998). ACER ConQuest: Generalized item response modeling software [Computer software]. Camberwell, Victoria: Australian Council for Educational Research.
58.
Wu, M.-C. (2000). A study of the effects of paralingustic factors on EFL listening performance. Unpublished master’s thesis, National Kaohsiung Normal University, Taiwan.
59.
Zimowski, M. F., Muraki, E., Mislevy, R. J., & Bock, R. D. (1996). BILOG-MG: Multiple-group IRT analysis and test maintenance for binary items [Computer software]. Chicago: Scientific Software International.
