Reliability generalization (RG) is a measurement meta-analytic method used to explore the variability in score reliability estimates and to characterize the possible sources of this variance. This article briefly summarizes some RG considerations. Included is a description of how reliability confidence intervals might be portrayed graphically. The article includes tabulations across various RG studies, including how frequently authors (a) report score reliabilities for their own data, (b) conduct reliability induction, or (c) do not even mention reliability.
Get full access to this article
View all access options for this article.
References
1.
Barnes, L.L.B. , Harp, D., & Jung, W. S. (2002). Reliability generalization of scores on the Spielberger State-Trait Anxiety Inventory. Educational and Psychological Measurement, 62, 603-618.
2.
Beretvas, S. N. , Meyers, J. L., & Leite, W. L. (2002). A reliability generalization study of the Marlow-Crowne Social Desirability Scale. Educational and Psychological Measurement, 62, 570-589.
3.
Capraro, M. M. , Capraro, R. M., & Henson, R. K. (2001). Measurement error of scores on the Mathematics Anxiety Rating Scale across studies. Educational and Psychological Measurement, 61, 373-386.
4.
Capraro, R. M. , & Capraro, M. M. (2002). Myers-Briggs Type Indicator scorereliability across studies: A meta-analytic reliability generalization study. Educational and Psychological Measurement, 62, 590-602.
5.
Caruso, J. C. (2000). Reliability generalization of the NEO personality scales. Educational and Psychological Measurement, 60, 236-254.
6.
Caruso, J. C. , & Edwards, S. (2001). Reliability generalization of the Junior Eysenck Personality Questionnaire. Personality and Individual Differences, 31, 173-184.
7.
Caruso, J. C. , Witkiewitz, K., Belcourt-Dittloff, A., & Gottleib, J. (2001). Reliability of scores from the Eysenck Personality Questionnaire: A reliability generalization study. Educational and Psychological Measurement, 61, 675-689.
8.
Crocker, L. , & Algina, J. (1986). Introduction to classical and modern test theory. New York: Holt, Rinehart & Winston.
9.
Fan, X. , & Thompson, B. (2001). Confidence intervals about score reliability coefficients, please: An EPM guidelines editorial. Educational and Psychological Measurement, 61, 517-531.
10.
Hanson, W. E. , Curry, K. T., & Bandalos, D. L. (2002). Reliability generalization of Working Alliance Inventory scalescores. Educational and Psychological Measurement, 62, 659-673.
11.
Henson, R. K. (2001). Understanding internal consistency reliability estimates: A conceptual primer on coefficient alpha. Measurement and Evaluation in Counseling and Development, 34, 177-189.
12.
Henson, R. K. , & Hwang, D. (2002). Variability and prediction of measurement error in Kolb’s Learning Style Inventory scores: A reliability generalization study. Educational and Psychological Measurement, 62, 712-727.
13.
Henson, R. K. , Kogan, L. R., & Vacha-Haase, T. (2001). A reliability generalization study of the Teacher Efficacy Scale and related instruments. Educational and Psychological Measurement, 61, 404-420.7
14.
Henson, R. K. , & Thompson, B. (in press). Characterizing measurement error in scores across studies: Some recommendations for conducting “Reliability Generalization” (RG) studies. Measurement and Evaluation in Counseling and Development.
15.
Kieffer, K. M. , & Reese, R. J. (in press). A reliability generalization study of the Geriatric Depression Scale. Educational and Psychological Measurement.
16.
Lane, G. G. , White, A. E., & Henson, R. K. (2002). Expanding reliability generalization methods with KR-21 estimates: An RG study of the Coopersmith Self-Esteem Inventory. Educational and Psychological Measurement, 62, 685-711.
17.
Nilsson, J. E. , Schmidt, C. K., & Meek, W. D. (2002). Reliability generalization: An examination of the Career Decision-Making Self-Efficacy Scale. Educational and Psychological Measurement, 62, 647-658.
18.
Onwuegbuzie, A. J. , & Daniel, L. G. (2000, November). Reliability generalization: The important of considering sample specificity, confidence intervals, and subgroup differences. Paper presented at the annual meeting of the Mid-South Educational Research Association, Bowling Green, KY.
19.
Pedhazur, E. J. , & Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale, NJ: Lawrence Erlbaum.
20.
Reese, R. J. , Kieffer, K. M., & Briggs, B. K. (2002). A reliability generalization study of select measures of adult attachment style. Educational and Psychological Measurement, 62, 619-646.
21.
Reinhardt, B. (1996). Factors affecting coefficient alpha: A mini Monte Carlo study. In B. Thompson (Ed.), Advances in social science methodology (Vol. 4, pp. 3-20). Greenwich, CT: JAI Press.
22.
Shields, L. , & Caruso, J. C. (in press). Reliability Generalization of the Alcohol Use Disorders Identification Test. Educational and Psychological Measurement.
23.
Thompson, B. (1994). Guidelines for authors. Educational and Psychological Measurement, 54, 837-847.
24.
Thompson, B. (Ed.). (2002). Score reliability: Contemporary thinking on reliability issues. Thousand Oaks, CA: Sage.
25.
Thompson, B. , & Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement, 60, 174-195.
26.
Traub, R. E. (1994). Reliability for the social sciences. London: Sage.
27.
Vacha-Haase, T. (1998). Rliability Generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58, 6-20.
28.
Vacha-Haase, T. , Kogan, L. R., Tani, C. R., & Woodall, R. A. (2001). Reliability generalization: Exploring variation of reliability coefficients of MMPI clinical scales scores. Educational and Psychological Measurement, 61, 45-59.
29.
Vacha-Haase, T. , Kogan, L. R., & Thompson, B. (2000). Sample compositions and variabilities in published studies versus those in test manuals: Validity of score reliability inductions. Educational and Psychological Measurement, 60, 509-522.
30.
Vacha-Haase, T. , Ness, C., Nilsson, J., & Reetz, D. (1999). Practices regarding reporting of reliability coefficients: A review of three journals. Journal of Experimental Education, 67, 335-341.
31.
Vacha-Haase, T. , Tani, C. R., Kogan, L. R., Woodall, R. A., & Thompson, B. (2001). Reliability generalization: Exploring reliability variations on MMPI/MMPI-2 validity scale scores. Assessment, 8, 391-401.
32.
Viswesvaran, C. , & Ones, D. S. (2000). Measurement error in “Big Five Factors” personality assessment: Reliability generalization across studies and measures. Educational and Psychological Measurement, 60, 224-235.
33.
Wallace, K. A. , & Wheeler, A.J. (2002) Reliability generalization of the Life Satisfaction Index. Educational and Psychological Measurement, 62, 674-684.
34.
Whittington, D. (1998). How well do researchers report their measures? An evaluation of measurement in published educational research. Educational and Psychological Measurement, 58, 21-37.
35.
Wilkinson, L. , & American Psychological Association (APA) Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604. (Reprint available through the APA Home Page: http://www.apa.org/journals/amp/amp548594.html)
36.
Yin, P. , & Fan, X. (2000). Assessing the reliability of Beck Depression Inventory scores: Reliability generalization across studies. Educational and Psychological Measurement, 60, 201-223.
37.
Youngstrom, E. A. , & Green, K. W. (in press). Reliability generalization of self-report of emotions when using the Differential Emotions Scale. Educational and Psychological Measurement.
