The present study was conducted to characterize (a) the features of the thousands of primary reports synthesized in 47 reliability generalization (RG) measurement meta-analysis studies and (b) typical methodological practice within the RG literature to date. With respect to the treatment of score reliability in the literature, in an astounding 54.6% of the 12,994 primary reports authors did not even mention reliability! Furthermore, in 15.7% of the primary reports authors did mention score reliability, but merely inducted previously reported values as if they applied to their data. Clearly, the admonitions of Wilkinson and the APA Task Force (1999) have yet to have their desired impacts with respect to reporting reliability estimates for one’s own data.
*Bachner, Y.G., & O’Rourke, N. (2007). Reliability generalization of responses by care providers to the Zarit Burden Interview. Aging & Mental Health, 11, 678-685. doi:10.1080/ 13607860701529965
2.
*Barnes, L.L.B., Harp, D., & Jung, W.S. ( 2002). Reliability generalization of scores on the Spielberger State-Trait Anxiety Inventory. Educational and Psychological Measurement, 62, 603-618. doi:10.1177/0013164402062004005
3.
Bem, S.L. ( 1981). Bem Sex-Role Inventory: Professional manual. Palo Alto, CA: Consulting Psychologists Press.
4.
*Beretvas, S.N., Meyers, J.L., & Leite, W.L. ( 2002). A reliability generalization study of the Marlowe- Crowne Social Desirability Scale. Educational and Psychological Measurement , 62, 570-589. doi:10.1177/0013164402062004003
5.
*Beretvas, S.N., Suizzo, M.-A., Durham, J.A., & Yarnell, L.M. ( 2008). A reliability generalization study of scores on Rotter’s and Nowicki-Strickland’s locus of control scales. Educational and Psychological Measurement, 68, 97-119. doi:10.1177/0013164407301529
*Campbell, J.S., Pulos, S., Hogan, M., & Murry, F. ( 2005). Reliability generalization of the Psychopathy Checklist Applied in youthful samples. Educational and Psychological Measurement, 65, 639-656. doi:10.1177/0013164405275666
8.
*Capraro, R.M., & Capraro, M.M. ( 2002). Myers-Briggs Type Indicator score reliability across studies: A meta-analytic reliability generalization study. Educational and Psychological Measurement, 62, 590-602. doi:10.1177/ 0013164402062004004
9.
*Capraro, M.M., Capraro, R.M., & Henson, R.K. ( 2001). Measurement error of scores on the Mathematics Anxiety Rating Scale across studies. Educational and Psychological Measurement, 61, 373-386. doi:10.1177/00131640 121971266
10.
Capraro, R.M., & Thompson, B. ( 2008). The educational researcher defined: What will future researchers be trained to do? Journal of Educational Research , 101, 247-253. doi:10.3200/ JOER.101.4.247-253
11.
*Caruso, J.C. ( 2000). Reliability generalization of the NEO Personality Scales . Educational and Psychological Measurement, 60, 236-254. doi:10.1177/00131640021970484
12.
*Caruso, J.C., & Edwards, S. ( 2001). Reliability generalization of the Junior Eysenck Personality Questionnaire. Personality and Individual Differences, 31, 173-184. doi:10.1016/S091-8869(00)00126-4
13.
*Caruso, J.C., Witkiewitz, K., Belcourt-Dittloff, A., & Gottlieb, J.D. (2001). Reliability of scores from the Eysenck Personality Questionnaire: A reliability generalization study . Educational and Psychological Measurement, 61, 675-689. doi:10.1177/00131640121971437
14.
Cohen, J. ( 1968). Multiple regression as a general data-analytic system . Psychological Bulletin, 70, 426-433. doi:10.1037/h0026714
15.
Dandan, G., & Houcan, Z. ( 2004). A redefinition of reliability and the study of reliability generalization. Psychological Science ( China), 27, 445-448.
16.
*Deditius-Island, H.K., & Caruso, J.C. ( 2002). An examination of the reliability of scores from Zuckerman’s Sensation Seeking Scales, Form V. Educational and Psychological Measurement, 62, 728-734. doi:10.1177/0013164402062004012
17.
Dimitrov, D.M. ( 2002). Reliability: Arguments for multiple perspectives and potential problems with generalizability across studies. Educational and Psychological Measurement, 62, 783-801. doi:10.1177/001316402236878
18.
*Dunn, T.W., Smith, T.B., & Montoya, J.A. ( 2006). Multicultural competency instrumentation: A review and analysis of reliability generalization. Journal of Counseling & Development, 84, 471-482.
19.
*Graham, J.M., & Christiansen, K. (2009). The reliability of romantic love: A reliability generalization meta-analysis. Personal Relationships, 16, 49-66. doi:10.1111/j.1475-6811.2009.01209.x
20.
*Graham, J.M., Liu, Y.J., & Jeziorski, J.L. (2006). The Dyadic Adjustment Scale: A reliability generalization meta-analysis. Journal of Marriage and Family, 68, 701-717. doi:10.1111/ j.1741-3737.2006.00284.x
21.
Gronlund, N.E., & Linn, R.L. ( 1990). Measurement and evaluation in teaching (6th ed.). New York, NY: Macmillan.
22.
*Hanson, W.E., Curry, K.T., & Bandalos, D.L. (2002). Reliability generalization of Working Alliance Inventory scale scores. Educational and Psychological Measurement, 62, 659-673. doi:10.1177/0013164402062004008
23.
*Hellman, C.M., Fuqua, D.R., & Worley, J. ( 2006). A reliability generalization study on the Survey of Perceived Organizational Support: The effects of mean age and number of items on score reliability. Educational and Psychological Measurement, 66, 631-642. doi:10.1177/ 0013164406288158
24.
*Hellman, C.M., Muilenburg-Trevino, E.M., & Worley, J.A. ( 2008). The belief in a just world: An examination of reliability estimates across three measures. Journal of Personality Assessment , 90, 399-401. doi:10.1080/ 00223890802108238
25.
*Henson, R.K., & Hwang, D.-Y. ( 2002). Variability and prediction of measurement error in a Kolb’s Learning Style Inventory Scores: A reliability generalization study. Educational and Psychological Measurement, 62, 712-727. doi:10.1177/ 0013164402062004011
26.
*Henson, R.K., Kogan, L.R., & Vacha-Haase, T. (2001). A reliability generalization study of the Teacher Efficacy Scale and related instruments. Educational and Psychological Measurement, 61, 404-420. doi:10.1177/00131640121971284
27.
Henson, R.K., & Thompson, B. ( 2002). Characterizing measurement error in scores across studies: Some recommendations for conducting "reliability generalization" (RG) studies . Measurement and Evaluation in Counseling and Development , 35, 113-127.
28.
Hogan, T.P., Benjamin, A., & Brezinski, K.L. (2000). Reliability methods: A note on the frequency of use of various types. Educational and Psychological Measurement, 60, 523-531.
29.
Hunter, J.E., & Schmidt, F.L. ( 1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
30.
*Huynh, Q.-L., Howell, R.T., & Benet-Martinez, V. (2009). Reliability of bidimensional acculturation scores: A meta-analysis. Journal of Cross-Cultural Psychology, 40, 256-274. doi:10.1177/0022022108328919
31.
*Kieffer, K.M., Cronin, C., & Fister, M.C. ( 2004). Exploring variability and sources of measurement error in Alcohol Expectancy Questionnaire reliability coefficients: A meta-analytic reliability generalization study. Journal of Studies on Alcohol , 65, 663-671.
32.
*Kieffer, K.M., & Reese, R.J. ( 2002). A reliability generalization study of the Geriatric Depression Scale. Educational and Psychological Measurement, 62, 969-994. doi:10.1177/0013 164402238085
33.
Knapp, T.R. ( 1978). Canonical correlation analysis: A general parametric significance testing system. Psychological Bulletin, 85, 410-416. doi:10.1037//0033-2909.85.2.410
34.
*Lane, G.G., White, A.E., & Henson, R.K. ( 2002). Expanding reliability generalization methods with KR-21 estimates: An RG study of the Coopersmith Self-Esteem Inventory. Educational and Psychological Measurement, 62, 685-711. doi:10.1177/0013164402062004010
35.
*Leach, L.F., Henson, R.K., Odom, L.R., & Cagle, L.S. ( 2006). A reliability generalization study of the Self-Description Questionnaire. Educational and Psychological Measurement, 66, 285-304. doi:10.1177/0013164405284030
36.
*Li, A., & Bagger, J. ( 2007). The Balanced Inventory of Desirable Responding (BIDR): A reliability generalization study. Educational and Psychological Measurement, 67, 525-544. doi:10.1177/001316440292087
37.
*Lopez-Pina, J.A., Sanchez-Meca, J., & Rosa-Alcazar, A.I. (2009). The Hamilton Rating Scale for Depression: A meta-analytic reliability generalization study. International Journal of Clinical and Health Psychology, 9, 143-159.
38.
McCrae, R.R., Kurtz, J.E., Yamagata, S., & Terracciano, A. (2011). Internal consistency, retest reliability, and their implications for personality scale validity. Personality and Social Psychology Review, 15, 28-50. doi:10.1177/1088868310366253
39.
*Miller, B.K., & Byrne, Z.S. ( 2009). Perceptions of organizational politics: A demonstration of the reliability generalization technique. Journal of Managerial Issues, 21, 280-300.
40.
*Miller, C.S., Shields, A.L., Campfield, D., Wallace, K.A., & Weiss, R.D. ( 2007). Substance use scales of the Minnesota Multiphasic Personality Inventory: An exploration of score reliability via meta-analysis. Educational and Psychological Measurement, 67, 1052-1065. doi:10.1177/0013164406299130
41.
*Miller, C.S., Woodson, J., Howell, R.T., & Shields, A.L. ( 2009). SASSI: Assessing the reliability of scores produced by the Substance Abuse Subtle Screening Inventory. Substance Use & Misuse, 44, 1090-1100.
42.
*Mji, A., & Alkhateeb, H.M. (2005). Combining reliability coefficients: Toward reliability generalization of the Conceptions of Mathematics Questionnaire . Psychological Reports, 96, 627-634. doi:10.2466/pr0.96.3.627-634
43.
*Nilsson, J.E., Schmidt, C.K., & Meek, W.D. ( 2002). Reliability generalization: An examination of the Career Decision-Making Self-Efficacy Scale. Educational and Psychological Measurement, 62, 647-658. doi:10.1177/ 0013164402062004007
44.
*O’Rourke, N. ( 2004). Reliability generalization of responses by care providers to the Center for Epidemiologic Studies-Depression Scale. Educational and Psychological Measurement, 64, 973-990. doi:10.1177/0013164404268668
45.
Pedhazur, E.J., & Schmelkin, L.P. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale, NJ: Erlbaum.
46.
*Reese, R.J., Kieffer, K.M., & Briggs, B.K. ( 2002). A reliability generalization study of select measures of adult attachment style. Educational and Psychological Measurement , 62, 619-646. doi:10.1177/0013164402062004006
47.
Reinhardt, B. ( 1996). Factors affecting coefficient alpha: A mini Monte Carlo study. In B. Thompson (Ed.), Advances in social science methodology (Vol. 4, pp. 3-20). Greenwich, CT: JAI Press.
48.
*Rexrode, K.R., Petersen, S., & O’Toole, S. (2008). The Ways of Coping Scale: A reliability generalization study. Educational and Psychological Measurement, 68, 262-280. doi:10.1177/ 0013164407310128
49.
Rodriguez, M.C., & Maeda, Y. ( 2006). Meta-analysis of coefficient alpha. Psychological Methods, 11, 306-322. doi:10.1037/1082-989X.11.3.306
50.
*Ross, M.E., Blackburn, M., & Forbes, S. ( 2005). Reliability generalization of the Patterns of Adaptive Learning Survey Goal Orientation Scales. Educational and Psychological Measurement, 65, 451-464. doi:10.1177/0013164404272496
51.
*Rouse, S.V. ( 2007). Using reliability generalization methods to explore measurement error: An illustration using the MMPI-2 PSY-5 Scales. Journal of Personality Assessment, 88, 264-275.
52.
*Ryngala, D.J., Shields, A.L., & Caruso, J.C. ( 2005). Reliability generalization of the Revised Children’s Manifest Anxiety Scale. Educational and Psychological Measurement , 65, 259-271. doi:10.1177/0013164404272495
53.
Schmidt, F.L., & Hunter, J.E. ( 1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62, 529-540. doi:10.1037//0021-9010.62.5.529
54.
*Shields, A.L., & Caruso, J.C. ( 2003). Reliability generalization of the Alcohol Use Disorders Identification Test. Educational and Psychological Measurement , 63, 404-413. doi:10.1177/0013164403063003004
55.
*Shields, A.L., & Caruso, J.C. ( 2004). A reliability induction and reliability generalization study of the CAGE Questionnaire. Educational and Psychological Measurement, 64, 254-270. doi:10.1177/0013164403261814
56.
Thompson, B. ( 1992). Two and one-half decades of leadership in measurement and evaluation. Journal of Counseling and Development, 70, 434-438.
57.
Thompson, B. ( 1994). Guidelines for authors. Educational and Psychological Measurement, 54, 837-847.
58.
Thompson, B. (Ed.). (2003). Score reliability: Contemporary thinking on reliability issues. Thousand Oaks, CA: Sage.
59.
*Thompson, B., & Cook, C. ( 2002). Stability of the reliability of LibQUAL + TM scores: A reliability generalization meta-analysis study. Educational and Psychological Measurement, 62, 735-743. doi:10.1177/0013164402062004013
60.
Thompson, B., & Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement , 60, 174-195. doi:10.1177/00131640021970448
61.
Urbina, S. ( 2004). Essentials of psychological testing. Hoboken, NJ: John Wiley.
62.
Vacha-Haase, T. ( 1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58, 6-20. doi:10.1177/00131640121971059
63.
Vacha-Haase, T., Henson, R.K., & Caruso, J.C. ( 2002). Reliability generalization: Moving toward improved understanding and use of score reliability. Educational and Psychological Measurement, 62, 562-569. doi:10.1177/ 0013164402062004002
64.
*Vacha-Haase, T., Kogan, L.R., Tani, C.R., & Woodall, R.A. ( 2001). Reliability generalization: Exploring variation of reliability coefficients of MMPI clinical scales scores. Educational and Psychological Measurement, 61, 45-59. doi:10.1177/00131640121971059
65.
Vacha-Haase, T., Kogan, L.R., & Thompson, B. ( 2000). Sample compositions and variabilities in published studies versus those in test manuals: Validity of score reliability inductions. Educational and Psychological Measurement, 60, 509-522. doi:10.1177/00131640021970682
66.
Vacha-Haase, T., Ness, C.M., Nilsson, J., & Reetz, D. ( 1999). Practices regarding reporting of reliability coefficients: A review of three journals. Journal of Experimental Education , 67, 335-341. doi:10.1080/00220979909598487
*Vassar, M., & Crosby, J.W. ( 2008). A reliability generalization study of coefficient alpha for the UCLA Loneliness Scale. Journal of Personality Assessment , 90, 601-607. doi:10.1080/ 00223890802388624
69.
*Victorson, D., Barocas, J., Song, J., & Cella, D. ( 2008). Reliability across studies from the Functional Assessment of Cancer Therapy-General (FACT-G) and its subscales: A reliability generalization . Quality of Life Research: An International Journal of Quality of Life Aspects of Treatment, Care & Rehabilitation, 17, 1137-1146. doi:10.1007/s11136-008-9398-2
70.
*Wallace, K.A., & Wheeler, A.J. ( 2002). Reliability generalization of the Life Satisfaction Index . Educational and Psychological Measurement, 62, 674-684. doi:10.1177/001316 4402062004009
71.
Wilkinson, L., & American Psychological Association (APA) Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604. doi:10.1037//0003066X.54.8.594
72.
Willson, V.L. ( 1980). Research techniques in AERJ articles: 1969 to 1978. Educational Researcher, 9(6), 5-10. doi:10.2307/1175221
73.
Yetkiner, Z.E., & Thompson, B. (in press). Demonstration of how score reliability is integrated into SEM and how reliability affects all statistical analyses. Multiple Linear Regression Viewpoints, 36(2).
74.
*Yin, P., & Fan, X. ( 2000). Assessing the reliability of Beck Depression Inventory scores: Reliability generalization across studies. Educational and Psychological Measurement, 60, 201-223. doi:10.1177/00131640021970466
75.
*Youngstrom, E.A., & Green, K.W. ( 2003). Reliability generalization of self-report of emotions when using the Differential Emotions Scale. Educational and Psychological Measurement, 63, 279-295. doi:10.1177/0013164403253226
76.
*Zangaro, G.A., & Soeken, K.L. ( 2005). Meta-analysis of the reliability and validity of Part B of the Index of Work Satisfaction across studies. Journal of Nursing Measurement, 13, 7-22. doi:10.1891/jnum.2005.13.1.7
77.
Zientek, L.R., & Thompson, B. ( 2009). Matrix summaries improve research reports: Secondary analyses using published literature. Educational Researcher , 38, 343-352. doi:10.3102/0013189X09339056
