Sage Journals: Discover world-class research

Abstract

Meta-analytic reliability generalizations (RGs) are limited by the scarcity of reliability reporting in primary articles, and currently, RG investigators lack a method to quantify the impact of such nonreporting. This article introduces a stepwise procedure to address this challenge. First, the authors introduce a formula that allows researchers to estimate the lower bound population average reliability for a desired instrument. Second, they present an equation to determine the Fail-Safe N for RG. This equation estimates the number of ``file drawer'' studies required to drop the aggregate score reliability of an instrument below a specified criterion value. Finally, the authors demonstrate the utility of these equations using published RG studies. Comments on the conclusions drawn from each RG application are provided.

Keywords

reliability reliability generalization Fail-Safe N meta-analysis

Get full access to this article

View all access options for this article.

References

Charter, R.A. (2003). Combining reliability coefficients: Possible application to meta-analysis and reliability generalization. Psychological Reports, 93, 643-647.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

Henson, R.K. (2001). Understanding internal consistency reliability estimates: A conceptual primer on coefficient alpha. Measurement and Evaluation in Counseling and Development, 34, 177-189.

Henson, R.K. (2004). Expanding reliability generalization: Confidence intervals and charter's combined reliability coefficient. Perceptual and Motor Skills, 99, 818-820.

Henson, R.K. , Kogan, L.R. , & Vacha-Haase, T. (2001). A reliability generalization study of the teacher efficacy scale and related instruments. Educational and Psychological Measurement, 61, 404-420.

Henson, R.K. , & Thompson, B. (2002). Characterizing measurement error in scores across studies: Some recommendations for conducting ``reliability generalization'' studies. Measurement and Evaluation in Counseling and Development , 35, 113-126.

Lane, G.G. , White, A.E. , & Henson, R.K. (2002). Expanding reliability generalization methods with KR-21 estimates: An RG study of the Coopersmith Self-Esteem Inventory. Educational and Psychological Measurement, 62, 685-711.

Nunnally, J.C. , & Bernstein, I.H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.

Rosenthal, R. (1979). The ``file drawer problem'' and tolerance for null results. Psychological Bulletin, 86, 638-641.

10.

Rosenthal, R. (1991). Meta-analytic procedures for social research (2nd ed.). Newbury Park, CA: Sage.

11.

Schmidt, F. , & Hunter, J.E. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology , 62, 529-540.

12.

Shields, A.L. , Howell, R.T. , Potter, J.S. , & Weiss, R.D. (in press). The Michigan Alcoholism Screening Test and its shortened forms: A meta-analytic inquiry into score reliability. Substance Use and Misuse.

13.

Vacha-Haase, T. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58, 6-20.

14.

Vacha-Haase, T. , Henson, R. , & Caruso, J.C. (2002). Reliability generalization: Moving toward improved understanding and use of score reliability. Educational and Psychological Measurement, 62, 562-569.

15.

Vacha-Haase, T. , Kogan, L.R. , & Thompson, B. (2000). Sample compositions and variabilities in published studies versus those in test manuals: Validity of score reliability inductions . Educational and Psychological Measurement, 60, 509-522.

16.

Viswesvaran, C. , & Ones, D.S. (2000). Measurement error in ``Big Five Factors'' personality assessment: Reliability generalization across studies and measures. Educational and Psychological Measurement, 60, 224-235.

17.

Wang, J. (2002). Reliability generalization: An HLM approach. Instructional Psychology, 29, 213-218.

18.

Yin, P. , & Fan, X. (2000). Assessing the reliability of Beck Depression Inventory scores: Reliability generalization across studies. Educational and Psychological Measurement, 60, 201-223.

The File Drawer Problem in Reliability Generalization

Abstract

Keywords

Get full access to this article

References