As measurement specialists, we have done a disservice to both ourselves and our profession by habitually referring to “the reliability of the test,” or saying that “the test is reliable.” This has created a mind-set implying that reliability, once proven, is immutable. More important, practitioners and scholars need not know measurement theories if they may simply rely on the reliability purportedly intrinsic within all uses of established measures. The present study investigated empirically exactly how dissimilar in both composition and variability samples inducting reliability coefficients from prior studies were from the cited prior samples from which coefficients were generalized.
Get full access to this article
View all access options for this article.
References
1.
Aiken,L.S. , West,S.G., Sechrest,L., & Reno,R.R. (with Roediger, H. L., Scarr, S., Kazdin, A. E., & Sherman, S. J.). (1990). The training in statistics, methodology, and measurement in psychology. American Psychologist, 45, 721-734.
2.
American Psychological Association . (1994). Publication manual of the American Psychological Association (4th ed.). Washington, DC: Author.
3.
Bem, S. L. (1974). The measurement of psychological androgyny. Journal of Consulting and Clinical Psychology, 42, 155-162.
4.
Bem, S. L. (1979). Theory and measurement of androgyny: A reply to the Pedhazur-Tetenbaum and Locksley-Colten critiques. Journal of Personality and Social Psychology, 37, 1047-1054.
5.
Bem, S. L. (1981). Bem Sex-Role Inventory: Professional manual. Palo Alto, CA: Consulting Psychologists Press.
6.
Constantinople, A. (1973). Masculinity-femininity: An exception to the famous dictum. Psychological Bulletin, 80, 389-407.
7.
Crocker, L. , & Algina, J. (1986). Introduction to classical and modern test theory. New York: Holt, Rinehart & Winston.
8.
Dawis, R. V. (1987). Scale construction. Journal of Counseling Psychology, 34, 481-489.
9.
Fleming, J. S. , & Courtney, B. E. (1984). The dimensionality of self esteem II: Hierarchical facet model for revised measurement scales. Journal of Personality and Social Psychology, 46, 404-421.
10.
Gronlund, N. E. , & Linn, R. L. (1990). Measurement and evaluation in teaching (6th ed.). New York: Macmillan.
11.
Meier, S. T. , & Davis, S. R. (1990). Trends in reporting psychometric properties of scales used in counseling psychology research. Journal of Counseling Psychology, 37, 113-115.
12.
Mittag, K. C. , & Thompson, B. (2000). A national survey of AERA members’ perceptions of statistical significance tests and other statistical issues. Educational Researcher, 29 (4), 21-27.
13.
Pedhazur, E. J. , & Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale, NJ: Lawrence Erlbaum.
14.
Pedhazur, E. J. , & Tetenbaum, T. J. (1979). Bem Sex-Role Inventory: A theoretical and methodological critique. Journal of Personality and Social Psychology, 37, 996-1016.
15.
Rosenberg, M. (1965). Society and the adolescent self-image. Princeton, NJ: Princeton University Press.
16.
Sawilowsky, S. S. (2000). Psychometrics vs datametrics: Comment on Vacha-Haase’s “reliability generalization” method and some EPM editorial policies. Educational and Psychological Measurement, 60, 157-173.
17.
Silber, E. , & Tippett, J. S. (1965). Self-esteem: Clinical assessment and measurement validation. Psychological Reports, 16, 1017-1071.
18.
Simmelink, S. , & Vacha-Haase, T. (1999, April). Reliability generalization with the Rosenberg Self-Esteem Instrument. Paper presented at the annual meeting of the Rocky Mountain Psychological Association, Fort Collins, CO.
19.
Simons, R. G. , Rosenberg, F., & Rosenberg, M. (1973). Disturbance in the self image at adolescence. American Sociological Review, 38, 553-568.
20.
Thompson, B. (1989). Meta-analysis of factor structure studies: A case study example with Bem’s androgyny measure. Journal of Experimental Education, 57, 187-197.
21.
Thompson, B. (1990). ALPHAMAX: A program that maximizes coefficient alpha by selective item deletion. Educational and Psychological Measurement, 50, 585-589.
22.
Thompson, B. (1992). Two and one-half decades of leadership in measurement and evaluation. Journal of Counseling and Development, 70, 434-438.
23.
Thompson, B. (1994). Guidelines for authors. Educational and Psychological Measurement, 54, 837-847.
24.
Thompson, B. (1999). Journal editorial policies regarding statistical significance tests: Heat is to fire as p is to importance. Educational Psychology Review, 11, 157-169.
25.
Thompson, B. , & Snyder, P. A. (1998). Statistical significance and reliability analyses in recent JCD research articles. Journal of Counseling and Development, 76, 436-441.
26.
Thompson, B. , & Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable. Educational and Psychological Measurement, 60, 174-195.
27.
Vacha-Haase, T. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58, 6-20.
28.
Vacha-Haase, T. , Ness, C., Nilsson, J., & Reetz, D. (1999). Practices regarding reporting of reliability coefficients: A review of three journals. Journal of Experimental Education, 67, 335-341.
29.
Vacha-Haase, T. , Nilsson, J. E., Reetz, D. R., Lance, T. S., & Thompson, B. (2000). Reporting practices and APA editorial policies regarding statistical significance and effect size. Theory & Psychology, 10, 413-425.
30.
Whittington, D. (1998). How well do researchers report their measures? An evaluation of measurement in published educational research. Educational and Psychological Measurement, 58, 21-37.
31.
Wilkinson, L. , & The APA Task Force on Statistical Inference. (1999). Statistical methods in psychology journals: Guidelines and explanations. American Psychologist, 54, 594-604. (Reprint available through the APA home page http://www.apa.org/journals/amp/amp548594.html)
32.
Willson, V. L. (1980). Research techniques in AERJ articles: 1969 to 1978. Educational Researcher, 9(6), 5-10.
