Because tests are not reliable, it is important to explore score reliability in virtually all studies. The present article proposes and illustrates a new method-reliability generalization-that can be used in a meta-analysis application similar to validity generalization. Reliability generalization characterizes (a) the typical reliability of scores for a given test across studies, (b) the amount of variability in reliability coefficients for given measures, and (c) the sources of variability in reliability coefficients across studies. The use of reliability generalization is illustrated here by analyzing 87 reliability coefficients reported for the two scales of the Bem Sex Role Inventory (BSRI).
Get full access to this article
View all access options for this article.
References
1.
*References marked with an asterisk indicate studies included in the meta-analysis.
2.
*Abdalla, I. A. (1995). Sex, sex-role self-concepts and career decision-making self-efficacy among Arab students. Social Behavior and Personality, 23, 389-401.
3.
*Alain, M. (1987). A French version of the Bem Sex-Role Inventory. Psychological Reports, 61, 673-674.
4.
*Antill, J. K. (1983). Sex role complementarity versus similarity in married couples. Journal of Personality and Social Psychology, 45, 145-155.
5.
*Ballard, R. D., & Elton, M. (1992). Gender orientation and the Bem Sex Role Inventory: A psychological construct revisited. Sex Roles, 27, 291-306.
6.
Bem, S. L. (1974). The measurement of psychological androgyny. Journal of Consulting and Clinical Psychology, 42, 155-162.
7.
Bem, S. L. (1979). Theory and measurement of androgyny: Areply to the Pedhazur-Tetenbaum and Locksley-Colten critiques. Journal of Personality and Social Psychology, 37, 1047-1054.
8.
Bem, S. L. (1981). Bem Sex-Role Inventory: Professional manual. Palo Alto, CA: Consulting Psychologists Press.
9.
*Bledsoe, J. C. (1983). Sex differences in female teachers' approval and disapproval behaviors as related to their self-definition of sex-role type. Psychological Reports, 53, 711-714.
10.
Brouse, S. H. (1985). Effect of gender role identity on patterns of feminine and self-concept scores from late pregnancy to early postpartum. Advances in Nursing Science, 7(3), 32-48.
11.
*Burke, R. J. (1994). Canadian business students' attitudes towards women as managers. Psychological Reports, 75, 1123-1129.
12.
*Butcher, J. E. (1989). Adolescent girls' sex role development: Relationship with sports participation, self-esteem, and age at menarche. Sex Roles, 20, 575-593.
13.
*Chusmir, L. H., & Koberg, C. S. (1989). Gender identity and sex role conflict among working women and men. Journal of Psychology, 122, 567-575.
14.
*Chusmir, L. H., & Koberg, C. S. (1990). Dual sex role identity and its relationship to sex role conflict. Journal of Psychology, 124, 545-555.
15.
Constantinople, A. (1973). Masculinity-femininity: An exception to the famous dictum. Psychological Bulletin, 80, 389-407.
16.
*Costos, D. (1986). Sex role identity in young adults: Its parental antecedents and relation to ego development. Journal of Personality and Social Psychology, 50, 602-611.
17.
*Costos, D. (1990). Gender role identity from an ego developmental perspective. Sex Roles, 22, 723-741.
18.
Crocker, L. , & Algina, J. (1986). Introduction to classical and modern test theory. New York: Holt, Rinehart & Winston.
19.
Damji, T. , & Lee, C. M. (1985). Gender role identity and perceptions of Ismaili Muslim men and women. Journal of Social Psychology, 135, 215-223.
20.
Dawis, R. V (1987). Scale construction. Journal of Counseling Psychology, 34, 481-489.
21.
*Dean-Church, L. (1993). Relation of sex-role orientation to life satisfaction in a healthy elderly sample. Journal of Social Behavior and Personality, 8, 133-140.
22.
Eason, S. (1991). Why generalizability theory yields better results than classical test theory: A primer with concrete examples. In B. Thompson (Ed.), Advances in educational research: Substantive findings, methodological developments (Vol. 1, pp. 83-98). Greenwich, CT: JAI.
23.
*Erez, M., Borochov, O., & Mannheim, B. (1989). Work values of youth: Effects of sex or sex role typing? Journal of Vocational Behavior, 34, 350-366.
24.
*Farner, H. S. (1985). Model of career and achievement motivation for women and men. Journal of Counseling Psychology, 32, 363-390.
25.
*Faulkender, P. J. (1987). Validity of using Bem Sex-Role Inventory norms on other samples: Analysis of a Southern sample. Psychological Reports, 60, 399-406.
26.
Feldman, S. S. , Nash, S. C., & Aschenbrenner, B. G. (1983). Antecedents of fathering. Child Development, 54, 1628-1636.
27.
*Fischer, J. L., & Narus, L. R. (1981). Sex-role development in late adolescence and adulthood. Sex Roles, 7, 97-106.
28.
*Francis, L. J., & Wilcox, C. (1996). Religion and gender orientation. Personality and Individual Differences, 20, 119-121.
29.
*Gijsbers van Wijk, C. M., & Kolk, A. M. (1996). Psychometric evaluation of symptom perception related measures. Personality and Individual Differences, 20, 55-70.
30.
*Goldsmith, W. M., & Ekhardt, B. N. (1984). Personality factors of men and women pastoral candidates: II. Sex-role preferences. Journal of Psychology and Theology, 12, 211-221.
31.
Gronlund, N. E. , & Linn, R. L. (1990). Measurement and evaluation in teaching (6th ed.). New York: Macmillan.
32.
*Harris, A. C. (1996). African American and Anglo-American gender identities: An empirical study. Journal of Black Psychology, 22, 182-194.
33.
Hunter, J. E. , & Schmidt, F. L. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
34.
*Jones, D. C., & Costin, S. E. (1995). Friendship quality during preadolescence and adolescence: The contributions of relationship orientations, instrumentality, and expressivity. Merrill-Palmer Quarterly, 41, 517-535.
35.
Jones, G. P. , & Dembo, M. H. (1989). Age and sex role differences in intimate friendships during childhood and adolescence. Merrill-Palmer Quarterly, 35, 445-462.
36.
*Juni, S., & Grimm, D. W. (1994). Sex roles as factors in defense mechanisms and object relations. Journal of Genetic Psychology, 155, 99-106.
37.
*Kirchmeyer, C. (1993). Multicultural task groups: An account of the low contribution level of minorities. Small Group Research, 24, 127-148.
38.
*Knight, K. H., Elfenbein, M. H., & Messina, J. A. (1995). A preliminary scale to measure connected and separate knowing: The Knowing Styles Inventory. Sex Roles, 33, 499-513.
39.
*Kolbe, R. H., & Langefeld, C. D. (1993). Appraising gender role portrayals in TV commercials. Sex Roles, 28, 393-417.
40.
*Krampen, G., Effertz, B., Jostock, U., & Muller, B. (1990). Gender differences in personality: Biological and/or psychological? European Journal of Personality, 4, 303-317.
41.
*Krampen, G., Galli, I., & Nigro, G. (1992). Sex-role orientations and control orientations of Southern Italian and West German university students. Journal of Cross-Cultural Psychology, 23, 240-250.
42.
*Kurdek, L. A., & Schmitt, J. P. (1986). Interaction of sex role self-concept with relationship quality and relationship beliefs in marmed, heterosexual cohabiting, gay, and lesbian couples. Journal of Personality and Social Psychology, 51, 365-370.
43.
*Langis, J., Sabourin, S., Lussier, Y., & Mathieu, M. (1994). Masculinity, femininity, and marital satisfaction: An examination of theoretical models. Journal of Personality, 62, 393-414.
44.
*Lau, S. (1989). Sex role orientation and domains of self-esteem. Sex Roles, 21, 415-422.
45.
*Leary, M. R., & Snell, W. E. (1988). The relationship of instrumentality and expressiveness to sexual behavior in males and females. Sex Roles, 18, 509-522.
46.
Levit, D. B. (1991). Gender differences in ego defenses in adolescence: Sex roles as one way to understand the differences. Journal of Personality and Social Psychology, 61, 992-999.
47.
*Lippa, R. (1991). Some psychometric characteristics of gender diagnosticity measures: Reliability, validity, consistency across domains, and relationship to the big five. Journal of Personality & Social Psychology, 61, 1000-1011.
48.
*Lippa, R., & Connelly, S. (1990). Gender diagnosticity: A new Bayesian approach to gender-related individual differences. Journal of Personality and Social Psychology, 59, 1051-1065.
49.
*Lobel, T. E., Gur, S., & Yerushalmi, H. (1989). Cheating behavior of sex-type and androgynous children in sex-stereotyped and non-sex-stereotyped tasks. Journal of Research in Personality, 23, 302-312.
50.
*Martin, H. J., & Ramanaiah, N. V. (1988). Confirmatory factor analysis of the Bem Sex-Role Inventory. Psychological Reports, 62, 343-350.
51.
*Matsui, T. (1994). Mechanisms underlying sex differences in career self-efficacy expectations of university students. Journal of Vocational Behavior, 45, 177-184.
52.
*Maznah, I., & Choo, P. F. (1986). The factor structure of the Bem Sex-Role Inventory (BSRI). International Journal of Psychology, 21, 3-41.
53.
*McCall, M. E., & Struthers, N. J. (1994). Sex, sex-role orientation and self-esteem as predictors of coping style. Journal of Social Behavior and Personality, 9, 801-810.
54.
*McChrystal, J., & Dolan, B. (1994). Sex-role identity and separation-individuation pathology. Counselling Psychology Quarterly, 7, 25-34.
55.
*McCreary, D. R., & Korabik, K. (1994). Examining the relationships between the socially desirable and undesirable aspects of agency and communion. Sex Roles, 31, 637-651.
56.
Meier, S. T. , & Davis, S. R. (1990). Trends in reporting psychometric properties of scales used in counseling psychology research. Journal of Counseling Psychology, 37, 113-115.
57.
*Nagoshi, C. T., Pitts, S. C., & Nakata, T. (1993). Intercorrelations of attitudes, personality, and sex role orientation in a college sample. Personality and Individual Differences, 14, 603-604.
58.
Pedhazur, E. J. , & Schmelldn, L. P. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale, NJ: Lawrence Erlbaum.
59.
Pedhazur, E. J. , & Tetenbaum, T. J. (1979). Bem Sex-Role Inventory: A theoretical and methodological critique. Journal of Personality and Social Psychology, 37, 996-1016.
60.
*Portello, J. Y., & Long, B. C. (1994). Genderrole orientation, ethical and interpersonal conflicts, and conflict handling styles of female managers. Sex Roles, 31, 683-701.
61.
*Ramanaiah, N. V., & Martin, H. J. (1984). Convergent and discriminant validity of selected masculinity and femininity scales. Sex Roles, 10, 493-504.
62.
Rowley, G. L. (1976). The reliability of observational measures. American Educational Research Journal, 13, 51-59.
63.
*Rubinstein, G. (1995). Right-wing authoritarianism, political affiliation, religiosity, and their relation to psychological androgyny. Sex Roles, 33, 569-586.
64.
*Salminen, S. (1990). Sex role and participation in traditionally inappropriate sports. Perceptual and Motor Skills, 71, 1216-1218.
65.
*Salminen, S. (1994). Sex roles and values of school children using self-esteem as a moderating factor. Adolescence, 29, 875-884.
66.
Schmidt, F. L. , & Hunter, J. E. (1977). Development of a general solution to the problem of validity generalization. Journal of Applied Psychology, 62, 529-540.
67.
Singh, S. , & Kaur, J. (1985). Motive to avoid and approach success: Two dimensions of the same motive. Indian Journal of Clinical Psychology, 12, 5-11.
68.
*Smoreda, Z. (1995). Power, gender stereotypes and perceptions of heterosexual couples. British Journal of Social Psychology, 34, 421-435.
69.
*Stericker, A. B., & Kurdek, L. A. (1982). Dimensions and correlates of third through eighth graders' sex-role self-concepts. Sex Roles, 8, 915-929.
70.
Thompson, B. (1984). Canonical correlation analysis: Uses and interpretation. Beverly Hills, CA: Sage.
71.
Thompson, B. (1989). Meta-analysis of factor structure studies: A case study example with Bem's androgyny measure. Journal of Experimental Education, 57, 187-197.
72.
Thompson, B. (1990). ALPHAMAX: A program that maximizes coefficient alpha by selective item deletion. Educational and Psychological Measurement, 50, 585-589.
73.
Thompson, B. (1992). Two and one-half decades of leadership in measurement and evaluation. Journal of Counseling and Development, 70, 434-438.
74.
Thompson, B. (1994a, April). Common methodology mistakes in dissertations, revisited. Paper presented at the annual meeting of the American Educational Research Association, New Orleans. (ERIC Document Reproduction Service No. ED 368 771)
75.
Thompson, B. (1994b). Guidelines for authors. Educational and Psychological Measurement, 54, 837-847.
76.
Thompson, B. (1995). Stepwise regression and stepwise discriminant analysis need not apply here: A guidelines editorial. Educational and Psychological Measurement, 55, 525-534.
77.
Thompson, B. (1997). The importance of structure coefficients in structural equation modeling confirmatory factor analysis. Educational and Psychological Measurement, 57, 5-19.
78.
Thompson, B. (in press). Canonical correlation analysis. In L. Grimm & P. Yamold (Eds.), Reading and understanding multivariate statistics (Vol. 2). Washington, DC: American Psychological Association.
79.
Thompson, B. , & Borrello, G. M. (1985). The importance of structure coefficients in regression research. Educational and Psychological Measurement, 45, 203-209.
80.
Thompson, B. , & Daniel, L. G. (1996). Factor analytic evidence for the construct validity of scores: An historical overview and some guidelines. Educational and Psychological Measurement, 56, 197-208.
81.
Thompson, B. , & Melancon, J. G. (1986). Factor structure of the Bem Sex-Role Inventory. Measurement and Evaluation in Counseling and Development, 19, 77-83.
82.
Thompson, B. , & Snyder, P. A. (in press). Statistical significance andreliability analyses in recent JCD research articles. Journal of Counseling and Development.
83.
*Thompson, E. H. (1991). The maleness of violence in dating relationships: An appraisal of stereotypes. Sex Roles, 24, 261-278.
84.
Turner, B. F. , & Turner, C. B. (1991). Bem Sex-Role Inventory stereotypes for men and women varying in age and race among National Register psychologists. Psychological Reports, 69, 931-944.
85.
*Uleman, J. S., & Weston, M. (1986). Does the BSRI inventory sex roles? Sex Roles, 15, 43-62.
86.
*Walsh, A. (1993). Love styles, masculinity/femininity, physical attractiveness, and sexual behavior: A test of evolutionary theory. Ethology and Sociobiology, 14, 25-38.
87.
*Weider, H. D. (1987). Differences in self-reported leadership behavior as a function of biological sex and psychological gender. Women's Studies in Communication, 10, 1-14.
88.
*Williams, D. E., & D'Alessandro, J. D. (1994). A comparison of three measures of androgyny and their relationship to psychological adjustment. Journal of Social Behavior and Personality, 9, 469-480.
89.
Willson, V. L. (1980). Research techniques in AERJ articles: 1969 to 1978. Educational Researcher, 9(6), 5-10.
90.
Wilson, D. , McMaster, J., Greenspan, R., Mboyi, L., Ncube, T., & Sibanda, B. (1990). Cross-cultural validation of the BEM Sex Role Inventory in Zimbabwe. Personality and Individual Differences, 11, 651-656.
91.
*Wilson, F. R., & Cook, E. P. (1984). Concurrent validity of four androgyny instruments. Sex Roles, 11, 813-837.
92.
*Wong, F. Y, McCreary, D. R., & Duffy, K. G. (1990). A further validation of the Bem Sex Role Inventory: A multitrait-multimethod study. Sex Roles, 22, 249-259.
93.
*Yanico, B. J. (1985). BSRI scores: Stability over four years for college women. Psychology of Women Quarterly, 9, 277-283.
94.
*Yamold, P. R., Bryant, F. B., & Litsas, F. (1989). Type A behaviour and psychological androgyny among Greek college students. European Journal of Personality, 3, 249-268.
