Traditionally, reliability generalization (RG) studies have used some form of the regression model to summarize score reliability of a measure across samples and examine conditions under which the reliability varies. Oftentimes, the assumptions underlying the use of multiple regression are not satisfied in RG studies. This article describes how the assumptions have been violated and introduces a more sophisticated technique, mixed-effects modeling, that can overcome many of the shortcomings of traditional approaches. A nontechnical introduction of mixed-effects models in the context of RG studies is provided along with an example using internal consistency reliability coefficients from the Beck Depression Inventory that compares results under the mixed-versus the fixed-effects models.
Get full access to this article
View all access options for this article.
References
1.
*Andersen, S. M., & Schwartz, A. H. (1992). Intolerance of ambiguity and depression: A cognitive vulnerability factor linked to hopelessness . Social Cognition, 10, 271-298 .
2.
*Anderson, G., Melin, L., Lindberg, P., & Scott, B. (1995). Dispositional optimism, dyspheria, health and coping with hearing impairment in elderly adults . Audiology, 34, 76-84 .
3.
*Barrera, M., Jr., & Oliver, J. M. (1981). Sex-ratio and gender differences in depression in an unelected adult population . Journal of Clinical Psychology, 37, 570-574 .
4.
*Beck, A. T. (1967). Depression: Causes and treatment. Philadelphia: University of Pennsylvania Press .
5.
*Beck, A. T., Guth, D., Steer, R. A., & Ball, R. (1997). Screening for major depression disorders in medical inpatients with the Beck Depression Inventory for primary care . Behavior Research and Therapy, 35, 785-791 .
6.
*Beck, A. T., & Steer, R. A. (1984). Internal consistencies of the original and revised Beck Depression Inventory . Journal of Clinical Psychology, 40, 1365-1367 .
7.
*Beck, A. T., Steer, R. A., & Brown, G. K. (1996). Manual for the Beck Depression Inventory–II. San Antonio, TX: The Psychological Corporation .
8.
Beck, A. T., Ward, C. H., Mendelson, M., Mock, J., & Erbaugh, J. (1961). An inventory for measuring depression . Archives of General Psychiatry, 4, 53-63 .
9.
*Beck, J. T., & Strong, S. R. (1982). Stimulating therapeutic change with interpretations: A comparison of positive and negative connotations . Journal of Counseling Psychology, 29, 551-559 .
10.
*Bosscher, R. J., Koning, H., & Van Meurs, R. (1986). Reliability and validity of the Beck Depression Inventory in a Dutch college population . Psychological Reports, 58, 696-798 .
11.
Bryk, A. S., & Raudenbush, S. W. (1992). Hierarchical linear models. Newbury Park, CA: Sage .
12.
*Bryson, S. E., & Pilon, D. J. (1984). Sex differences in depression and the method of administering the Beck Depression Inventory . Journal of Clinical Psychology, 40, 529-534 .
13.
*Byerly, F. G., & Carlson, W. A. (1982). Comparison among inpatients, outpatients, and normals on three self-report depression inventories . Journal of Clinical Psychology, 30, 797-804 .
14.
*Campbell, I. M., Burgess, P. M., & Finch, S. J. (1984). A factorial analysis of BDI scores . Journal of Clinical Psychology, 40, 992-996 .
15.
Capraro, M. M., Capraro, R. M., & Henson, R. K. (2001). Measurement error of scores on the mathematics anxiety rating scale across studies . Educational and Psychological Measurement, 61, 373-386 .
16.
*Carr, J. G., Gilroy, F. D., & Sherman, M. F. (1996). Silencing the self and depression among women . Psychology of Women Quarterly, 20, 375-392 .
17.
Caruso, J. C. (2000). Reliability generalization of the NEO personality scales . Educational and Psychological Measurement, 60, 236-254 .
18.
Caruso, J. C., Witkiewitz, K., Belcourt-Dittloff, A., & Gottlieb, J. D. (2001). Reliability of scores from the Eysenck personality questionnaire: A reliability generalization study . Educational and Psychological Measurement, 61, 675-689 .
19.
*Cascardi, M., & O’Leary, L. D. (1992). Depressive symptomatology, self-esteem, and self-blame in battered women . Journal of Family Violence, 7, 249-259 .
20.
*Catanzaro, S. J. (1993). Mood regulation expectancies, anxiety sensitivity, and emotional distress . Journal of Abnormal Psychology, 102, 327-330 .
21.
*Chan, D. W. (1991). The Beck Depression Inventory: What difference does the Chinese version make? Psychological Assessment, 3, 616-622 .
22.
*Cook, M. L., & Peterson, C. (1986). Depressive irrationality . Cognitive Therapy and Research, 10, 293-298 .
23.
*Cox, B. J., Swinson, R. P., Kuch, K., & Reichman, J. T. (1993). Self-reported differentiation of anxiety and depression in an anxiety disorders sample . Psychological Assessment, 5, 484-486 .
24.
*Dobson, K. S., & Breiter, H. J. (1983). Cognitive assessment of depression: Reliability and validity of three measures . Journal of Abnormal Psychology, 92, 107-109 .
25.
*Elevin, B. E., Llabre, M. M., & Weiner, W. J. (1988). Parkinson’s disease and depression: Psychometric properties of the Beck Depression Inventory . Journal of Neurology, Neurosurgery, and Psychiatry, 51, 1401-1404 .
26.
*Ellis, T., Rudd, M. D., Rajab, M. H., & Wehrly, T. (1996). Cluster analysis of MCMT scores of suicidal psychiatric patients: Four personality profiles . Journal of Clinical Psychology, 52, 411-422 .
27.
*Feather, N. T. (1983). Some correlates of attributional style: Depressive symptoms, self-esteem and protestant ethic values . Personality and Social Psychology Bulletin, 9, 125-135 .
28.
*Feather, N. T. (1987). The rosy glow of self-esteem: Depression, masculinity, and causal attributions . Australian Journal of Psychology, 39, 25-41 .
29.
*Feather, N. T., & Bond, M. J. (1983). Time structure and purposeful activity among employed and unemployed university graduates . Journal of Occupational Psychology, 56, 241-254 .
30.
Field, A. P. (2001). Meta-analysis of correlation coefficients: A Monte Carlo comparison of fixed- and random-effects methods . Psychological Methods, 6, 161-180 .
31.
Fisher, R. A. (1928). Statistical methods for research workers (2nd ed.). London: Oliver & Boyd .
32.
*Foelker, G. A., Jr., Shewchuk, R. M., & Niederehe, G. (1987). Confirmatory factor analysis of the short form of the Beck Depression Inventory in elderly community samples . Journal of Clinical Psychology, 43, 111-118 .
33.
*Gallagher, D., Nies, G., & Thompson, L. W. (1982). Reliability of the Beck Depression Inventory with older adults . Journal of Consulting and Clinical Psychology, 50, 152-153 .
34.
*Gatewood-Colwell, G., Kaczmarek, M., & Ames, M. H. (1989). Reliability of the Beck Depression Inventory for a White and Mexican-American gerontic population . Psychological Reports, 65, 1163-1166 .
35.
*Giambra, L. M. (1977). Independent dimensions of depression: A factor analysis of three self-report measures . Journal of Clinical Psychology, 33, 928-935 .
36.
*Golin, S., & Hartz, M. A. (1979). A factor analysis of the Beck Depression Inventory in a mildly depressed population . Journal of Clinical Psychology, 35, 322-325 .
37.
*Gorenstein, C., & Pompeia, S. (1995). Scores of Brazilian University students on the Beck Depression and the State-Trait Anxiety Inventories . Psychological Reports, 77, 635-641 .
38.
*Gotlib, I. H. (1984). Depression and general psychopathology in university students . Journal of Abnormal Psychology, 93, 19-30 .
39.
*Gould, J. (1982). A psychometric investigation of the standard and short form Beck Depression Inventory . Psychological Reports, 51, 1167-1170 .
40.
*Hankin, B. L., Roberts, J., & Gotlib, I. H. (1997). Elevated self-standards and emotional distress during adolescence: Emotional specificity and gender differences . Cognitive Therapy and Research, 21, 663-679 .
41.
Harwell, M. (1997) An empirical study of Hedges’homogeneity test . Psychological Methods, 2, 219-231 .
42.
*Hatzenbuehler, L. C., Parpal, M., & Matthews, L. (1983). Classifying college students as depressed or nondepressed using the Beck Depression Inventory: An empirical analysis . Journal of Consulting and Clinical Psychology, 51, 360-366 .
43.
Hedges, L. V., & Olkin, I. (1985). Statistical methods for meta-analysis. San Diego, CA: Academic Press .
44.
Hedges, L. V., & Vevea, J. L. (1998). Fixed- and random-effects models in meta-analysis . Psychological Methods, 3, 486-504 .
45.
Henson, R. K., Kogan, L. R., & Vacha-Haase, T. (2001). A reliability generalization study of the teacher efficacy scale and related instruments . Educational and Psychological Measurement, 61, 404-420 .
46.
*Hjelle, L. A., & Bernard, M. (1994). Private self-consciousness and the retest reliability of self-reports . Journal of Research in Personality, 28, 52-67 .
47.
*Hojat, M., & Schapurian, R. (1986). Psychometric properties of a Persian version of the short form of the Beck Depression Inventory for Iranian college students . Psychological Reports, 59, 331-338 .
48.
*Joe, G. W., Knezek, L., Watson, D., & Simpson, D. D. (1991). Depression and decision making among intravenous drug users . Psychological Reports, 68, 339-347 .
49.
*Johnson, J. G., Crofton, A., & Feinstein, S. B. (1996). Enhancing attributional style and positive life events increased hopefulness among depressed psychiatric inpatients . Motivation and Emotion, 20, 285-297 .
50.
Kalaian, H. A., & Raudenbush, S. W. (1996). A multivariate mixed linear model for meta-analysis . Psychological Methods, 1, 227-235 .
51.
*Kirkham, M. A., Schinke, S. P., Schilling, R. F. II, Meltzer, N. J., & Norelius, K. L. (1986). Cognitive, behavioral skills, social supports, and child abuse potential among mothers of handicapped children . Journal of Family Violence, 1, 235-245 .
52.
*Knight, R. G. (1984). Some general population norms for the short form Beck Depression Inventory . Journal of Clinical Psychology, 40, 751-753 .
53.
Kreft, I., & de Leeuw, J. (1998). Introducing multilevel modeling. London: Sage .
54.
*Kwon, S. M., & Dei, T.P.S. (1992). Differential causal roles of dysfunctional attitudes and automatic thought in depression . Cognitive Therapy and Research, 16, 309-328 .
55.
*Leahy, J. M. (1992). Validity and reliability of the Beck Depression Inventory-Short Form in a group of adult bereaved females . Journal of Clinical Psychology, 48, 64-68 .
56.
*Leahy, J. M. (1993). A comparison of depression in women bereaved of a spouse, child or a parent . Omega, 26, 207-217 .
57.
*Lee, A. M., & Lee, S. (1996). Disordered eating and its psychosocial correlates among Chinese adolescent females in Hong Kong . International Journal of Eating Disorders, 20, 177-183 .
58.
*Leigh, I. W., Robins, C. J., & Welkowitz, J. (1988). Modification of the Beck Depression Inventory for use with a deaf population . Journal of Clinical Psychology, 44, 728-732 .
59.
*Marcovitz, R. J., & Smith, J. E. (1983). Patients’ perceptions of curative factors in short-term group psychotherapy . International Journal of Group Psychotherapy, 33, 21-39 .
60.
*Matsumi, J., & Kameoka, V. A. (1986). Reliabilities and concurrent validities of popular self-report measures of depression, anxiety, and social desirability . Journal of Consulting and Clinical Psychology, 54, 328-333 .
61.
*McCabe, K. M. (1997). Sex differences in the long term effects of divorce on children: Depression and heterosexual relationship difficulties in the young adult years . Journal of Divorce and Remarriage, 27, 123-135 .
62.
*Michelson, L., & Mavissakalian, M. (1983). Temporal stability of self-report measures in agoraphobia research . Behavior Research and Therapy, 21, 695-698 .
63.
*Moilanen, D. L. (1995). Validity of Beck’s cognitive therapy of depression with nonreferred adolescents . Journal of Counseling and Development, 73, 438-442 .
64.
*Oliver, J. M., & Baumgart, E. P. (1995). The dysfunctional attitude scale: Psychometric properties and relations to depression in an unselected adult population . Cognitive Therapy and Research, 9, 161-167 .
65.
*Oliver, J. M., & Burkham, R. (1979). Depression in university students: Duration, relation to calendar time, prevalence, and demographic correlates . Journal of Abnormal Psychology, 88, 667-670 .
66.
*Oliver, J. M., & Simmons, M. E. (1985). Affective disorders and depression as measures by the diagnostic interview schedule and the Beck Depression Inventory in an unselected adult population . Journal of Clinical Psychology, 41, 469-477 .
67.
*Palinkas, L. A., Wingard, P. L., & Barrett-Connor, E. (1996). Depressive symptoms in overweight and obese older adults: A test of the “jolly fat” hypothesis . Journal of Psychosomatic Research, 40, 59-66 .
68.
*Peterson, C. (1979). Uncontrollability and self-blame in depression: Investigation of the paradox in a college population . Journal of Abnormal Psychology, 88, 620-624 .
69.
*Peterson, C., & Avila, M.E.D. (1995). Optimistic explanation style and the perception of health problems . Journal of Clinical Psychology, 51, 128-132 .
70.
*Peterson, C., & Barrett, L. (1987). Explanatory style and academic performance among university freshman . Journal of Personality and Social Psychology, 53, 603-607 .
71.
*Peterson, C., Rosenbaum, A. C., & Conn, M. K. (1985). Depression mood reactions to breaking up: Testing the learned helplessness model of depression . Journal of Social and Clinical Psychology, 3, 161-169 .
72.
*Pulos, S. (1996). Validity of the Beck Depression Inventory with eating disorder patients . Educational and Psychological Measurement, 56, 139-141 .
73.
*Radnitz, C. L., et al. (1997). Use of the Beck Depression Inventory in veterans with spinal cord injury . Rehabilitation Psychology, 42, 93-101 .
74.
*Rapp, S. R., Walsh, D. A., Parisi, S., & Wallace, C. (1988). Detecting depression in elderly medical inpatients . Journal of Consulting and Clinical Psychology, 56, 509-513 .
75.
Raudenbush, S. W. (1988). Educational applications of hierarchical linear models: A review . Journal of Educational Statistics, 13, 85-116 .
76.
Raudenbush, S. W., & Bryk, A. S. (1985). Empirical Bayes meta-analysis . Journal of Educational Statistics, 10, 75-98 .
77.
*Rees, A., Hardy, G. E., & Barkham, M. (1997). Covariance in the measurement of depression/anxiety and three cluster C personality disorders (avoidant, dependent, obsessive-compulsive) . Journal of Affective Disorders, 45, 143-153 .
78.
*Reynolds, W. M., & Gould, J. W. (1981). A psychometric investigation of the standard and short form of the Beck Depression Inventory . Journal of Consulting and Clinical Psychology, 49, 306-307 .
79.
Rosenthal, R. (1979). The “file drawer problem” and tolerance for null results . Psychological Bulletin, 86, 1165-1168 .
80.
Rosenthal, R. (1985). Parametric measures of effect size. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 231-244). San Diego, CA: Academic Press .
81.
*Schaefer, A., et al. (1985). Comparison of the validities of the Beck, Zung, and MMPI Depression Scales . Journal of Consulting and Clinical Psychology, 53, 415-418 .
82.
Schmidt, F. L., & Hunter, J. E. (1977). Development of a general solution to the problem of validity generalization . Journal of Applied Psychology, 62, 529-540 .
83.
*Schotte, C.K.W., Maes, M., Cluydts, R., DeDoncker, D., & Cosyns, P. (1997). Construct validity of the Beck Depression Inventory in a depressive population . Journal of Affective Disorders, 46, 115-125 .
84.
*Scogin, F., Beutler, L., Corbishley, A., & Hamblin, D. (1988). Reliability and validity of the short form Beck Depression Inventory with older adults . Journal of Clinical Psychology, 44, 853-857 .
85.
*Shek, D.T.L. (1990). Reliability and factorial structure of the Chinese version of the Beck Depression Inventory . Journal of Clinical Psychology, 46, 35-43 .
86.
*Somoza, E., Steer, R. A., Beck, A. T., & Clark, D. A. (1994). Differentiating major depression and panic disorders by self-report and clinical rating scales: ROC analysis and information theory . Behavior Research and Therapy, 32, 771-782 .
87.
*Steer, R. A., Beck, A. T., Riskind, J. H., & Brown, G. (1987). Relationships between the Beck Depression Inventory and the Hamilton Psychiatric Rating Scale for depression in depressed outpatients . Journal of Psychopathology and Behavioral Assessment, 9, 327-339 .
88.
*Steer, R. A., & Clark, D. A. (1997). Psychometric characteristics of the Beck Depression Inventory-II with college students . Measurement and Evaluation in Counseling and Development, 30, 128-136 .
89.
*Steer, R. A., McElroy, M. G., & Beck, A. T. (1982). Structure of depression in alcoholic men: A partial replication . Psychological Reports, 50, 723-728 .
90.
*Steer, R. A., Shaw, B. F., Beck, A. T., & Fine, E. W. (1977). Structure of depression in Black alcoholic men . Psychological Reports, 41, 1235-1241 .
91.
*Strober, M., Green, J., & Carlson, G. (1981). Utility of the Beck Depression Inventory with psychiatrically hospitalized adolescents . Journal of Consulting and Clinical Psychology, 49, 482-483 .
92.
*Subkoviak, M., et al. (1995). Measuring interpersonal forgiveness in late adolescence and middle adulthood . Journal of Adolescence, 18, 641-655 .
93.
*Tanaka, J. S., & Huba, G. J. (1987). Assessing the stability of depression in college students . Multivariate Behavioral Research, 22, 5-19 .
94.
*Tangney, J. P., Wagner, P., & Granizow, R. (1992). Proneness to shame, proneness to guilt, and psychopathology . Journal of Abnormal Psychology, 101, 469-478 .
95.
*Tashakkori, A., Barefoot, J., & Mehryar, A. H. (1989). What does the Beck Depression Inventory measure in college students? Evidence from a non-Western culture . Journal of Clinical Psychology, 45, 595-602 .
96.
*Teri, L. (1982). The use of the Beck Depression Inventory with adolescents . Journal of Abnormal Child Psychology, 10, 277-284 .
97.
Thompson, B., & Vacha-Haase, T. (2000). Psychometrics is datametrics: The test is not reliable . Educational and Psychological Measurement, 60, 174-195 .
98.
Vacha-Haase, T. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies . Educational and Psychological Measurement, 58, 6-20 .
99.
Vacha-Haase, T., Kogan, L. R., Tani, C. R., & Woodall, R. A. (2001). Reliability generalization: Exploring variation of reliability coefficients of MMPI clinical scale scores . Educational and Psychological Measurement, 61, 45-59 .
100.
Viswesvaran, C., & Ones, D. S. (2000). Measurement error in “big five factors” personality assessment: Reliability generalization across studies and measures . Educational and Psychological Measurement, 60, 224-235 .
101.
*Vrendenburg, K., Krames, L., & Flett, G. L. (1985). Reexamining the Beck Depression Inventory: The long and short form of it . Psychological Reports, 56, 767-778 .
102.
*Webster-Stratton, C. (1985). Predictor of treatment outcome in parent training for conduct-disordered children . Behavior Therapy, 16, 223-243 .
103.
*Welch, G., Hall, A., & Walkey, F. (1990). The replicable dimensions of the Beck Depression Inventory . Journal of Clinical Psychology, 46, 817-827 .
104.
*Westaway, M., & Wolmarans, L. (1992). Depression and self-esteem: Rapid screening for depression in Blacks, low literacy and hospitalized tuberculosis patients . Social Science and Medicine, 35, 1311-1315 .
105.
*Whishman, M. A., & Kwon, P. (1992). Parental representations, cognitive distortions, and mild depression . Cognitive Therapy and Research, 16, 557-568 .
106.
Yin, P., & Fan, X. (2000). Assessing the reliability of Beck depression inventory scores: Reliability generalization across studies . Educational and Psychological Measurement, 60, 201-223 .
107.
*Zea, M. C., Belgrave, F. Z., Townsend, T. G., Jarama, S. L., & Banks, S. R. (1996). The influence of social support and active coping on depression among African Americans and Latinos with disabilities . Rehabilitation Psychology, 41, 225-242 .
108.
*Zemore, R. (1983). Development of a self-report measure of depression-proneness . Psychological Reports, 52, 211-216 .
109.
*Zemore, R., & Bretell, D. (1983). Depression-proneness, low self-esteem, unhappy outlook, and narcissistic vulnerability . Psychological Reports, 52, 223-230 .
110.
*Zimmerman, M. (1986). The stability of the revised Beck Depression Inventory in college students: Relationship with life events . Cognitive Therapy and Research, 10, 37-43 .
