The present article addresses reliability issues in light of recent studies and debates focused on psychometrics versus datametrics terminology and reliability generalization (RG) introduced by Vacha-Haase. The purpose here was not to moderate arguments presented in these debates but to discuss multiple perspectives on score reliability and how they may affect research practice, editorial policies, and RG across studies. Issues of classical error variance and reliability are discussed across models of classical test theory, generalizability theory, and item response theory. Potential problems with RG across studies are discussed in relation to different types of reliability, different test forms, different number of items, misspecifications, and confounding independent variables in a single RG analysis.
Get full access to this article
View all access options for this article.
References
1.
Alsawalmeh, Y. M. , & Feldt, L. S. (1999). Testing the equality of independent alpha coefficients adjusted for test length. Educational and Psychological Measurement, 59, 373-383.
2.
American Educational Research Association , American Psychological Association, & National Council on Measurement in Education. (1985). Standards for educational and psychological testing. Washington, DC: American Psychological Association.
3.
Anastasi, A. , & Urbina, S. (1997). Psychological testing (4th ed.). Upper Saddle River, NJ: Prentice Hall.
4.
Barnette, J. J. (2000). Effects of stem and Likert response option reversals on survey internal consistency: Ifyou feel the need, there is a better alternative to using those negatively worded stems. Educational and Psychological Measurement, 60, 361-370.
5.
Bentler, P. M. (1995). EQS structural equations program manual. Enrico, CA: Multivariate Software.
6.
Berk, R. A. (1980). A consumer’s guide to criterion referenced reliability. Journal of Educational Measurement, 17, 323-349.
7.
Brennan, R. L. (1983). Elements of generalizability theory. Iowa City, IO: ACT.
8.
Brennan, R. L. , & Kane, M. T. (1977). An index of dependability for mastery tests. Journal of Educational Measurement, 14, 277-289.
9.
Brennan, R. L. , & Prediger, D. J. (1981). Coefficient kappa: Some uses, misuses, and alternatives. Educational and Psychological Measurement, 20, 37-46.
10.
Capraro, M. M. , Capraro, R. M., & Henson, R. K. (2001). Measurement error of scores on the Mathematics Anxiety Rating Scale across studies. Educational and Psychological Measurement, 61, 373-386.
11.
Caruso, J. C. (2000). Reliability generalization of the NEO personality scales. Educational and Psychological Measurement, 60, 236-254.
12.
Caruso, J. C. , Witkiewitz, K., Belcourt-Dittloff, A., & Gottlieb, J. (2001). Reliability of scores from the Eysenck Personality Questionnaire: A reliability generalization (RG) study. Educational and Psychological Measurement, 61, 675-682.
13.
Charter, R. A. (2001). It is time to bury the Spearman-Brown “prophecy” formula for some common applications. Educational and Psychological Measurement, 61, 690-696.
14.
Clauser, B. E. , Clyman, S. G., & Swanson, D. B. (1999). Components of rater error in a complex performance assessment. Journal of Educational Measurement, 36, 29-45.
15.
Cohen, J. (1960). A coefficient of agreement for nominal scales.Educational and Psychological Measurement, 20, 37-46.
16.
Collins, L. M. (1996). Is reliability obsolete? A commentary on “Are simple gain score obsolete?”Applied Psychological Measurement, 20, 289-292.
17.
Conger, A. J. (1980). Integration and generalization of kappas for multiple raters. Psychological Bulletin, 88, 322-328.
18.
Cooil, B. , & Rust, R. T. (1994). Reliability and expected loss: A unifying principle. Psychometrika, 59, 203-216.
19.
Cook, C. , Heath, F., Thompson, R. L., & Thompson, B. (2001). Score reliability in Webor Internet-based surveys: Unnumbered graphic rating scales versus Likert-type scales. Educational and Psychological Measurement, 61, 697-706.
20.
Crick, J. E. , & Brennan, R. L. (1983). Manual for GENOVA: A generalized analysis of variance system. Iowa City, IO: ACT.
21.
Crocker, L. , & Algina, J. (1986). Introduction to classical and modern test theory. New York: Holt, Rinehart & Winston.
22.
Cronbach, L. J. (1951). Coefficient alpha and the internal structure of a test. Psychometrika, 16, 297-334.
23.
Cronbach, L. J. , & Furby, L. (1970). How we should measure “change”—Or should we?Psychological Bulletin, 74, 68-80.
24.
Cronbach, L. J. , Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: Wiley.
25.
Dillon, W. R. , & Mulani, N. (1984). A probabilistic latent class model for assessing inter-rater reliability. Multivariate Behavioral Research, 19, 439-458.
26.
Dimitrov, D. M. (2001a, October). Reliability and standard error of measurement related to parameters of items and ability distributions. Paper presented at the annual meeting of the MidWestern Educational Research Association, Chicago.
27.
Dimitrov, D. M. (2001b, October). Reliability of Rasch measurement with skewed ability distributions. Paper presented at the International Conference on Objective Measurement, Chicago.
28.
Dimitrov, D. M. , Rumrill, P., Fitzgerald, S., & Hennessey, M. (2001). Reliability in rehabilitation measurement. Work: A Journal of Prevention, Assessment, and Rehabilitation, 16, 159-164.
29.
Erlich, O. , & Shavelson, R. J. (1978). The search for correlations between measures of teacher behavior and student achievement: Measurement problem, conceptualization problem, or both?Journal of Educational Measurement, 15, 77-89.
30.
Fan, X. , & Thompson, B. (2001). Confidence intervals about score reliability coefficients, please: An EPM guidelines editorial. Educational and Psychological Measurement, 61, 517-531.
31.
Feldt, L. S. , & Brennan, R. L. (1989). Reliability. In R. L. Linn (Ed.), Educational measurement (pp. 105-146). New York: Macmillan.
32.
Frisbie, D. A. , & Druva, C. A. (1986). Estimating reliability of multiple true-false tests. Journal of Educational Measurement, 23, 99-105.
33.
Goodenough, F. L. (1949). Mental testing: Its history, principles, and applications. New York: Rinehart.
34.
Gronlund, N. E. (1988). How to construct achievement tests (4th ed.). Englewood Clifs, NJ: Prentice Hall.
35.
Gronlund, N. E. , & Linn, R. (1990). Measurement and evaluation in teaching (6th ed.). New York: Macmillan.
36.
Hambleton, R. K. , & Novick, M. R. (1973). Toward an integration of theory and method for criterion-referenced tests. Journal of Educational Measurement, 10, 159-170.
37.
Henson, R. K. , Kogan, L. R., & Vacha-Haase, T. (2001). A reliability generalization study on the Teacher Efficacy Scale and related instruments. Educational and Psychological Measurement, 61, 404-420.
38.
Hunter, J. E. , & Schmidt, F. L. (1990). Methods of meta-analysis. Newbury Park, CA: Sage.
39.
Huynh, H. (1976). On the reliability of decisions in domain-referenced testing. Journal of Educational Measurement, 13, 253-264.
40.
Jöreskog, K. G. (1971). Statistical analysis ofsets ofcongeneric tests. Psychometrika, 36, 109-133.
41.
Jöreskog, K. G. , & Sörbom, D. (1989). LISREL 7: A guide to the program and applications (2nd ed.). Chicago: SPSS.
42.
Kane, M. (1996). The precision ofmeasurements. Applied Measurement in Education, 9, 355-379.
43.
Kolen, M. J. , Hanson, B. A., & Brennan, R. L. (1992). Conditional standard errors of measurement for scale scores. Journal of Educational Measurement, 29, 285-307.
44.
Komaroff, E. (1997). Effect of simultaneous violations of essential tau-equivalent and uncorrelated errors on coefficient alpha.Applied Psychological Measurement, 21, 337-348.
45.
Kristof, W. (1970). On the sampling theory of reliability estimation. Journal of Mathematical Psychology, 7, 371-377.
46.
Linacre, J. M. (1994). Many-facet Rasch measurement (2nd ed.). Chicago: MESA Press.
47.
Linn, R. L. , & Slindle, J. A. (1977). The determination of the significance of change between pre- and posttesting periods. Review of Educational Research, 47, 121-150.
48.
Lord, F. M. (1957). Do tests of the same length have the same standard error ofmeasurement?Educational and Psychological Measurement, 17, 510-521.
49.
Lord, F. M. (1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.
50.
Lord, F. M. (1984). Standard error of measurement at different ability levels. Journal of Educational Measurement, 21, 239-243.
51.
Lord, F. M. , & Novick, R. (1968). Statistical theories of mental tests scores. Reading, MA: Addison-Wesley.
52.
May, K. , & Nicewander, W. A. (1993). Reliability and information functions for percentile ranks. Psychometrika, 58, 313-325.
53.
Novick, M. R. , & Lewis, C. (1967). Coefficient alpha and the reliability of composite measurements. Psychometrika, 32, 1-13.
54.
Nunnally, J. C. , & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.
55.
Payne, J. I. (1975). Principles of social science measurement. College Station, TX: Lytton.
56.
Pedney, T. N. , & Hubert, L. (1975). An empirical comparison of several interval estimation procedures for coefficient alpha. Psychometrika, 40, 169-181.
57.
Peng, C. J. , & Subkoviak, M. J. (1980). A note on Huynh’s normal approximation procedure for estimating criterion referenced reliability. Journal of Educational Measurement, 17, 359-368.
58.
Raykov, T. (1997). Estimation of composite reliability for congeneric measures. Applied Psychological Measurement, 21, 173-184.
59.
Raykov, T. (1998). A method for obtaining standard errors and confidence intervals of composite reliability for congeneric items. Applied Psychological Measurement, 22, 369-374.
60.
Raykov, T. (2000). A method for examining stability in reliability. Multivariate Behavioral Research, 35, 289-305.
61.
Raykov, T. (2001). Bias of coefficient alpha for fixed congeneric measures with correlated errors. Applied Psychological Measurement, 25, 69-76.
62.
Rogosa, D. (1995). Myths and methods: “Myths about longitudinal research” plus supplemental questions. In J. M. Gottman (Ed.), The analysis of change (pp. 3-66). Hillsdale, NJ: Lawrence Erlbaum.
63.
Samejima, F. (1977). The use of the information function in tailored testing. Applied Psychological Measurement, 1, 233-247.
64.
Sawilowsky, S. S. (2000). Psychometrics versus datametrics: Comments on Vacha-Haase’s “reliability generalization” method and some EPM editorial policies. Educational and Psychological Measurement, 60, 157-173.
65.
Shavelson, J. S. , & Webb, N. M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage.
66.
Suen, H. K. (1990). Principles of test theories. Hillsdale, NJ: Lawrence Erlbaum.
67.
Sun, A. , Valiga, M. J., & Gao, X. (1997). Using generalizability theory to assess the reliability of student ratings of academic advising. Journal of Experimental Education, 65, 367-379.
68.
Thissen, D. (1990). Reliability and measurement precision. In H. Wainer (Ed.), Computerized adaptive testing: A primer (pp. 161-185). Hillsdale, NJ: Lawrence Erlbaum.
69.
Thompson, B. (1992). Two and one-halfdecades of leadership in measurement and evaluation. Journal of Counseling and Development, 70, 434-438.
70.
Thompson, B. (1994). Guidelines for authors. Educational and Psychological Measurement, 54, 837-847.
71.
Thompson, B. , & Daniel, L. G. (1996). Factor analytic evidence for the construct validity of scores: An historical overview and some guidelines. Educational and Psychological Measurement, 56, 197-208.
72.
Thompson, B. , & Vacha-Haase, T. (2000). Psychometrics is datametrics: The test in not reliable. Educational and Psychological Measurement, 60, 174-195.
73.
Thorndike, R. L. (1982). Applied psychometrics. Boston: Houghton Mifflin.
74.
Tisak, J. , & Tisak, M. S. (1996). Longitudinal models of reliability and validity: A latent curve approach. Applied Psychological Measurement, 20, 275-288.
75.
Traub, R. E. (1994). Measurement methods for the social sciences—Reliability for the social sciences: Theory and applications (Vol. 3). Thousand Oaks, CA: Sage.
76.
Vacha-Haase, T. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58, 6-20.
77.
Vacha-Haase, T. , Kogan, L., Tani, C. R., & Woodall, R. A. (2001). Reliability generalization: Exploring reliability coefficients of MMPI clinical scales scores. Educational and Psychological Measurement, 61, 45-59.
78.
Van der Linden, W. J. , & Mellenbergh, G. J. (1977). Optimal cutting scores using a linear loss function. Applied Psychological Measurement, 1, 593-599.
79.
Viswesvaran, C. , & Ones, D. (2000). Measurement error in “big five factors” personality assessment: Reliability generalization across studies and measures. Educational and Psychological Measurement, 60, 224-235.
80.
Wang, T. (1998). Weights that maximize reliability under a congeneric model. Applied Psychological Measurement, 22, 179-187.
81.
Williams, R. H. , & Zimmerman, D. W. (1996). Are simple gain scores obsolete?Applied Psychological Measurement, 20, 59-69.
82.
Winer, B. J. (1971). Statistical principals in experimental design (2nd ed.). New York: McGraw-Hill.
83.
Wright, B. D. (2001). Separation, reliability, and skewed distributions. Rasch Measurement Transactions, 14(4), 786-786.
84.
Yin, P. , & Fan, X. (2000). Assessing the reliability of Beck Depression Inventory scores: Reliability generalization across studies. Educational and Psychological Measurement, 60, 201-223.
85.
Zimmerman, D. W. , & Williams, R. H. (1982). Gain scores in research can be highly reliable. Journal of Educational Measurement, 19, 149-154.
86.
Zimmerman, D. W. , Zumbo, B. D., & Lalonde, C. (1993). Coefficient alpha as an estimate of test reliability under violation of two assumptions. Educational and Psychological Measurement, 53, 33-49.
