The present article reviews issues regarding test reliability, which is psychometric terminology, and score reliability, which is score-centric terminology. These issues have arisen, in part, due to some EPM editorial policies and Vacha-Haase’s “reliability generalization” proposal. The article includes (a) a brief historical review of reliability terminology, (b) discussion on the emergence of datametrics (loosely defined as the application of psychometry to scores as opposed to an instrument) including a review of textbook authors’uses of psychometric versus datametric terminology, (c) discussion of problems with datametrics, and (d) a critique of Vacha-Haase’s proposed meta-analytic reliability generalization via dummy-coded regression. The article concludes with a brief summary that presents several suggestions.
Get full access to this article
View all access options for this article.
References
1.
Aiken, L. R. (1997). Psychological testing and assessment (9th ed.). Boston: Allyn & Bacon.
2.
Airasian, P. W. (1991). Classroom assessment. New York: McGraw-Hill.
3.
Baker, S. K. , Plasencia-Peinado, J., & Lezcano-Lytle, V. (1998). The use of curriculum-based measurement with language-minority students. In M. R. Shinn (Ed.), Advanced applications of curriculum-based measurement (pp. 175-213). New York: Guilford.
4.
Bott, P. A. (1996). Testing and assessment in occupational and technical education. Boston: Allyn & Bacon.
5.
Cangelosi, J. S. (1990). Designing tests for evaluating student achievement. New York: Longman.
6.
Cangelosi, J. S. (1991). Evaluating classroom instruction. New York: Longman.
7.
Carey, L. M. (1994). Measuring and evaluating school learning (2nd ed.). Boston: Allyn & Bacon.
8.
Cattell, J. M. (1890). Mental tests and measurements. Mind, 15, 373-381.
9.
Chase, C. I. (1999). Contemporary assessment for educators. New York: Longman.
10.
Crocker, L. , & Algina, J. (1986). Introduction to classical and modern test theory. Fort Worth, TX: Holt, Rinehart & Winston.
11.
Cunningham, G. K. (1986). Educational and psychological measurement. New York: Macmillan.
12.
Drummond, R. J. (1996). Appraisal procedures for counselors and helping professionals (3rd ed.). Englewood Cliffs, NJ: Merrill.
13.
Ebel, R. L. (1972). Essentials of educational measurement (2nd ed.). Englewood Cliffs, NJ: Prentice Hall.
14.
Evans, S. S. , Evans, W. H., & Mercer, C. D. (1986). Assessment for instruction. Boston: Allyn & Bacon.
15.
Gallagher, J. D. (1993). Classroom assessment for teachers. Upper Saddle River, NJ: Merrill.
16.
Gay, L. R. , & Airasian, P. (2000). Educational research (6th ed.). Upper Saddle River, NJ: Prentice Hall.
17.
Gellman, E. S. (1995). School testing: What parents and educators need to know. Westport, CT: Praeger.
18.
Gillford, B. R. (1993). Policy perspectives on educational testing. Boston: Kluwer Academic.
19.
Goodenough, F. L. (1936). A critical note on the use of the term “reliability” in mental measurement. Journal of Educational Psychology, 27, 173-178.
20.
Goodenough, F. L. (1949). Mental testing: Its history, principles, and applications. New York: Rinehart.
21.
Gredler, M. E. (1996). Program evaluation. Englewood Cliffs, NJ: Prentice Hall.
22.
Gredler, M. E. (1999). Classroom assessment and learning. New York: Longman.
23.
Gregory, R. J. (1996). Psychological testing: History, principles, and applications (2nd ed.). Boston: Allyn & Bacon.
24.
Gronlund, N. E. (1988). How to construct achievement tests (4th ed.). Englewood Cliffs, NJ: Prentice Hall.
25.
Gronlund, N. E. (1993). How to make achievement tests and assessments (5th ed.). Boston: Allyn & Bacon.
26.
Gronlund, N. E. (1998). Assessment of student achievement (6th ed.). Boston: Allyn & Bacon.
27.
Gronlund, N. E. , & Linn, R. (1990). Measurement and evaluation in teaching (6th ed.). New York: Macmillan.
28.
Hanna, G. S. (1993). Better teaching through better measurement. Fort Worth, TX: Harcourt Brace Jovanovich.
29.
Hart, D. (1994). Authentic assessment: A handbook for educators. Menlo Park, CA: Addison-Wesley.
30.
Hillman, S. B. , & Sawilowsky, S. S. (1991). Multidimensional differences between adolescent substance abusers and users. Psychological Reports, 68, 115-122.
31.
Hopkins, K. D. (1998). Educational and psychological measurement and evaluation (8th ed.). Boston: Allyn & Bacon.
32.
Hopkins, K. D. , & Antes, R. L. (1989). Classroom testing construction (2nd ed.). Itasca, IL: F. E. Peacock.
33.
Hopkins, K. D. , & Antes, R. L. (1990). Classroom measurement and evaluation (3rd ed.). Itasca, IL: F. E. Peacock.
34.
Hopkins, K. D. , Stanley, J. C., & Hopkins, B. R. (1990). Educational and psychological measurement and evaluation (7th ed.). Englewood Cliffs, NJ: Prentice Hall.
35.
Howell, K. W. , Fox, S. L., & Morehead, M. K. (1993). Curriculum-based evaluation: Teaching and decision making. Pacific Grove, CA: Brooks/Cole.
36.
Hoy, C. , & Gregg, N. (1994). Assessment: The special educator’s role. Pacific Grove, CA: Brooks/Cole.
37.
Kaplan, R. M. , & Saccuzzo, D. P. (1993). Psychological testing: Principles, applications, and issues (3rd ed.). Pacific Grove, CA: Brooks/Cole.
38.
Kelley, D. L. (1999). Measurement made accessible: A research approach using qualitative, quantitative, and quality improvement methods. Thousand Oaks, CA: Sage.
39.
Kendall, M. G. , & Buckland, W. R. (1971). A dictionary of statistical terms (3rd ed.). Edinburgh, UK: Oliver & Boyd.
40.
Kramer, J. J. , & Conoley, J. C. (1992). The eleventh mental measurement yearbook. Lincoln, NE: Buros Institute of Mental Measurement.
41.
Kubiszyn, T. , & Borich, G. (1996). Educational testing and measurement (5th ed.). New York: HarperCollins.
42.
Levit, D. B. (1991). Gender differences in ego defenses in adolescence: Sex roles as one way to understand the differences. Journal of Personality and Social Psychology, 61, 992-999.
43.
Linn, J. E. , & Gronlund, M. A. (1995). Measurement and assessment in teaching (7th ed.). Englewood Cliffs, NJ: Merrill.
44.
Lyman, H. B. (1991). Test scores & what they mean (5th ed.). Englewood Cliffs, NJ: Prentice Hall.
45.
McDaniel, E. (1994). Understanding educational measurement. Madison, WI: WCB Brown & Benchmark.
46.
McMillan, J. H. (1997). Classroom assessment: Principles and practice for effective instruction. Boston: Allyn & Bacon.
47.
Mehrens, W. A. , & Lehmann, I. J. (1987). Using standardized tests in education (4th ed.). New York: Longman.
48.
Mitchell, J. V. (1983). Tests in print III. An index to tests, test reviews, and the literature on specific tests. Lincoln, NE: Buros Institute of Mental Measurement.
49.
Mitchell, R. (1992). Testing for learning: How new approaches to evaluation can improve American schools. New York: Free Press.
50.
Monroe, W. S. , DeVoss, J. C., & Kelly, F. J. (1917). Educational tests and measurements. Boston: Houghton Mifflin.
51.
Nitko, A. J. (1996). Educational assessment of students. Englewood Cliffs, NJ: Merrill.
52.
Nunnally, J. C. , & Bernstein, I. H. (1994). Psychometric theory (3rd ed.). New York: McGraw-Hill.
53.
Nuttall, D. L. , & Wilmott, A. S. (with Backhouse, J. K., & Morrison, R. B.). (1972). British examinations: Techniques of analysis. Slough, UK: National Foundation for Educational Research in England and Wales.
54.
O’Dell, C. W. (1935). Statistical method in education. New York: Appleton-Century.
55.
Oosterhof, A. (1990). Classroom applications of educational measurement. Columbus, OH: Merrill.
56.
Oosterhof, A. (1996). Developing and using classroom assessments. Englewood Cliffs, NJ: Prentice Hall.
57.
Payne, D. A. (1992). Measuring and evaluating educational outcomes. New York: Merrill.
58.
Payne, J. L. (1975). Principles of social science measurement. College Station, TX: Lytton.
59.
Pedhazur, E. J. , & Schmelkin, L. P. (1991). Measurement, design, and analysis: An integrated approach. Hillsdale, NJ: Lawrence Erlbaum.
60.
Popham, W. J. (1990). Modern educational measurement: A practitioner’s perspective (2nd ed.). Englewood Cliffs, NJ: Prentice Hall.
61.
Popham, W. J. (1993). Educational evaluation (3rd ed.). Boston: Allyn & Bacon.
62.
Puckett, M. B. , & Black, J. K. (1994). Authentic assessment of the young child: Celebrating development and learning. New York: Merrill.
63.
Rowley, G. L. (1976). The reliability of observational measures. American Educational Research Journal, 13, 51-59.
64.
Rugg, H. O. (1917). Statistical methods applied to education: A textbook for students of education in the quantitative study of school problems. Boston: Houghton Mifflin.
65.
Salvia, J. , & Hughes, C. (1990). Curriculum-based assessment: Testing what is taught.New York: Macmillan.
66.
Salvia, J. , & Ysseldyke, J. E. (1988). Assessment: In special and remedial education (4th ed.). Boston: Houghton Mifflin.
67.
Sax, G. (1989). Principles of educational and psychological measurement and evaluation (3rd ed.). Belmont, CA: Wadsworth.
68.
Silverman, D. (1993). Interpreting qualitative data: Methods for analysing talk, text, and interaction. London: Sage.
69.
Stanley, J. C. (1971). Reliability. In R. L. Thorndike (Ed.), Educational measurement (2nd ed., pp. 356-442). Washington, DC: American Council on Education.
70.
Suen, H. K. (1990). Principles of test theories. Hillsdale, NJ: Lawrence Erlbaum.
71.
Sweetland, R. C. , & Keyser, D. J. (1986). Tests: A comprehensive reference for assessment in psychology, education and business (2nd ed.). Kansas City, MO: Test Corporation of America.
72.
Terman, L. M. (1916). The measurement of intelligence: An explanation of and a complete guide for the use of the Stanford revision and extension of the Binet-Simon Intelligence Scale. Boston: Houghton Mifflin.
73.
Thissen, D. (1990). Reliability and measurement precision. In H. Wainer (Ed.) with N. J. Dorans, R. Flaugher, B. F. Green, R. J. Mislevey, L. Steinberg, & D. Thissen, Computerized adaptive testing: A primer (pp. 161-186). Hillsdale, NJ: Lawrence Erlbaum.
74.
Thompson, B. (1994). Guidelines for authors. Educational and Psychological Measurement, 54, 837-847.
75.
Thorndike, R. M. (1997). Measurement and evaluation in psychology and education (6th ed.). Upper Saddle River, NJ: Merrill.
76.
Thorndike, R. M. , Cunningham, G. K., Thorndike, R. L., & Hagen, E. P. (1991). Measurement and evaluation in psychology and education (5th ed.). New York: Macmillan.
77.
Traub, R. E. (1994). Measurement methods for the social sciences—Reliability for the social sciences: Theory and applications (Vol. 3). Thousand Oaks, CA: Sage.
78.
Tryon, R. C. (1957). Reliability and behavior domain validity: Reformulation and historical critique. Psychological Bulletin, 54, 229-249.
79.
Tuckman, B. W. (1988). Testing for teachers (2nd ed.). San Diego, CA: Harcourt Brace Jovanovich.
80.
Tuckman, B. W. (1994). Conducting educational research (4th ed.). Fort Worth, TX: Harcourt Brace.
81.
Vacha-Haase, T. (1998). Reliability generalization: Exploring variance in measurement error affecting score reliability across studies. Educational and Psychological Measurement, 58, 6-20.
82.
Wallace, G. , Larsen, S. C., & Elksnin, L. K. (1992). Educational assessment of learning problems: Testing for teaching (2nd ed.). Boston: Allyn & Bacon.
83.
Walsh, W. B. , & Betz, N. E. (1995). Tests and assessment (3rd ed.). Englewood Cliffs, NJ: Prentice Hall.
84.
Weiss, C. H. (1998). Evaluation (2nd ed.). Upper Saddle River, NJ: Prentice Hall.
85.
Wise, P. S. (1989). The use of assessment techniques by applied psychologists. Belmont, CA: Wadsworth.
86.
Wood, R. (1991). Assessment and testing: A survey of research commissioned by the University of Cambridge Local Examinations Syndicate. Cambridge, UK: Cambridge University Press.
87.
Worthen, B. R. , Borg, W. R., & White, K. R. (1993). Measurement and evaluation in the schools. New York: Longman.
88.
Worthen, B. R. , White, K. R., Fan, X., & Sudweeks, R. R. (1999). Measurement and assessment in schools (2nd ed.). New York: Longman.
