In this paper Jöreskog's model of congeneric tests is used to analyze agreement between raters. Raters are treated as measuring instruments. The model of congeneric tests, of which classical prallelism and tau-equivalence are shown to be special cases, is applied to teachers' ratings of students' responses on open-end questions. The findings suggest that the ratings are tau-equivalent and that the ratings given by the teachers are reliable.
Get full access to this article
View all access options for this article.
References
1.
Abelson, R. P.Scales derived by consideration of variance components of multiway tables. In: Gulliksen, H. and Messick, S. S. (Eds.) Psychological Scaling: Theory and Applications. New York, 1960, 169-181.
2.
Boruch, R. F., Larkin, J. D., Wolins, L., and MacKinney, A. C.Alternative methods of analysis: multitrait-multimethod data. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 1970, 30, 833-853.
3.
Ebel, R. L.Estimation of the reliability of ratings. Psychometrika, 1951, 16, 407-424.
4.
Fleiss, J. L.Assessing the accuracy of multivariate observations. Journal of the American Statistical Association , 1966, 61, 403-412.
5.
Gulliksen, H.Theory of mental tests. New York: Wiley, 1950 .
6.
Gulliksen, H.Methods for determining equivalence of measures. Psychological Bulletin, 1968, 70, 534-544.
7.
Jöreskog, K. G.A general method for analysis of covariance structures. Biometrika, 1970, 57, 239-251.
8.
Jöreskog, K. G.Statistical analysis of sets of congeneric tests. Psychometrika, 1971, 36, 109-133.
9.
Jöreskog, K. H., Gruvaeus, G. T., and Van Thillo, M.ACOVS — A general computer program for analysis of covariance structures . Educational Testing Service, RB-70-15, Princeton, 1970.
10.
La Forge, R.Components of reliability. Psychometrika, 1965, 30, 187-195.
11.
Lord, F. M. and Novick, M. R.Statistical theories of mental test scores. Reading, Mass.:Addison-Wesley, 1968.
12.
Maxwell, A. E. and Pilliner, A. E. G.Deriving coefficients of reliability and agreement for ratings. British Journal of Mathematical and Statistical Psychology, 1968, 21, 105-116.
13.
Mellenbergh, G. J.Een onderzoek naar het beoordelen van open vragen. Nederlands Tijdschrift voor de Psychologie, 1971, 26, 102-120.
14.
Stanley, J. C.Analysis of unreplicated three-way classifications with applications to rater bias and trait idependence. Psychometrika, 1961, 26, 205-219.
