Based on recently published papers, it might be tempting to routinely apply the beta-binomial model to obtain a single administration estimate of the reliability of a mastery test. Using real data, the paper illustrates two practical problems with esti mating reliability in this manner. The first is that the model might give a poor fit to data, which can seriously affect the reliability estimate, and the sec ond is that inadmissible estimates of the param eters in the beta-binomial model might be ob tained. Two possible solutions are described and il lustrated.
Algina, J., & Noe, M.J.A study of the accuracy of Subkoviak's single administration estimate of the coefficient of agreement using two true-score estimates. Journal of Educational Measurement, 1978, 15, 101-110.
3.
Cohen, J.A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 1960, 20, 37-46.
4.
Divgi, D.R.Group dependence of some reliability indices for mastery tests. Applied Psychological Measurement, 1980, 4, 213-218.
5.
Griffiths, D.A.Maximum likelihood estimation for the beta-binomial distribution and an application to the household distribution of the total number of cases of a disease. Biometrics, 1973, 29, 637-648.
6.
Gross, A.L., & Shulman, V.The applicability of the beta-binomial for criterion-referenced testing. Journal of Educational Measurement, 1980, 17, 195-202.
7.
Hambleton, R.K., Swaminathan, H., Algina, J., & Coulson, D.Criterion-referenced testing and measurement : A review of technical issues and developments. Review of Educational Research, 1978, 48, 1-47.
8.
Huynh, H.On the reliability of decisions in domain-referenced testing. Journal of Educational Measurement, 1976,13, 253-264.
9.
Huynh, H.Statistical inference for two reliability indices in mastery testing based on the beta-binomial model. Journal of Educational Statistics , 1979, 4, 231-246.
10.
Irwin, J.O.The generalized Waring distribution applied to accident data. Journal of the Royal Statistical Society, Series A, 1968,131, 205-225.
11.
Johnson, N., & Kotz, S.Discrete distributions. New York: Wiley, 1969.
12.
Keats, J.A.Some generalizations of a theoretical distribution of mental test scores. Psychometrika, 1964, 29, 215-231.
13.
Lord, F.M., & Novick, M.R.Statistical theories of mental test scores. Reading MA : Addison-Wesley, 1968.
14.
Marshall, J.L., & Haertel, E.H.A single-administration reliability index for criterion-referenced tests: The mean split-half coefficient of agreement. Paper presented at the annual meeting of the American Educational Research Association , April 1975.
15.
Sibuya, M.Generalized hypergeometric, digamma, and trigamma distributions. Annals of the Institute of Statistical Mathematics, 1979, 31, 373-390.
16.
Subkoviak, M.J.Estimating reliability from a single administration of a criterion-referenced test. Journal of Educational Measurement, 1976, 13, 265-276.
17.
Subkoviak, M.Decision-consistency approaches. In R. Berk (Ed.), Criterion-referenced measurement: The state of the art. Baltimore: The Johns Hopkins University Press, 1980.
18.
Traub, R., & Rowley, G.L.Reliability of test scores and decisions. Applied Psychological Measurement, 1980, 4, 517-545.
19.
Wilcox, R.R.Estimating the parameters of the beta-binomial distribution. Educational and Psychological Measurement, 1979, 31, 527-535.
20.
Wilcox, R.R.A review of the beta-binomial model and its extensions. Journal of Educational Statistics, in press.
