The precision of simple difference or “gain” scores is described in terms of their confidence intervals on the latent trait scale and of significance probabilities under the H₀ of no change. For this, two approaches are compared: one employs the asymptotic normal distribution of the maximum likelihood estimator of the person parameter, the other is based on the exact conditional distribution of the gain score, given the total number-correct score over the two time points. In either case, a detailed assessment of the precision of change measurements results. For illustration, results are presented of three test scales. The present methods yield more relevant and much more detailed psychometric information than the traditional estimation of reliability as a sole indicator of measurement precision. Other areas of application, namely, the comparison of the abilities of two examinees or the aggregation of individual signi.cance levels within groups of examinees, are also mentioned.
Agresti, A., & Coull, B.A. (1998). Approximate is better than “exact” for interval estimation of binomial proportions. American Statistician, 52, 119-125.
2.
Andersen, E.B. (1973). Conditional inference and models for measuring. Copenhagen, Denmark: Mentalhygiejnisk Forlag.
3.
Andersen, E.B. (1980). Discrete statistical models with social science applications. Amsterdam: North-Holland.
4.
Andersen, E.B. (1995). Polytomous Rasch models and their estimation. In G.H. Fischer & I.W. Molenaar (Eds.), Rasch models: Foundations, recent developments, and applications (pp. 271-291). New York: Springer-Verlag.
5.
Andrich, D. (1978). A rating formulation for ordered response categories. Psychometrika, 43, 561-573.
6.
Anscombe, F.J. (1948). The validity of comparative experiments. Journal of the Royal Statistical Society, Series A, 181-200.
7.
Barndorff-Nielsen, O. (1978). Information and exponential families in statistical theory. New York: Wiley.
8.
Chang, H.-H. (1996). The asymptotic posterior normality of the latent trait for polytomous items. Psychometrika, 61, 445-463.
9.
Chang, H.-H., & Stout, W. (1993). The asymptotic posterior normality of the latent trait in an IRT model. Psychometrika, 58, 37-52.
10.
Christodoulides, P. (1993). Individuelle politische Profile und der Einfluss der politischen Propaganda auf die politische Ideologie [Individual political profiles and the effects of political propaganda on political ideology]. Unpublished master’s thesis, University of Vienna.
11.
Collins, L.M. (1996). Is reliability obsolete? A commentary on “Are simple gain scores obsolete?”Applied Psychological Measurement, 20, 289-292.
12.
Colonius, H. (1979). Zur Eindeutigkeit der Parameter im Rasch-Modell [On the uniqueness of the parameters in the Rasch model]. Psychologische Beiträge, 21, 414-416.
13.
Cronbach, L.J., & Furby, L. (1970). How should we measure change, or should we?Psychological Bulletin, 74, 68-80.
14.
Eddington, A.S. (1913). On a formula for correcting statistics for the effects of a known probable error of observation. Royal Astronomical Society Monthly Notices, 73, 359-360.
15.
Eggen, T.J.H.M. (2000). On the loss of information in conditional maximum likelihood estimation of item parameters. Psychometrika, 65, 337-362.
16.
Embretson, S.E. (1991). A multidimensional latent trait model for measuring learning and change. Psychometrika, 56, 495-515.
17.
Fischer, G.H. (1995a). Derivations of polytomous Rasch models. In G.H. Fischer & I.W. Molenaar (Eds.), Rasch models: Foundations, recent developments, and applications (pp. 293-305). New York: Springer-Verlag.
18.
Fischer, G.H. (1995b). Derivations of the Rasch model. In G.H. Fischer & I.W. Molenaar (Eds.), Rasch models: Foundations, recent developments, and applications (pp. 15-38). New York: Springer-Verlag.
19.
Fischer, G.H. (1995c). Linear logistic models for change. In G.H. Fischer & I.W. Molenaar (Eds.), Rasch models: Foundations, recent developments, and applications (pp. 157-180). New York: Springer-Verlag.
20.
Fischer, G.H. (1995d). Some neglected problems in IRT. Psychometrika, 60, 459-487.
21.
Fischer, G.H. (1997). Unidimensional linear logistic Rasch models. In W.J. van der Linden & R.K. Hambleton (Eds.), Handbook of modern item response theory. New York: Springer-Verlag.
22.
Fischer, G.H. (2001). Gain scores revisited under an IRT perspective. In A. Boomsma, M.A.J. Van Duijn, & T.A.B. Snijders (Eds.), Essays on item response theory (pp. 43-68) (Lecture Notes in Statistics 157). New York: Springer-Verlag.
23.
Fischer, G.H., & Ponocny-Seliger, E. (1998). Structural Rasch modeling. Handbook of the usage of LPCM-WIN 1.0. Groningen, the Netherlands: ProGAMMA.
24.
Fischer, G.H., & Prieler, J.A. (2000). An IRT methodology for the assessment of change. In J. Raven, J.C. Raven, & J.H. Court, Standard progressive matrices. Raven Manual: Section 3 (pp. 113-121). Oxford: Oxford Psychologists Press.
25.
Fisher, R.A. (1932). Statistical methods for research workers (5th ed.). Edinburgh, UK: Oliver & Boyd.
26.
Gittler, G. (1990). Dreidimensionaler Würfeltest [The three-dimensional cubes test]. Weinheim, Germany: Beltz Test.
27.
Gittler, G. (1992). Testpsychologische Aspekte der Raumvorstellungsforschung. Kritik, Lösungsansätze und Befunde [Psychometric issues in spatial ability research. Critique, theoretical considerations, and empirical findings]. Weinheim, Germany: Beltz Test.
28.
Gittler, G. (1994). Intelligenzförderung durch Schulunterricht: Darstellende Geometrie und räumliches Vorstellungsvermögen [The promotion of intelligence via teaching: Mechanical geometry and spatial abilities]. In G. Gittler, M. Jirasko, U. Kastner-Koller, C. Korunka, & A. Al-Roubaie (Eds.), Die Seele ist ein weites Land [The mind is a wide country] (pp. 105-122). Vienna: WUV-Wiener Universitätsverlag.
29.
Glas, C.A.W., & Verhelst, N.D. (1995). Tests of fit for polytomous Rasch models. In G.H. Fischer & I.W. Molenaar (Eds.), Rasch models: Foundations, recent developments, and applications (pp. 325-352). New York: Springer-Verlag.
30.
Gulliksen, H. (1950). Theory of mental tests. New York: Wiley.
31.
Hamerle, A. (1979). Über die messtheoretischen Grundlagen von Latent-Trait Modellen [On the measurement-theoretic foundation of latent trait models]. Archiv für Psychologie, 132, 19-39.
32.
Harris, C.W. (Ed.). (1963). Problems in measuring change. Madison: University of Wisconsin Press.
33.
Kane, M. (1996). The precision of measurements. Applied Measurement in Education, 9, 355-379.
34.
Klauer, K.C. (1991). An exact and optimal standardized person test for assessing consistency with the Rasch model. Psychometrika, 56, 213-228.
35.
Lancaster, H.O. (1961). Significance tests in discrete distributions. Journal of the American Statistical Association, 56, 223-234.
36.
Lehmann, E.L. (1986). Testing statistical hypotheses (2nd ed.). New York: John Wiley.
37.
Liou, M. (1993). Exact person tests for assessing model-data fit in the Rasch model. Applied Psychological Measurement, 17, 187-195.
38.
Liou, M., & Chang, C.-H. (1992). Constructing the exact significance level for a person fit statistic. Psychometrika, 47, 169-181.
39.
Lord, F.M. (1956). The measurement of growth. Educational and Psychological Measurement, 16, 421-437.
40.
Lord, F.M. (1963). Elementary models for measuring change. In C.W. Harris (Ed.), Problems in measuring change (pp. 21-38). Madison: University of Wisconsin Press.
41.
Lord, F.M., & Novick, M.R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
42.
Kane, M. (1996). The precision of measurement. Applied Measurement in Education, 9, 335-379.
43.
Masters, G.N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149-174.
44.
Mellenbergh, G.J. (1999). A note on simple gain score precision. Applied Psychological Measurement, 23, 87-89.
45.
Mellenbergh, G.J., & Van den Brink, W.P. (1996). The measurement of individual change. Psychological Methods, 3, 470-485.
46.
Mood, A.M., Graybill, F.A., & Boes, D.C. (1974). Introduction to the theory of statistics. Singapore: McGraw-Hill.
47.
Muraki, E. (1993). Information functions of the graded partial credit model. Applied Psychological Measurement, 17, 351-363.
48.
Overall, J.E., & Woodward, J.A. (1975). Unreliability of difference scores: A paradox for measurement of change. Psychological Bulletin, 82, 85-86.
49.
Pfanzagl, J. (1971). Theory of measurement. Würzburg, Germany: Physica-Verlag.
50.
Ponocny, I. (2000). Exact person fit indexes for the Rasch model for arbitrary alternatives. Psychometrika, 65, 29-42.
51.
Ponocny, I. (2001). Non-parametric goodness-of-fit tests for the Rasch model. Psychometrika, 66, 437-460.
52.
Ponocny, I., & Ponocny-Seliger, E. (1999). T-Rasch 1.0. Groningen, the Netherlands: ProGAMMA.
53.
Prieler, J. (2000). Evaluation eines Ausleseverfahrens für Unteroffiziere beim Österreichischen Bundesheer [Evaluation of a selection procedure for noncommissioned officers in the Austrian army]. Unpublished dissertation, University of Vienna, Department of Psychology.
54.
Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen: The Danish Institute of Educational Research. (Expanded edition, 1980, Chicago: University of Chicago Press)
55.
Rasch, G. (1961). On general laws and the meaning of measurement in psychology. In Proceedings of the IV. Berkeley Symposium on mathematical statistics and probability; Vol. IV (pp. 321-333). Berkeley: University of California Press.
56.
Rasch, G. (1965). Statistisk Seminar [Statistical seminar] (Notes taken by J. Stene). Copenhagen: University of Copenhagen, Department of Statistics.
57.
Rogosa, D.R., & Willett, J.B. (1983). Demonstrating the reliability of the difference score in the measurement of change. Journal of Educational Measurement, 20, 335-343.
58.
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika Monograph (Suppl. No. 17).
59.
Santner, T.J., & Duffy, D.E. (1989). The statistical analysis of discrete data. New York: Springer-Verlag.
Schervish, M.J. (1996). P values: What they are and what not. The American Statistician, 50, 203-206.
62.
Stevens, W.L. (1950). Fiducial limits of the parameter of a discontinuous distribution. Biometrika, 37, 117-129.
63.
Tanzer, N.K., Gittler, G., & Ellis, B.B. (1995). Cross-cultural validation of item complexity in a LLTM-calibrated spatial test. European Journal of Psychological Assessment, 11, 170-183.
64.
Tanzer, N.K., Gittler, G., & Sim, C.Q.E. (1994). A cross-cultural comparison of a Rasch calibrated spatial ability test between Austrian and Singaporean adolescents. In A.M. Bouvy, F.J.R. van de Vijver, P. Boski, & P. Schmitz (Eds.), Journey into cross-cultural psychology. Selected papers from the Eleventh International Conference of the International Association for Cross-Cultural Psychology in Liège, Belgium, 14.-18.7.1992 (pp. 96-110). Amsterdam: Swets & Zeitlinger.
65.
Trumpler, R.J., & Weaver, H.F. (1953). Statistical astronomy. Berkeley: University of California Press.
66.
Vollset, S.E. (1993). Confidence intervals for a binomial proportion. Statistics in Medicine, 12, 809-824.
67.
Willett, J.B. (1989). Some results on reliability for the longitudinal measurement of change: Implications for the design of studies of individual growth. Educational and Psychological Measurement, 49, 587-602
68.
Williams, R.H., & Zimmerman, D.W. (1996). Are simple gain scores obsolete?Applied Psychological Measurement, 20, 59-69.
69.
Witting, H. (1985). Mathematische Statistik I [Mathematical statistics, Vol. I]. Stuttgart, Germany: Teubner.
Zimmerman, D.W., & Williams, R.H. (1982a). Gain scores in research can be highly reliable. Journal of Educational Measurement, 19, 149-154.
72.
Zimmerman, D.W., & Williams, R.H. (1982b). On the high predictive potential of change and growth measures. Educational and Psychological Measurement, 42, 961-968.
