Clinical, medical, and health psychologists use difference scores obtained from pretest–posttest designs employing the same test to assess intraindividual change possibly caused by an intervention addressing, for example, anxiety, depression, eating disorder, or addiction. Reliability of difference scores is important for interpreting observed change. This article compares the well-documented traditional method and the unfamiliar, rarely used item-level method for estimating difference-score reliability. We simulated data under various conditions that are typical of change assessment in pretest–posttest designs. The item-level method had smaller bias and greater precision than the traditional method and may be recommended for practical use.
BereiterC. (1963). Some persisting dilemmas in the measurement of change. In HarrisC. W. (Ed.), Problems in measuring change (pp. 3–20). University of Wisconsin Press.
CohenJ.OnunakuN.ClothierS.PoppeJ. (2005). Helping young children succeed: Strategies to promote early childhood social and emotional development[Research and policy report]. National Conference of State Legislature.
8.
CronbachL. J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16(3), 297–334. https://doi.org/10.1007/BF02310555
9.
CronbachL. J.FurbyL. (1970). How we should measure “change”: Or should we?Psychological Bulletin, 74(1), 68–80. https://doi.org/10.1037/h0029382
GuZ.EmonsW. H. M.SijtsmaK. (2018). Review of issues about classical change scores: A multilevel modeling perspective on some enduring beliefs. Psychometrika, 83(3), 674–695. https://doi.org/10.1007/s11336-018-9611-3
HalvorsenM. S.HagtvetK. A.MonsenJ. T. (2006). The reliability of self-image change scores in psychotherapy research: An application of generalizability theory. Psychotherapy: Theory, Research, Practice, Training, 43(3), 308–321. https://doi.org/10.1037/0033-3204.43.3.308
14.
JabrayilovR. R.EmonsW. H. M.SijtsmaK. (2016). Comparison of classical test theory and item response theory in individual change assessment. Applied Psychological Measurement, 40(8), 559–572. https://doi.org/http://doi.org/10.1177/0146621616664046
15.
JacksonP. H.AgunwambaC. C. (1977). Lower bounds for the reliability of the total score on a test composed of non-homogeneous items: I: Algebraic lower bounds. Psychometrika, 42(4), 567–578. https://doi.org/10.1007/BF02295979
16.
LambertM. J.HansenN. B.UmphressV.LunnenK.OkiishiJ.BurlingameG.ReisingerC. W. (1996). Administration and Scoring Manual for the OQ-45.2. American Professional Credentialing Services.
17.
LambertM. J.WhippleJ. L.HawkinsE. J.VermeerschD. A.NielsenS. L.SmartD. W. (2003). Is it time for clinicians to routinely track patient outcome? A meta-analysis. Clinical Psychology Science Practice, 10, 288–302.
18.
LindgrenB. (1993). Statistical theory (4th ed.). Chapman Hall/CRC Press.
19.
LordF. M.NovickM. R. (1968). Statistical theories of mental test scores. Information Age.
20.
McConnelK.ValdesS. (1998). Testing temporal reliability and carry-over effect: The role of correlated responses in test-retest reliability studies. Environmental and Resource Economics, 12, 357–374. https://doi.org/10.1023/A:1008264922331
21.
MillerT. B.KaneM. (2001). The precision of change scores under absolute and relative interpretations. Applied Measurement in Education, 14(4), 307–327. https://doi.org/10.1207/S15324818AME1404_1
OosterwijkP. R.van der ArkL. A.SijtsmaK. (2016). Numerical differences between Guttman’s reliability coefficients and the GLB. In Van der ArkL.BoltD.WangW.-C.DouglasM. (Eds.), Quantitative psychology research: The 80th Annual Meeting of the Psychometric Society, Beijing, China, 2015 (pp. 155–172). Springer. https://doi.org/10.1007/978-3-319-38759-8_12
24.
OosterwijkP. R.van der ArkL. A.SijtsmaK. (2017). Overestimation of reliability by Guttman’s λ4, λ5, and λ6 and the greatest lower bound. In van der ArkL. A.WibergM.CulpepperS. A.DouglasJ. A.WangW. C. (Eds.), Quantitative psychology. IMPS 2016. Springer Proceedings in Mathematics & Statistics (Vol. 196, pp. 159–172). Springer. https://doi.org/10.1007/978-3-319-56294-0_15
25.
RaykovT. (2001). Bias of coefficient for fixed congeneric measures with correlated errors. Applied Psychological Measurement, 25, 69–76. https://doi.org/10.1177/01466216010251005
26.
RogosaD. R.BrandtD.ZimowskiM. (1982). A growth curve approach to the measurement of change. Psychological Bulletin, 92(3), 726–748. https://doi.org/10.1037/0033-2909.92.3.726
27.
RogosaD. R.WillettJ. B. (1983). Demonstrating the reliability of the difference score in the measurement of change. Journal of Educational Measurement, 20(4), 335–343. https://doi.org/10.1111/j.1745-3984.1983.tb00211.x
28.
SamejimaF. (1970). Estimation of latent ability using a response pattern of graded scores. Psychometrika, 35(1), 139–139. https://doi.org/10.1007/BF02290599
29.
SijtsmaK. (2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74, 107–120. https://doi.org/10.1007/s11336-008-9101-0
30.
TabogaM. (2012). Lectures on probablity theory and mathematical statistics (2nd ed.). CreateSpace Independent Publishing Platform.
31.
Ten BergeJ. M. F.SočanG. (2004). The greatest lower bound to the reliability of a test and the hypothesis of unidimensionality. Psychometrika, 69(4), 613–625. https://doi.org/10.1007/BF02289858
32.
WoodwardJ. A.BentlerP. M. (1978). A statistical lower bound to population reliability. Psychological Bulletin, 85(6), 1323–1326. https://doi.org/10.1037/0033-2909.85.6.1323
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
