Pre/post designs that rely on self-report measures are potentially contaminated by response shift
bias, a change in respondents' understanding of the phenomena being measured between the
pre- and posttests. Retrospective pretests, or then tests, have been shown to be useful in
identifying response shifts. Pre/post/then designs are highly utilitarian designs that have the
potential of providing more accurate assessments of respondent change. This article reviews the
literature on response shift bias and the utility of retrospective pretests m detecting such bias.
Three studies that used the pre/post/then design to evaluate self-reported changes in caseworker
abilities (knowledge or skill) relative to practice evaluation, basic interviewing skills, and/or
assessment are described.
Get full access to this article
View all access options for this article.
References
1.
Aiken, L.S. (1986). Retrospective self-reports by clients differ from original reports: Implications for the evaluation of drug treatment programs . The International Journal of the Addictions, 21(7), 767-788.
2.
Benjamin, E.R. (1982). Using the post-then method of evaluation. Training, 19, 72.
3.
Bloom, M., & Fischer, J. (1982). Evaluating practice: Guidelines for the accountable professional. Englewood Cliffs, NJ:Prentice-Hall.
4.
Bray, J.H., Maxwell, S.E., & Howard, G.S. (1984). Methods of analysis with response-shift bias . Educational and Psychological Measurement, 44, 781-804.
5.
Brown, B.W., Jr., & Hollander, M. (1977). Statistics: A biomedical introduction. New York: Wiley.
6.
Campbell, D.T., & Stanley, J.C. (1966). Experimental and quasi-experimental designs for research. Chicago: Rand McNally .
7.
Cook, T.D., & Campbell, D.T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Chicago : Rand McNally.
8.
Doueck, H.J., & Bondanza, A. (1990). Training social work staff to evaluate practice: A pre/post/then comparison. Administration in Social Work , 14(1), 119-133.
9.
Hays, W.L. (1973). Statistics for the social sciences (2nd ed.). New York: Holt, Rinehart and Winston.
10.
Howard, G.S. (1980). Response-shift bias: A problem in evaluating interventions with pre/post self-reports. Evaluation Review, 4(1), 93-106.
11.
Howard, G.S., & Dailey, P.R. (1979). Response shift bias: A source of contamination of self report measures. Journal of Applied Psychology, 64(2), 144-150.
12.
Howard, G.S., Dailey, P.R., & Gulanick, N.A. (1979). The feasibility of informed pretests in attenuating response shift bias. Applied Psychological Measurement, 3(4), 481-494
13.
Howard, G.S., Milham, J., Slaten, S., & O'Donnell, L. (1981). Influence of subject response style effects on retrospective measures. Applied Psychological Measurement , 5(1), 89-100.
14.
Howard, G.S., Ralph, K.M., Gulanick, N.A., Maxwell, S.E., Nance, D.W., & Gerber, S.K. (1979). Internal invalidity in pretest-posttest self-report evaluations and a re-evaluation of retrospective pretests. Applied Psychological Measurement, 3(1), 1-23.
15.
Howard, G.S., Schmeck, R.R., & Bray, J.H. (1979). Internal invalidity in studies employing self-report instruments: A suggested remedy. Journal of Educational Measurement , 16, 129-135.
16.
Koele, P., & Hoogstraten, J. (1988a). A method for analyzing retrospective pretest/posttest designs: I. Theory. Bulletin of the Psychonomic Society, 26(1), 51-54.
17.
Koele, P., & Hoogstraten, J. (1988b). A method for analyzing retrospective pretest/posttest designs: II. Application. Bulletin of the Psychonomic Society, 26(2), 124-125.
18.
Nisbett, R.E., & Wilson, T.D. (1977). "Telling more than we can know:" Verbal reports on mental processes. Psychological Review, 84, 231-259.
19.
Rhodes, J.E., & Jason, L.A. (1987). The retrospective pretest: An alternative approach in evaluating drug prevention programs. Journal of Drug Education , 17(4), 345-355.
20.
Rubin, A. (1992, May). Methodological issues in using single-case designs to evaluate groups for battered women. Paper presented at the 8th Annual Symposium on Empirical Foundations of Group Work, Madison, WI.
21.
Sprangers, M. (1988). Subject bias and the retrospective pretest in retrospect. Bulletin of the Psychonomic Society, 27(1), 11-14.
22.
Sprangers, M., & Hoogstraten, J. (1987). Response-style effects, response-shift bias, and a bogus-pipeline. Psychological Reports, 61, 579-585.
23.
Sprangers, M., & Hoogstraten, J. (1988). Response-style effects, response-shift bias, and a bogus-pipeline: A replication. Psychological Reports, 62, 11-16.
24.
Sprangers, M., & Hoogstraten, J. (1989). Pretesting effects in retrospective pretest-posttest designs. Journal of Applied Psychology, 74(2), 265-272.
