We tested for inflationary bias introduced through retrospective pretests by analyzing traditional pretest, retrospective pretest, and posttest evaluation data collected on a first-line supervisory leadership training program, involving 196 supervisors and their subordinates, across 17 organizational settings. Retrospective pretest ratings by both trained (supervisors) and untrained (subordinates) respondents were significantly lower than traditional pretest ratings, resulting in substantially inflated training effect sizes when posttests were compared with retrospective pretests rather than with traditional pretests. Further analysis revealed evidence of both respondents' application of an implicit theory of change (i.e., assumption that posttraining scores should generally be higher than pretraining scores) and a tendency to rate their own improvement as greater than that of others. Implications for program evaluation are discussed.
Birkenbach, X.C. (1986). Measuring alpha, beta, and gamma change. South African Journal of Psychology, 16, 1-7.
2.
Bray, J.H., & Howard, G.S. (1980). Methodological considerations in the evaluation of a teacher-training program. Journal of Educational Psychology , 72, 62-70.
3.
Campbell, D.T., & Stanley, J.C. (1963). Experimental and quasi-experimental designs for research. Boston, MA: Houghton Mifflin .
4.
Conway, M., & Ross, M. (1984). Getting what you want by revising what you had . Journal of Personality and Social Psychology, 47, 738-748.
5.
Cook, T.D., & Campbell, D.T. (1979). Quasi-experimentation: Design & analysis issues for field settings. Boston, MA: Houghton Mifflin.
6.
Cook, T.D., Campbell, D.T., & Peracchio, L. (1990). Quasi experimentation. In M. D, Dunnette & L. M, Hough (Eds.), Handbook of industrial and organizational psychology (2nd ed., Vol. 1, pp. 491-576). Palo Alto, CA: Consulting Psychologists Press.
7.
Decker, P.J., & Nathan, B.R. (1985). Behavior modeling training: Principles and applications . New York: Praeger.
8.
Goldstein, A.P., & Sorcher, M.A. (1974). Changing supervisory behavior. New York: Pergoman Press.
9.
Golembiewski, R.T., Billingsley, K., & Yeager, S. (1976). Measuring change and persistence in human affairs: Types of change generated by OD designs. Journal of Applied Behavioral Science, 12, 133-157.
10.
Hill, L.G., & Betz, D.L. (2005). Revisiting the retrospective pretest. American Journal of Evaluation, 26, 501-517.
11.
Hoogstraten, J. (1982). The retrospective pretest in an educational training context. Journal of Experimental Education, 50, 200-204.
12.
Hoogstraten, J. (1985). Influence of objective measures on self-reports in a retrospective pretest-posttest design. Journal of Experimental Education, 53, 207-210.
13.
Howard, G.S., & Dailey, P.R. (1979). Response-shift bias: A source of contamination of self-report measures. Journal of Applied Psychology, 64, 144-150.
14.
Howard, G.S., Dailey, P.R., & Gulanick, N.A. (1979). The feasibility of informed pretests in attenuating response-shift bias. Applied Psychology Measurement, 3, 1-23.
15.
Howard, G.S., Ralph, K.M., Gulanick, N.A., Maxwell, S.E., Nance, D.W., & Gerber, S.K. (1979). Internal invalidity in pretest-posttest self-report evaluations and a re-evaluation of retrospective pretests. Applied Psychological Measurement, 3, 1-23.
16.
Howard, G.S., Schmeck, R.R., & Bray, J.H. (1979). Internal invalidity in studies employing self-report instruments: A suggested remedy. Journal of Educational Measurement , 16, 129-135.
17.
Krosnick, J.A. (1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213-236.
18.
Lam, T.C.M., & Bengo, P. (2003). A comparison of three retrospective self-reporting methods of measuring change in instructional practice. American Journal of Evaluation, 24, 65-80.
19.
Menon, G. (1994). Judgments of behavioral frequencies: Memory search and retrieval strategies. In Schwarz. N, & Sudman. S (Eds.), Autobiographical memory and the validity of retrospective reports (pp. 161-172). New York: Springer-Verlag.
20.
Paulhus, D.L. (2002). Socially desirable responding: The evolution of a construct. In H. I. Braun, & D. N. Jackson (Eds.), The role of constructs in psychological and educational measurement (pp. 49-69). Mahwah, NJ: Lawrence Erlbaum.
21.
Pohl, N.F. (1982). Using retrospective pre-ratings to counteract response-shift confounding. Journal of Experimental Education , 51, 211-214.
22.
Pratt, C.C., Mcguigan, W.M., & Katzev, A.R. (2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21, 341-350.
23.
Rhodes, J.E., & Jason, L.A. (1987). The retrospective pretest: An alternative approach in evaluating drug intervention programs. Journal of Drug Education, 17, 345-356.
24.
Rice, R.E., & Contractor, N.S. (1990). Conceptualizing effects of office information systems: A methodology and application for the study of alpha, beta, and gamma changes. Decision Sciences, 21, 301-317.
25.
Ross, M. (1989). Relation of implicit theories to the construction of personal histories. Psychological Review, 96, 341-357.
26.
Schwarz, N. (2007). Cognitive aspects of survey methodology. Applied Cognitive Psychology, 21, 277-287.
27.
Schwarz, N., & Oyserman, D. (2001). Asking questions about behavior: Cognition, communication, and questionnaire construction. American Journal of Evaluation , 22, 127-160.
28.
N. Schwarz, & S. Sudman (Eds.), (1994). Autobiographical memory and the validity of retrospective reports. New York: Springer-Verlag.
29.
Sibthorp, J., Paisley, K., Gookin, J., & Ward, P. (2007). Addressing response-shift bias: Retrospective pretests in recreation research and evaluation. Journal of Leisure Research, 39, 295-315.
30.
Sprangers, M., & Hoogstraten, J. (1988). On delay and reassessment of retrospective preratings. Journal of Experimental Education , 56, 148-153.
31.
Sprangers, M., & Hoogstraten, J. (1989). Pretesting effects in retrospective pretest-posttest designs. Journal of Applied Psychology, 74, 265-272.
32.
Sprangers, M.A.G., & Hoogstraten, J. (1991). Subject bias in three self-report measures of change. Methodika, 5, 1-13.
33.
Stieglitz, R.-D. (1990). Studies on the validity of the retrospective pretest in therapy research. Zeitschrift fur Klinische Psychologie , 19, 144-150.
34.
Taylor, P.J., Russ-Eft, D., & Taylor, H. (in press). The transfer of management training from alternative perspectives . Journal of Applied Psychology.
35.
Terborg, J.R., & Davis, G.A. (1982). Evaluation of a new method for assessing change to planned job redesign and applied to Hackman and Olham's job characteristics model. Organizational Behavior and Human Performance, 29, 112-128.
36.
Tourangeau, R. (2000). Remembering what happened: Memory errors and survey reports. In Stone. A. A, Turkkan. J. S, Bachrach. C. A, Jobe. J. B, Kurtzman. H. S, & Cain. V. S (Eds.), The science of self-report: Implications for research and practice. Mahwah, NJ: Lawrence Erlbaum.
37.
Tourangeau, R., Rips, L.J., & Rasinski, K. (2000). The psychology of survey response. Cambridge: Cambridge University Press.
38.
Tversky, A., & Kahneman, D. (1974). Judgment under uncertainty: Heuristics and biases . Science, 185, 1124-1131.
39.
Wilson, A.E., & Ross, M. (2001). From chump to champ: People's appraisals of their earlier and present selves. Journal of Personality and Social Psychology, 80, 572-584.
40.
Wilson, D.B., & Lipsey, M.W. (2001). The role of method in treatment effectiveness research: Evidence from meta-analysis. Psychological Methods , 6, 413-429.
