This study analyzed data from four evaluation designs incorporating the retrospective pretest (i.e., thentest), analyzing the effects of self-report pretesting and post-program survey format on a set of self-report measures. Validity of self-report data was assessed by comparing the criterion validity of then ratings to established benchmarks and by including a control condition. This study found that designs incorporating separate posttest and thentest surveys yielded the most comparable levels of criterion validity for then ratings and the least biased measures of program effectiveness. Conversely, designs that incorporated a single post-program survey, with adjacent post and then items, yielded the least comparable levels of criterion validity for then ratings and the most biased measures of program effectiveness. This study also found a slight interaction between the effects of self-report pretesting and post-program survey format. Implications for program evaluation are discussed.
Betz, D.L., & Hill, L.G. ( 2006). Real world evaluation. Journal of Extension , 44. Retrieved November 20, 2009 , from http://www.joe.org/joe/2006april/rb9.php
3.
Blanchard, K.H., Zigarmi, D., & Zigarmi, P. ( 1985). Leadership and the one minute manager. New York, NY: Morrow Publishers.
4.
Campbell, D.T., & Stanley, J.C. ( 1963). Experimental and quasi-experimental designs for research on teaching. In N. L. Gage (Ed.), Handbook of research on teaching. Chicago, IL: RandMcNally.
5.
Cohen, J. ( 1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
6.
Cronbach, L.J., & Furby, L. ( 1970). How should we measure ‘‘change’’ or should we? Psychological Bulletin, 74, 68-80.
7.
Deutsch, M., & Collins, M.E. ( 1951). Interracial housing: A psychological evaluation of a social experiment. Minneapolis, MN: University of Minnesota Press.
8.
Dunlap, W.P., Cortina, J.M., Vaslow, J.B., & Burke, M.J. ( 1996). Meta-analysis of experiments with matched groups or repeated measures. Psychological Methods, 1, 170-177.
9.
Grice, H.P. ( 1975). Logic and conversation. In P. Cole & J. L. Morgan (Eds.), Syntax and semantics (pp. 41-48). New York, NY: Academic Press.
10.
Haccoun, R.R., & Hamtiaux, T. ( 1994). Optimizing knowledge tests for inferring learning acquisition levels in single group training evaluation designs: The internal referencing strategy. Personnel Psychology, 47, 593-604.
11.
Hill, L.G., & Betz, D.L. ( 2005). Revisiting the retrospective pretest. American Journal of Evaluation, 26, 501-507.
12.
Hinkle, D.E., Wiersma, W., & Jurs, S.G. ( 2003). Applied statistics for the behavioral sciences (5th ed.). Boston: Houghton Mifflin.
13.
Hoogstraten, J. ( 1985). Influence of objective measures on self-reports in a retrospective pretest-posttest design. Journal of Experimental Education, 53, 207-210.
14.
Howard, G.S. ( 1980). Response-shift bias: A problem in evaluating programs with pre/post self-reports. Evaluation Review, 4, 93-106.
15.
Howard, G.S., Ralph, K.M., Gulanick, N.A., Maxwell, S.E., Nance, D.W., & Gerber, S.K. ( 1979). Internal validity in pretest-posttest self-report evaluations and a re-evaluation of retrospective pretests. Applied Psychological Measurement, 3, 1-23.
16.
Howard, G.S., Schmeck, R.R., & Bray, J.H. ( 1979). Internal validity in studies employing self-report instruments: A suggested remedy. Journal of Educational Measurement, 16, 129-135.
17.
Klatt, J., & Taylor-Powell, E. (2005, October). Synthesis of literature relative to the retrospective pretest design. Paper presented at the 2005 Joint CES/AEA Conference , Toronto, Canada.
18.
Lam, T.C.M., & Bengo, P. ( 2003). A comparison of three retrospective self-reporting methods of measuring change in instructional practice. American Journal of Evaluation, 24, 65-80.
19.
Lamb, T.A., & Tschillard, R. (2005, Spring). Evaluating learning in professional development workshops: Using the retrospective pretest. Journal of Research in Professional Learning. Retrieved October 19, 20005, from http://www.nsdc.org/library/publications/research/lamb.pdf
20.
Linn, R.L., & Slinde, J.A. ( 1977). The determination of the significance of change between pre- and post testing periods. Review of Educational Research , 47, 121-150.
21.
May, H. ( 2004). Making statistics more meaningful for policy research and program evaluation. American Journal of Evaluation, 25, 525-540.
22.
Mezoff, B. ( 1981). How to get accurate self-reports of training outcomes . Training and Development Journal, 35, 56-61.
23.
Nisbett, R.E., & Wilson, T.D. ( 1977). Telling more than we can know: Verbal reports on mental processes. Psychological Review, 84, 231-259.
24.
Pearson, R.W., Ross, M., & Dawes, R.M. ( 1992). Personal recall and the limits of retrospective questions in surveys. In J. M. Tanur (Ed.), Questions and survey questions, Meaning, memory, expression, and social interactions in survey (pp. 65-94). New York, NY: Russell Sage.
25.
Pelfrey, W.V., & Sr., & Pelfrey, W. V. Jr. (2009). Curriculum evaluation and revision in a nascent field: The utility of the retrospective pretest-posttest model in a homeland security program of study. Evaluation Review , 1, 54-82.
26.
Pitts, S.C., West, S.G., & Tein, J. ( 1996). Longitudinal measurement models in evaluation research: Examining stability and change. Evaluation and Program Planning , 19, 333-350.
27.
Pratt, C.C., McGuigan, W.M., & Katzev, A.R. ( 2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21, 341-349.
28.
Raidl, M., Johnson, S., Gardiner, K., Denham, M., Spain, K., & Lanting, R. ( 2004). Use retrospective surveys to obtain complete data sets and measure impact in extension programs. Journal of Extension , 42. Retrieved October 19, 2005 , from http://www.joe.org/joe/2004april/rb2.shtml
29.
Ross, M. ( 1989). Relation of implicit theories to the construction of personal histories. Psychological Review, 96, 341-357.
30.
Schwarz, N. ( 1996). Cognition and communication, Judgmental biases, research methods, and the logic of conversation. Mahwah, NJ: Lawrence Erlbaum.
31.
Schwarz, N. ( 1999). Self-reports: How the questions shape the answers. American Psychologist, 54, 93-105.
32.
Sears, R.R., Maccoby, E.E., & Levin, H. ( 1957). Patterns of child rearing. Evanston, IL: Row Peterson.
33.
Shipper, F. ( 1995). A study of the psychometric properties of the managerial skill scales of the survey of management practices. Educational and Psychological Measurement, 55, 468-479.
34.
Sprangers, M., & Hoogstraten, J. (1989). Pretesting effects in retrospective pretest-posttest designs. Journal of Applied Psychology, 74, 265-272.
35.
Taylor, P.J., Russ-Eft, D.F., & Taylor, H. ( 2009). Gilding the outcome by tarnishing the past: Inflationary biases in retrospective pretest. American Journal of Evaluation , 30, 31-34.
36.
Terborg, J.R., & Davis, G.A. ( 1982). Evaluation of a new method for assessing change to planned job redesign as applied to Hackman and Oldham’s job characteristic model . Organizational Behavior and Human Performance, 29, 112-128.
37.
Thorndike, R.L. ( 1949). Personnel selection. New York, NY : John Wiley.
38.
Umble, K., Upshaw, V., Orton, S., & Matthews, K. ( 2000, June). Using the post-then method to assess learner change . Presentation at the AAHE Assessment Conference , Charlotte, NC, USA.
39.
Walk, R.D. ( 1956). Self-ratings of fear in a fear-invoking situation. Journal of Abnormal and Social Psychology, 52, 171-178.
40.
Willis, G. ( 2005). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: Sage .
41.
Wilson, C.L. ( 1978). The Wilson multi-level management surveys: Refinement and replication of the scales. JSAS: Catalog of Selected Documents in Psychology, 8, 1707. Washington, DC: American Psychology Association.
42.
Wilson, C.L. ( 2006). The Clark Wilson group surveys: Management practices . Silver Spring, MD: The Clark Wilson Group .
43.
Zigarmi, D., Edeburn, C., & Blanchard, K. ( 1997). Getting to know the LBAII®, Research, validity and reliability of the self and other forms (4th ed.). Escondido, CA: Blanchard Training and Development, Inc.
