Measuring Change in a Short-Term Educational Program Using a Retrospective Pretest Design

Abstract

The Pennsylvania Governor's School for International Studies is an intensive summer program designed to give talented high school students a challenging introduction to the study of international affairs. One focus of the evaluation seeks to understand the effect of the program on the students' perception of their knowledge concerning core issues. Across the long history of the program, a variety of measures were used (and subsequently discarded) to assess changes in knowledge and perception of competence. Four years ago the program instituted a retrospective pre-post design. Results from these years, indicate that these students have consistently overestimated their pre-test understanding of core competencies emphasized in the program and that they seem better able to assess their knowledge gains and their initial inflated sense of knowledge as a result of the program. This article offers an overview of the development, application, use and analysis of a retrospective pre-post instrument to address response shift bias.

Keywords

pre—post test response-shift bias retrospective pre—post test gifted education perceived competence

Get full access to this article

View all access options for this article.

References

Bamberger, M. , Rugh, J. , Church, M. , & Fort, L. (2004). Shoestring evaluation: Designing impact evaluations under budget, time, and data constraints. American Journal of Evaluation, 25, 5-37.

Bandura, A. (1977). Self-efficacy: Toward a unifying theory of behavioral change. Psychological Review, 84, 191-215.

Bray, J.H. , Maxwell, S.E. , & Howard, G.S. (1984). Methods of analysis with response shift bias. Educational and Psychological Measurement, 44, 781-804.

Campbell, D.T. , & Stanley, J.C. (1963). Experimental and quasi-experimental designs for research. Boston: Houghton Mifflin Co.

Cantrell, P. (2003). Traditional vs. retrospective pretests for measuring science teaching efficacy beliefs in preservice teachers. School Science and Mathematics, 103, 177-185.

Choi, N. (2004). Sex role group differences in specifying academic and general self-efficacy. The Journal of Psychology, 138, 149-159.

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.

Cooke, B. (1998). Beyond basic training. Evaluation Exchange, 4, 8-9.

Cronbach, L.J. (1951). Coefficient alpha and the internal structure of tests. Psychometrika, 16, 297-334.

10.

D'Eon, M. , Sadownik, L. , Harrison, A. , & Nation, J. (2008). Using self-assessments to detect workshop success: Do they work? American Journal of Evaluation, 29, 92-98.

11.

Gutek, B.A. , & Winter, S.J. (1992). Consistency of job satisfaction across situations: Fact or framing artifact? Journal of Vocational Behavior, 41, 61-78.

12.

Hill, L.G. & Betz, D.L. (2005). Revisiting the retrospective pretest. American Journal of Evaluation, 26, 501-517.

13.

Hoogstraten, J. (1982). The retrospective pre-test in an educational training context. Journal of Experimental Education, 50, 200-204.

14.

Howard, G.S. (1980). Response-shift bias: A problem in evaluating interventions with pre/post self-reports. Evaluation Review, 4, 93-106.

15.

Howard, G.S. , Ralph, K.M. , Gulanick, N.A. , Maxwell, S.E. , Nance, D.W. , & Gerber, S.K. (1979). Internal invalidity in pretest-posttest self-report evaluations and a re-evaluation of retrospective pretests. Applied Psychological Measurement, 3, 1-23.

16.

Howard, G.S. , Schmeck, R.R. , & Bary, J.H. (1979). Internal invalidity in studies employing self-report instruments: A suggested remedy. Journal of Educational Measurement , 16, 129-135.

17.

Klatt, J. , & Taylor-Powell, E. (2005a). Using the retrospective post-then-pre design, quick tips #27. Program Development and Evaluation. Madison, WI: University of Wisconsin-Extension .

18.

Krosnick, J.A. (1999). Survey research. Annual Review of Psychology, 50, 537-567.

19.

Krosnick, J.A. , & Fabrigar, L.R. (1998). Designing good questionnaires: Insights from psychology. New York: Oxford University Press.

20.

Lent, R.W. , Brown, S.D. , & Gore, P.A. (1997). Discriminant and predictive validity of academic self-concept, academic self-efficacy and mathematics specific self-efficacy . Journal of Counseling Psychology, 44, 307-315.

21.

Manthei, R.J. (1997). The response-shift bias in a counselor education programme. British Journal of Guidance and Counseling, 25, 229-237.

22.

Paulhus, D.L. (2002). Socially desirable responding: The evolution of a construct. In H. I. Braun , D. N. Jackson , & D. E. Wiley (Eds.), The role of constructs in psychological and educational measurement (pp. 49-69). Mahwah, NJ: Erlbaum.

23.

Pratt, C.C. , Mcguigan, W.M. , & Katzev, A.R. (2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21, 341-349.

24.

Ross, M. (1989). Relation of implicit theories to the construction of personal histories. Psychological Review, 96, 341-357.

25.

Schwarz, N. (2007). Cognitive aspects of survey methodology. Applied Cognitive Psychology, 21, 277-287.

26.

Taylor, P. , Russ, Eft , D. , & Taylor, H. (2009). Gilding the outcome by tarnishing the past: Inflationary biases in retrospective pretests. American Journal of Evaluation, 30, 31-43.

27.

Wolfinger, R. , & Chang, M. (2007). Comparing the SAS GLM and MIXED procedures for repeated measures. SAS Institute). Retrieved March 6, 2008, from http://support.sas.com\rnd\app\papers\mixedglm.pdf