Measuring Program Outcomes: Using Retrospective Pretest Methodology

Abstract

This study used longitudinal data from 307 mothers with firstborn infants participating in a home-visitation, child-abuse prevention program. A self-report measure of specific constructs the program hoped to affect showed that the retrospective pretest methodology produced a more legitimate assessment of program outcomes than did the traditional pretest-posttest methodology. Results showed that when response shift bias was present, traditional pretest-posttest comparisons resulted in an underestimation of program effects that could easily be avoided by the retrospective pretest methodology. With demands for documenting program outcomes increasing, retrospective pretest designs are shown to be a simple, convenient, and expeditious method for assessing program effects in responsive interventions. The limits of retrospective pretests, and methods for strengthening their use, are discussed.

Get full access to this article

View all access options for this article.

References

Bray, J. W. , Maxwell, S. E. , & Howard, G. S. (1984). Methods of analysis with response shift-bias. Educational and Psychological Measurement, 44, 781-804.

Brooks, L. , & Gersh, T. L. (1998). Assessing the impact of diversity initiatives using the retrospective pretest design. Journal of College Student Development, 34, 383-385.

Conway, M. , & Ross, M. (1984). Getting what you want by revising what you had. Journal of Personality and Social Psychology, 47, 738-748.

Family Resource Coalition of America . (1998). How are we doing?: A program self-assessment toolkit for the family support field. Chicago, IL: author.

Goedhart, H. , & Hoogstraten, J. (1992). The retrospective pretest and the role of pretest information in evaluation studies. Psychological Reports, 70, 699-704.

Hatry, H. (1997). Where the rubber meets the road: Performance measurement for state and local public agencies. New Directions for Evaluation, 75, 31-44.

Hawkins, S. A. , & Hastie, R. (1990). Hindsight: Biased judgements of past events after the outcomes are known. Psychological Bulletin, 107, 311-327.

Howard, G. S. (1982). Improving methodology via research on research methods. Journal of Counseling Psychology, 29(3), 318-326.

Howard, G. S. , & Dailey, P. R. (1979). Response-shift bias: A source of contamination of self-report measures. Journal of Applied Psychology, 64, 144-150.

10.

Howard, G. S. , Millham, J. , Slaten, S. , & O’Donnel, L. (1981). Influence of subject response style effects on retrospective measures. Applied Psychological Measurement, 5, 89-100.

11.

Howard, G. S. , Ralph, K. M. , Gulanick, N. A. , Maxwell, S. E. , Nance, D. , & Gerber, S. L. (1979). Internal invalidity in pretest-posttest self-report evaluations and the re-evaluation of retrospective pretests. Applied Psychological Measurement, 3, 1-23.

12.

Marshak, S. H. , deSilva, P. , & Silberstein, J. (1998). Evaluation of a peer-taught nutrition education program for low-income parents. Journal of Extension, 27, 19-21.

13.

Newcomer, K. (1997). Using performance measurement to improve public and non-profit programs. New Directions for Evaluation, 75, 5-14.

14.

Rossi, P. H. , & Freeman, H. E. (1993). Evaluation: A systematic approach (5th ed.). Newbury Park, CA: Sage.

15.

Schwarz, N. , & Sudman, S. (Eds.). (1993). Autobiographical memory and the validity of retrospective reports. New York: Springer-Verlag.

16.

Sprangers, M. , & Hoogstraten, J. (1991). Subject bias in three self-report measures of change. Methodika, 5, 1-13.

17.

Terborg, J. R. , Howard, G. S. , & Maxwell, S. E. (1980). Evaluating planned organizational change: A method for assessing alpha, beta and gamma change. Academy of Management Review, 5, 109-121.

18.

Toedter, L. J. , Lasker, J. N. , & Campbell, D. T. (1990). The comparison group problem in bereavement studies and the retrospective pretest. Evaluation Review, 14, 75-90.

19.

United Way of America . (1996). Measuring program outcomes: A practical approach. Alexandria, VA: Author