Situational Enhancement of Design Validity: The Case of Training Evaluation at the World Bank Institute

Abstract

The critical role played by research design has been widely discussed among evaluators. Generally there is a consensus that experimental and certain quasi-experimental designs are preferable for their ability to eliminate or minimize threats to internal validity. Least preferred are the class of nonexperimental designs, particularly the one-group, pretest-posttest design. This design, while convenient, is regarded as incapable of controlling for most standard threats to internal validity. In practical applications, however, this design may be stronger than expected. The context, or situation, in which the design is applied may make one or more standard threats to validity implausible. An example from evaluations at the World Bank Institute is provided to demonstrate the effectiveness of this design within a given situation. It is suggested that evaluators carefully review the plausibility of each threat to design validity within the context to which it is applied as a means of adding to their repertoire of existing evaluation tools.

Get full access to this article

View all access options for this article.

References

Bollen, K. A. (1989). Structural equations with latent variables. New York: Wiley.

Campbell, D. T. , & Stanley, J. C. (1963). Experimental and quasi-experimental designs for research. Chicago: Rand McNally.

Cook, T. D. , & Campbell, D. T. (1979). Quasi-experimentation: Design & analysis issues for field settings. Boston: Houghton Mifflin.

Cronbach, L. J. (1982). Designing evaluations of educational and social programs. San Francisco: Jossey-Bass.

Cronbach, L. J. (1986). Social inquiry by and for earthlings. In D. W. Fiske & R. A. Shweder (Eds.), Metatheory in social sciences. Chicago: University of Chicago Press.

Marcantonio, R. J. , & Cook, T. D. (1994). Convincing quasi-experiments: The interrupted time series and regression-discontinuity designs. In J. S. Wholey , H. P. Hatry , & K. E. Newcomer (Eds.), Handbook of practical program evaluation. San Francisco: Jossey-Bass.

Mark, M. M. (1986). Validity typologies and the logic and practice of quasi-experimentation. In W. M. K. Trochim (Ed.), Advances in quasi-experimental design and analysis. New Directions for Program Evaluation, 31. San Francisco: Jossey-Bass.

Mohr, L. B. (1995). Impact analysis for program evaluation (2nd ed.). Thousand Oaks, CA: Sage.

Orwin, R. G. (1997). Twenty-one years old and counting: The interrupted time series design comes of age. In E. Chelimsky & W. R. Shadish (Eds.), Evaluation for the 21st century: A handbook. Thousand Oaks, CA: Sage.

10.

Shadish, W. R. , & Epstein, R. (1987). Patterns of program evaluation practice among members of the Evaluation Research Society and evaluation network. Evaluation Review, 11 (3), 555-590.