This article presents a method to evaluate pretest effects on posttest scores in the absence of an un-pretested control group using published results of pretesting effects due to Willson and Putnam. Confidence intervals around the expected theoretical gain due to pretesting are computed, and observed gains or differential gains are compared with the original reported results. Examples from the American Educational Research Journal and the Journal of Educational Psychology are presented that illustrate both support for the original interpretation and evidence for reinterpretation of the results as statistically significant or nonsignificant opposite to the original conclusion. Implications for using the results of meta-analyses in substantive research are discussed.
Campbell, D.T., & Stanley, J.C. ( 1963). Experimental and quasi-experimental designs for research . Chicago, IL: Rand McNally.
2.
Carlson, K.D., & Schmidt, F.L. ( 1999). Impact of experimental design on effect size: Findings from the research literature on training. Journal of Applied Psychology, 84, 851-862.
3.
Crocker, L., & Algina, J. ( 1986). Introduction to classical and modern test theory. Belmont, CA: Wadsworth.
4.
Dugard, P., & Todman, J. ( 1995). Analysis of pre-test post-test control group designs in educational research. Educational Psychology, 15, 181-198.
5.
Glass, G.V. ( 1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10), 3-8.
6.
Glass, G.V., McGaw, B., & Smith, M.L. ( 1981). Meta-analysis in social research. Beverly Hills, CA: Sage.
7.
Gunasekera, T.W. ( 1997). Effects of pretest sensitization associated with cooperative learning strategies on the achievement level of adult mathematics students . Dissertation Abstracts International, 58(03), 718A.
8.
Heinsman, D.T., & Shadish, W.R. ( 1996). Assignment methods in experimentation: When do nonrandomized experiments approximate answers from randomized experiments? Psychological Methods, 1, 154-169.
9.
Jaeger, R.M. ( 1975). Some new developments and discoveries for evaluative analysis. CEDR Quarterly, 9(3), 3-7.
10.
Jimerson, S., Carlson, E., Robert, M., Egeland, B., & Sroufe, L.A. ( 1997). A prospective, longitudinal study of the correlates and consequences of early grade retention. Journal of School Psychology , 35(1), 3-25.
May, K., & Hittner, J.B. ( 2003). On the relation between power and reliability of difference scores. Perceptual and Motor Skills, 97, 905-908.
13.
Morris, S.B. ( 2008). Estimating effect sizes for pretest-posttest-control group designs. Organizational Research Methods, 11, 364-386.
14.
Rittle-Johnson, B., & Star, J.R. ( 2007). Does comparing solution methods facilitate conceptual and procedural Knowledge? An experimental study on learning to solve equations . Journal of Educational Psychology, 99, 561-574.
15.
Sime, M.E., & Boyce, G.C. ( 1969). Overt response, knowledge of results and learning. Programmed Learning and Educational Technology, 6(1), 12-19.
16.
Stevens, J. ( 2002). Applied multivariate statistics for the social sciences (4th ed.). Mahwah, NJ: Erlbaum.
17.
Vaughn, S., Cirino, P.T., Linan-Thompson, S., Mathes, P.G., Carlson, C.D., Hagan, E.C., ... Francis, D.J. ( 2006). Effectiveness of a Spanish intervention and an English intervention for English-language learners at risk for reading problems. American Educational Research Journal, 43, 449-479.
18.
Whitehurst, G.J. ( 2002). Evidence-based education (EBE). Retrieved from http://ies.ed.gov/director/pdf/200210.pdf
19.
Willson, V.L. ( 1984). Adding results to a meta-analysis: Theory and example . Journal of Research in Science Teaching, 21, 649-658.
20.
Willson, V.L., & Hughes, J.L. ( 2008). Who is retained in first grade? A psychological perspective . Elementary School Journal, 109, 251-266.
21.
Willson, V.L., & Kim, E.S. (in press). Pretest sensitization. In N. Salkind (Ed.), Encyclopedia of research design. Thousand Oaks, CA: Sage.
22.
Willson, V.L., & Putnam, R.R. ( 1982). A meta-analysis of pretest sensitization effects in experimental design. American Educational Research Journal, 19, 249-258.
23.
Zimmerman, D.W., & Williams, R.H. (1998). Reliability of gain scores under realistic assumptions about properties of pre-test and post-test scores . British Journal of Mathematical and Statistical Psychology , 51, 343-351.
