This study analyzed the magnitude of experimental intervention outcomes as a function of violations in internal and external validity for studies that included students with learning disabilities. The results indicated that treatment outcomes were significantly affected by the following violations: teacher effects, establishing criterion levels of instructional performance, reliance on experimental measures, using different measures between pretest and posttest, using a sample heterogenous in age, and using incorrect units of analysis. Furthermore, the underreporting of information related to ethnicity, locale of the study, psychometric data, and teacher applications positively inflated the magnitude of treatment outcomes. A weighted hierarchical regression analysis revealed that composite scores of the aforementioned high-risk variables accounted for 16% of the total variance in effect size. The implications for interpreting intervention research to practice are discussed.
Get full access to this article
View all access options for this article.
References
1.
Bollen, K.A. (1989). Structural equations with latent variables. New York: Wiley-Interscience.
2.
Borg, W. (1984). Dealing with threats to internal validity that randomization does not rule out. Educational Researcher, 13(10), 11-14.
3.
Bronowski, A. (1978). The common sense of science. Cambridge, MA: Harvard University Press.
4.
Bryk, A., & Raubenbush, S.W. (1992). Hierarchical linear models: Applications and data analysis. Newbury Park, CA: Sage.
5.
Chalmers, T.C., Berrier, J., Sacks, H.S., Levin, H., Reitman, D., & Nagalingham, R. (1987). Meta-analysis of clinical trials as a scientific discipline. II. Replicate variability and comparison of studies that agree and disagree. Statistics in Medicine , 6, 733-744.
6.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
7.
Cook, T.D., & Campbell, D.T. (1979). Quasi-experimentation: Design and analysis issues for field settings. Boston: Houghton Mifflin.
8.
Elbaum, B., Vaughn, S., Hughes, M.T., Moody, S.W., & Schumm, J.S. (2000). How reading outcomes of students with disabilities are related to instructional grouping formats: A meta-analytic review. In R. Gersten, E. Schiller, & S. Vaughn (Eds.), Contemporary Special education research: Syntheses of the knowledge base on critical issues (pp. 105-135). Mahwah, NJ: Erlbaum.
9.
Gersten, R., Baker, S., & Lloyd, J. (2000). Designing high-quality research in special education: Group experimental design. The Journal of Special Education , 34, 2-18.
10.
Gleser, L.J., & Olkin, I. (1994). Stochastically dependent effect sizes. In H. Cooper & L. V Hedges (Eds.), The handbook of research synthesis (pp. 339-355). New York: Russell Sage Foundation.
11.
Harwell, M.R. (1992). Summarizing Monte Carlo results in methodological research. Journal of Educational Statistics, 17, 297-313.
12.
Hedges, L.V. (1994). Fixed effects models. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 285-299). New York: Russell Sage Foundation.
13.
Hedges, L.V., & Olkin, I. (1985). Statistical methods for meta-analysis. San Diego: Academic Press.
14.
Lou, Y., Abrami, P.C., Spence, J.C., Poulsen, C., Chambers, B., & d'Apollonia, S. (1996). Within class grouping: A meta-analysis. Review of Educational Research, 66,423-458.
15.
Lysynchuk, L.M., Pressley, M., d'Ailly, H., Smith, M., & Cake, H. (1989). A methodological analysis of experimental studies of comprehension strategy instruction. Reading Research Quarterly , 24, 458-470.
Matt, G.E., & Cook, T.D. (1994). Threats to the validity of research syntheses . In H. Cooper & L. V Hedges (Eds.), The handbook of research synthesis (pp. 503-520). New York: Russell Sage Foundation. Parker, R.M. (1990). Power, control, and validity in research. Journal of Learning Disabilities , 23, 613-620.
18.
Polloway, E.A., Cronin, M.E., Patton, J. (1986). The efficacy of group versus one-to-one instruction . Remedial and Special Education, 7, 22-30.
19.
Pressley, M., & Harris, K.R. (1994). Increasing the quality of educational intervention research. Educational Psychology Review, 6, 191-208.
20.
Rosenthal, R. (1994). Parametric measures of effect size. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 231-244). New York: Russell Sage Foundation.
21.
Slavin, R.E. (1986). Best-evidence synthesis: An alternative to meta-analytic and traditional reviews. Educational Researcher, 15, 5-11.
22.
Swanson, H.L., & Hoskyn, M. (1998). Experimental intervention research on students with learning disabilities: A meta-analysis of treatment outcomes. Review of Educational Research, 68, 277-321.
23.
Swanson, H.L., Hoskyn, M., Lee, C., Mc-Mahan, C., & O'Shaughnessy, T. (1998). A selective synthesis of single-subject design intervention research on students with learning disabilities . In T. Scruggs & M. Mastropieri (Eds.), Advances in learning and behavioral disabilities (Vol. 12, pp. 79-127). Greenwich, CT: JAI Press.
24.
Troia, G. (1999). Phonological awareness intervention research: A critical review of the experimental methodology. Reading Research Quarterly, 34, 28-52.
25.
Wortman, P.M. (1994). Judging research quality. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 97-109). New York: Russell Sage Foundation.
26.
Wortman, P.M., & Bryant, F.B. (1985). School desegregation and black achievement: An integrative review. Sociological Methods and Research, 13, 289-324.
