Sage Journals: Discover world-class research

Abstract

Experimental and quasi-experimental designs are used in educational research to establish causality and develop effective practices. These research designs rely on a counterfactual model that, in simple form, calls for a comparison between a treatment group and a control group. Developers of educational practices often assume that the population from which control groups are drawn is unchanging in its behavior or performance. This is not always the case. Populations and study samples can change over time—sometimes dramatically so. We illustrate this important point by presenting data from 5 randomized control trials of the efficacy of Kindergarten Peer-Assisted Learning Strategies, a supplemental, peer-mediated reading program. The studies were conducted across 9 years and involved 2,591 students. Findings demonstrate a dramatic increase in the performance of control students over time, and suggest the need for a more nuanced understanding of the counterfactual model and its role in establishing evidence-based practices.

Keywords

classroom research counterfactual educational policy evidence-based practice Kindergarten Peer-Assisted Learning Strategies (K-PALS)literacy research methodology

Get full access to this article

View all access options for this article.

References

Boruch

Rui

(2008). From randomized controlled trials to evidence grading schemes: Current state of evidence-based practice in social sciences. Journal of Evidence-Based Medicine, 1(1), 41–49.

Boyles

(2012). Fish oil fizzles for fighting heart attack, stroke. WebMD. Retrieved from http://www.webmd.com/heart-disease/news/20120911/fish-oil-fizzles-for-fighting-heart-attack-stroke

Burr

M. L.

Sweetnam

Fehily

A. M.

(1997). Diet and reinfarction. European Heart Journal, 15, 1152–1153.

Cordray

D. S.

Pion

G. M.

(2006). Treatment strength and integrity: Models and methods. In Bootzin

R. R.

McKnight

P. E.

(Eds.), Strengthening research methodology: Psychological measurement and evaluation (pp. 103–124). Washington, DC: American Psychological Association.

Coyne

M. D.

Little

Rawlinson

Simmons

Oi-man

Kim

. . . Civetelli

(2013). Replicating the impact of a supplemental beginning reading intervention: The role of instructional context. Journal of Research on Educational Effectiveness, 6(1), 1–23.

Darling-Hammond

(1996). The quiet revolution: Rethinking teacher development. Educational Leadership, 53(6), 4–10.

Delquadri

J. C.

Greenwood

C. R.

Whorton

Carta

J. J.

Hall

R. V.

(1986). Classwide peer tutoring. Exceptional Children, 52, 535–542.

Education Sciences Reform Act of 2002, P.L. 107–279, 116 Stat. 1940 (2002).

Elmore

R. F.

(1996). Getting to scale with good educational practice. Harvard Educational Review, 66(1), 1–26.

10.

Evidence-Based Medicine Working Group. (1992). Evidence-based medicine: A new approach to teaching the practice of medicine. Journal of the American Medical Association, 268, 2420–2425.

11.

Feinstein

A. R.

Horwitz

R. I.

(1997). Problems in the “evidence” of “evidence-based medicine.” The American Journal of Medicine, 103, 529–535.

12.

Fuchs

L. S.

(2005). Peer-assisted learning strategies: Promoting word recognition, fluency, and reading comprehension in young children. Journal of Special Education, 39(1), 34-44.

13.

Fuchs

L. S.

Thompson

Al Otaiba

Yen

McMaster

K. L.

. . . Braun

(2001). Peer assisted learning strategies: Kindergarten reading. Nashville, TN: Vanderbilt University.

14.

Fuchs

L. S.

Thompson

Al Otaiba

Yen

Yang

. . . O’Conner

R. E.

(2001). Is reading important in reading-readiness programs? A randomized field trial with teachers as program implementers. Journal of Educational Psychology, 93(2), 251-267.

15.

Fuchs

L. S.

Thompson

Al Otaiba

Yen

Yang

N. J.

. . . O’Connor

R. E.

(2002). Exploring the importance of reading programs for kindergarteners with disabilities in mainstream classrooms. Exceptional Children, 68(3), 295-311.

16.

Fuchs

L. S.

Thompson

Svenson

Yen

Al Otaiba

(2001). Peer-assisted learning strategies in reading: Extensions for kindergarten, first grade, and high school. Remedial and Special Education, 22, 15-21.

17.

Fuchs

McMaster

K. L.

Saenz

Kearns

Fuchs

L. S.

Yen

. . . Schatschneider

(2010). Bringing educational innovation to scale: Top-down, bottom-up, or a third way? Paper presented at the Institute of Education Sciences Conference, National Harbor, MD.

18.

Gamse

Boulay

Fountain

A. R.

Unlu

Maree

McCall

McCormick

(2011). Reading First Implementation Study 2008–2009: Final report. Washington, DC: U.S. Department of Education.

19.

Gann

(2012). Fish oil delivers few heart benefits, study finds. ABC News. Retrieved from http://abcnews.go.com/Health/fish-oil-delivers-heart-benefits-study/story?id=16103420

20.

GISSI-Prevenzione Investigators. (1999). Dietary supplementation with n-3 polyunsaturated fatty acids and vitamin E after mycardial infarction: Results of the GISSI-Prevenzione trial. Lancet, 354, 447–455.

21.

Hawthorne

(2012, September 11). Fish oil no lifesaver, study finds. World News with Diane Sawyer. Retrieved from http://abcnews.go.com/Health/Wellness/fish-oil-lifesaver-study-finds/story?id=17211288-.UGNxevk-ui8

22.

Hernán

M. A.

VanderWeele

T. J.

(2011). Compound treatments and transportability of causal inference. Epidemiology, 22(3), 368–377.

23.

Humphreys

(2013, January 15). How “breakthrough medical findings” disappear. The Reality-Based Community. http://www.samefacts.com

24.

Individuals With Disabilities Education Improvement Act (IDEA) of 2004, P. L. No. 108–446, 118 Stats. 2647. (2004).

25.

Institute of Education Sciences. (2011). What Works Clearinghouse: Procedures and standards handbook (Version 2.1). Washington, DC: Department of Education.

26.

Kennedy

(2004). A brief history of disease, science and medicine: From the ice age to the genome project. Mission Viejo, CA: Asklepiad Press.

27.

Kris-Etherton

P. M.

Harris

W. S.

Lawrence

J. A.

(2002). Fish consumption, fish oil, omega-3 fatty acids, and cardiovascular disease. Circulation, 106, 2747–2757.

28.

Lewis

(1973). Causation. The Journal of Philosophy, 70(17), 556–567.

29.

Maghimbeigi

Eshraghian

M. R.

Mohammad

McArdle

(2008). Multilevel zero-inflated negative binomial regression modeling for over-dispersed count data with extra zeros. Journal of Applied Statistics, 35, 1193–1202. doi: 10.1177/0962280206071840

30.

McMaster

K. L.

Fuchs

L. S.

(2007). Promises and limitations of peer-assisted learning strategies in reading. Learning Disabilities: A Contemporary Journal, 5(97-112).

31.

McMaster

K. L.

Fuchs

Saenz

Lemons

C. J.

Kearns

Yen

. . . Fuchs

L. S.

(2010). Scaling up PALS: Importance of implementing evidence-based practice with fidelity and flexibility. New Times for DLD: Division for Learning Disabilities, 28(1), 1-4.

32.

National Reading Panel. (2000). Teaching children to read: An evidence-based assessment of the scientific research literature on reading and its implications for reading instruction. Washington, DC: U.S. Department of Health and Human Services.

33.

Nelson

M. C.

Cordray

D. S.

Hulleman

C. S.

Sommer

E. C.

(2012). A procedure for assessing intervention fidelity in experiments testing educational and behavioral interventions. The Journal of Behavioral Health Services & Research, 39(4), 374–396.

34.

No Child Left Behind Act of 2001, P. L. No. 107–110, § 1–1076, 115 Stat. 1425. (2001).

35.

Patterson

(2002, May 5). What doctors don’t know (almost everything). The New York Times Magazine, 74–79.

36.

Richardson

(1996). The role of attitudes and beliefs in learning to teach. In Sikula

(Ed.), Handbook of research on teacher education (Vol. 2, pp. 102–119). New York: Macmillan.

37.

Rizos

E. C.

Ntzani

E. E.

Bika

Kostapanos

M. S.

Elisaf

M. S.

(2012). Association between Omega-3 fatty acid supplementation and risk of major cardiovascular disease events: A systematic review and meta-analysis. Journal of the American Medical Association, 308, 1024–1033.

38.

Sackett

D. L.

Rosenberg

W. M. C.

Muir Gray

J. A.

Haynes

R. B.

Richardson

W. S.

(1996). Evidence based medicine: What it is an what it isn’t. BMJ, 312, 71–72. doi: 10.1136/bmj.312.7023.71

39.

Schwartz

Gatto

N. M.

Campbell

U. B.

(2011). What would have been is not what would be: Counterfactuals of the past and potential outcomes of the future. In Shrout

P. E.

Keyes

K. M.

Ornstein

(Eds.), Causality and psychopathology: Finding the determinants of disorders and their cures (pp. 25–46). Oxford, UK: Oxford University Press

40.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. New York: Houghton Mifflin.

41.

Singh

R. B.

Niaz

M. A.

Sharma

J. P.

Kumar

Rastogi

Mshiri

(1997). Randomized, double-blind, placebo-controlled trial of fish oil and mustard oil in patients with suspected acute myocardial infarction: The Indian experiment of infarct survival-4. Cardiovascular Drugs and Therapy, 11, 485–491.

42.

Stata Corporation. (2009). Stata / IC 10.1 for Macintosh. College Station, TX: Author.

43.

Tryon

W. W.

Lewis

(2008). An inferential confidence interval method of establishing statistical equivalence that corrects Tryon’s (2001) reduction factor. Psychological Methods, 13(3), 272–277.

44.

Van der Werf

Bax

Betriu

Blomstron-Mundqvist

Crea

Falk

. . . Weis

(2008). Management of acute myocardial infarction in patients presenting with persistent ST-segment elevation. European Heart Journal, 29, 2909–2945.

45.

Vaughn

Swanson

E. A.

Roberts

Wanzek

Stillman-Spisak

Solis

Simmons

(2013). Improving reading comprehension and social studies knowledge in middle school. Reading Research Quarterly, 48(1), 77–93.

46.

Whitehurst

(2003). The Institute of Education Sciences: New wine, new bottles. Chicago: American Educational Research Association. Retrieved from http://www2.ed.gov/rschstat/research/pubs/ies.pdf

47.

Woodcock

R. W.

(1998). Woodcock Reading Mastery Test–Revised. Circle Pines, MN: American Guidance Service.

Evidence-Based Practices in a Changing World

Abstract

Keywords

Get full access to this article

References