Sage Journals: Discover world-class research

Abstract

In this article, the authors examine the interconnections among education researchers’ misconceptions about the quality of the education research literature. Specifically, the authors catalog the sequence of events that, when taken together with a series of contradictory graphs presented by influential scholars in prominent settings, may have gratuitously damaged the already fragile reputation of education research as a field. The authors’ “take-home” messages emphasize the importance of both critically evaluating the claims and warrants of our scholar colleagues and being aware of our natural tendencies, often unconscious, toward confirmation bias.

Keywords

Campbell Collaboration confirmation bias education research evidence based practice experimental design experimental validity randomized control trials randomized field trials

Get full access to this article

View all access options for this article.

References

American Institutes for Research. (2008). Board of directors. Retrieved March 20, 2008, from http://www.air.org/overview/board.aspx

Boruch

De Moya

Snyder

(2002). The importance of randomized field trials in education and related areas. In Mosteller

Boruch

(Eds.), Evidence matters: Randomized trials in education research (pp. 50–79). Washington, DC: Brookings Institution Press.

Campbell Collaboration. (n.d.). About us in English [Leaflet]. Oslo, Norway: Author.

Campbell

D. T.

Stanley

J. C.

(1963). Experimental and quasi- experimental designs for research on teaching. In Gage

N. L.

(Ed.), Handbook of research on teaching (pp. 171–246). Chicago: Rand McNally.

Coalition for Evidence-Based Policy. (2003). Identifying and implementing educational practices supported by rigorous evidence: A user friendly guide. Washington, DC: Author.

Coalition for Evidence-Based Policy. (2010). Help desk. Retrieved September 30, 2010, from http://evidencebasedpolicy.org/wordpress

Colorado Department of Education. (2005). Fast facts: Evidence-based practice. Retrieved September 1, 2008, from http://www.cde.state.co.us/cdesped/download/pdf/FF-EBP_MH_ADHD.pdf

Cook

T. D.

(1999). Considering the major arguments against random assignment: An analysis of intellectual culture surrounding evaluation in American schools of education. Paper presented at the Harvard Faculty Seminar on Experiments in Education. Retrieved September 1, 2009, from http://www.northwestern.edu/ipr/publications/workingpapers/wpS99.html

Cook

T. D.

(2001a). A critical appraisal of the case against using experiments to assess school (or community) effects. Education Next, 3. Retrieved from http://media.hoover.org/documents/ednext20013unabridged_cook.pdf

10.

Cook

T. D.

(2001b). Sciencephobia: Why education researchers reject randomized experiments. Education Next, 1, 62–68.

11.

Council for Excellence in Government. (2008). Coalition for Evidence-Based Policy: Advisory board biographies. Retrieved January 15, 2008, from http://www.excelgov.org/index.php?keyword=a432fbc71d7564

12.

Evans

J. St. B. T.

(1989). Bias in human reasoning: Causes and consequences. Hillsdale, NJ: Erlbaum.

13.

Feuer

M. J.

Towne

Shavelson

R. J.

(2002). Scientific culture and educational research. Educational Researcher, 31(8), 4–14.

14.

Glass

G. V

Peckham

P. D.

Sanders

J. R.

(1972). Consequences of failure to meet assumptions underlying the fixed effects analysis of variance and covariance. Review of Educational Research, 42, 237–288.

15.

Graesser

A. C.

(2009). Inaugural editorial for Journal of Educational Psychology. Journal of Educational Psychology, 101, 259–261. doi:10.1037/a001488310.1037/a0014883

16.

Guttman

(1954). Some necessary conditions for common-factor analysis. Psychometrika, 19, 149-161. doi:10.1007/BF0228916210.1007/BF02289162

17.

Hood

P. D.

(2003). Scientific research and evidence-based practice. Retrieved November 29, 2009, from http://www.wested.org/online_pubs/scientrific.research.pdf

18.

Hsieh

P.-H.

Acee

Chung

W.-H.

Hsieh

Y.-P.

Kim

Thomas

G. D.

. (2005). Is educational intervention research on the decline? Journal of Educational Psychology, 97, 523–529. doi:10.1037/0022-0663.97.4.52310.1037/0022-0663.97.4.523

19.

Hsu

P. L.

(1938). Contribution to the theory of “Student’s” t-test as applied to the problem of two samples. Statistical Research Memoirs, 2, 1–24.

20.

Institute for Education Sciences. (2007). Grover J. (Russ) Whitehurst. Retrieved November 29, 2009, from http://ies.ed.gov/director/

21.

Institute for Education Sciences. (2008). Priorities for the Institute of Education Sciences as approved by the National Board for Education Sciences. Retrieved November 29, 2009, from http://ies.ed.gov/director/board/priorities.asp

22.

Institute for Policy Research. (2008). IPR home page: People. Thomas D. Cook: Curriculum vitae. Retrieved November 29, 2009, from http://www.northwestern.edu/ipr/people/tcook.html

23.

Kaiser

H. F.

(1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20, 141–151. doi:10.1177/00131644600200011610.1177/001316446002000116

24.

Mosteller

Boruch

(Eds.). (2002). Evidence matters: Randomized trials in education research. Washington, DC: Brookings Institution Press.

25.

Nickerson

R. S.

(1998). Confirmation bias: A ubiquitous phenomenon in many guises. Review of General Psychology, 2, 175–220. doi:10.1037/1089-2680.2.2.17510.1037/1089-2680.2.2.175

26.

No Child Left Behind Act of 2001, Pub. L. No. 107-110 (2001).

27.

Petrosino

Boruch

R. F.

Rounding

McDonald

Chalmers

(2000). The Campbell Collaboration Social, Psychological, Educational and Criminological Trials Register (C-2 SPECTR) to facilitate the preparation and maintenance of systematic reviews of social and educational interventions. Evaluation and Research in Education, 14, 206–219. doi:10.1080/0950079000866697310.1080/09500790008666973

28.

Petrosino

Boruch

R. F.

Soydan

Duggan

Sanchez-Meca

(2001). Meeting the challenges of evidence-based policy: The Campbell Collaboration. ANNALS of the American Academy of Political and Social Science, 578, 14–34. doi:10.1177/000271620157800100210.1177/0002716201578001002

29.

Raudenbush

S. W.

(2002, February). Identifying scientifically-based research in education. Paper presented at the Scientifically Based Research Seminar. http://www.ssicentral.com/hlm/techdocs/ScientificallyBasedResearchSeminar.pdf

30.

Robinson

D. H.

Levin

J. R.

Thomas

G. D.

Pituch

K. A.

Vaughn

(2007). The incidence of “causal” statements in teaching-and -learning research journals. American Educational Research Journal, 44, 400–413. doi:10.3102/000283120730217410.3102/0002831207302174

31.

Rosenbaum

P. R.

(1995). Randomized experiments. New York: Springer-Verlag.

32.

Sandler

Apple

M. W.

(2010). A culture of evidence, a politics of objectivity: The evidence-based practices movement in educational policy. In Leonardo

(Ed.), Handbook of cultural politics and education (pp. 325–340). Rotterdam, The Netherlands: Sense Publishers.

33.

Schuerman

Soydan

Macdonald

Forslund

de Moya

Boruch

(2002). The Campbell Collaboration. Research on Social Work Practice, 12, 309–317. doi:10.1177/10497315020120020710.1177/104973150201200207

34.

Shadish

W. R.

Cook

T. D.

Campbell

D. T.

(2002). Experimental and quasi-experimental designs for generalized causal inference. Boston: Houghton Mifflin.

35.

Shavelson

R. J.

Phillips

D. C.

Towne

Feuer

M. J.

(2003). On the science of education design studies. Educational Researcher, 32, 25–28. doi:10.3102/0013189X03200102510.3102/0013189X032001025

36.

Shavelson

R. J.

Towne

(Eds.). (2002). Scientific research in education. Washington, DC: National Academy Press.

37.

Slavin

R. E.

(2002). Evidence-based education policies: Transforming educational practice and research. Educational Researcher, 31, 15–21. doi:10.3102/0013189X03100701510.3102/0013189X031007015

38.

Slavin

R. E.

(2008). Perspectives on evidence-based research in education. Educational Researcher, 37, 5–14. doi:10.3102/0013189X0831411710.3102/0013189X08314117

39.

Society for Research on Educational Effectiveness. (2008). Leadership: Advisory board. Retrieved November 29, 2009, from http://www.educationaleffectiveness.org/pages/leadership/advisory.shtml

40.

Society for Research on Educational Effectiveness. (2010). About SREE. Retrieved November 29, 2009, from http://www.sree.org/pages/about .php

41.

Thompson

(2008). Standards in conducting and publishing research in education. Mid-Western Educational Researcher, 21(1), 10–16. Available at from http://people.cehd.tamu.edu/~bthompson/mwera.htm

42.

Tolstoy

(1899). What is art? ( Maude

, Trans.). New York: Thomas Y. Crowell. Retrieved from http://books.google.com/books?id=0SYVAAAAYAAJ&pg=PA124

43.

Towne

Wise

L. L.

Winters

T. M.

(Eds.). (2004). Advancing scientific research in education. Washington, DC: National Academies Press.

44.

U.S. Department of Education. (n.d.). What Works Clearinghouse: About us. Retrieved November 29, 2009, from http://ies.ed.gov/ncee/wwc/aboutus

45.

U.S. Department of Education. (2002). Lead & manage my school: Student achievement and school accountability conference. Retrieved November 29, 2009, from http://www.ed.gov/admins/lead/account/sasaconference02.html

46.

U.S. Department of Education. (2005). IES biennial report: Grants, cooperative agreements, and contracts over $100,000-FY2002 through FY2004, October 30, 2007. Retrieved from http://www.ed.gov/about/reports/annual/otherplanrpts.html

47.

U.S. Department of Education. (2008). Proven methods. Reading first: Student achievement, teacher empowerment, national success. Retrieved November 29, 2009, from http://www.ed.gov/nclb/methods/reading/readingfirst.html

48.

Wason

P. C.

(1968). Reasoning about a rule. Quarterly Journal of Experimental Psychology, 20, 273–281. doi:10.1080/1464074680840016110.1080/14640746808400161

49.

Waytowich

V. L.

Onwuegbuzie

A. J.

Jiao

Q. G.

(2006). Character- istics of doctoral students who commit citation errors. Library Review, 55, 195–208. doi:10.1108/0024253061065599310.1108/00242530610655993

50.

WestEd. (2008). Staff: Anthony Petrosino. Retrieved November 29, 2009, from http://www.wested.org/cs/we/view/u/2043

51.

Whitehurst

G. J.

(2002a). Archived evidence-based education (EBE). Retrieved November 29, 2009, from http://www.ed.gov/offices/OERI/presentations/evidencebase.html

52.

Whitehurst

G. J.

(2002b). Evidence-based education (EBE). PowerPoint presented at the Student Achievement and School Accountability Conference. Retrieved November 29, 2009, from http://www.ed.gov/nclb/methods/whatworks/eb/edlite-index.html

53.

Whitehurst

G. J.

(2003, April). The Institute of Education Sciences: New wine, new bottles. Paper presented at the annual meeting of the American Educational Research Association, Chicago.

54.

Whitehurst

G. J.

(2004). Making education evidence-based: Premises, principles, pragmatics and politics. Retrieved November 29, 2009, from http://www.northwestern.edu/ipr/events/lectures/DPPL-Whitehurst.pdf.

55.

Yin

Fan

(2001). Estimating R² shrinkage in multiple regression: A comparison of different analytical methods. Journal of Experimental Education, 69, 203-224. doi:10.1080/0022097010960065610.1080/00220970109600656

Propagation of Misinformation About Frequencies of RFTs/RCTs in Education

Abstract

Keywords

Get full access to this article

References