Sage Journals: Discover world-class research

Abstract

Concerns about the conduct of research are pervasive in many fields, including education. In this preregistered study, we replicated and extended previous studies from other fields by asking education researchers about 10 questionable research practices and five open research practices. We asked them to estimate the prevalence of the practices in the field, to self-report their own use of such practices, and to estimate the appropriateness of these behaviors in education research. We made predictions under four umbrella categories: comparison to psychology, geographic location, career stage, and quantitative orientation. Broadly, our results suggest that both questionable and open research practices are used by many education researchers. This baseline information will be useful as education researchers seek to understand existing social norms and grapple with whether and how to improve research practices.

Keywords

ethics globalization open science psychology questionable research practices replication research methodology survey research

Get full access to this article

View all access options for this article.

References

Agnoli

Wicherts

J. M.

Veldkamp

C. L. S.

Albiero

Cubelli

(2017). Questionable research practices among Italian research psychologists, PLOS ONE, 12(3), e0172792. https://doi.org/10.1371/journal.pone.0172792

Allen

Mehler

D. M.

(2019). Open science challenges, benefits and tips in early career and beyond. PLOS BIOLOGY, 17(5), e3000246. https://doi.org/10.1371/journal.pbio.3000246

Begley

C. G.

Ellis

L. M.

(2012). Raise standards for preclinical cancer research. Nature, 483, 531–533. https://doi.org/10.1038/483531a

Butler

Delaney

Spoelestra

(2017). The gray zone: Questionable research practices in the business school. Academy of Management Learning & Education, 16(1), 94–109. https://doi.org/10.5465/amle.2015.0201

Campbell

H. A.

Micheli-Campbell

M. A.

Udyawer

(2019). Early career researchers embrace data sharing. Trends in Ecology & Evolution, 34(2), 95–98. https://doi.org/10.1016/j.tree.2018.11.010

Camerer

C. F.

Dreber

Holzmeister

T. H.

Huber

Johannesson

Kirchler

Nave

Nosek

B. A.

Pfeiffer

Altmejd

Buttrick

Chan

Chen

Forsell

Gampa

Heikensten

Hummer

Imai

. . .Wu

(2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour, 2, 637–644. https://doi.org/10.1038/s41562-018-0399-z

Chambers

(2019). The registered reports revolution Lessons in cultural reform. Significance, 16(4), 23–27. https://doi.org/10.1111/j.1740-9713.2019.01299.x

Chambers

C. D.

Feredoes

Muthukumaraswamy

S. D.

Etchells

P. J.

(2014). Instead of “playing the game” it is time to change the rules: Registered reports at AIMS Neuroscience and beyond. AIMS Neuroscience, 1(1), 4–17. https://doi.org/10.3934/Neuroscience.2014.1.4

Choshen-Hillel

Shaw

Caruso

E. M.

(2020). Lying to appear honest. Journal of Experimental Psychology: General, 149(9), 1719–1735. https://doi.org/10.1037/xge0000737

10.

Cook

B. G.

Collins

L. W.

Cook

S. C.

Cook

L. H.

(2016). A replication by any other name: A systematic review of replicative intervention studies. Remedial and Special Education, 37(4), 223–234. https://doi.org/10.1177/0741932516637198

11.

Cook

B. G.

Lloyd

J. W.

Mellor

Nosek

B. A.

Therrien

W. J.

(2018). Promoting open science to increase the trustworthiness of evidence in special education. Exceptional Children, 85(1), 104–118. https://doi.org/10.1177/0741932516637198

12.

Elmore

(1996). Getting to scale with good educational practice. Harvard Educational Review, 66(1), 1–27. https://doi.org/10.17763/haer.66.1.g73266758j348t33

13.

Fanelli

(2012). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891–904. https://doi.org/10.1007/s11192-011-0494-7

14.

Fanelli

Costas

Larivière

(2015). Misconduct policies, academic culture and career stage, not gender or pressures to publish, affect scientific integrity. PLOS ONE, 10(6), e0127556. https://doi.org/10.1073/pnas.1302997110

15.

Farnham

Kurz

Öztürk

M. A.

Solbiati

Myllyntaus

Meekes

Pham

T. M.

Paz

Langiewicz

Andrews

Kanninen

Agbemabiese

Guler

A. T.

Durieux

Jasim

Viessmann

Frattini

Yembergenova

Benito

C. M.

. . .Hettne

(2017). Early career researchers want Open Science. Genome Biology, 18, Article 221. https://doi.org/10.1186/s13059-017-1351-7

16.

Fiedler

Schwarz

(2016). Questionable research practices revisited. Social Psychological & Personality Science, 7(1), 45–52. https://doi.org/10.1177/1948550615612150

17.

Fraser

Parker

Nakagawa

Barnett

Fidler

(2018). Questionable research practices in ecology and evolution. PLOS ONE, 13(7), e0200303. https://doi.org/10.1371/journal.pone.0200303

18.

Gehlbach

Robinson

C. D.

(2018). Mitigating illusory results through preregistration in education. Journal of Research on Educational Effectiveness, 11(2), 296–315. https://doi.org/10.1080/19345747.2017.1387950

19.

Halevy

Shalvi

Verschuere

(2014). Being honest about dishonesty: Correlating self-reports and actual lying. Human Communication Research, 40(1), 54–72. https://doi.org/10.1111/hcre.12019

20.

Hussar

Zhang

Hein

Wang

Roberts

Cui

Smith

Bullock Mann

Barmer

Dilig

(2020). The condition of education 2020 (NCES 2020-144). U.S. Department of Education. https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2020144

21.

John

L. K.

Loewenstein

Prelec

(2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532. https://doi.org/10.1177/0956797611430953

22.

Kaplan

R. M.

Irvin

V. L.

(2015). Likelihood of null effects of large NHLBI clinical trials has increased over time. PLOS ONE, 10(8), e0132382. https://doi.org/10.1371/journal.pone.0132382

23.

Kerr

N. L.

(1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196–217. https://doi.org/10.1207/s15327957pspr0203_4

24.

Kidwell

M. C.

Lazarević

L. B.

Baranski

Hardwicke

T. E.

Piechowski

Falkenberg

L. S.

Kennett

Slowik

Sonnleitner

Hess-Holden

Errington

T. M.

Fiedler

Errington

T. M.

(2016). Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLOS BIOLOGY, 14(5), e1002456. https://doi.org/10.1371/journal.pbio.1002456

25.

Klein

R. A.

Vianello

Hasselman

Adams

B. G.

Adams

R. B.

Jr. Alper

Aveyard

Axt

J. R.

Babalola

M. T.

Bahník

Š.

Batra

Berkics

Bernstein

M. J.

Berry

D. R.

Bialobrzeska

Binan

E. D.

Bocian

Brandt

M. J.

Busching

. . .Nosek

B. A.

(2018). Many labs 2: Investigating variation in replicability across sample and setting. Advances in Methods and Practices in Psychological Science, 1(4), 443–490. https://doi.org/10.1177/2515245918810225

26.

Lakens

(2013). Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology, 4, Article 863. https://doi.org/10.3389/fpsyg.2013.00863

27.

Lakens

(2017). Equivalence tests: A practical primer for t tests, correlations, and meta-analyses. Social Psychological and Personality Science, 8(4), 355–363. https://doi.org/10.1177/1948550617697177

28.

LeBel

E. P.

Borsboom

Giner-Sorolla

Hasselman

Peters

K. R.

Ratliff

K. A.

Smith

C. T.

(2013). PsychDisclosure.org: Grassroots support for reforming reporting standards in psychology. Perspectives on Psychological Science, 8(4), 424–432. https://doi.org/10.1177/1745691613491437

29.

Lortie-Forgues

Inglis

(2019). Most rigorous large-scale educational RCTs are uninformative: Should we be concerned? Educational Researcher, 48(3), 158–166. https://doi.org/10.3102/0013189X19832850

30.

Makel

M. C.

Plucker

J. A.

(2014). Facts are more important than novelty: Replication in the education sciences. Educational Researcher, 43(6), 304–316. https://doi.org/10.3102/0013189X14545513

31.

Makel

M. C.

Smith

K. N.

McBee

M. T.

Peters

S. J.

Miller

E. M.

(2019). A path to greater credibility: Large-scale collaborative education research. AERA Open, 5(4). https://doi.org/10.1177/2332858419891963

32.

McBee

M. T.

Makel

M. C.

Peters

S. J.

Matthew

M. S.

(2018). A call for open science in giftedness research. Gifted Child Quarterly, 62(4), 374–388. doi: 10.1177/0016986218784178

33.

Nosek

B. A.

Alter

Banks

G. C.

Borsboom

Bowman

S. D.

Breckler

S. J.

Buck

Chambers

C. D.

Chin

Christensen

Contestabile

Dafoe

Eich

Freese

Glennerster

Goroff

Green

D. P.

Hesse

Humphreys

. . .Yarkoni

(2015). Promoting an open research culture. Science, 348(6242), 1422–1425. https://doi.org/10.1126/science.aab2374

34.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716

35.

Pashler

Harris

C. R.

(2012). Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science, 7(6), 531–536. https://doi.org/10.1177/1745691612463401

36.

Pigott

T. D.

Valentine

J. C.

Polanin

J. R.

Williams

R. T.

Canada

D. D.

(2013). Outcome-reporting bias in education research. Educational Researcher, 42(8), 424–432. https://doi.org/10.3102/0013189X13507104

37.

Polanin

J. R.

Tanner-Smith

E. E.

Hennessy

E. A.

(2016). Estimating the difference between published and unpublished effect sizes: A meta-review. Review of Educational Research, 86(1), 207–236. https://doi.org/10.3102/0034654315582067

38.

Prinz

Schlange

Asadullah

(2011). Believe it or not: How much can we rely on published data on potential drug targets? Nature Reviews Drug Discovery, 10, 712–713. https://doi.org/10.1038/nrd3439-c1

39.

Rosenthal

(1979). The file drawer problem and tolerance for null results. Psychological Bulletin, 86(3), 638–641. https://doi.org/10.1037//0033-2909.86.3.638

40.

Rotton

Foos

P. W.

Vanmeek

Levitt

(1995). Publication practices and the file drawer problem: A survey of published authors. Journal of Social Behavior and Personality, 10(1), 1–13.

41.

Sham

Smith

(2014). Publication bias in studies of an applied behavior-analytic intervention: An initial analysis. Journal of Applied Behavior Analysis, 47(3), 663–678. https://doi.org/10.1002/jaba.146

42.

Simmons

J. P.

Nelson

L. D.

Simonsohn

(2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632

43.

Simmons

J. P.

Nelson

L. D.

Simonsohn

(2018). False-positive citations. Perspectives on Psychological Science, 13(2), 255–259. https://doi.org/10.1177/1745691617698146

44.

Spellman

B. A.

(2012). Introduction to the special section: Data, data, everywhere . . . especially in my file drawer. Perspectives on Psychological Science, 7(1), 58–59. https://doi.org/10.1177/1745691611432124

45.

van der Zee

Reich

. (2018). Open education science. AERA Open, 4(3), 1–15. https://doi.org/10.1177/2332858418787466

46.

Vazire

(2018). Implications of the credibility revolution for productivity, creativity, and progress. Perspectives on Psychological Science, 13(4), 411–417. https://doi.org/10.1177/1745691617751884

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.47 MB

Both Questionable and Open Research Practices Are Prevalent in Education Research

Abstract

Keywords

Get full access to this article

References

Supplementary Material