Potential for Meta-Scientific Inquiry to Improve the Usefulness of HRD Research Outcomes for Practice

Abstract

The Problem

Meta-science, that is, science about science, is an application of scientific method to explain and control the methods and enterprise of science. The field of inquiry of meta-science emerged originally in reaction to problems with reproducibility and replicability of seemingly settled science. These same problems could affect research evidence conducted by human resource development (HRD) researchers and, as a result, could limit the applicability of HRD research outcomes to HRD practice.

The Solution

Discussed in this article are examples of meta-scientific issues and solutions that ultimately affect the transition of knowledge derived through research to practice in the field of HRD. Problems with the reproducibility and replication of research conducted in the field are detailed as are meta-scientific issues involved in judging the quality of scientific work, methodology-agnostic applications of meta-scientific inquiry, requirements for sharing data and code, and the need to let theory guide research.

The Stakeholders

Researchers would benefit from the considerations of meta-scientific concerns in the design, conduct, and reporting of research to improve the replicability and reproducibility of research outcomes. Knowledge of meta-scientific principles and cautions by adopters of research outcomes would inject healthy skepticism into decisions about whether research outcomes are worthy for application in practice or to guide future research.

Keywords

knowledge management measurement/metrics evidence-based practice evaluation

Get full access to this article

View all access options for this article.

References

Adolph

K. E.

Gilmore

R. O.

Freeman

Sanderson

Millman

(2012). Toward open behavioral science. Psychological Inquiry, 23, 244-247. doi:10.1080/1047840X.2012.705133

APS announces third replication project. (2014, October 28). Retrieved from https://www.psychologicalscience.org/publications/observer/obsonline/aps-announces-third-replication-project.html

Arlot

Celisse

(2010). A survey of cross-validation procedures for model selection. Statistics Surveys, 4, 40-79. doi:10.1214/09-SS054

Baggerly

K. A.

Coombes

K. R.

(2009). Deriving chemosensitivity from cell lines: Forensic bioinformatics and reproducible research in high-throughput biology. The Annals of Applied Statistics, 3, 1309-1334. doi:10.1214/09-aoas291

Barker

(2007). The UK research assessment exercise: The evolution of a national research evaluation system. Research Evaluation, 16, 3-12. doi:10.3152/095820207x190674

Bergstrom

C. T.

(2010). How to improve the use of metrics. Nature, 465, 870-872. doi:10.1038/465870a

Bohannon

(2015). Many psychology papers fail replication test. Science, 349, 910-911. doi:10.1126/science.349.6251.910

Brandimarte

(2014). Handbook in Monte Carlo simulation: Applications in financial engineering, risk management, and economics. New York, NY: Wiley.

Campbell

D. T.

(1979). Assessing the impact of planned social change. Evaluation and Educational Planning, 2(1), 67-90. doi:10.1016/0149-7189(79)90048-x

10.

OpenSDP. (n.d.). Retrieved from https://opensdp.github.io/data/

11.

Davey Smith

. (2002). Data dredging, bias, or confounding. British Medical Journal, 325, 1437-1438. doi:10.1136/bmj.325.7378.1437

12.

Dearborn

Marks

Trimble

(2018). The changing influence of journal data sharing policies on local RDM practices. International Journal of Digital Curation, 12, 376-389. doi:10.2218/ijdc.v12i2.583

13.

Fanelli

(2011). Negative results are disappearing from most disciplines and countries. Scientometrics, 90, 891-904. doi:10.1007/s11192-011-0494-7

14.

Fleck

(2013). The impact factor fetishism. European Journal of Sociology, 54, 327-356. doi:10.1017/s0003975613000167

15.

Freese

Peterson

(2017). Replication in social science. Annual Review of Sociology, 43, 147-165. doi:10.1146/annurev-soc-060116-053450

16.

Gilbert

D. T.

King

Pettigrew

Wilson

T. D.

(2016). Comment on “estimating the reproducibility of psychological science.” Science, 351, 1037-1037.

17.

Gilmore

R. O.

Adolph

K. E.

Millman

D. S.

(2016). Curating identifiable data for sharing: The databrary project. Proceedings of the 2016 New York Scientific Data Summit. doi:10.1109/NYSDS.2016.7747817

18.

Given

(2008). The SAGE encyclopedia of qualitative research methods. London, England: SAGE. doi:10.4135/9781412963909.n464

19.

Guba

E. G.

(1987). What have learned about naturalistic evaluation? American Journal of Evaluation, 8, 23-43. doi:10.1177/109821408700800102

20.

Head

M. L.

Holman

Lanfear

Kahn

A. T.

Jennions

M. D.

(2015). The extent and consequences of p-hacking in science. PLoS Biology, 13(3), e1002106. doi:10.1371/journal.pbio.1002106

21.

Herndon

Ash

Pollin

(2013). Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff. Cambridge Journal of Economics, 38, 257-279. doi:10.1093/cje/bet075

22.

Ioannidis

J. P. A.

(2005). Why most published research findings are false. CHANCE, 18, 40-47. doi:10.1080/09332480.2005.10722754

23.

Ioannidis

J. P. A.

(2018). Meta-research: Why research on research matters. PLoS Biology, 16(3), e2005468. doi:10.1371/journal.pbio.2005468

24.

IPUMS. (n.d.). Available from https://www.ipums.org/

25.

Journal Instructions. (n.d.). Retrieved from http://wiki.datadryad.org/Journal_instructions

26.

Leek

J. T.

Peng

R. D.

(2015). Opinion: Reproducible research can still be wrong: Adopting a prevention approach. Proceedings of the National Academy of Sciences of the United States of America, 112, 1645-1646. doi:10.1073/pnas.1421412111

27.

Lucero

Zayatz

(2010). Lecture Notes in Computer Science: The microdata analysis system at the U.S. Census Bureau. doi:10.1007/978-3-642-15838-4_21

28.

Makel

M. C.

Plucker

J. A.

(2014a). Creativity is more than novelty: Reconsidering replication as a creativity act. Psychology of Aesthetics, Creativity, and the Arts, 8, 27-29. doi:10.1037/a0035811

29.

Makel

M. C.

Plucker

J. A.

(2014b). Facts are more important than novelty. Educational Researcher, 43, 304-316. doi:10.3102/0013189x14545513

30.

Manski

C. F.

(2011). Policy analysis with incredible certitude. The Economic Journal, 121, F261-F289. doi:10.1111/j.1468-0297.2011.02457.x

31.

Meehl

P. E.

(1990). Why summaries of research on psychological theories are often uninterpretable. Psychological Reports, 66, 195-244. doi:10.2466/pr0.1990.66.1.195

32.

Muller

J. Z.

(2018). The tyranny of metrics. Princeton, NJ: Princeton University Press.

33.

National Longitudinal Surveys. (n.d). Retrieved from https://www.bls.gov/nls/home.htm

34.

NORC at the University of Chicago. (n.d). General Social Survey: Confidence in scientific community, 1973-2018. Retrieved from https://gssdataexplorer.norc.org/variables/458/vshow

35.

Nosek

B. A.

Spies

J. R.

Motyl

(2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7, 615-631. doi:10.1177/1745691612459058

36.

Nuijten

M. B.

Hartgerink

C. H. J.

van Assen

M. A. L. M.

Epskamp

Wicherts

J. M.

(2016). The prevalence of statistical reporting errors in psychology (1985–2013). Behavior Research Methods, 48, 1205-1226. doi:10.3758/s13428-015-0664-2

37.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), Article aac4716. doi:10.1126/science.aac4716

38.

Open Science Framework. (n.d). Available from https://osf.io/

39.

Research. (n.d). Retrieved from http://metrics.stanford.edu/research

40.

Sale

J. E. M.

Lohfeld

L. H.

Brazil

(2002). Revisiting the quantitative-qualitative debate: Implications for mixed methods research. Quality & Quantity, 36, 43-53. doi:10.1023/A:1014301607592

41.

Slevitch

(2011). Qualitative and quantitative methodologies compared: Ontological and epistemological perspectives. Journal of Quality Assurance in Hospitality & Tourism, 12, 73-81. doi:10.1080/1528008x.2011.541810

42.

Spellman

B. A.

(2012). Data, data, everywhere . . . especially in my file drawer. Perspectives on Psychological Science, 7, 58-59. doi:10.1177/1745691611432124

43.

Teixeira

Silva

J. A.

(2017). The ethics of peer and editorial requests for self-citation of their work and journal. Medical Journal Armed Forces India, 73, 181-183. doi:10.1016/j.mjafi.2016.11.008

44.

Templ

(2017). Synthetic data. In Templ

(Ed.), Statistical disclosure control for microdata (pp. 157–179). Berlin, Germany: Springer. doi:10.1007/978-3-319-50272-4_6

45.

Umesh

R. A.

(2013). Sharing the ideas of meta–science to improve quality of research. Kathmandu University Medical Journal, 11, 75-77. doi:10.3126/kumj.v11i1.11032

46.

Van Bavel

J. J.

Mende-Siedlecki

Brady

W. J.

Reinero

D. A

. (2016). Contextual sensitivity in scientific reproducibility. Proceedings of the National Academy of Sciences of the United States of America, 113, 6454-6459. doi:10.1073/pnas.1521897113

47.

Vanpaemel

Vermorgen

Deriemaecker

Storms

(2015). Are we wasting a good crisis? The availability of psychological research data after the storm. Collabra, 1(1), Article 3. doi:10.1525/collabra.13

48.

Wicherts

J. M.

Borsboom

Kats

Molenaar

(2006). The poor availability of psychological research data for reanalysis. American Psychologist, 61, 726-728. doi:10.1037/0003-066x.61.7.726

49.

Wilhite

A. W.

Fong

E. A.

(2012). Coercive citation in academic publishing. Science, 335, 542-543. doi:10.1126/science.1212540

50.

Yong

(2012). Replication studies: Bad copy. Nature, 485, 298-300. doi:10.1038/485298a