Sage Journals: Discover world-class research

Abstract

The current crisis of confidence in psychological science has spurred on field-wide reforms to enhance transparency, reproducibility, and replicability. To solidify these reforms within the scientific community, student courses on open science practices are essential. Here we describe the content of our Research Master course “Good Research Practices” which we have designed and taught at the University of Amsterdam. Supported by Chambers’ recent book The 7 Deadly Sins of Psychology, the course covered topics such as QRPs, the importance of direct and conceptual replication studies, preregistration, and the public sharing of data, code, and analysis plans. We adopted a pedagogical approach that: (a) reduced teacher-centered lectures to a minimum; (b) emphasized practical training on open science practices; and (c) encouraged students to engage in the ongoing discussions in the open science community on social media platforms.

Keywords

Open science education meta-science replication crisis

Introduction

Over the last eight years, psychological research has been in the midst of a “crisis of confidence” (e.g., Pashler & Wagenmakers, 2012; Simmons, Nelson, & Simonsohn, 2011). Central to the crisis is the increasing realization that common research practices may in fact be deeply problematic. Examples include poor study design (i.e., low statistical power; Button et al., 2013; Ioannidis, 2005), the field's reluctance to conduct direct replication studies (Pashler & Harris, 2012; Schmidt, 2009), and a bias to selectively report positive results (Francis, 2013; Scargle, 1999). Moreover, many researchers self-admit to the use of so-called questionable research practices (QRPs; John, Loewenstein, & Prelec, 2012). Hidden from the reader, QRPs exploit researchers' degrees of freedom in study design and analysis in order to produce significant findings. For instance, researchers may decide to stop data collection when the result reaches significance, exclude data points based on their impact on the results, or report unexpected findings as having been predicted from the start (Kerr, 1998; Simmons et al., 2011). The detrimental effect of these practices is evident from recent surveys and large-scale replication projects. For instance, a survey among over 1,500 scientists revealed that 90% believe there is indeed a crisis, with 52% observing a “significant crisis” (Baker, 2016). These perceptions are substantiated by large-scale replication efforts, which demonstrated replication rates from 36% to 77% (Camerer et al., 2018; Klein et al., 2014, 2018; Open Science Collaboration, 2015).

To combat the crisis of confidence, the scientific community has begun to adopt research standards that reduce cherry-picking and significance chasing. For instance, an effective practice that has quickly gained popularity is preregistration. When preregistering their studies, researchers outline their analysis plan before the data are collected. Because the analysis pipeline cannot be tailored to the data, researchers protect themselves against hindsight bias and other QRPs that may unwittingly contaminate the results. Researchers can choose to either preregister their study independently or integrate preregistration with the peer-review process (i.e., in the form of a Registered Report; Chambers, 2013). In addition, the scientific community has launched various initiatives to increase transparency. For instance, to encourage data sharing, Morey et al. (2016) started the Peer Reviewers' Openness (PRO) initiative. PRO signatories agree to provide a full review only for articles that share data and materials in a public repository, or provide reasons why this is not possible. Journals have also promoted transparency standards, for instance by signing up to the Transparency and Openness Promotion guidelines (TOP; Nosek et al., 2015), or by providing open science badges for preregistration and sharing of data and materials (Kidwell et al., 2016). Open science advocates have argued that the methodological reforms within the scientific community have been so substantial as to warrant descriptions such as “Revolution 2.0” (Spellman, 2015) or “Credibility Revolution” (Vazire, 2018).

In addition to the reforms within the research community itself, researchers have emphasized the need to overhaul methodological education. For instance, in the survey by Baker (2016), three of the five factors considered most promising for increasing the reproducibility in science were directly related to improvements in scientific training (i.e., “better statistical understanding,” “better mentoring/supervision,” and “better teaching”). Central among the proposed changes are offering lectures on the crisis of confidence and open scientific practices (Chopik, Bremner, Defever, & Keller, 2018; Funder et al., 2014; Munafò et al., 2017).

We believe that a course on good research practices deserves a place in the standard psychology curriculum, and that open scientific practices should be an inherent part of the methodological training of students for several reasons. First, without the proper education, students' opinions on the crisis of confidence tend to be “quite radical, superficial, or even emotional” (Chopik et al., 2018, p. 159). Educating students about the ongoing methodological changes allows them to develop informed opinions on these topics. Second, when students—the next generation of scientists—understand open science practices, they can confidently introduce them in their future labs. Third, students who pursue an academic career will ultimately be evaluated on whether they adhere to these practices. As journals and university policies are making increasing demands on transparency criteria, educating students about these practices seems advisable, if not imperative. Lastly, regardless of students' future career plans, advancing the methodological curriculum also benefits the students' development on a more general level. By following a course on good scientific practices, students learn to recognize scientific studies that meet certain quality standards reflected by, for instance, being preregistered, having open materials and data, being published as a Registered Report, including a power analysis, or reporting effect sizes. As such, a course on open science enhances students' skills to critically evaluate research, be it from the published literature or conducted by themselves, for instance as part of a thesis requirement.

Since 2015 we have offered the open science course “Good Research Practices” at the University of Amsterdam. The course covers the current crisis of confidence in psychological science and outlines attempts by the scientific community to increase the reliability and transparency in the field. “Good Research Practices” is a Research Master course; students generally know basic statistics and have had practical experience with the empirical cycle. This background makes it easier to understand the challenges and advantages of implementing open science practices. Nevertheless, the course is not technical in nature and mostly demands common sense—hence, the material may also be useful for a course for undergraduate students.

In this article, we aim to provide an overview of our “Good Research Practices” in order to assist lecturers who intend to develop a similar course. Below we discuss the course objectives, describe our pedagogical approach, and illustrate the contents of two classes in more detail. Furthermore, we will list the lecture topics together with suggested literature for students. Readers interested in the full course catalogue and materials can access it in our online appendix (accessible via https://osf.io/v3z7q/).¹

General Information

“Good Research Practices” is designed as a seven-week course including a total of 14 two-hour classes. A total of 43 Research Master Psychology students at the University of Amsterdam participated in the course last year (academic year 2018/2019), for which they were awarded six ECTS credits after completion (equivalent to 180 hours of work). Grading was based on a combination of bi-weekly quizzes about the background literature, and on the quality of their short presentations and in-class assignments.

Course Objectives

In general, a course on good research practices should teach students how to critically review the scientific literature and how to conduct open, transparent, and reliable research. In addition, we wanted to immerse students in current debates and recent developments in the open science community. Specifically, our course had four objectives, as follows.

Our first objective was for students to reflect on various types of questionable research practices. In particular, we emphasize that researchers are not immune to biases (e.g., hindsight bias and confirmation bias) that cause them to selectively report analyses that yield publishable findings. To protect themselves against their own biases, researchers must rely on scientific practices that minimize hidden degrees of freedom (Wagenmakers, Wetzels, Borsboom, van der Maas, & Kievit, 2012). As primary course literature we used the book The 7 Deadly Sins of Psychology by Chambers (2017), which presents a clearly written, authoritative, and comprehensive account of the causes and proposed solutions for the current crisis in psychological science.

Our second objective was to engage students in current debates and recent developments in the open science community. Social media platforms constitute a prominent stage for science communication and debates on research methods reforms. These platforms include Twitter, scientific blogs, and podcasts. As part of the curriculum, we encouraged students to stay informed about ongoing discussions and new developments within the open science movement, and to educate their peers in weekly “Newsflashes” on interesting debates, articles, or events. A list with Twitter handles, podcasts, and scientific blogs we recommended is available via https://osf.io/mcqa5/.²

Our third objective was to let our students contribute to the curriculum themselves. We believe that students learn more when they are stimulated to actively participate in the course (e.g., Jang, Reeve, & Halusic, 2016; Reeve, 2016). Therefore, we adopted a flipped-classroom setting to reduce teacher-centered classes to a minimum. In this setting, students give short lectures, design in-class assignments, and lead group discussions.

Our fourth objective was to provide multiple perspectives on the open science movement. Therefore, we invited a series of guest speakers to present their most recent research projects, their perspectives on the developments within the scientific community, and their opinions on possible ways to resolve the crisis. Since the course was designed to illustrate the necessity and benefits of open science, we exclusively invited proponents of the open science movement. At the same time, we tried to select speakers who differ in their level of seniority, and who approach methodological reforms from different angles. In the current installment of the course, the guest speakers included a former student from the Research Master program (Bobby Lee Houtkoop), a science journalist (Hans van Maanen), metascience researchers (Balazs Aczel, Nick Brown, and Olmo van den Akker), and Chris Chambers, who is the chair of the Registered Reports committee at the Center for Open Science (https://cos.io/) and leading force within the open science movement.

Pedagogical Approach

In line with our course objectives, we alternated regular classes with classes organized by students. Lectures in regular classes were given either by us or one of our guest speakers, and focused on the substantial impact that QRPs may have on the reliability of research findings. In particular, we explained why certain research practices can be considered “bad science” (Goldacre, 2009), and how such practices can be detected, and—importantly—avoided. The classes also featured specific in-class assignments and group discussions to deepen students' understanding. In addition, the regular classes covered recent developments and debates within the open science movement; specifically, we reserved the last 20 minutes of each regular class for a “Newsflash” item, where students gave lighting presentations about relevant events, discussions, or articles they encountered on social media platforms that week. It should be noted that discussions following the lightning presentations were led by one of the lecturers who could provide context and insight about the presented topics. These guided discussions are recommended, since students might not be aware that they are exposed to only a selective group of people who typically dominate these debates, and who may not be representative of the entire scientific community.

Classes given by students were structurally similar to regular classes. However, at about 10 minutes each, the student lectures were much shorter than regular lectures, leaving considerable time for active learning during the in-class assignments. Shorter student lectures also allowed us to have multiple groups present each week.

To encourage creativity and originality, students were instructed to base their lectures on relevant topics that had not already been elaborately discussed in their assigned readings. With respect to the in-class assignments, we emphasized that the exercises should have practical value for their peers, that is, the exercises should be training material for open science practices.³ Examples of the current year's in-class assignments are: tutorials on how to preregister a study or share data on the Open Science Framework (https://osf.io); trying out software tools that examine possible anomalies in individual articles (e.g., statcheck; Epskamp & Nuijten, 2016, or SPRITE; Heathers, Anaya, van der Zee, & Brown, 2018); or detecting hidden analytic flexibility in entire research fields (e.g., with a p-curve analysis as proposed by Simonsohn, Nelson, & Simmons, 2014). To illustrate our pedagogical approach, the next two sections describe a regular class and a student-organized class.

Example of a Regular Class: The Sin of Data Hoarding

The fifth week of the course focused on “The Sin of Data Hoarding” (Chambers, 2017), that is, the chapter on data sharing (for a recent special issue see Simons, 2018). As an expert on this topic we invited Bobby Lee Houtkoop, a former student from the same program. Houtkoop recently conducted and published a survey study to reveal reasons why researchers are reluctant to share their data, and what can be done to overcome this reluctance (Houtkoop et al., 2018). In her lecture, Houtkoop discussed the dominant scientific culture in which data sharing is not the norm, even though data sharing offers unequivocal advantages for both the author and the scientific community. In cancer research, for instance, it was found that studies for which data were publicly shared received higher citation rates compared to studies for which data were not available (Piwowar, Day, & Fridsma, 2007). In addition, data sharing may improve the reputation or perceived integrity of the researcher. The scientific community benefits from data sharing since: (a) it increases the longevity of the data; (b ) data can be reanalyzed and reused efficiently (e.g., for meta-analyses); and (c) statistical or reporting errors are more likely to be found (Vanpaemel, Vermorgen, Deriemaecker, & Storms, 2015; Wicherts, Borsboom, Kats, & Molenaar, 2006). Houtkoop then presented the methods and results of the survey study. The survey results demonstrated that data are shared only infrequently. Most respondents acknowledged the benefits and importance of data sharing in general; however, they perceived data sharing as less beneficial for their own research projects. Among the perceived barriers to data sharing are the respondents' belief that data sharing is not a common practice in their fields, their preference to share data only upon request, their perception that data sharing requires additional work, and their perceived lack of training in data sharing. Houtkoop's study sparked a lively discussion among the students about future research, about initiatives that encourage data sharing, but also about limitations of the study. In particular, the students were critical about potential biases in the results due the low response rate of the survey (i.e., a response rate of only about 5% which, however, translated into a sample of 600 respondents) and the self-selection of the respondents.

The end of the class featured a “Newsflash.” In that particular week, the science community was excitedly debating the results of the “Many Labs 2” project (Klein et al., 2018) which had just been published. In this project, the participating research teams conducted high-powered preregistered replications of 28 classic and contemporary findings across many samples and settings. The replication efforts showed that only 54% (i.e., 15 studies) could be replicated. In the newsflash, students discussed the article by Klein et al. (2018), the related news article published in The Atlantic titled “Psychology's replication crisis is running out of excuses” (Yong, 2018), and the BBC radio episode on the replication crisis (BBC Radio 4, 2018).

Example of a Student Class: The Sin of Data Hoarding

The student lecture continued where Houtkoop's study left off. The student presenters emphasized the benefits of data sharing and created a tutorial for their peers on how to archive and share data of simple empirical studies on the Open Science Framework (see also Soderberg, 2018). The objective of this lecture was to encourage their peers to ask their future thesis supervisors' permission to share the collected data in a public repository. The in-class assignment revolved around the Peer Reviewers' Openness initiative (PRO; Morey et al., 2016) mentioned in the introduction. Specifically, the students let their peers create a set of questions for the signatories of the PRO initiative, inquiring about signatories' post-PRO experiences with journals and editors, their attitude towards data sharing in their own research, and whether and how the signatories would improve the initiative. Students were divided into small groups and were instructed to read the article by Morey et al. (2016) on the PRO initiative. Then, each group had to propose concrete questions for the PRO signatories. In a plenary discussion, the students reviewed the questions, selected the ones they found most relevant, and created a survey. Since this exercise generated items that seemed informative and useful, the students who prepared the class decided to continue and execute the survey as a separate research project. Currently, the PRO initiative survey has elicited responses from over 120 of the current 340 signatories for whom email information could be retrieved (i.e., 37.4%).

Topics Covered

Table 1 lists the topics covered in the lectures, including the guest lectures. The table also contains pointers for students to the relevant literature. Most topics follow the chapters of Chambers (2017); however, we added topics that we deemed relevant in the current research debate. For instance, we dedicated one lecture to the recently published and much-debated article by Benjamin et al. (2018), who proposed a more stringent significance threshold for new scientific discoveries. We also discussed analysis blinding—a promising and underused method that allows researchers maximum flexibility while preserving the confirmatory status of the analyses.

Table 1

Topics Covered and Suggested Literature for the Course “Good Research Practices”

Topic	Description
The crisis of confidence	In two classes we covered some of the main events that led to the crisis of confidence: multiple instances of scientific fraud, the wide acceptance of QRPs among researchers, and the preference of journals to publish novel and positive findings. Suggested literature: Pashler & Harris, 2012; Spellman, Gilbert, & Corker, 2018.
Biases in scientific research	This class covered cognitive biases, such as confirmation bias and hindsight bias, that lead researchers to unwittingly present unexpected findings in their data as if they were hypothesized from the beginning. Suggested literature: Chambers, 2017, Chapter 1.
Lack of transparency (with Balazs Aczel)	In this class we argued that whenever part of the scientific process remains hidden from view, the trustworthiness of the associated conclusions is eroded, since QRPs cannot be detected. To combat this issue, researchers will be able to use a transparency checklist (which is nearing completion) that facilitates the disclosure of the transparency and openness-related factors of their study. This lecture was given by Balazs Aczel, the leading researcher of this project. Suggested literature: Simmons et al., 2011; De Groot, 1956/2014.
Hidden flexibility in data analysis	In this class we stressed the point that the reliability of research findings is ensured only when researchers adhere to the empirical cycle. Specifically, we argued that if researchers do not strictly separate between the stage of hypothesis generation and the stage of hypothesis testing, the predictive interpretation is lost. Suggested literature: Chambers, 2017, Chapter 2.
Blinded analyses	In this class we discussed analysis blinding as a valuable addition to study preregistration to avoid hidden flexibility in data analysis. Analysis blinding, just as preregistration, prevents implicit or explicit forms of significance-chasing, but it retains the possibility for the data analyst to account for unexpected features of the data. Suggested literature: MacCoun & Perlmutter, 2015, 2018; Dutilh, Sarafoglou, & Wagenmakers, 2018.
Unreliability of scientific findings	Science depends on direct replications of scientific studies to determine the validity of alleged effects. In this lecture we discussed recent large-scale replication efforts and the impact they had on psychological science. Suggested literature: Chambers, 2017, Chapter 3.
Data hoarding (with Bobby Lee Houtkoop)	This class covered the importance of data sharing and discussed reasons why researchers are still reluctant to share their data. This lecture was given by Bobby Lee Houtkoop, the leading researcher of a recently published survey study that identified these perceived barriers and possible remedial action. Suggested literature: Chambers, 2017, Chapter 4; Houtkoop et al., 2018.
Scientific fraud (with Nick Brown)	In this class we discussed how to detect anomalies in research articles, for instance, by reconstructing plausible samples from descriptive statistics. This lecture was given by Nick Brown, who was involved in the development of these techniques. Suggested literature: Brown & Heathers, 2017; Heathers et al., 2018; Levelt, Drenth, & Noort, 2012; Stapel, 2014.
Overselling scientific findings (with Hans van Maanen)	In this class we discussed how the over-generalization or exaggeration of study conclusions in abstracts and press releases distort the representation of scientific findings in the media. As an expert on this topic we invited science journalist Hans van Maanen, who is known for his columns in the Dutch newspaper De Volkskrant in which he eviscerates published research. Suggested literature: Chambers, 2017, Chapter 5.
Redefining statistical significance	In this class we discussed the recently published paper by Benjamin et al. (2018) in which the authors propose to lower the α-levels for claims of new discoveries from 0.05 to 0.005. Suggested literature: Benjamin et al., 2018 Blogpost articles: Wagenmakers, 2019, Redefine Statistical Significance (Parts I–XVII) YouTube debate on Benjamin et al. (2017): BITSS, 2017.
Statistical errors (with Olmo van den Akker)	Statistical reporting errors can lead to erroneous substantive conclusions. In this class we discussed how researchers can minimize the chance of statistical reporting errors by using software that automatically detects inconsistencies. This lecture was given by Olmo van den Akker, who is part of the Meta-Research Center at Tilburg University that is specialized in scientific misconduct and reproducibiity. Suggested literature: Chambers, 2017, Chapter 6; Epskamp & Nuijten, 2016; Greenland et al., 2016; Nuijten, Hartgerink, van Assen, Epskamp, & Wicherts, 2016.
Registered Reports (with Chris Chambers)	Apart from publishing the course textbook The 7 Deadly Sins of Psychology, Chambers has participated in drafting the TOP guidelines and is the chair of the Registered Reports committee supported by the Center for Open Science. In his class, Chris Chambers shared his experiences of how he first proposed the Registered Report format to the Cortex editorial board, how the initiative was implemented in the journal, and how Registered Reports are having a growing influence on the scientific community. Suggested literature: Chambers, 2017, Chapter 8; Chambers, 2013.

Student Evaluation and Recommendations for Future Courses

Student feedback was highly positive. Students particularly appreciated: (a) the guest lectures; (b) the group discussions about ongoing debates and recent articles; (c) the assigned literature (i.e., the course book and the additional articles), which was perceived as relevant and enjoyable; and (d) the teaching of important practical skills. The perceived work load was deemed appropriate, and students liked the fact that the course was designed to encourage regular work through quizzes and assignments.

Students were most critical about our emphasis on negative facets during regular classes, that is, QRPs and the crisis of confidence. Some students stated that discussing these aspects so frequently made them pessimistic about the current state of science. Furthermore, the students felt the two-hour classes were too short. In particular, students were disappointed that often only one group rather than two groups (as anticipated) could present during the student classes. This lack of time also repeatedly forced us to skip the weekly “Newsflashes.”

We believe the student feedback is constructive and helpful. We agree with the students that scheduling an additional hour for each class will reduce the time pressure. With regard to the focus on negative facets, we believe that the recognition of QRPs and “bad science” (Goldacre, 2009) is essential to motivate the methodological reorientation towards more transparency and rigor; on the other hand, our main objective was to inspire students to embrace open research practices, not to instill a sense of despair. As nicely put by Michèle Nuijten (2019), we want to “turn students into skeptics, not cynics.” Therefore, the next installment of our course will devote a larger proportion of time to the positive changes within the scientific community. For instance, we suggest to reconstruct the lecture “Unreliability of Scientific Findings.” During this lecture, we focused mainly on the importance of conducting direct replications to determine the validity of alleged effects, and emphasized the lack thereof in the scientific literature. However, this lecture offers the opportunity to highlight recent large-scale replication efforts and multi-lab collaborations, such as the Open Science Collaboration (2015), the Many Labs projects (Klein et al., 2014, 2018), the ManyBabies project (Frank et al., 2017), and the Psychological Science Accelerator (Moshontz et al., 2018). In addition to a lecture which gives students a general overview on these collaborative efforts, it would be particularly interesting to invite a guest speaker who participated in one of these collaborations to share his or her experiences in working and publishing in such an environment.

Additionally, we would like to replace the lecture “Scientific Fraud” with a lecture on “Open Science within the University of Amsterdam” to educate our students on the concrete steps our university has taken to improve reproducibility, transparency, and openness. For instance, the ethical committee of the psychology department demands a detailed methods and analysis plan as precondition to grant ethical approval for any research project; similarly, students are requested to write their introduction, methods, and analysis plan of their internship and thesis projects before data collection. Additionally, we would like to highlight the methodological and statistical consulting which is offered to both researchers and students, as well as several open science initiatives that were launched recently.⁴

Concluding Remarks

Across 14 lectures, the course “Good Research Practices” taught psychology students about the causes of the crisis of confidence and about recent attempts by the scientific community to increase transparency, reproducibility, and replicability. In addition, students acquired practical skills on how to conduct research that is open, transparent, and reliable. We believe that this learning success was primarily due to the active role we gave students in our course. By being instructed to create lectures and in-class assignments that go beyond the assigned literature, students were able to choose articles covering topics that they consider most relevant for their future research projects. Furthermore, the students developed a sense of ownership for the lectures and in-class assignments, which facilitated ambitious student projects such as the PRO initiative survey.

As the scientific culture changes, practical knowledge on open scientific practices is becoming an increasingly important scientific skill. A course on this topic helps students not only to develop critical thinking, but also to get excited about conducting research that distinguishes sharply between its exploratory and confirmatory components. We hope that courses on open science practices inspire the future generation of psychological researchers to deliver psychology from the deadly sins that have so stained it in the past.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by a Netherlands Organisation for Scientific Research (NWO) grant to AS (406-17-568), a Veni grant from the NWO to DM (451-15-010), as well as a Vici grant from the NWO to EJW (016.Vici.170.083).

Notes

Author biographies

Alexandra Sarafoglou is a PhD candidate in the department of Psychological Methods at the University of Amsterdam. She founded the Open Science Community Amsterdam, and is a member of the JASP development team (). She has recently received an NWO research talent grant to develop methods for analysis blinding, validate them empirically and make them accessible in the open-source softwares R and JASP and through teaching materials. Her teaching interests include Bayesian inference and open scientific practices. Since 2016, she has been a teaching assistant on the course “Good Research Practices.”

Suzanne Hoogeveen is a PhD candidate in the department of Social Psychology at the University of Amsterdam. Together with Alexandra Sarafoglou she founded the Open Science Community Amsterdam. She specializes in replication research in the field of psychology of religion. Her teaching interests include the psychology of religion and spirituality, and open scientific practices. Since 2017, she has been a teaching assistant on the course “Good Research Practices.”

Dora Matzke is an assistant professor in the department of Psychological Methods at the University of Amsterdam. She is an active member in the open science and reproducibility community. Her areas of specialization include formal models of response inhibition, multinomial processing tree models, and Bayesian inference, and she was recently awarded the Early Career Award from the Society for Experimental Psychology and Cognitive Science. Her teaching interests include Bayesian inference, statistics, and open scientific practices. Since 2015, she has been a lecturer on the course “Good Research Practices.”

Eric-Jan Wagenmakers is a full professor in the department of Psychological Methods at the University of Amsterdam. He has participated in drafting the Transparency and Openness Promotion (TOP) guidelines and has been a strong advocate of preregistration. He was also a member of the 2018 KNAW committee on replication research. His teaching interests include Bayesian inference, and open scientific practices. Since 2015, he has been the main lecturer on the course “Good Research Practices.”

References

Baker

(2016) 1,500 scientists lift the lid on reproducibility. Nature News 533: 452.

BBC Radio 4. (2018). The replication crisis [Radio Broadcast]. BBC. Retrieved from https://www.bbc.co.uk/programmes/m00013p9.

Benjamin

D. J.

Berger

J. O.

Johannesson

Nosek

B. A.

Wagenmakers

E.-J.

Berk

Johnson

V. E.

(2018) Redefine statistical significance. Nature Human Behaviour 2: 6–10.

BITSS. (2017). 2017 BITSS Annual Meeting—Day 1 [YouTube movie]. Berkeley Initiative for Transparency in the Social Sciences (BITSS). Retrieved from https://www.youtube.com/watch?v=4IgSVxkXMaM.

Brown

Heathers

(2017) The GRIM test: A simple technique detects numerous anomalies in the reporting of results in psychology. Social Psychological and Personality Science 8: 363–369.

Button

K. S.

Ioannidis

J. P. A.

Mokrysz

Nosek

B. A.

Flint

Robinson

E. S. J.

Munafò

M. R.

(2013) Power failure: Why small sample size undermines the reliability of neuroscience. Nature Reviews Neuroscience 14: 1–12.

Camerer

C. F.

Dreber

Holzmeister

T.-H.

Huber

Johannesson

(2018) Evaluating replicability of social science experiments in Nature and Science. Nature Human Behaviour 2: 637–644.

Chambers

C. D.

(2013) Registered reports: A new publishing initiative at. Cortex. Cortex 49: 609–610.

Chambers, C. D. (2017). The seven deadly sins of psychology: A manifesto for reforming the culture of scientific practice. Princeton, NJ: Princeton University Press.

10.

Chopik

Bremner

Defever

Keller

(2018) How (and whether) to teach undergraduates about the replication crisis in psychological science. Teaching of Psychology 45: 158–163.

11.

Davis, W. E., Sellers, P. D., Miller, E., Machluf, K., Peters, P., Scott J, Salvatore, J., … Steltenpohl, C. (2017, Sep). Integrating Open Science in the undergraduate psychology curriculum. OSF. Retrieved from osf.io/f82ej.

12.

De Groot

A. D.

(1956/2014) The meaning of “significance” for different types of research. Translated and annotated by E.-J. Wagenmakers, D. Borsboom, J. Verhagen, R. Kievit, M. Bakker, A. Cramer, D. Matzke, D. Mellenbergh, & H. L. J. van der Maas. Acta Psychologica 148: 188–194.

13.

Dutilh, G., Sarafoglou, A., & Wagenmakers, E.-J. (2018). Flexible yet fair: Blinding analyses in experimental psychology. Manuscript submitted for publication. Retrieved from https://osf.io/hg8m7/.

14.

Epskamp, S., & Nuijten, M. B. (2016). statcheck: Extract statistics from articles and recompute p-values (R package version 1.3.0).

15.

Francis

(2013) Replication, statistical consistency, and publication bias. Journal of Mathematical Psychology 57: 153–169.

16.

Frank

M. C.

Bergelson

Bergmann

Cristia

Floccia

Gervain

Yurovsky

(2017) A collaborative approach to infant research: Promoting reproducibility, best practices, and theory-building. Infancy 22: 421–435.

17.

Funder

D. C.

Levine

J. M.

Mackie

D. M.

Morf

C. C.

Sansone

Vazire

West

S. G.

(2014) Improving the dependability of research in personality and social psychology: Recommendations for research and educational practice. Personality and Social Psychology Review 18: 3–12.

18.

Goldacre, B. (2009). Bad science. London, UK: Fourth Estate.

19.

Greenland

Senn

S. J.

Rothman

K. J.

Carlin

J. B.

Poole

Goodman

S. N.

Altman

D. G.

(2016) Statistical tests, p values, confidence intervals, and power: A guide to misinterpretations. European Journal of Epidemiology 31: 337–350.

20.

Heathers, J., Anaya, J., van der Zee, T., & Brown, N. J. (2018). Recovering data from summary statistics: Sample Parameter Reconstruction via Iterative TEchniques (SPRITE). Manuscript submitted for publication. Retrieved from https://peerj.com/preprints/26968.pdf.

21.

Houtkoop

B. L.

Chambers

Macleod

Bishop

D. V.

Nichols

T. E.

Wagenmakers

E.-J.

(2018) Data sharing in psychology: A survey on barriers and preconditions. Advances in Methods and Practices in Psychological Science 1: 70–85.

22.

Ioannidis

(2005) Why most published research findings are false. PLoS Medicine 2: 696–701.

23.

Jang

Reeve

Halusic

(2016) A new autonomy-supportive way of teaching that increases conceptual learning: Teaching in students' preferred ways. The Journal of Experimental Education 84: 686–701.

24.

John

Loewenstein

Prelec

(2012) Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science 23: 524–532.

25.

Kerr

(1998) HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review 2: 196–217.

26.

Kidwell

M. C.

Lazarević

L. B.

Baranski

Hardwicke

T. E.

Piechowski

Falkenberg

L.-S.

Nosek

B. A.

(2016) Badges to acknowledge open practices: A simple, low cost, effective method for increasing transparency. PLOS Biology 14: e1002456.

27.

Klein

Ratliff

Vianello

Adams

R. B.

Jr. Bahník

Bernstein

Nosek

(2014) Investigating variation in replicability: A “Many Labs” replication project. Social Psychology 45: 142–152.

28.

Klein

Vianello

Hasselman

Adams

Alper

Nosek

(2018) Many Labs 2: Investigating variation in replicability across sample and setting. Advances in Methods and Practices in Psychological Science 1: 443–490.

29.

Levelt, W., Drenth, P., & Noort, E. (2012). Flawed science: The fraudulent research practices of social psychologist Diederik Stapel. Retrieved from https://pure.mpg.de/rest/items/item_1569964/component/file_1569967/content.

30.

MacCoun

Perlmutter

(2015) Hide results to seek the truth: More fields should, like particle physics, adopt blind analysis to thwart bias. Nature 526: 187–190.

31.

MacCoun, R., & Perlmutter, S. (2018). Blind analysis as a correction for confirmatory bias in physics and in psychology. In S. O. Lilienfeld & I. Waldman (Eds.), Psychological science under scrutiny: Recent challenges and proposed solutions (pp. 297–322). New York, NY: John Wiley and Sons.

32.

Morey

R. D.

Chambers

C. D.

Etchells

P. J.

Harris

C. R.

Hoekstra

Lakens

Zwaan

R. A.

(2016) The peer reviewers' openness initiative: Incentivizing open research practices through peer review. Royal Society Open Science 3: 150547.

33.

Moshontz

Campbell

Ebersole

C. R.

IJzerman

Urry

H. L.

Forscher

P. S.

Chartier

C. R.

(2018) The Psychological Science Accelerator: Advancing psychology through a distributed collaborative network. Advances in Methods and Practices in Psychological Science 1: 501–515.

34.

Munafò

Nosek

Bishop

Button

Chambers

Du Sert

Ioannidis

(2017) A manifesto for reproducible science. Nature Human Behaviour 1: 0021.

35.

Nosek

Alter

Banks

G. C.

Borsboom

Bowman

S. D.

Breckler

S. J.

Yarkoni

(2015) Promoting an open research culture. Science 348: 1422–1425.

36.

Nuijten, M. B. (2019). Teaching Open Science: Turn students into skeptics, not cynics. Conference talk at ICPS in Paris. Retrieved from https://static1.squarespace.com/static/5b03f9039f87701b3fd495cf/t/5c87bff753450a6e2f7502a2/1552400378346/Nuijten_TeachingOpenScience.pdf.

37.

Nuijten

M. B.

Hartgerink

C. H.

van Assen

M. A.

Epskamp

Wicherts

J. M.

(2016) The prevalence of statistical reporting errors in psychology (1985–2013). Behavior Research Methods 48: 1205–1226.

38.

Open Science Collaboration, T (2015) Estimating the reproducibility of psychological science. Science 349: aac4716.

39.

Open Science MOOC Team. (2019). Open Science MOOC [Website] https://opensciencemooc.eu.

40.

Pashler

Harris

C. R.

(2012) Is the replicability crisis overblown? Three arguments examined. Perspectives on Psychological Science 7: 531–536.

41.

Pashler

Wagenmakers

E.-J.

(2012) Editors' introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science 7: 528–530.

42.

Piwowar

Day

Fridsma

(2007) Sharing detailed research data is associated with increased citation rate. Plos ONE 2: e308.

43.

Reeve, J. (2016). Autonomy-supportive teaching: What it is, how to do it. In W. Liu, J. Wang, & R. Ryan (Eds.), Building autonomous learners: Perspectives from research and practice using self–determination theory (pp. 129–152). Singapore: Springer.

44.

Scargle, J. (1999). Publication bias (the “file–drawer problem”) in scientific inference. arXiv.

45.

Schmidt

(2009) Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology 13: 90–100.

46.

Simmons

Nelson

Simonsohn

(2011) False–positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science 22: 1359–1366.

47.

(2018) Challenges in making data available [Invited Forum]. In: Simons

(ed) Advances in Methods and Practices in Psychological Science 1.

48.

Simonsohn

Nelson

L. D.

Simmons

J. P.

(2014) P-curve: A key to the file-drawer. Journal of Experimental Psychology: General 143: 534–547.

49.

Soderberg

(2018) Using OSF to share data: A step-by-step guide. Advances in Methods and Practices in Psychological Science 1: 115–120.

50.

Spellman

B. A.

(2015) A short (personal) future history of revolution 2.0. Perspectives on Psychological Science 10: 886–899.

51.

Spellman, B. A., Gilbert, E. A., & Corker, K. S. (2018). Open science. In J. Wixted & E.-J. Wagenmakers (Eds.), Stevens' handbook of experimental psychology and cognitive neuroscience (4th ed.), Volume 5: Methodology (pp. 297–322). New York, NY: Wiley.

52.

Stapel, D. (2014). Faking science: A true story of academic fraud. Translated by Nicholas JL Brown. [Jan 14, 2014]. Retrieved from http://beinspired.no/wp-content/uploads/2016/03/FakingScience-20141214.pdf.

53.

Vanpaemel

Vermorgen

Deriemaecker

Storms

(2015) Are we wasting a good crisis? The availability of psychological research data after the storm. Collabra 1: 1–5.

54.

Vazire

(2018) Implications of the credibility revolution for productivity, creativity, and progress. Perspectives on Psychological Science 13: 411–417.

55.

Wagenmakers, E.-J. (2019). Bayesian Spectacles [Scientific Blog]. https://www.bayesianspectacles.org.

56.

Wagenmakers

E.-J.

Wetzels

Borsboom

van der Maas

Kievit

(2012) An agenda for purely confirmatory research. Perspectives on Psychological Science 7: 632–638.

57.

Wicherts

Borsboom

Kats

Molenaar

(2006) The poor availability of psychological research data for reanalysis. American Psychologist 61: 726–728.

58.

Yong, E. (2018). Psychology's replication crisis is running out of excuses [Online news article]. The Atlantic. Retrieved from https://www.theatlantic.com/science/archive/2018/11/ psychologys-replication-crisis-real/576223/.

Teaching Good Research Practices: Protocol of a Research Master Course

Abstract

Keywords

Introduction

General Information

Course Objectives

Pedagogical Approach

Example of a Regular Class: The Sin of Data Hoarding

Example of a Student Class: The Sin of Data Hoarding

Topics Covered

Student Evaluation and Recommendations for Future Courses

Concluding Remarks

Footnotes

Declaration of Conflicting Interests

Funding

Notes

Author biographies

References