Sage Journals: Discover world-class research

Abstract

The preregistration of research plans and hypotheses may prevent publication bias and questionable research practices. We incorporated a modified version of the preregistration process into an undergraduate capstone research course. Students completed a standard preregistration form during the planning stages of their research projects as well as surveys about their knowledge of preregistration. Based on survey results, our senior-level psychology students lacked knowledge of importance of the preregistration movement in the sciences but could anticipate some of its benefits. Our review of the completed preregistration assignment suggested that students struggle with data analysis decision-making but generally perceive preregistration as a helpful planning tool. We discuss the value of a preregistration assignment for generating discussions of research practice and ethics.

Keywords

Preregistration questionable research practices undergraduate preregistration assignment

Introduction

The Open Science movement encourages research collaboration and strives to increase the accessibility and transparency of scientific work. The Center for Open Science (COS), co-founded by psychologist Brian Nosek, is at the forefront of the movement. In support of its mission to “increase openness, integrity, and reproducibility of research” (COS, n.d.), COS offers a suite of services that support preregistration, a process whereby researchers “specify, in as much detail as they can, their plans for a study (e.g., number and nature of subjects, stimulus materials, procedures, measures, rules for excluding data, plans for data analysis predictions/hypotheses, etc.) and post those plans in a time-stamped, locked file in an online repository that can be accessed by editors and reviewers (and, ultimately, by readers)” (Association for Psychological Science, n.d.). When researchers choose to preregister a study, they can further elect to make it public and searchable. Open Science Framework (OSF) Registries (OSF Registries, n.d.) is an example of an online repository, a searchable database of open registries across scientific disciplines. As of May 16, 2019, there were 290,831 searchable registrations in the repository.

Preregistration existed in various, though often partial, forms before the advent of OSF (Nosek & Lindsey, 2018). For example, the thesis proposal stage of graduate training requires identification and commitment to a research methodology and statistical strategy. In grant applications, researchers typically specify research design and data analysis plans, including a projected sample size based on power analysis. Nosek, Ebersole, DeHaven, and Mellor (2018) advocate for widespread adoption of preregistration as a way to clearly distinguish prediction from postdiction. By separating, through advance documentation, the steps of hypothesis generation and hypothesis testing, researchers combat the individual hindsight bias and systemic publication incentives that can result in reports of post-hoc results as “I-knew-it-all-along” predictions, also known as HARKing (Hypothesizing After the Results are Known; Kerr, 1998). A prominent concern about preregistration is that the practice will have deleterious effects on exploratory research (Goldin-Meadow, 2016). Preregistration advocates respond that the practice simply reduces “the frequency at which exploratory research masquerades as having an a priori design” (Gonzales & Cunningham, 2015).

Knowledge and Use of Preregistration Remains Limited

Prominent journals in psychology are implementing incentive structures that incorporate preregistration (e.g., Psychological Science, Perspectives on Psychological Science). It is still early to measure many of the effects of these incentives, but one result of a January 2014 implementation of open science badges at Psychological Science was an improved rate of data sharing (Kidwell et al., 2016). Preregistration is poised to shape the future of publication in academic psychology, and although most students will never publish in psychology journals, preregistration is a practice worth understanding. Incorporating preregistration into research methods training gives students the opportunity to move from factual knowledge to procedural knowledge and the ability to select appropriate procedures and the circumstances for their use. In addition, the preregistration process makes explicit the points at which researchers can slip into questionable practices, including failing to report an element of a study such as a dependent measure or a condition, rounding p-values to achieve significance, and HARKing. Without proper training, students may unintentionally use what John, Loewenstein, and Prelec (2012) describe as “steroids of scientific competition” (p. 524). Not only can preregistration practices benefit students’ own research, but they may enable students to be good consumers of others’ research, a skill every person should have.

The Open Science movement has advanced rapidly, but training and teaching tools are not yet widely available (for very recent examples, see Chopik, Bremner, Defever, & Keller, 2018; Crüwell et al., 2019; Toelch & Ostwald, 2018). A 2017 survey of 1,277 European researchers, primarily from the natural and social sciences, showed that most of the researchers had limited knowledge of Open Science tools and practices (European Commission, 2017); furthermore, the authors of a recent article reported that US psychology instructors are unsure about how and where to integrate relevant conversations into the curriculum (Chopik et al., 2018). In what follows, we discuss our first attempt to expose baccalaureate psychology students to Open Science practices, preregistration in particular; we hope our example will inspire and equip other psychology instructors to do the same.

Incorporating Preregistration in a Capstone Research Course

At our rural southern US university, the Psychology Department is one of the largest on campus, with approximately 200 students. Senior Seminar is a required course for our psychology program; fourth-year baccalaureate degree-seeking students work in groups of two or three to design, conduct, and present a research project. Weekly assignments are scaffolded as students move through the research process (e.g., idea generation, literature search, ethics review, data collection, data analysis, dissemination of findings). The semester culminates with a poster presentation, oral presentation, and a manuscript written in the required style of the American Psychological Association (APA). Below we describe our experience implementing a preregistration assignment.

At the beginning of the Spring 2019 (January—May) semester, we asked the Senior Seminar students (16.67% male) to complete a three-item survey. The goal of the survey was to identify pre-existing knowledge and beliefs, if any, about preregistration. It began with the question “Have you heard of preregistration, publicly declaring and registering your research design/plans/hypotheses before you begin collecting data?” We gave no additional information about preregistration. Consistent with the lack of familiarity observed by others (Chopik et al., 2018), of the 36 students, 35 chose the response “No, I’ve never even heard of it.” One student chose the response “Yes, I’ve heard of it but don’t know much about it.” The second question prompted students who answered “yes” to the first question to describe “what you know about the preregistration process.” All students skipped this question.

The next question was “Why do you think a scientist might choose to preregister a research design or hypothesis before collecting data?” Every student provided at least one idea. We grouped the responses into eight themes (see Table 1). Many of the responses captured a potential benefit of preregistration, including good record-keeping and opportunities for feedback, but only four students accurately predicted the primary motives for preregistration: transparency and accountability. By preregistering, a researcher makes clear to themselves and others (i.e., readers, reviewers) the a priori hypotheses and planned analyses.

Table 1.

Themed Responses from Students to the Question: Why Preregister a Research Design?

Response Type	Number of Students Reporting
Receive feedback/find out if the study design is doable	12
Protect an idea	8
Avoid replication/find out if the study has already been done	7
Collaboration	5
Prevent the changing of a hypothesis	4
Move more quickly through the research or ethics-approval process	3
Make the public aware of the research	2
Keep good records	2

Note. Responses were from 36 students; some provided more than one response.

One-third of the students (n = 12) believed preregistration could be an opportunity to receive feedback on a study design, and this is true for one form of preregistration known as reviewed preregistration, or the Registered Report. In this type of preregistration the researcher sends the detailed research plan to a specific journal before launching the study. Peer review is conducted on the methodology and strong designs are given a provisional acceptance for publication, assuming follow-through on the registered procedure (Nosek & Lakens, 2014). Having identified in our fourth-year students an overall lack of familiarity with the concept of preregistration, we selected two points in the course to incorporate preregistration practices.

The Preregistration Template Assignment

Preregistration typically includes two steps: (a) completion of a preregistration template; and (b) uploading to a secure, time-stamped online depository. Our students completed only the first step. We assigned completion of a preregistration form as one of three tasks required in preparation for a Design Meeting. The Design Meeting is a 30-minute discussion occurring between the instructor and the research team during the third week of the course. It allows us to screen research ideas for feasibility, determine sound connections to past research, and ensure design strength.

The format of a preregistration template serves as a useful organizational tool for the information discussed during the Design Meeting. Although a common criticism of preregistration is that it is (just) a time-consuming administrative exercise, we think the paperwork is hardly burdensome. The time commitment is more of an investment, or reallocation of energy resources, in that careful thinking and planning at the research design stage saves time at the data analysis stage. To design our assignment we examined several preregistration forms. Some preregistration protocols (and their corresponding templates) best serve professional researchers (e.g., Documenting Research in the Empirical Social Sciences; “DRESS Protocol,” 2019). We chose to use the Project TIER (Teaching Integrity in Empirical Science; McIntyre & Le, 2017) template, which is adapted from other preregistration forms available through the Open Science Framework. It combines comprehensiveness and user-friendliness.

The template (see Supplemental material) prompts for and organizes information about variable operationalization, hypotheses, sampling, research design, and the data analysis plan. We provided students with the template and general instructions, but did not walk them through each item (a decision we return to later). Students completed two additional tasks alongside the preregistration template. Preregistration does not require a rationale for the research question and hypotheses, so we asked each research team to write a justification for their hypothesis that incorporated at least five appropriate peer-reviewed references. The TIER preregistration template also lacks a place for procedural details. Therefore, we tasked the research teams with writing a detailed procedural overview, like a script of what researchers and participants would do during the study.

Strengths and Weaknesses of the Preregistration Template

Each of the 15 student research teams had an individual face-to-face Design Meeting, with the completed preregistration template, hypothesis justification, and procedural overview due in hard copy at the meeting. The Design Meeting grade, which constitutes less than 5% of the final course grade, was based primarily on active discussion from all group members during the meeting and thorough, but not necessarily accurate, completion of the required materials. During the discussions, afterward when grading template completion, and through a two-item survey (e.g., “What did you find most difficult about the preregistration process”; “What did you find to be most helpful about the preregistration process?”) administered in the class period following the Design Meetings, we looked for ways to improve the usability of the preregistration template and assignment.

First, we discovered the necessity of guiding students through the template, item by item, with examples, in advance. We (incorrectly) assumed students would understand all elements of the template based on their knowledge from a previous required research methods course, but on the survey, nine students reported general confusion about use of the form and the preregistration process, and eight students said they were confused by particular terms used on the form, in particular the questions about exploratory research questions and exploratory variables. The template also included the question “Did you create new, or modify existing, variables for this study?” with instructions to select all options that apply, including “some variables were created for this study,” “some variables were modified from their original form,” and “some, or all, variables have been used in prior, published research, and no modifications were made” (see Supplemental material). It was clear from our Design Meeting conversations that students need examples and practice in discerning between these options. Measuring heart rate or using a subscale from an existing personality inventory may be straightforward choices, but many of our groups struggled to articulate the ways in which they planned to modify and utilize existing variables. For example, a group studying ageism and cyberbullying embedded their manipulation check and dependent variable measure in an existing cyberbullying incident report, and then wondered if this constituted creation of a variable.

In the meetings we discovered that having the section on variables come before the section on research questions and hypotheses in the template interrupted the flow of our conversation. As instructors, we find it useful to understand the research question before moving to operationalization, and suspect that the reverse order on the template (see Supplemental material) hindered students in the planning process. They may have felt compelled to move through the template in order, not recognizing the value of refining a hypothesis before proceeding to operationally defining variables.

On the survey, six students cited a data analysis decision (selecting a sample size or a p-value) or the data analysis section of the template in its entirety (see Supplemental material), as the most difficult portion of the preregistration process. From past experience with this course we know that the establishment of exclusion criteria, and thresholds for judging a participant’s performance as an outlier in particular, are challenging decisions. At semester’s end we routinely see presentations and reports in which the researchers describe eliminating participants from the dataset because they failed to report a demographic variable, or judging a participant as an outlier because she was the only participant to receive a perfect score on the dependent variable test. The preregistration template highlights connections between methodological decisions and statistical decisions and may prevent misguided post-hoc statistical decision-making. In fact, we anecdotally noted that in subsequent one-on-one meetings focused on data analysis, our research groups returned to their preregistration template to justify decisions about excluding participants based on failed manipulation checks and other criteria.

When grading, we noted an additional, surprising, source of confusion on the template. The very first fillable item on the form is investigator name and affiliation and, separately, the names and affiliations of collaborators. Students seemed unsure about the difference between the two, as evidenced by the myriad combinations of researcher names they attempted to include for each item. Noting this, we plan to build into our course a discussion about collaboration in research and how to determine and acknowledge appropriate credit for contributions to a research project.

From our conversations and survey responses we learned that students were not discouraged by the preregistration process per se, but rather by the realization that designing a study takes considerable time and focused decision-making. One student described the difficulty as “running into problems with the design and having to change ideas multiple times.” When asked on the post-Design Meeting survey about the most helpful aspect of the preregistration process, every student reported a variation on the idea that preregistration required them to think carefully, at length, and in advance about research design decisions. We know that such deliberate, critical thinking can happen without a preregistration template; indeed, in past iterations of the course we simply asked for a bullet-point list of hypotheses, variable operationalizations, and procedural information. However, extensive research demonstrates that the use of outlines, note-taking guides, and other conceptual organizers can aid student learning (e.g., Barstow et al., 2017; Bui & McDaniel, 2015; Chen, Teo, & Zhou, 2017). For our novice students, we expect that the highly structured preregistration template promotes a more systematic approach to research design than a bullet-point list; this expectation is worth investigating in the future.

The Preregistration Template, Take Two

At approximately the midterm point of the course (subsequent to institutional review board (IRB) submission but prior to data collection) we typically conduct a workshop on the use of the statistical software package SPSS. With data collection on the horizon, we use this workshop as an opportunity for informal large and small group discussions about exploratory hypotheses, data analysis plans appropriate for the research design, target sample sizes, rules for when to stop collecting data, and the ethics of data analysis. Students are used to thinking about ethics in the context of research like the Stanford Prison Experiment, and undergraduate psychology coursework reinforces this perception of ethical research as focusing on treatment of participants. But with IRBs providing oversight of ethics in research design, the typical ethics violation of a research psychologist in the 21st century may be one of questionable research practices rather than participant deception or mistreatment. We chose this class meeting as a second opportunity to incorporate preregistration practices and asked students to update the preregistration template prior to the SPSS workshop.

Recommendations and Lessons Learned

For instructors considering the addition of a preregistration assignment to a course, we recommend conducting a background knowledge survey and discussion, not just about preregistration, but about the terminology found on a preregistration template. Using the template revealed possible gaps in earlier coursework related to research methods terminology, such as exploratory hypotheses. In future iterations of the course we will guide our students item by item through the template to clarify concepts and provide one or two completed examples.

We also envision the possibility of an end-of-semester check for preregistration template adherence. To more closely replicate Open Science processes we could ask students to address any deviations from the preregistration template in their final APA-style manuscript. From many years of instructing the course we can anticipate some of the most likely deviations: not meeting the target sample size; in the face of null results, choosing to conduct an additional exploratory statistical test to find “something interesting”; excluding participants for vague or unsubstantiated reasons. It is sometimes necessary or useful to deviate from plans made during the preregistration process, and this is an important conversation to have with student researchers throughout a project. However, explicit expectations for adherence to the preregistration template should discourage students from mining their data post hoc for a significant effect. When they do not find a significant effect, a few students may conclude that research is boring (because they associate excitement with p < .05), but the tradeoff is a cohort of students with a more robust understanding of ethical decision-making in the psychological sciences. In addition, an important lesson for students is that null findings are also meaningful and contribute to a fuller understanding of a topic.

We encourage implementation of a preregistration activity in other types of courses and research experiences. Even in courses, like ours, that require students to draft the method and/or results sections of a manuscript before collecting or analyzing data, supplementing with a preregistration activity or assignment may have additional benefits. The template can facilitate the writing process and prevent students from leaving out critical information (e.g., stopping rules, manipulation checks, exclusion criteria). Skills and knowledge derived from the activity, such as research design and planning, are also applicable to qualitative research courses, to students’ independent research projects, and to students’ understanding of research from a consumer point of view as well. It is important to prepare students, as both research consumers and potential research generators, with the information and skills necessary for the changing landscape of research practices; armed with knowledge of, and experience with, preregistration, they may become advocates and practitioners of open science practices.

Supplemental Material

Supplemental material for Research Preregistration as a Teaching and Learning Tool in Undergraduate Psychology Courses

Supplemental Material for Research Preregistration as a Teaching and Learning Tool in Undergraduate Psychology Courses by Sarai Blincoe and Stephanie Buchert in Psychology Learning & Teaching

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Supplemental Material

Supplemental material is available for this article online.

Author biographies

Dr. Sarai Blincoe is an associate professor in the Department of Psychology and the Assistant Dean of Curriculum and Assessment in the Cook-Cole College of Arts & Sciences at Longwood University in Farmville, VA. She earned her PhD in experimental psychology from the University of Kentucky. She regularly teaches undergraduate courses in quantitative methods, social psychology, and the psychology of warfare and terrorism. Her research interests include disrespect, trust, and topics in the scholarship of teaching and learning (SoTL).

Dr. Stephanie Buchert serves as chair and associate professor in the Department of Psychology at Longwood University in Farmville, VA. She holds a PhD and a Master’s degree in experimental psychology from Kent State University of Ohio. She regularly teaches undergraduate courses in research methods, quantitative methods, cognitive psychology, and developmental psychology. Her areas of research interest include adult and child metacognition, cognitive development, language development, and pedagogy.

References

Association for Psychological Science. (n.d.). Preregistration of research plans. Retrieved from http://www.psychologicalscience.org/publications/psychological_science/preregistration.

Barstow

Fazio

Lippman

Falakmasir

Schunn

C. D.

Ashley

K. D.

(2017) The impacts of domain-general vs. domain-specific diagramming tools on writing. International Journal of Artificial Intelligence in Education 27: 671–693. http://doi.org/10.1007/s40593-016-0130-z .

Bui

D. C.

McDaniel

M. A.

(2015) Enhancing learning during lecture note-taking using outlines and illustrative diagrams. Journal of Applied Research in Memory and Cognition 4: 129–135. http://doi.org/10.1016/j.jarmac.2015.03.002 .

COS. (n.d.). Center for Open Science mission statement. Retrieved from http://cos.io/about/mission/.

Chen

P. H.

Teo

Zhou

(2017) Effects of guided notes on enhancing college students’ lecture note-taking quality and learning performance. Current Psychology: A Journal for Diverse Perspectives on Diverse Psychological Issues 36: 719–732. http://doi.org/10.1007/s12144-016-9459-6 .

Chopik

W. J.

Bremner

R. H.

Defever

A. M.

Keller

V. N.

(2018) How (and whether) to teach undergraduates about the replication crisis in psychological science. Teaching of Psychology 45: 158–163. http://doi.org/10.1177/0098628318762900 .

Crüwell, S., van Doorn, J., Etz, A., Makel, M. C., Moshontz, H., Niebaum, J. C., Schulte-Mecklenbeck, M. (2019). 7 easy steps to open science: An annotated reading list. https://doi.org/10.31234/osf.io/cfzyx.

The DRESS protocol. (2019, February 14). Retrieved from https://www.projecttier.org/tier-protocol/dress-protocol/.

European Commission. Directorate-General for Research and Innovation. Open Science and ERA Policy Unit. (2017). Providing researchers with the skills and competencies they need to practise Open Science – Report of the Working Group on Education and Skills under Open Science. Retrieved from: https://cdn1.euraxess.org/sites/default/files/policy_library/ectd_os_skills_report_final_complete_2207_1.pdf.

10.

Goldin-Meadow, S. (2016, September). Why preregistration makes me nervous. Retrieved from http://www.psychologicalscience.org/observer/why-preregistration-makes-me-nervous.

11.

Gonzales, J. E., & Cunningham, C. A. (2015, August). The promise of pre-registration in psychological research: Encouraging a priori research and decreasing publication bias. Retrieved from http://www.apa.org/science/about/psa/2015/08/pre-registration.

12.

John

L. K.

Loewenstein

Prelec

(2012) Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science 23: 524–532. http://doi.org/10.1177/0956797611430953 .

13.

Kerr

N. L.

(1998) HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review 2: 196–217. http://doi.org/10.1207/s15327957pspr0203_4 .

14.

Kidwell

M. C.

Lazarevi

L. B.

Baranski

Hardwicke

T. E.

Piechowski

Falkenberg

L. S.

Nosek

B. A.

(2016) Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLoS Biology 14: e1002456 . http://doi.org/10.1371/journal.pbio.1002456.

15.

McIntyre, K. P. & Le, B. (2017). OSL/PT preregistration template. Retrieved from http://osf.io/fjy79/.

16.

Nosek

B. A.

Ebersole

C. R.

DeHavena

A. C.

Mellor

D. T.

(2018) The preregistration revolution. PNAS 115: 2600–2606. doi: 10.1073/pnas.1708274114.

17.

Nosek

B. A.

Lakens

(2014) Registered reports: A method to increase the credibility of published results. Social Psychology 45: 137–141. http://doi.org/10.1027/1864-9335/a000192 .

18.

Nosek, B. A., & Lindsay, D. S. (2018, March). Preregistration becoming the norm in psychological science. Retrieved from http://www.psychologicalscience.org/observer/preregistration-becoming-the-norm-in-psychological-science.

19.

OSF Registries (n.d.). Retrieved from http://osf.io/registries.

20.

Toelch

Ostwald

(2018) Digital open science—Teaching digital tools for reproducible and transparent research. PLOS Biology 16: e2006022 . https://doi.org/10.1371/journal.pbio.2006022.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

1.47 MB