Sage Journals: Discover world-class research

Abstract

Research shows that questionable research practices (QRPs) are present in undergraduate final-year dissertation projects. One entry-level Open Science practice proposed to mitigate QRPs is “study preregistration,” through which researchers outline their research questions, design, method, and analysis plans before data collection and/or analysis. In this study, we aimed to empirically test the effectiveness of preregistration as a pedagogic tool in undergraduate dissertations using a quasi-experimental design. A total of 89 UK psychology students were recruited, including students who preregistered their empirical quantitative dissertation (n = 52; experimental group) and students who did not (n = 37; control group). Attitudes toward statistics, acceptance of QRPs, and perceived understanding of Open Science were measured both before and after dissertation completion. Exploratory measures included capability, opportunity, and motivation to engage with preregistration, measured at Time 1 only. This study was conducted as a Registered Report; Stage 1 protocol: https://osf.io/9hjbw (date of in-principle acceptance: September 21, 2021). Study preregistration did not significantly affect attitudes toward statistics or acceptance of QRPs. However, students who preregistered reported greater perceived understanding of Open Science concepts from Time 1 to Time 2 compared with students who did not preregister. Exploratory analyses indicated that students who preregistered reported significantly greater capability, opportunity, and motivation to preregister. Qualitative responses revealed that preregistration was perceived to improve clarity and organization of the dissertation, prevent QRPs, and promote rigor. Disadvantages and barriers included time, perceived rigidity, and need for training. These results contribute to discussions surrounding embedding Open Science principles into research training.

Keywords

preregistration Open Science reproducibility undergraduate training dissertations research training

In recent years, psychology has put reproducibility, replicability, and transparency at the forefront of the research agenda (Asendorpf et al., 2013; Munafò et al., 2017; Open Science Collaboration, 2015). Fueled by replication concerns in the general scientific literature, an era of “Open Science” has prompted a plethora of ideas and recommendations to envision a new future for science (Pashler & Wagenmakers, 2012). A move to study preregistration, open materials, and open data are proposed to combat “questionable research practices” (QRPs; John et al., 2012) that plague the literature, such as p-hacking (Head et al., 2015), “hypothesizing after results are known” (Kerr, 1998), and selective reporting (John et al., 2012) or “undisclosed flexibility” (Simmons et al., 2011). Furthermore, an incentive shift to high-quality, “slow” science is picking up momentum (Frith, 2020). Despite these practices being increasingly endorsed and embraced by the scientific community (however, for an alternative perspective, see Szollosi et al., 2019), scant research has assessed the pedagogic value of Open Science practices in improving teaching and learning.

Much of the recent shift to Open Science practices has been championed by grassroots, collaborative initiatives (e.g., see Button et al., 2020; Pownall, 2020a). In recent years, psychologists have developed initiatives such as the Society for the Improvement of Psychological Science (https://improvingpsych.org), the open-source reporting forum PsychDisclosure (LeBel et al., 2013), and the journal club led by early career researchers, ReproducibiliTea (Orben, 2019), all with the aim of improving the rigor and reproducibility of psychological science. Beyond these, organizations and initiatives are centered around the improvement of psychological science, stressing the importance of rigorous, robust methods (e.g., Crüwell et al., 2019; Munafò et al., 2017; Simmons et al., 2011; Tennant et al., 2016; Wagenmakers et al., 2012). For example, Klein et al. (2018) noted the importance of preparing and sharing research in a way that values transparency and noted how this can be done incrementally to improve research efficiency and credibility. Likewise, Devezer et al. (2020) focused on recommendations to improve methodological problems in science reform, such as the adoption of a formal approach that embeds statistical rigor and nuance into science reform.

Open Science in Undergraduate Training

The recent shifts toward novel and creative ways of promoting uptake of Open Science practices offer the opportunity to reevaluate core aspects of undergraduate training and wider scientific-research practices. For example, there have been some emergent initiatives that have specifically concentrated on how to embed teaching on the “replication crisis” and Open Science practices into undergraduate teaching (e.g., Button et al., 2016; Chopik et al., 2018; Frank & Saxe, 2012; Janz, 2016). There has also been a keen interest in interventions to improve understanding of QRPs in, for example, graduate psychology training (Sacco & Brown, 2019; Sarafoglou et al., 2020). However, the impact that these have on students’ learning and perceptions is yet to be empirically investigated.

The Value of Preregistration

One method of reducing QRPs and enhancing research transparency is study preregistration. Study preregistration comprises a time-stamped, uneditable protocol that transparently outlines a study’s research questions, design, hypotheses, methods, and analysis plan before data collection and/or analysis (Nosek et al., 2018; van’t Veer & Giner-Sorolla, 2016). The process of preregistration encourages researchers to plan the decisions that have traditionally been made after data collection (e.g., exclusion criteria, analysis details) beforehand, using a wide host of platforms such as OSF (https://osf.io/) and AsPredicted (https://aspredicted.org/). Preregistration increases transparency about the authors’ original intentions (LeBel & Peters, 2011) and should, in theory, limit selective reporting of results (Nuzzo, 2015).

Here, we propose that preregistration is one entry-level way of establishing a level of rigor and robustness into the undergraduate dissertation process (as per Pownall, 2020b). The potential value of preregistration in this context has been noted by educators. For example, the Framework of Open and Reproducible Research Training (FORRT; www.forrt.org) includes preregistration as one of the six pillars of effective reproducibility training, including at the undergraduate level. Others have suggested that “most study programmes should offer easy ways of implementing preregistration in empirical research seminars” (Olson et al., 2019) because of the potential for preregistration to promote “critical reflections of research practices” and improve students’ statistics literacy (Olson et al., 2019). As Pownall (2020b) also argued, the process of embedding preregistration of undergraduate dissertations largely complements current practices in dissertation supervision. Sacco and Brown (2019) noted that preregistration is thus useful when conducting research with the view to publish the results with undergraduate students (see also Blincoe & Buchert, 2020). In this study, we examine the value of study preregistration in the undergraduate curriculum to assess whether this can improve attitudes toward statistics (e.g., students’ perceived difficulty of statistics, value of statistics, and perceived competence in statistics) and QRPs and students’ perceived understanding of Open Science.

The undergraduate dissertation

In the UK, final-year psychology dissertations consist typically of an independent empirical project that requires students to design a protocol, collect data, and analyze the results. According to the accreditation standards of the British Psychological Society (2019), undergraduate psychology dissertations in the UK require students to “individually demonstrate a range of research skills including planning, considering and resolving ethical issues, analysis and dissemination of findings.” Final-year projects are thus typically self-contained research studies that are constrained by the scope and availability of resources but are supervised closely by an experienced academic. Much pedagogic research has demonstrated that given the level of autonomy that students have over their final-year dissertation, students typically struggle with some of the components of this mandatory part of their degree. For example, it has been reported widely that undergraduate students face anxiety, disengagement, and stress related to their final-year dissertation (e.g., Devonport & Lane, 2006). Indeed, research has shown that undergraduate students often experience difficulty with their dissertation because of pedagogic issues such as debilitating statistics anxiety (e.g., Onwuegbuzie & Wilson, 2003), underconfidence with their writing ability (Greenbank et al., 2008), and challenges navigating supervisory relationships (Day & Bobeva, 2007).

Contemporary research has also indicated that QRPs are prevalent in undergraduate research projects (Krishna & Peter, 2018; Kvetnaya et al., 2019; Sorokowski et al., 2019). For example, Krishna and Peter (2018) assessed the prevalence of QRPs in final-year undergraduate dissertations and found that students typically engage in QRPs related to reporting and analyzing their results. Likewise, Olson et al. (2019) studied the prevalence of QRPs of taught master’s students’ theses and found inconsistency of p-value reporting, although it was not clear that this was a result of intentional p-hacking. Research outside of psychology has also indicated that from dissertation to publication, the ratio of supported to unsupported hypotheses more than doubles (O’Boyle et al., 2017). Recently, there has been a focus on addressing QRPs that feature in undergraduate final-year projects through consortia-based approaches (Button et al., 2020; Kvetnaya et al., 2019; Munafò et al., 2017) and through focusing on replication studies with undergraduate projects (e.g., de Leeuw et al., 2019; Jekel et al., 2020).

The use of QRPs in the undergraduate dissertation likely stems from many different sources: Resource and time constraints mean that many undergraduate experiments are typically underpowered (Button et al., 2016); students perceive that there is a pressure from supervisors to “find” significant results, which are more likely to lead to a publication (Wagge et al., 2019); and in our own experience, students also worry that a “lack of significant” results will adversely affect their grades. QRPs may also stem from a lack of awareness that they are problematic (e.g., Banks et al., 2016). This is related to the pressures put on academics to publish novel, positive results (Franco et al., 2014) because of the “publish or perish” culture that pervades academia (Grimes et al., 2018), which might filter down to their students. Indeed, an undergraduate publication is seen as an advantage when applying for highly competitive places on taught master’s and doctoral training (Button, 2018). If these studies are then selectively published, they contaminate the scientific literature with unreliable results. Understanding undergraduate students’ use and acceptance of QRPs is useful because students’ research behavior reflects the quality of Open Science teaching and adoption of rigorous practices more broadly (Olson et al., 2019). Some emergent research has begun to investigate the research practices of early career researchers (Nicholas et al., 2017), including uptake of Open Science practices (Stürmer et al., 2017).

Consideration of the prevalence of QRPs in the undergraduate dissertation has led to interventions to reduce them. Button et al. (2020), for example, described and evaluated an approach to improving rigor of undergraduate dissertations via a consortium approach to science. This approach also echoes Detweiler-Bedell and Detweiler-Bedell’s (2019) team-based approach to undergraduate research supervision. Creaven et al. (2021) stressed the importance of embedding a concern for rigor, transparency, and openness into the undergraduate dissertation, stressing how the undergraduate dissertation should be thought of as an important learning activity that offers many pedagogical benefits to students. Likewise, Blincoe and Buchert (2020) proposed that preregistration may be a useful pedagogical tool for undergraduate psychology students. Despite some useful and recent conversations that discuss the need to embed an Open Science approach into undergraduate research training (Button et al., 2020; Creaven et al., 2021; Pownall, 2020b), an empirical exploration into how Open Science practices may benefit both students and the Open Science movement has been notably absent from these conversations. Indeed, although much work has considered how to promote uptake of preregistration practices of early career (Zecˇević et al., 2020) and more established researchers (Kidwell et al., 2016; Munafò et al., 2017), little research has explicitly focused on the utility of preregistration for undergraduate students’ research practices despite recommendations that preregistration could facilitate engagement with the dissertation process (e.g., Nosek et al., 2018), reduce statistics anxiety, and improve students’ experience of their dissertation (Creaven et al., 2021; Pownall, 2020b).

The Present Study

We aimed to investigate empirically the pedagogical effectiveness of preregistration in undergraduate-dissertation provision, that is, how the process of preregistration may be useful at tackling some of the core pedagogical challenges that students face in their dissertation research (including attitudes toward statistics), while also considering how engaging with the process of preregistration can aid understanding of Open Science issues more generally. Our core research questions aimed to evaluate whether preregistration is a useful pedagogic practice to improve students’ attitudes toward statistics (i.e., perceptions of the value and difficulty of statistics and students’ perceived competence in statistics), awareness of QRPs, and perceived understanding of Open Science in this cohort. To achieve this, we employed a 2 (Group: Preregistration vs. Control) × 2 (Time: Time 1, Before Dissertation vs. Time 2, After Dissertation) mixed design with group as the between-participants factor and time as the within-participants factor. We had three confirmatory hypotheses based on a significant two-way interaction between group and time. For all of the hypotheses, we predicted a significant Time × Group interaction such that participants in the preregistration group would show improvements above and beyond those that occur because of time differences (Time 1 vs. Time 2):

Hypothesis 1: Because of the thoughtful engagement with statistical processes that the preregistration process requires (Lindsay et al., 2016), we predicted that students who preregister their dissertation will have higher scores on the four constructs of the Survey of Attitudes Toward Statistics (SATS-28) from Time 1 to Time 2.

Hypothesis 1a: Students who preregister their dissertation will have higher (i.e., more positive) affect toward statistics compared with students who do not preregister their dissertation from Time 1 to Time 2.

Hypothesis 1b: Students who preregister their dissertation will have higher self-reported competence with statistics compared with students who do not preregister their dissertation from Time 1 to Time 2.

Hypothesis 1c: Students who preregister their dissertation will have higher perceived value of statistics compared with students who do not preregister their dissertation from Time 1 to Time 2.

Hypothesis 1d: Students who preregister their dissertation will have less perceived difficulty of statistics at Time 2 compared with students who do not preregister their dissertation from Time 1 to Time 2.

Hypothesis 2: Given that the preregistration process prompts wider consideration of the QRPs that preregistration aims to avoid, we predicted that students who preregister their undergraduate dissertations will have a reduced self-reported acceptance of 11 selected QRPs compared with students who do not preregister their dissertation (Time 1 responses compared with Time 2 responses).

Hypothesis 3: Relatedly, given that the preregistration process forms part of a wider conversation about open and transparent science, we expect that students who preregister their undergraduate dissertations will have higher perceived confidence in their understanding of 12 selected Open Science terminology terms compared with students who do not preregister their dissertation (Time 1 responses compared with Time 2 responses).

Finally, as an exploratory measure with no predetermined hypotheses, we also assessed students’ capability, opportunity, and motivation toward preregistration at Time 1 and qualitative responses regarding the perceived barriers and facilitators of preregistration at Time 2.

Method

Transparency statement

All materials and data are publicly available on OSF (https://osf.io/5qshg/), and our study meets Level 6 of the Peer Community in Registered Reports bias control (https://rr.peercommunityin.org/help/guide_for_authors). In the sections that follow, we report all measures, manipulations, and exclusions. This study was conducted as a Registered Report; preregistered Stage 1 protocol is available at https://osf.io/9hjbw (date of in-principle acceptance: September 21, 2021).

Design and participants

The study comprised a 2 (Group: Preregistration vs. Control) × 2 (Time: Before Dissertation vs. After Completion) mixed-factors design. To be eligible for inclusion, participants were required to confirm that they were a final-year undergraduate student studying psychology at a UK institution and planning an empirical quantitative undergraduate dissertation. Participants must have not already preregistered their proposed undergraduate study at Time 1 and confirmed this in the beginning of the study. This was to ensure that the study contributes directly to existing pedagogic policy discussions regarding embedding Open Sciences in the undergraduate dissertation (e.g., course accreditation standards by the British Psychological Society, 2019). To be eligible to participate at Time 2, participants must have completed Time 1 measures (and have a corresponding participant ID number to match up responses). To be included in the preregistration group at Time 2, participants indicated that their preregistration included a “data analysis plan” (see Time 2 measures).

Our planned sample size was based solely on resource and time considerations, including the time window for participant recruitment and available funds for participant compensation (see Lakens, 2021). We initially aimed to recruit 240 final-year undergraduate students. We planned to recruit psychology students and expected an approximately 20% attrition at Time 2 given prior research sampling from online platforms (Palan & Schitter, 2018). We planned to recruit 200 participants to include an experimental group of approximately 100 having initiated a preregistration of their final-year quantitative project and a control group of 100 not initiating a preregistration. Simulation-based power analyses conducted using the superpower shiny package (Lakens & Caldwell, 2021; https://arcstats.io/shiny/anova-exact/) with 10,000 simulations indicated that this sample size would have 80% statistical power to detect a moderate effect size for the two-way interaction between group and time (η_p² = .04) and a small-moderate effect of d = 0.40 for the focal pairwise comparison between preregistration and control at Time 2 (code/output available at https://osf.io/y9vz7/) with α = .05.

At Time 1, there were initially 354 participants with complete data (i.e., responses with survey progress of 100%). Of these participants, 187 passed the various attention checks (see Method). After removing five direct duplicates (i.e., whereby a participant had clearly completed the study twice or submitted the survey twice), there were 182 participants left to invite back at Time 2. At Time 2, 139 participants initially responded to the survey. Of these participants, 108 both had 100% progress and passed the attention checks (see Procedure). Fifteen participants at Time 2 did not match with participants in Time 1, and there were four participants removed because of duplicates (i.e., identical responses and ID codes), leaving 89 complete participants with Time 1 and Time 2 data left for analysis. Therefore, our final sample comprised 89 participants (age: M = 21.84 years, SD = 3.457; 77.5% female; n = 60 White British); 52 students confirmed they had preregistered their dissertation (preregistration group), and 37 did not preregister (control group). On the basis of the lowest cell size (n = 37), we found in sensitivity power analyses that we could reliably detect an effect size of η_p² = .10 for the Group × Time interaction and pairwise comparisons of d ≥ 0.66 with 80% statistical power, which was higher than planned. All participants provided informed consent. Ethical approval was granted from the University of Leeds School of Psychology Ethics Committee on July 8, 2021 (reference: PSYC-266; https://osf.io/5rtch/).

Recruitment plan

We purposefully sampled students via Prolific Academic (using custom prescreening) and university participant pools (SONA Systems) and through social media adverts, ensuring they met the inclusion criteria. Inclusion criteria were included in all recruitment materials, and participants confirmed they met these in the first page of the study’s procedure, via checklist boxes. After reading a brief definition of preregistration, participants were asked to confirm at Time 1 and 2 whether they preregistered their undergraduate dissertation. We used cross logic quota sampling in Qualtrics (see Qualtrics, https://www.qualtrics.com/support/surveyplatform/survey-module/survey-tools/quotas/) to roughly monitor group allocation at Time 1, although this was done using the preregistration plan questions (see below), which could differ from the final preregistration-group allocation at Time 2 (i.e., some participants could plan to preregister but do not actually preregister at Time 2). Because preregistration is typically at the supervisor’s discretion and not widely implemented in undergraduate-degree programs, we also engaged in targeted recruitment to the preregistration condition through appropriate Open Science teaching channels: These included organizational stakeholders such as the UK Reproducibility Network and the British Psychological Society and UK institutions who incorporate preregistration as part of their undergraduate curriculum (see Table 1). We also used social media channels to recruit participants. All participants recruited via Prolific Academic were paid the equivalent of £6.50 per hour for their time; participants were paid the equivalent of £6.50 per hour at each time point; completion time of each was estimated to be 15 to 20 min. Participants recruited via Prolific were contacted for Time 2 via Prolific’s contact participants function; participants recruited elsewhere were contacted via email.

Table 1.

A Sample of Universities Sampled Who Offer Preregistration in the Final-Year Curriculum

University	Preregistration approach
Bath Spa University	Students complete an internal preregistration in Semester 1.
University of Glasgow	Open Science forms an integral part of core undergraduate teaching.
Royal Holloway University	Internal preregistration is embedded into dissertation supervision.
University of Surrey	Optional preregistration, dependent on agreement between student and supervisor

Procedure

Data were collected online using Qualtrics (https://www.qualtrics.com/uk/) through the various recruitment strategies above. At Time 1, participants were enrolled for their final year but had not initiated their dissertation project or their preregistration (September–November 2021). This provided a baseline in which to compare responses at Time 2 (after dissertation; May–July 2022).

Participants first provided demographic information (age, gender, ethnicity, institution of study) before confirming that they were in the final year of their bachelor’s of science undergraduate psychology degree and planned to undertake a quantitative dissertation project in the 2021–2022 year (yes/no). Participants who answered “no” were informed that they did not meet the inclusion criteria for the study. We then collected data related to students’ self-reported academic attainment in the mandatory statistics module of their degree in second year and their average grade in the second/penultimate year of their degree. This was scored on a categorical scale that is in line with the UK conventions of academic grades awarding: first-class classification (> 70%), 2:1 classification (60%–69%), 2:2 classification (50%–59%), third-class classification 40% to 49%, and fail (< 40%). This was to control for potential baseline differences between our two groups.

Participants were then provided with a brief definition of preregistration, adapted from Lindsay et al. (2016):

Preregistering a research project involves creating a record of your study plans before you look at the data. The plan is date-stamped and uneditable. The main purpose of preregistration is to make clear which hypotheses and analyses were decided on before you have accessed your data and which were more exploratory and driven by the data.

Then, to ensure participants had not yet preregistered their project at Time 1, we asked participants whether they planned to preregister their undergraduate dissertation (yes/no/unsure) and whether the undergraduate dissertation had already been preregistered (yes/no). All participants at Time 1 then completed the same measures. The items relating to participants’ plans were not used to categorize participants into groups and instead were used to guide quota sampling.

Measures (Time 1)

SATS-28

To assess whether preregistration improves attitudes toward statistics, students completed the SATS-28. This 28-item scale includes items related to statistics affect (e.g., “I am scared by statistics”), cognitive competence (e.g., “I can learn statistics.”), value (e.g., “Statistics is worthless”), and difficulty (e.g., “Statistics is highly technical”). These items were scored on a Likert scale from 1 (completely disagree) to 7 (completely agree); 19 items were reverse-scored. A total score was computed for each of the subscales: statistics affect, cognitive competence, value, and difficulty. Reverse-scored items were recoded so that higher scores indicate more positive affect, higher competence, higher value, and lower difficulty. This scale has been found to have acceptable internal reliability (Cronbach’s αs = .64–.85) for each of the subscales; Dauphinee et al., 1997) and for the scale as an overall index (α = .91; Ayebo et al., 2019). The internal reliability of each subscale was excellent at both Time 1 (affect: α = .92; competency: α = .91; value: α = .88; difficulty: α = .79) and Time 2 (affect: α = .91; competency: α = .87; value: α = .91; difficulty: α = .76) in the current study.

Acceptance of QRPs

To assess whether preregistration influences attitudes toward QRPs, students rated their views on 15 research decisions (11 of which are QRPs, four of which are neutral/acceptable) on a sliding scale from 1 (sensible) to 7 (problematic; Krishna & Peter, 2018). These included items such as “selectively reporting studies” and “deciding to exclude data after looking at results” (QRPs) and “reporting effect sizes” (neutral/acceptable). The neutral/acceptable items were not analyzed but, instead, were used to mask the nature of this questionnaire. We computed all 11 items pertaining to QRPs into one total indicating general acceptance of QRPs such that higher scores indicate less acceptance of QRPs. The internal reliability of this questionnaire was adequate in the current study (Time 1: α = .72; Time 2: α = .70).

Perceived understanding of Open Science

As per other literature (Krishna & Peter, 2018; Stürmer et al., 2017), to test perceived understanding of Open Science practices and terminology, students indicated their confidence in their ability to understand 12 key terms (e.g., “Replication Crisis,” “p-hacking,” “open data,” “file drawer effect”) on a Likert scale from 1 (not at all confident) to 7 (entirely confident). These concept-recall items were compiled into a total score of Open Science perceived understanding. The internal reliability of this questionnaire was excellent in the current study (Time 1: α = .90; Time 2: α = .91).

Attention and bot checks

As an attention check (i.e., to ensure that participants were actively paying attention to the survey materials and to prevent spam/bot respondents), we added an item, “Please select strongly disagree to this question,” in the COM-B measure to assure data quality. This was repeated in Time 1 and Time 2. As a second attention check, we used a protocol from the Prolific guidelines and asked participants, “Please enter the word ‘purple’ in the textbox below,” accompanied by a text box. Any participant who failed both of these attention checks (i.e., who did not select strongly disagree and correctly enter the word “purple”) was excluded from the final analyses. We also employed Qualtrics’ “prevent multiple submissions” and “prevent indexing” (i.e., block search engines from including the study URL in search results) security options to minimize chances of fraud/bot responses.

Exploratory measures

Capability, opportunity, and motivation toward preregistration

In line with Norris and O’Connor (2019), we also applied a behavior-change approach to assess the facilitators and barriers to study preregistration at Time 1 only. The capability, opportunity, motivation, behavior (COM-B) model (Michie et al., 2011) posits that a behavior occurs only if an individual has sufficient capability, opportunity, and motivation to perform it. Capability includes psychological capability (i.e., knowing how to perform the behavior) and physical capability (i.e., being physically able to perform the behavior). Opportunity includes social opportunity (i.e., being around others who are performing the behavior) and physical opportunity (i.e., having the time and resources to perform the behavior). Motivation includes reflective motivation (i.e., plans and beliefs to perform the behavior) and automatic motivation (i.e., desires, impulses, and inhibitions toward the behavior; Michie et al., 2011). The brief measure of COM-B developed by Keyworth et al. (2020) was employed. This measure contains six items; two items address each of the three components of the COM-B on a 5-point Likert scale ranging from 0 (strongly disagree) to 5 (strongly agree). Note that the 5-point scale is a deviation from our Stage 1 Registered Report, which proposed to use an 11-point Likert scale. This deviation was due to researcher oversight in the building of the Qualtrics survey. Each item is accompanied by an explanation of what the COM-B component referred to in the questions means. For example, “I have the PHYSICAL opportunity to preregister my undergraduate dissertation” is accompanied by the explanation defined by Keyworth et al.: For example, “What is PHYSICAL opportunity? The environment provides the opportunity to engage in the activity concerned (e.g sufficient time, the necessary materials, reminders).” A total score was computed for each subscale. The internal reliability of these items was excellent for the opportunity subscale (α = .90) and the capability subscale (α = .91) and satisfactory for motivation (α = .57) in the current study. This exploratory measure was chosen to explore how a behavior-change model may be applied to engagement in Open Science practices (e.g., as per Norris & O’Connor, 2019).

After dissertation (Time 2)

The same sample of students was asked to complete all of the above measures, except for the COM-B, again at Time 2, which represents a follow-up after their dissertation was completed in approximately May 2022. At Time 1, participants reported whether they planned to preregister their dissertation, and at Time 2, participants first reported whether they did actually preregister (yes/no). Participants’ responses to this question at Time 2 were used to allocate participants to the preregistration versus no-preregistration groups. For example, if participants responded at Time 1 that they planned to preregister but at Time 2 they did not, they were allocated to the no-preregistration control group for the final analyses. At Time 2, we also asked participants who preregistered to self-report the extent to which they followed their preregistration plan (1 = not at all, 2 = somewhat, 3 = entirely). We also asked participants at Time 2 to identify what their preregistration included from a list. This list included 14 items taken from the OSF standard preregistration template (Bowman et al., 2020), including items such as “information about study background,” “testable hypotheses,” “design plan,” and “sample size.” Crucially, one item was “data analysis plan.” Participants who did not indicate that a data analysis plan was included in their preregistration were removed from the study. The rest of this preregistration data were used descriptively in our study.

In addition, participants were also asked four questions to assess whether they had implemented other Open Science practices associated with their dissertation: (a) creating an OSF account or uploading (b) material (open material), (c) code/scripts (open code), and (d) data (open data) to a public archive. This was used descriptively to gain more insight into other contextual factors that are associated with preregistration. Qualitative responses of students’ experiences of the preregistration process, including enablers and barriers, were also collected through three open-ended questions: “Please list all of the advantages you perceive of preregistration,” “Please list all of the disadvantages,” and “Do you see any barriers to preregistration?”

Perceptions of supervisory support

Finally, given the literature that suggests that perceived supervisor support affects students’ experiences of their dissertation research (Roberts & Seaman, 2018) and that supervisor belief affects preregistration behavior (Spitzer & Mueller, 2023), to assess students’ perceptions of their supervisory support at Time 2, we used a 14-item measure of perceptions of supervisor support. This scale includes items such as “I am satisfied with the support I have received from my supervisor” and “My supervisor was knowledgeable about research design/process as related to my project.” One item was “I felt pressure from my supervisor to find significant results in my dissertation” (reverse-scored). These were measured on a scale from 1 (strongly disagree) to 5 (strongly agree). Answers were aggregated into one overall score of supervisory support and used as a covariate in further analyses (α = .95).

Risk and mitigations

At Stage 1 of this Registered Report, we acknowledged certain risks associated with our study and aimed to mitigate these with the following measures. The first risk was participant attrition from Time 1 to Time 2, leading to incomplete data across measures. We aimed to mitigate this by accounting for average attrition rates in our planned sample as per other longitudinal studies conducted on Prolific (7%–24%; Palan & Schitter, 2018) and using a varied recruitment approach. At Time 2, participants not recruited via Prolific were entered into a prize draw to incentivize participation. Likewise, recruitment of the preregistration group required a level of buy-in from institutions that embed a preregistration model into their undergraduate-dissertation process. Members of the research team had contacts with these institutions listed in Table 1, which should mitigate barriers to student access in the preregistration group. We ran a sensitivity power analyses on the complete data and used this to contextualize our discussions and interpretation of final results. Our final sample size is smaller than planned, largely because of our stringent attention checks and matching of data from Time 1 to Time 2; we discuss this in the Limitations section.

Second, at Stage 1, we had also factored in discrepancies in definitions of preregistration practices by providing all students with a student-friendly, accessible definition of preregistration from the literature (Lindsay et al., 2016). This should mean that students were able to readily identify whether they engaged in this specific process above and beyond other processes in the dissertation timeline (e.g., discussing a protocol with their supervisor or writing an ethics application). Asking students to confirm at Time 2 that they had preregistered their study should also have alleviated any problems with students erroneously being allocated to the wrong condition at Time 1.

Finally, our study may have had confounding variables that we aimed to reduce. For example, it is likely that institutions that actively embed preregistration into the dissertation process may also teach Open Science practices more generally within their curriculum, which may be a confound when evaluating the effectiveness of study preregistration. This was first checked by establishing whether there are differences in students’ Open Science attitudes and knowledge at Time 1. Second, we mitigated this by investigating the interaction between group and time on all of our outcome variables. Specifically, we expect that despite any differences between groups at Time 1, there will be a significant interaction indicating that engaging with the preregistration process has an additive effect on students’ attitudes, behaviors, and perceptions of Open Science (i.e., it improves scores beyond improvement that occurs because of differences in time point).

It could also be possible for ceiling effects to occur in the preregistration group at Time 1, particularly given the aforementioned concern about contextual factors that affect students’ knowledge of Open Science and QRPs. This could mean that differences from Time 1 to Time 2 are “masked” because of high scores at Time 1 for the preregistration group. Although we cannot methodologically mitigate this concern, we discussed it in detail following data collection and use this to guide interpretation of our results. Finally, we avoided missing data adversely affecting our statistical power by using a “requested entry” option on Qualtrics, so participants were unable to progress in the survey without first confirming that they were happy that they had answered all the questions they wished to (if some were left unanswered).

Analysis Strategy

Our full analysis strategy, registered at Stage 1, is shown in Table 2.

Table 2.

Research Questions, Accompanying Hypotheses, and a Priori Analysis Plan

Research question	Hypotheses	Sampling plan	Analysis plan	Rationale for deciding the sensitivity of the test for confirming or disconfirming the hypothesis	Interpretation given different outcomes	Theory that could be shown wrong by the outcomes	Outcome
1. Is preregistration a useful pedagogic practice to improve students’ perceived understanding of research methods and statistics in the undergraduate dissertation?	We generally predict that attitudes toward statistics will improve over time as a result of engaging with the final-year dissertation process itself but that preregistration will have an additive effect on this. Students in the preregistration group will show a marked improvement compared with students in the control (Hypothesis 1).	We planned to recruit 240 final-year undergraduate psychology students and anticipated approximately 20% attrition at Time 2 based on prior research sampling from online platforms (Palan & Schitter, 2018). The final planned sample size is therefore 200 participants, although see Participants section for final sample size. Also see design and participants for power analysis in more detail.	2 (Group: Preregistration vs. Control) × 2 (Time: Time 1 vs. Time 2) mixed ANOVA with attitudes toward statistics as the dependent variable	Simulation-based power analyses conducted using the superpower shiny package (Lakens & Caldwell, 2021) with 10,000 simulations indicate that this sample size will have 80% statistical power to detect an effect size of η_p² = .04 for the two-way interaction between Group and Time and 80% power to detect small-moderate effects of d = .40 for the focal pairwise comparison between preregistration and control at Time 2 (code/output: https://osf.io/y9vz7/).We will also run a sensitivity analysis to compare our achieved sample size with planned sample size (see Participants and design section for further details).	This could find that preregistration does affect students’ statistics attitudes, as we predict, or it could suggest that preregistration does not add benefits above and beyond differences that occur because of time (from Time 1 to Time 2).No main effect of time would suggest that students do not change in their attitudes toward statistics as they progress through their academic studies in final year. However, our Bayesian analyses will also reveal the strength of evidence we have to make these conclusions.	Theoretically, the notion that preregistration confers a tangible, pedagogical benefit to students in their dissertation process could be (un)supported by all of our proposed analyses.Explanations for all results will be presented in the discussion.	We generally found no evidence to suggest that preregistration affects attitudes toward statistics.
2. Does the process of preregistration enhance awareness and acceptance of QRPs?	We predict that preregistration will reduce acceptance of QRPs as “sensible” for the preregistration compared with the control group (Hypothesis 2).		2 (Group: Preregistration vs. Control) × 2 (Time: Time 1 vs. Time 2) mixed ANOVA with acceptance of QRPs as the dependent variable	For our Bayesian analyses, we will adopt a BF₁₀ < .17 as evidence for the null, which is a conservative criteria for this analysis that will allow us to test support for the null or alternative hypotheses.	Likewise, this analysis tests whether a preregistration process improves students’ awareness of QRPs; therefore, this analysis could find that preregistration does positively affect students’ awareness of QRPs, as we predict, or it could suggest that preregistration does not add benefits above and beyond differences that occur because of time (from Time 1 to Time 2).		We found no evidence to suggest that preregistration may affect acceptance of QRPs among students.
3. Does the process of preregistration improve perceived understanding of Open Science practices?	We predict that preregistration will improve perceived understanding of Open Science practices and terminology compared with the control group (Hypothesis 3).		2 (Group: Preregistration vs. Control) × 2 (Time: Time 1 vs. Time 2) mixed ANOVA with awareness of Open Science practices as the dependent variable		As above, this analysis allows us to test whether preregistration improves students’ perceived understanding of Open Science practices. Similar to the above, a significant main effect of group would indicate that preregistration does or does not affect students’ Open Science perceived understanding, independent from time effects.		Students who preregistered showed an increase from Time 1 to Time 2 on perceived understanding of Open Science. There were no other effects or interactions detected.
					Interactions of the ANOVA could find that preregistration does positively affect students’ perceived understanding of Open Science, as we predict, or it could suggest that preregistration does not add benefits above and beyond differences that occur because of time (from Time 1 to Time 2).
4. Do students recognize the benefits of the preregistration process in their undergraduate dissertation, and are there any barriers/challenges to its implementation?	This research question is exploratory. We will first explore whether preregistration is associated with capability, opportunity, and motivation for preregistration by comparing the preregistration. We will then conduct qualitative content analysis on participants’ free-text responses at Time 2.	This research question is exploratory, and the same sample detailed above will be used to address this question.	A t test comparing preregistration group vs. control group at Time 1 with COM-B scores as the dependent variable.Qualitative analysis using qualitative content analysis for free-text responses.	This research question is exploratory. Qualitative research typically does not share concerns of generalizability with quantitative research, so our planned sample size for this study will be sufficient for our qualitative research question, given the epistemological underpinnings of this approach.	This set of exploratory analyses allows us to test whether students have the sufficient capability, opportunity, and motivation to complete preregistration.Qualitative analyses will shine light into whether students recognize any barriers or challenges to provide more nuance to the quantitative analysis.		Students who preregistered reported higher capability, opportunity, and motivation to preregister compared with students who did not.Table 4 summarizes the qualitative content analysis findings.

Note: ANOVA = analysis of variance; QRPs = questionable research practices; COM-B capability, opportunity, motivation, behavior model.

Results

Baseline characteristics of perceived supervisory support and prior statistics attainment at Time 1 did not significantly differ between the preregistration and control groups (see Table 3; both ps > .05). Because there were no baseline differences between groups on perceptions of supervisor and prior statistics attainment (categorized by second-year statistics grades), these were not entered as covariates in the following analyses.

Table 3.

Baseline Characteristics Between the Preregistration and Control Groups (Means and Standard Deviations)

	Preregistration	Control
Perceptions of supervisor support	5.19 (1.32)	4.92 (1.56)
Prior statistics attainment	1.81 (.84)	1.78 (.63)

Note: Perceptions of supervisor support were measured using a 14-item measure on a 5-point Likert scale (Roberts & Seaman, 2018).

A series of 2 (Group: Preregistration vs. Control) × 2 (Time: Time 1 vs. Time 2) mixed analyses of variance were conducted on attitudes toward statistics (SATS-28; Hypothesis 1), attitudes toward QRPs (Hypothesis 2), and perceived understanding of Open Science (Hypothesis 3). For our complete analysis plan, see Table 2. Bonferroni corrections were applied to elucidate pairwise comparisons, and statistical significance was denoted as p < .05. Bayes factors were calculated for all analyses to evaluate strength of evidence (Dienes, 2011). In line with recommendations for early research (Schönbrodt et al., 2017), BF₁₀ > 6 was considered evidence for the alternative hypothesis, and null results with BF₁₀ < .17 was considered evidence for the null hypotheses. There is no previous literature to guide an informed prior, and thus, Bayesian analyses were computed using the default Jeffreys-Zellner-Siow (JZS) prior (r = .707; Rouder et al., 2009) in JASP (JASP Team, 2020). The JZS prior is a noninformative default and objective prior designed to minimize assumptions about the expected effect size.

As an exploratory analysis, we also conducted a between-participants t test on Time 1 responses to the COM-B questionnaire to assess enablers and barriers to preregistration between the preregistration and no-preregistration groups.

Descriptives about preregistration practice

Of the 52 students who preregistered their dissertation, 27 students (51.92%) reported that they somewhat followed the analysis plan set out in the preregistration, and 25 (48.1%) followed the plan exactly. No students reported that they did not follow the analysis plan in the preregistration, and thus all participants were retained in the analyses. Students preregistered most commonly on a university preregistration template (55.8%, n = 29), followed by the OSF (34.6%, n = 18) and the AsPredicted templates (7.7%, n = 4). Of the 89 complete participants, 66 students (74.2%) reported that they completed their dissertation individually, and 23 (25.8%) completed as part of a group. Some students engaged with other Open Science practices in their dissertation, including open materials (71.15%, n = 37), open code (21.15%, n = 11), and open data sharing (42.31%, n = 22).

Attitudes toward statistics

We predicted that there would be a main effect of time such that over time, students’ perceptions of statistics would improve (i.e., their scores on this scale would go down) in both groups (for our full analysis plan, see Table 2). We also predicted that there would be a two-way interaction between group and time with the preregistration condition exerting an additive effect on this to show more marked improvement in statistics attitudes. However, contrary to hypotheses, there were no significant main effects or interactions between preregistration groups on the four dimensions of statistics attitudes. Specifically, for statistics affect, there was no significant main effect of group, F(1, 87) = 1.108, p = .295, η_p² = .013, BF₁₀ = .605; no significant main effect of time, F(1, 87) = 0.542, p = .464, η_p² = .006, BF₁₀ = .226; and no significant main effect of the Group × Time interaction, F(1, 87) = 0.616, p = .435, η_p² = .007, BF₁₀ = .215. For students’ statistics cognitive competence, there was no significant main effect of group, F(1, 87) = 0.552, p = .460, η_p² = .006, BF₁₀ = .507; no significant main effect of time F(1, 87) = 1.522, p = .221, η_p² = .017, BF₁₀ = .343; and no significant main effect of the Group × Time interaction F(1, 87) = 0.046, p = .830, η_p² < .01, BF₁₀ = .237. For perceived value of statistics, there was no significant main effect of group, F(1, 87) = 0.860, p = .356, η_p² = .01, BF₁₀ = .477; no significant main effect of time, F(1, 87) = 0.057, p = .812, η_p² < .01, BF₁₀ = .166; and no significant main effect of the Group × Time interaction, F(1, 87) = 0.001, p = .975, η_p² = < .01, BF₁₀ = .234. Finally, for perceived statistics difficulty, there was no significant main effect of group F(1, 87) = 0.998, η_p² = .011, p = .320, BF₁₀ = .510; no significant main effect of Time, F(1, 87) = 0.004, p = .953, η_p² < .01, BF₁₀ = .165; and no significant main effect of the Group × Time interaction, F(1, 87) = 2.171, p = .144, η_p² = .024, BF₁₀ = .598. Note that all of the effect sizes here are small, and thus, all fall below the threshold for which we were able to detect an effect size on the basis of our sensitivity power analyses at 80% power (see Limitations).

Acceptance of QRPs

Contrary to hypotheses, we were unable to detect a significant main effect of time, F(1, 87) = 2.504, p = .117, η_p² = .028, BF₁₀ = .523, or a significant main effect of preregistration group F(1, 87) = 2.033, p = .157, η_p² = .023, BF₁₀ = .729, on acceptance of QRPs. We were also unable to detect a significant Time × Group interaction, F(1, 87) = 0.006, p = .939, η_p² < .01. BF₁₀ = .213; as noted above, this may be due to issues with statistical power rather than the absence of a significant effect. However, beyond the null hypothesis significance testing results, the Bayes factor here also lends support for the null result. Note also that the effect sizes here all fall below the threshold that we were able to detect according to our sensitivity analysis (i.e., moderate to large effects with 80% power).

Perceived understanding of Open Science

We predicted a Group × Time interaction whereby participants in the preregistration group would improve their perceived understanding from Time 1 to Time 2 compared with the no-preregistration group. There was a significant main effect of time, F(1, 87) = 24.238, p < .001, η_p² = .218, BF₁₀ = 12,556.604, such that students generally showed an increase in understanding of Open Science from Time 1 (M = 4.36, SD = 1.3) to Time 2 (M = 4.93, SD = 1.25). The Bayes factor here indicates a substantial difference, which lends strong support for the hypothesis. This effect size is also larger than the threshold effect size that we were able to detect on the basis of our sample size and sensitivity analysis. We did not, however, detect a significant main effect of preregistration group, F(1, 87) = 1.726, p = .192, η_p² = .019, BF₁₀ = .587, although we did for the Time × Group interaction, F(1, 87) = 4.663, p = .034, η_p² = .051, BF₁₀ = 1.751. The effect size of the interaction was also larger than our threshold effect size according to our sensitivity analysis (at 80% power). In line with our hypotheses, pairwise comparisons indicated that participants who preregistered showed a significant increase in understanding of Open Science from Time 1 (M = 4.4, SD = 1.38) to Time 2 (M = 5.17, SD = 1.25; p < .001; see Fig. 1). There was no significant difference between students who did not preregister from Time 1 (M = 4.30, SD = .214) to Time 2 (M = 4.60, SD = .201), p = .074.

Fig. 1.

Two-way interaction between preregistration group and time on perceived understanding of Open Science.

Exploratory analyses

COM-B

A between-participants t test showed that participants who preregistered their dissertation reported significantly higher opportunity to preregister at Time 1 (i.e., before they actually completed their preregistration; M = 4.32, SD = 1.01) compared with students who did not preregister (M = 3.24, SD = 1.03), t(87) = 4.90, p < .001, Cohen’s d = 1.05, BF₁₀ = 3,617.18. As the Bayes factor indicates, this lends considerable evidence to the alternative hypothesis. Likewise, participants who preregistered their dissertation reported significantly higher motivation to preregister at Time 1 (M = 3.46, SD = 0.94) compared with students who did not preregister (M = 2.70, SD = 0.88), t(87) = 3.84, p < .001, d = 0.83, BF₁₀ = 103.807. The effect sizes for opportunity and motivation were both comfortably beyond the effect-size threshold that we were powered to detect, according to our sensitivity analysis (at 80% power). Students who preregistered also reported significantly higher capability to preregister (M = 4.09, SD = 1.042) compared with students who did not (M = 3.51, SD = 0.96), t(87) = 2.64, p = .009, d = 0.57, BF₁₀ = 4.466. Although, this effect size was smaller than the effect size we were powered to detect. Note that we proposed to measure the COM-B on an 11-point Likert scale at Stage 1 and deviated to a 5-point scale at Stage 2. This does not affect the interpretation of the results but does mean that variation (i.e., the standard deviations reported here) is likely to be lower than if we had used a broader scale.

As a final exploratory analysis, we explored whether there were differences in capability, motivation, and opportunity for preregistration between the students who indicated at Time 1 that they initially planned to preregister and then at Time 2 did not (n = 8) versus students who did preregister (n = 29). Independent-samples t tests showed that there was no difference in reported opportunity, t(7.52) = 1.79, p = .057, or motivation, t(35) = 0.58, p = .28, but there was a small but significant difference between capability such that students who planned to preregister and then did preregister rated their capability to be higher (M = 4.48, SD = 0.738) than students who planned to preregister but did not (M = 4.0, SD = 0.6), t(35) = 1.7, p = .049.

Qualitative analysis

Students’ responses to the open-ended questions at Time 2 were analyzed using qualitative content analysis to identify advantages, disadvantages, and barriers to preregistration in students. This involved one author reading and coding the free-text responses for their content before discussing with the rest of the core authorship team (C. R. Pennington, E. Norris, and K. Clark). M. Pownall, in consultation with the rest of this research team, then generated categories and subcategories for the data before counting frequency within the responses. This allowed an exploratory investigation into students’ firsthand accounts of the advantages, disadvantages, and barriers of preregistration.

Table 4 shows the results of this content analysis. Three core categories were found for the perceived advantages of preregistration, each with subcategories. These were perceptions of preregistration for (a) improving clarity and organization, (b) reducing bias, and (c) promoting rigor and integrity. In terms of perceived disadvantages, two core categories were identified: (a) the time and effort required to preregister and (b) perceived rigidity of preregistration. Finally, the majority of participants did not report that they knew of any barriers but frequently noted need for support (including supervisory support and top-down wider support for preregistration) as a barrier to preregistration. For each category, there were also miscellaneous categories that were not frequent enough to represent core categories, but these are still presented in Table 4 for completeness.

Table 4.

Content Analysis of Students’ Free-Text Responses to Advantages, Disadvantages, and Barriers of Preregistration

Domain	Category	Subcategories	Frequency	Illustrative quotes
Advantages	Clarity and organization	Enhances students’ clarity with the research process	29	“Aided me in clarity when undergoing my dissertation, specifically stats”“Helps you to organise your thoughts”
		Prompts recordkeeping and planning	15	“You have a record of everything you were planning on doing that you can refer back to later when writing about your work”“Gives clear guidance to the university etc. as to what you are doing.”
		Promotes thoroughness and thoughtfulness	12	“You know exactly what you are studying and what you are researching”“You must think carefully about your hypothesis when designing an experiment”
	Reducing bias	Prevents p-hacking and HARKing	44	“Preregistering your study gives you a concrete plan you have to follow, which deters behaviours such as creating new hypotheses after data collection.”“Avoids any problems which could arise from data analysis (e.g., p-hacking etc.)”
	Reducing bias	Reduces pressure to find significant results	4	“It also helps with destigmatising null results as it demonstrates how studies that are performed correctly and to a good standard can achieve null yet still meaningful results. It can also encourage people to conduct studies without the pressure of having to gain significant results.”“Avoid the publication of only significant results and meaningful results only. Allows people to see exactly what you intend to do and if anything has changed there’s a reason for a it”
		Avoids fabrication of data	3	“To ensure no falsification of data”“Avoids any potential falsification”
	Rigor and integrity	Good research practices	16	“Encourages good research practices and scientific integrity.”“Allows for more better practices in science”
	Rigor and integrity	Promotes transparency and replicability	16	“Allows for the study to be replicated easily by another person”“Adopts an open approach towards the study design details, promoting replicability.”
	Miscellaneous	Avoids scooping	2	“You get to ‘claim’ your idea first”“Could also act as a way to establish ‘ownership’ of a novel concept, safeguarding against research ideas theft.”
	Miscellaneous	Grades	1	“You get good grades”
Disadvantages	Time and effort	Time-consuming	20	“The time required for its submission process.”“Time consuming”
		Early effort required	10	“More effort for researchers and it is also questionable how many people will actually check and control for the information and time stamps of the pre-registration.”
		Negatively affects confidence	5	“Things can go wrong in unexpected ways, it can feel like the research is failing if I can’t stick to what I pre-registered”
		Fear of scooping	5	“People might be able to steal others research ideas and beat them to publication.”“Possibility that reviewers may scoop my research.”
	Perceived rigidity	Lack of flexibility	16	“Reduction of freedom to change items. Inability to adjust open ended research questions.”“There may be points whilst writing a dissertation where thoughts and perceptions change and pre-registration somewhat denies the flexibility to change research focus and data collection.”
		Little scope to update following training	15	“Doesn’t allow for you to change your mind as you learn more (e.g., if you’re an undergraduate student still learning different methods of data analysis).”“Makes it unable to change little things in study in future like sample size as it would differ from the preregistration.”
		Restricts creativity	7	“Can force you into a less exploratory and more fixed approach can force you to organise things way earlier than you want to time consuming.”“If you think of something interesting half way through you should really probably leave it out of the paper.”
Barriers	Need for support	Training needs	22	“For me as a student it definitely was the fact that I wasntt very educated about pre-registration and therefore didn’t know how or when to do it.”“Sometimes you’re not educated enough to make a proper judgement before seeing the data.”
		Top-down implementation and support	10	“It wasn’t available during our undergraduate project.”“If it is not a course requirement”
		Need for supervisory support	4	“Lack of support from supervisor”“Lack of mentor/project partner support”
	Miscellaneous	Unsure of barriers	32	“Don’t know”
	Miscellaneous	Practical barriers	2	“It is difficult to know what to write within the manuscript.”

Note: HARKing = hypothesizing after results are known.

Discussion

The aim of this study was to provide the first empirical investigation into the pedagogical impact of study preregistration on undergraduate students in the final-year dissertation. Students who preregistered their dissertations showed an increase in perceived understanding of Open Science terms (e.g., the replication crisis, p-hacking, open data, file-drawer effect) compared with students who did not preregister, but other outcomes did not appear to be significantly influenced by the preregistration process (e.g., attitudes toward statistics and acceptance of QRPs). Informed by the COM-B model of behavior change, results also indicated that at the start of the academic year (i.e., at Time 1), students who later preregistered their dissertation also reported significantly higher capability, opportunity, and motivation to preregister, suggesting that these may be key factors in the uptake of preregistration. This also provides initial evidence for the value of a COM-B behavior-change approach to open-science behavior uptake (see Norris & O’Connor, 2019). Qualitative analyses showed further that students generally perceived preregistration to confer some advantages to their dissertation, such as improved rigor, thoughtfulness, and enhanced clarity of the dissertation process. However, they also noted some barriers, including the need for support, the extra time and effort required for preregistration, and a perceived lack of flexibility and creativity within the research analysis. We note that these apparent obstacles echo those documented by published researchers whom, for example, have noted inflexibility, time consumption, and fear of scooping as barriers to preregistration (Toth et al., 2021). In this way, students’ views appear largely reflective of wider considerations of preregistration in research practices (and indeed, these may be passed down through the supervisor–student relationship).

Implications

This study has much to contribute to the Open Science movement because it is the first study, to our knowledge, that empirically considers how one entry-level Open Science practice might be useful in tackling some of the challenges that undergraduate students face in their dissertation-research process. Our findings suggest that the process of preregistration can bolster students’ confidence with understanding Open Science concepts more broadly, which suggests that this practice may indeed be a useful way of providing an entry point into the wider Open Science conversation. However, findings also generally found no evidence to suggest that preregistration affected attitudes toward statistics and acceptance of QRPs, contrary to our hypotheses. Preregistration may also have benefits beyond those that are captured in the measures of the present study, and thus this warrants further research. For example, engagement in the preregistration process may likely improve outcomes such as students’ trust in the research they are conducting, inspire ambitions to pursue a career in research, and improve research literacy above and beyond attitudes toward statistics. These potential variables are all worthy of investigation in future studies to further interrogate how preregistration and, indeed, Open Science tools more broadly may confer advantages to undergraduate students.

Furthermore, our study also has broad implications for communities of Open Science, too. Supporters of Open Science have eloquently and convincingly made the moral and theoretical argument for embedding Open Science within undergraduate teaching and supervision. However, there is a notable lack of empirical, experimental research that gathers data to assess whether students actually benefit from engagement with these practices. To our knowledge, this study is the first to use quasi-experimental methods to begin to investigate this research question. This study thus responds directly to the calls of Pownall et al. (2023) to adopt the principles of Open Science (e.g., robust methodologies, preregistration, open data sharing, collaborative science) to pedagogical research about the value of Open Science. As Pownall et al. noted, to date, the majority of evidence available to educators and scholars who wish to make decisions about the incorporation of Open Science into their pedagogy typically relies on anecdotal and local-level evaluations of practice, which lack control groups and the ability to draw broader conclusions.

Limitations

We must acknowledge certain limitations of the present study. First, our sample size was smaller than we initially planned, largely because of the attrition from Time 1 to Time 2 of the survey and the implementation of rigorous data-quality checks. This meant that instead of being able to detect effect sizes of approximately d = 0.40 for the pairwise comparisons of interest, we were able to detect effect sizes of d ≥ 0.66 with 80% power (i.e., medium-large effect sizes). This means that we were powered enough to be able to detect only medium-large effects. Therefore, it is possible that null results reported here were because of an inability for us to detect smaller significant effects with our smaller than planned sample size rather than the absence of a true effect. Therefore, future research should aim to conceptually replicate our findings with larger sample sizes that are better equipped to detect smaller effect sizes. The issue of sample size is a challenge inherent within all quasi-experimental and longitudinal research, and we implemented multiple approaches to mitigate this, such as close contact with study participants through their supervisors and follow-up emails to participate (see Recruitment). Therefore, we call now to other pedagogical scholars to take these reported findings as one early investigation into the impact of preregistration and urge the discipline to continue to provide high-quality, rigorous, nationally representative data to shine empirical light onto Open Science tools and their value. That is, current findings should be regarded as a useful first step in the exploration of preregistration and its pedagogic value, and we call on other researchers to shine further empirical light onto Open Science tools in education.

Other limitations include the discrepancies in student experiences, particularly when collecting data cross-institutionally. For example, students and supervisors who develop a detailed, rigorous preregistration and engage in the process more with their supervisor might report greater benefits compared with students and supervisors who develop a poor-quality, less detailed preregistration. Indeed, there is emerging literature to suggest that the specificity of preregistrations differs between researchers (Bakker et al., 2020). However, it is beyond the scope of this research to assess each preregistration for quality and rigor. Likewise, adherence to preregistration protocols is another indicator of preregistration value (i.e., if researchers do not strictly adhere to their analysis plan, it may not be useful in reducing QRPs or, in our context, improving statistics attitudes). No participants in our sample indicated that they did not follow their preregistration plan at all in their dissertation, but the extent to which students closely and actively used their preregistration is unknown; this suggests that more research is needed into the implementation of preregistration in a pedagogical context. Practical reasons for this may also be informed by our qualitative data here, which report perceived (dis)advantages to preregistration, including time restraints, perceptions of preregistration requiring high effort, and fears of limited flexibility in the analysis. Furthermore, many participants in our sample used “university templates” to preregister their dissertations. Although we asked participants to confirm that they set out an analysis plan in the preregistration, some templates may be more stringent than others, and these in themselves might differentially affect the pedagogical outcomes of their use. Future work could also focus on how preregistration may be useful for different types of dissertations, including qualitative studies and analyses of secondary data.

Conclusion

Taken together, our quantitative and qualitative findings have demonstrated that although study preregistration did not significantly affect students’ attitudes toward statistics or their acceptance of QRPs, students who preregistered reported significantly greater perceived understanding of Open Science from Time 1 to Time 2 compared with students who did not preregister. Furthermore, students who preregistered reported significantly greater capability, opportunity, and motivation to preregister, suggesting that the COM-B model of behavior change might be a useful theoretical approach to understand Open Science uptake. Specifically, this suggests that when there is sufficient opportunity, capability, and motivation to engage with the preregistration process, there may be beneficial downstream consequences for students, including bolstered understanding of Open Science and science reform. Students also reported a range of positive potential benefits of preregistration, including heightened transparency, improved clarity with the dissertation data-analysis process, and reduction of the lure to engage in QRPs (e.g., p-hack their results to obtain significant findings). However, before preregistration is integrated into dissertations as standard, some key barriers should be considered, such as time pressures, perceived rigidity of preregistration, and need for adequate training, as other researchers have recently noted (Spitzer & Mueller, 2023). We hope that this study will contribute to the ongoing reappraisal of Open Science to progress conversations about the robustness, replicability, and reliability of psychological science. In recent years, there have been productive and important considerations of how to maximize the potential of Open Science practices (see Gervais, 2021; Suls et al., 2022), and the present study contributes to these ongoing metascientific efforts.

Our findings also contribute to the case that Open Science should be embedded into higher education for improved student scientific literacy and confidence (for a review, see Pownall et al., 2023). In response to the UK House of Commons Science and Technology Committee’s call for evidence of the contributors to research integrity, FORRT argued the importance of the pedagogical consequences of how students are taught, mentored, and supervised (Azevedo et al., 2021). A wide range of resources have recently been developed to support student learning of Open Science, including a student guide (Pennington, 2023), the FORRT project materials (Azevedo et al., 2019), and the Collaborative Replications and Education Project (Wagge et al., 2019).

These efforts aim to strengthen student knowledge and engagement in research to become more savvy consumers of science (Korbmacher et al., 2023). There is now a need for researchers to continue this line of work, critically and empirically investigating how barriers to Open Science can be negated with students (and, indeed, more broadly) to continue embedding high-quality, rigorous, thoughtful research practices into the undergraduate dissertation and beyond.

Footnotes

Acknowledgements

Stage 1 Peer Community in Registered Reports recommendation is available at https://rr.peercommunityin.org/articles/rec?id=48. Stage 2 Peer Community in Registered Reports recommendation is available at .

Transparency

Action Editor: David A. Sbarra

Editor: David A. Sbarra

Author Contribution(s)

All authors except M. Pownall, C. R. Pennington, E. Norris, and K. Clark are in randomized order.

Madeleine Pownall: Conceptualization; Data curation; Formal analysis; Funding acquisition; Investigation; Methodology; Project administration; Resources; Validation; Visualization; Writing – original draft; Writing – review & editing.

Charlotte R. Pennington: Conceptualization; Funding acquisition; Investigation; Methodology; Validation; Writing – original draft; Writing – review & editing.

Emma Norris: Conceptualization; Funding acquisition; Investigation; Validation; Writing – original draft; Writing – review & editing.

Marie Juanchich: Investigation; Writing – review & editing.

David Smailes: Investigation; Writing – review & editing.

Sophie Russell: Investigation; Writing – review & editing.

Debbie Gooch: Investigation; Writing – review & editing.

Thomas Rhys Evans: Investigation; Writing – review & editing.

Sofia Persson: Investigation; Writing – review & editing.

Matthew H. C. Mak: Investigation; Writing – review & editing.

Loukia Tzavella: Investigation; Writing – review & editing.

Rebecca Monk: Investigation; Writing – review & editing.

Thomas Gough: Investigation; Writing – review & editing.

Christopher S. Y. Benwell: Investigation; Writing – review & editing.

Mahmoud Elsherif: Investigation; Writing – review & editing.

Emily Farran: Investigation; Writing – review & editing.

Thomas Gallagher-Mitchell: Investigation; Writing – review & editing.

Luke T. Kendrick: Investigation; Writing – review & editing.

Julia Bahnmueller: Investigation; Writing – review & editing.

Emily Nordmann: Investigation; Writing – review & editing.

Mirela Zaneva: Investigation; Writing – review & editing.

Katie Gilligan-Lee: Investigation; Writing – review & editing.

Marina Bazhydai: Investigation; Writing – review & editing.

Andrew Jones: Investigation; Writing – review & editing.

Jemma Sedgmond: Investigation; Writing – review & editing.

Iris Holzleitner: Investigation; Writing – review & editing.

James Reynolds: Investigation; Writing – review & editing.

Jo Moss: Investigation; Writing – review & editing.

Daniel Farrelly: Investigation; Writing – review & editing.

Adam J. Parker: Investigation; Writing – review & editing.

Kait Clark: Conceptualization; Formal analysis; Funding acquisition; Investigation; Methodology; Validation; Writing – original draft; Writing – review & editing.

ORCID iDs

Madeleine Pownall

Charlotte R. Pennington

Marie Juanchich

Thomas Rhys Evans

Rebecca Monk

Mahmoud Elsherif

Emily Nordmann

Mirela Zaneva

Adam J. Parker

References

Asendorpf

J. B.

Conner

De Fruyt

De Houwer

Denissen

J. J.

Fiedler

Funder

D. C.

Kliegl

Nosek

B. A.

Perugini

Roberts

B. W.

Schmitt

van Aken

M. A. G.

Weber

Wicherts

J. M.

(2013). Recommendations for increasing replicability in psychology. European Journal of Personality, 27(2), 108–119. https://doi.org/10.1037/14805-038

Ayebo

Bright

Ballam

(2019). Examining the factor structure of the survey of attitudes towards statistics among undergraduate health science students. International Electronic Journal of Mathematics Education, 15(1), Article em0560. https://doi.org/10.29333/iejme/5942

Azevedo

Liu

Pennington

C. R.

Pownall

Evans

T. R.

Parsons

Elsherif

Micheli

Westwood

, & FORRT. (2021). Towards a culture of open scholarship: The role of pedagogical communities. MetaArXiv. https://doi.org/10.31222/osf.io/ek9m5

Azevedo

Parsons

Micheli

Strand

J. F.

Rinke

Guay

Elsherif

Quinn

Wagge

J. R.

Steltenpohl

Kalandadze

Vasilev

Ferreira de Oliveira

Aczel

Miranda

Galang

C. M.

Baker

B. J.

Pennington

C. R.

Marques

, . . . FORRT. (2019). Introducing a Framework for Open and Reproducible Research Training (FORRT). OSF Preprints. https://doi.org/10.31219/osf.io/bnh7p

Bakker

Veldkamp

C. L.

van Assen

M. A.

Crompvoets

E. A.

Ong

H. H.

Nosek

B. A.

Soderberg

C. K.

Mellor

Wicherts

J. M.

(2020). Ensuring the quality and specificity of preregistrations. PLOS Biology, 18(12), Article e3000937. https://doi.org/10.1371/journal.pbio.3000937

Banks

G. C.

Rogelberg

S. G.

Woznyj

H. M.

Landis

R. S.

Rupp

D. E.

(2016). Editorial: Evidence on questionable research practices: The good, the bad, and the ugly. Journal of Business and Psychology, 31(3), 323–338. https://doi.org/10.1007/s10869-016-9456-7

Blincoe

Buchert

(2020). Research preregistration as a teaching and learning tool in undergraduate psychology courses. Psychology Learning & Teaching, 19(1), 107–115.

Bowman

S. D.

DeHaven

A. C.

Errington

T. M.

Hardwicke

T. E.

Mellor

D. T.

Nosek

B. A.

Soderberg

C. K.

(2020). OSF prereg template. MetaArXiv. https://doi.org/10.31222/osf.io/epgjd

British Psychological Society. (2019). Standards for the accreditation of undergraduate, conversion and integrated Masters programmes in psychology.

10.

Button

(2018). Reboot undergraduate courses for reproducibility. Nature, 561(7723), 287–288.

11.

Button

K. S.

Chambers

C. D.

Lawrence

Munafò

M. R.

(2020). Grassroots training for reproducible science: A consortium-based approach to the empirical dissertation. Psychology Learning & Teaching, 19(1), 77–90.

12.

Button

K. S.

Lawrence

N. S.

Chambers

C. D.

Munafò

M. R.

(2016). Instilling scientific rigour at the grassroots. The Psychologist, 29(3), 158–159.

13.

Chopik

W. J.

Bremner

R. H.

Defever

A. M.

Keller

V. N.

(2018). How (and whether) to teach undergraduates about the replication crisis in psychological science. Teaching of Psychology, 45(2), 158–163.

14.

Creaven

A. M.

Button

Woods

Nordmann

(2021). Maximising the educational and research value of the undergraduate dissertation in psychology. PsyArXiv. https://doi.org/10.31234/osf.io/deh93

15.

Crüwell

van Doorn

Etz

Makel

M. C.

Moshontz

Niebaum

J. C.

Schulte-Mecklenbeck

(2019). Seven easy steps to Open Science. Zeitschrift für Psychologie, 227(4), 237–248. https://doi.org/10.1027/2151-2604/a000387

16.

Dauphinee

T. L.

Schau

Stevens

J. J.

(1997). Survey of attitudes toward statistics: Factor structure and factorial invariance for women and men. Structural Equation Modeling, 4(2), 129–141.

17.

Day

Bobeva

(2007). Applying performance management principles to a learning environment for undergraduate dissertations: A case study. International Journal for Quality and Standards, 1(1), 217–239.

18.

de Leeuw

J. R.

Andrews

Livingston

Franke

Hartshorne

Hawkins

Wagge

. (2019). Using replication studies to teach research methods in cognitive science. Perspectives on Psychological Science, 7(6), 600–604.

19.

Detweiler-Bedell

J. B.

(2019). Undergraduate research teams that build bridges, produce publishable research, and strengthen grant proposals. Frontiers in Psychology, 10, Article 133. https://doi.org/10.3389/fpsyg.2019.00133

20.

Devezer

Navarro

D. J.

Vandekerckhove

Buzbas

E. O.

(2020). The case for formal methodology in scientific reform. BiorXiv. https://doi.org/10.1101/2020.04.26.048306

21.

Devonport

T. J.

Lane

A. M.

(2006). Cognitive appraisal of dissertation stress among undergraduate students. The Psychological Record, 56(2), 259–266.

22.

Dienes

(2011). Bayesian versus orthodox statistics: Which side are you on? Perspectives on Psychological Science, 6(3), 274–290.

23.

Franco

Malhotra

Simonovits

(2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502–1505.

24.

Frank

M. F.

Saxe

(2012). Teaching replication. Perspectives on Psychological Science, 7, 600–604. https://doi.org/10.1177/1745691612460686

25.

Frith

(2020). Fast lane to slow science. Trends in Cognitive Sciences, 24(1), 1–2.

26.

Gervais

W. M.

(2021). Practical methodological reform needs good theory. Perspectives on Psychological Science, 16(4), 827–843.

27.

Greenbank

Penketh

Schofield

Turjansky

(2008). The undergraduate dissertation: ‘Most likely you go your way and I’ll go mine.’ The International Journal for Quality and Standards, 3(22), 1–24.

28.

Grimes

D. R.

Bauch

C. T.

Ioannidis

J. P.

(2018). Modelling science trustworthiness under publish or perish pressure. Royal Society Open Science, 5(1), Article 171511. https://doi.org/10.1098/rsos.171511

29.

Head

M. L.

Holman

Lanfear

Kahn

A. T.

Jennions

M. D.

(2015). The extent and consequences of p-hacking in science. PLOS Biology, 13(3), Article e1002106. https://doi.org/10.1371/journal.pbio.1002106

30.

Janz

(2016). Bringing the gold standard into the classroom: Replication in university teaching. International Studies Perspectives, 17(4), 392–407.

31.

JASP Team. (2020). JASP (Version 0.14.1) [Computer software].

32.

Jekel

Fiedler

Allstadt Torras

Mischkowski

Dorrough

A. R.

Glöckner

(2020). How to teach Open Science principles in the undergraduate curriculum—The Hagen Cumulative Science Project. Psychology Learning & Teaching, 19(1), 91–106.

33.

John

L. K.

Loewenstein

Prelec

(2012). Measuring the prevalence of questionable research practices with incentives for truth telling. Psychological Science, 23(5), 524–532.

34.

Kerr

N. L.

(1998). HARKing: Hypothesizing after the results are known. Personality and Social Psychology Review, 2(3), 196–217.

35.

Keyworth

Epton

Goldthorpe

Calam

Armitage

C. J.

(2020). Acceptability, reliability, and validity of a brief measure of capabilities, opportunities, and motivations (“COM-B”). British Journal of Health Psychology, 25, 474–501. https://doi.org/10.1111/bjhp.12417

36.

Kidwell

M. C.

Lazarević

L. B.

Baranski

Hardwicke

T. E.

Piechowski

Falkenberg

L. S.

Kennett

Slowik

Sonnleitner

Hess-Holden

Errington

Fielder

Nosek

B. A.

(2016). Badges to acknowledge open practices: A simple, low-cost, effective method for increasing transparency. PLOS Biology, 14(5), Article e1002456. https://doi.org/10.1371/journal.pbio.1002456

37.

Klein

Hardwicke

T. E.

Aust

Breuer

Danielsson

Mohr

A. H.

IJzerman

Nilsonne

Vanopaemel

Vazire

(2018). A practical guide for transparency in psychological science. Collabra: Psychology, 4(1), Article 20. https://doi.org/10.1525/collabra.158

38.

Korbmacher

Azevedo

Pennington

Hartmann

Pownall

Schmidt

Elsherif

Breznau

Robertson

Kalandaze

Baker

B. J.

O’Mahony

Olsnes

J. Ø.-S.

Shaw

J. J.

Gjoneska

Yamada

Röer

J. P.

Murphy

. . . Evans

(2023). The replication crisis has led to positive structural, procedural, and community changes. Communications Psychology, 1, Article 3. https://doi.org/10.1038/s44271-023-00003-2

39.

Krishna

Peter

S. M.

(2018). Questionable research practices in student final theses – Prevalence, attitudes, and the role of the supervisor’s perceived attitudes. PLOS ONE, 13, Article e0203470. https://doi.org/10.1371/journal.pone.0203470

40.

Kvetnaya

Frank

Brachem

Hill

Falk Florentin Schramm

Eiberger

(2019). Questionable research practices and Open Science in undergraduate empirical projects: Results from a nationwide survey amongst German psychology students [Paper presentation]. Proceedings of Open Practices in Education conference. http://www.dipfdocs.de/volltexte/2020/19034/pdf/Heck_2019_OPINE_Proceedings_A.pdf#page=42

41.

Lakens

(2021). Sample size justifications. PsyArXiv. https://doi.org/10.31234/osf.io/9d3yf

42.

Lakens

Caldwell

(2021). Simulation-based power analysis for factorial analysis of variance designs. Advances in Methods & Practices in Psychological Science, 4(1). https://doi.org/10.1177/2515245920951503

43.

LeBel

E. P.

Borsboom

Giner-Sorolla

Hasselman

Peters

K. R.

Ratliff

K. A.

Smith

C. T.

(2013). PsychDisclosure.org: Grassroots support for reforming reporting standards in psychology. Perspectives on Psychological Science, 8(4), 424–432.

44.

LeBel

E. P.

Peters

K. R.

(2011). Fearing the future of empirical psychology: Bem’s (2011) evidence of psi as a case study of deficiencies in modal research practice. Review of General Psychology, 15(4), 371–379. https://doi.org/10.1037/a0025172

45.

Lindsay

D. S.

Simons

D. J.

Lilienfeld

S. O.

(2016, November 30). Research preregistration 101. APS Observer. https://www.psychologicalscience.org/observer/research-preregistration-101

46.

Michie

Van Stralen

M. M.

West

(2011). The behaviour change wheel: A new method for characterising and designing behaviour change interventions. Implementation Science, 6, Article 42. https://doi.org/10.1186/1748-5908-6-42

47.

Munafò

M. R.

Nosek

B. A.

Bishop

D. V.

Button

K. S.

Chambers

C. D.

Du Sert

N. P.

Simonsohn

Wagenmakers

E. J.

Ware

J. J.

Ioannidis

J. P.

(2017). A manifesto for reproducible science. Nature Human Behaviour, 1, Article 0021. https://doi.org/10.1038/s41562-016-0021

48.

Nicholas

Watkinson

Boukacem-Zeghmouri

Rodríguez-Bravo

Abrizah

Świgoń

Herman

(2017). Early career researchers: Scholarly behaviour and the prospect of change. Learned Publishing, 30(2), 157–166. https://doi.org/10.1002/leap.1098

49.

Norris

O’Connor

D. B.

(2019). Science as behaviour: Using a behaviour change approach to increase uptake of Open Science. Psychology & Health, 34(12), 1397–1406. https://doi.org/10.1080/08870446.2019.1679373

50.

Nosek

B. A.

Ebersole

C. R.

DeHaven

A. C.

Mellor

D. T.

(2018). The preregistration revolution. Proceedings of the National Academy of Sciences, USA, 115(11), 2600–2606.

51.

Nuzzo

(2015). How scientists fool themselves–and how they can stop. Nature News, 526(7572), 182–185.

52.

O’Boyle

E. H.

Jr. Banks

G. C.

Gonzalez-Mulé

(2017). The chrysalis effect: How ugly initial results metamorphosize into beautiful articles. Journal of Management, 43(2), 376–399.

53.

Olson

Mosen

Voracek

KIrchler

(2019). Research practices and statistical reporting quality in 250 economic psychology master’s theses: A meta-research investigation. Royal Society Open Science, 6, Article 190738. https://doi.org/10.1098/rsos.190738

54.

Onwuegbuzie

A. J.

Wilson

V. A.

(2003). Statistics anxiety: Nature, etiology, antecedents, effects, and treatments—A comprehensive review of the literature. Teaching in Higher Education, 8(2), 195–209.

55.

Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), Article aac4716. https://doi.org/10.1126/science.aac4716

56.

Orben

(2019). A journal club to fix science. Nature, 573(7775), 465–466.

57.

Palan

Schitter

(2018). Prolific.ac—A subject pool for online experiments. Journal of Behavioral and Experimental Finance, 17, 22–27.

58.

Pashler

Wagenmakers

E. J.

(2012). Editors’ introduction to the special section on replicability in psychological science: A crisis of confidence? Perspectives on Psychological Science, 7(6), 528–530.

59.

Pennington

(2023). A student’s guide to Open Science: Using the replication crisis to reform psychology. McGraw-Hill Education.

60.

Pownall

(2020a, June 20). Early career researchers in Open Science: Vanguard or cannon fodder? The Psychologist. https://www.bps.org.uk/psychologist/early-career-researchers-open-science-vanguard-or-cannon-fodder

61.

Pownall

(2020b). Pre-registration in the undergraduate dissertation: A critical discussion. Psychology Teaching Review, 26(1), 71–76.

62.

Pownall

Azevedo

König

L. M.

Slack

H. R.

Evans

Flack

Grinschgl

Elsherif

M. M.

Gilligan-Lee

K. A.

de Oliveira

C. M. F.

Gjoneska

Kalandadze

Button

Ashcroft-Jones

Terry

Albayrak-Aydemir

Deˇchteˇrenko

Alzahawi

Baker

B. J.

, . . . FORTT. (2023). Teaching open and reproducible scholarship: A critical review of the evidence base for current pedagogical methods and their outcomes. Royal Society Open Science, 10(5), Article 221255. https://doi.org/10.1098/rsos.221255

63.

Roberts

L. D.

Seaman

(2018). Good undergraduate dissertation supervision: Perspectives of supervisors and dissertation coordinators. International Journal for Academic Development, 23(1), 28–40.

64.

Rouder

Speckman

Sun

Morey

(2009). Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic Bulletin & Review, 16(2), 255–237. https://doi.org/10.3758/pbr.16.2.225

65.

Sacco

D. F.

Brown

(2019). Assessing the efficacy of a training intervention to reduce acceptance of questionable research practices in psychology graduate students. Journal of Empirical Research on Human Research Ethics, 14, 209–218. https://doi.org/10.1177/1556264619840525

66.

Sarafoglou

Hoogeveen

Matzke

Wagenmakers

E.-J.

(2020). Teaching good research practices: Protocol of a research master course. Psychology, Learning & Teaching, 19, 46–59. https://doi.org/10.1177/1475725719858807

67.

Schönbrodt

F. D.

Wagenmakers

E. J.

Zehetleitner

Perugini

(2017). Sequential hypothesis testing with Bayes factors: Efficiently testing mean differences. Psychological Methods, 22(2), 322–339.

68.

Simmons

J. P.

Nelson

L. D.

Simonsohn

(2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366.

69.

Sorokowski

Groyecka

Błaszczyński

Fra˛ckowiak

Kobylarek

(2019). Registered reports as a method to increase credibility of science: Experimental study among psychological students. Journal of Education Culture and Society, 10(2), 67–75.

70.

Spitzer

Mueller

(2023). Registered report: Survey on attitudes and experiences regarding preregistration in psychological research. PLOS ONE, 18(3), Article e0281086. https://doi.org/10.1371/journal.pone.0281086

71.

Stürmer

Oeberst

Trötschel

Decker

(2017). Early-career researchers’ perceptions of the prevalence of questionable research practices, potential causes, and Open Science. Social Psychology, 48(6), 365–371. https://doi.org/10.1027/1864-9335/a000324

72.

Suls

Rothma

A. J.

Davidson

K. W.

(2022). Now is the time to assess the effects of open science practices with randomized control trials. American Psychologist, 77(3), 467–475.

73.

Szollosi

Kellen

Navarro

Shiffrin

van Rooij

Van Zandt

Donkin

(2019). Is preregistration worthwhile? Trends in Cognitive Science, 24(2), 94–95. https://doi.org/10.1016/j.tics.2019.11.009

74.

Tennant

J. P.

Waldner

Jacques

D. C.

Masuzzo

Collister

L. B.

Hartgerink

C. H.

(2016). The academic, economic and societal impacts of Open Access: An evidence based review. F1000Research, 5, Article 632. https://doi.org/10.12688/f1000research.8460.3

75.

Toth

A. A.

Banks

G. C.

Mellor

O’Boyle

E. H.

Dickson

Davis

D. J.

DeHaven

Bochantin

Borns

(2021). Study preregistration: An evaluation of a method for transparent reporting. Journal of Business and Psychology, 36(4), 553–571. https://doi.org/10.1007/s10869-020-09695-3

76.

van’t Veer

A. E.

Giner-Sorolla

. (2016). Pre-registration in social psychology—A discussion and suggested template. Journal of Experimental Social Psychology, 67, 2–12.

77.

Wagenmakers

E. J.

Wetzels

Borsboom

van der Maas

H. L.

Kievit

R. A.

(2012). An agenda for purely confirmatory research. Perspectives on Psychological Science, 7(6), 632–638.

78.

Wagge

J. R.

Brandt

M. J.

Lazarevic

L. B.

Legate

Christopherson

Wiggins

Grahe

J. E.

(2019). Publishing research with undergraduate students via replication work: The collaborative replications and education project. Frontiers in Psychology, 10, Article 247. https://doi.org/10.3389/fpsyg.2019.00247

79.

Zecˇević

Houghton

Noone

Lee

Matvienko-Sikar

Toomey

(2020). Exploring factors that influence the practice of Open Science by early career health researchers: A mixed methods study. HRB Open Research, 3, Article 56. https://doi.org/10.12688/hrbopenres.13119.2

Evaluating the Pedagogical Effectiveness of Study Preregistration in the Undergraduate Dissertation

Abstract

Keywords

Open Science in Undergraduate Training

The Value of Preregistration

The undergraduate dissertation

The Present Study

Method

Transparency statement

Design and participants

Recruitment plan

Procedure

Measures (Time 1)

SATS-28

Acceptance of QRPs

Perceived understanding of Open Science

Attention and bot checks

Exploratory measures

Capability, opportunity, and motivation toward preregistration

After dissertation (Time 2)

Perceptions of supervisory support

Risk and mitigations

Analysis Strategy

Results

Descriptives about preregistration practice

Attitudes toward statistics

Acceptance of QRPs

Perceived understanding of Open Science

Exploratory analyses

COM-B

Qualitative analysis

Discussion

Implications

Limitations

Conclusion

Footnotes

Acknowledgements

Transparency

ORCID iDs

References