Abstract
Psychology programs often emphasize inferential statistical tests over a solid understanding of data and research design. This imbalance may leave graduates underequipped to effectively interpret research and employ data to answer questions. We conducted a two-round modified Delphi to identify the research-methods skills that the UK psychology community deems essential for undergraduates to learn. Participants included 103 research-methods instructors, academics, students, and nonacademic psychologists. Of 78 items included in the consensus process, 34 reached consensus. We coupled these results with a qualitative analysis of 707 open-ended text responses to develop nine recommendations for organizations that accredit undergraduate psychology programs—such as the British Psychological Society. We recommend that accreditation standards emphasize (1) data skills, (2) research design, (3) descriptive statistics, (4) critical analysis, (5) qualitative methods, and (6) both parameter estimation and significance testing; as well as (7) give precedence to foundational skills, (8) promote transferable skills, and (9) create space in curricula to enable these recommendations. Our data and findings can inform modernized accreditation standards to include clearly defined, assessable, and widely encouraged skills that foster a competent graduate body for the contemporary world.
Keywords
For graduates from psychology programs to thrive, they must become effective thinkers in a data-laden world. Research-methods education in psychology programs, however, often emphasizes inferential statistical tests over a deep understanding of data and research design (TARG Meta-Research Group, 2022), which could lead to the problematic use and interpretation of statistics. Moreover, open research practices are not yet embedded in many curricula, and qualitative research methods often remain underemphasized. In the UK, the British Psychological Society (BPS; 2019) sets the requirements for the vast majority of undergraduate psychology programs through their accreditation standards. Here, we conducted a consensus process, open to the UK psychology community, with the aim to strengthen the research-methods section of the BPS accreditation standards.
In response to the “replication crisis,” psychology researchers have increasingly adopted open science practices and considered statistical power, sample size, and the use of estimation (Cumming & Calin-Jageman, 2016). These advances, however, are not yet well reflected in psychology curricula. At least five studies have assessed the content of university-level psychology programs in the United States and generally concluded that there have been few updates to the curricula over the past 2 or 3 decades (Aiken et al., 1990, 2008; Anglin & Edlund, 2020; Friedrich et al., 2000, 2018). A similar study in the UK found that only 19% of universities had publicly available syllabi describing the content taught in each of their statistics modules in undergraduate psychology (TARG Meta-Research Group, 2022). Although these syllabi rarely contained a lesson-by-lesson breakdown, most mentioned specific inferential statistical tests (e.g., analyses of variance); about half mentioned probability and randomness, effect size, and statistical power; and few mentioned concepts such as confidence intervals, multiple comparisons, meta-analysis, replication, Bayesian statistics, frequentist statistics, and practical significance. Another study surveyed psychology students and instructors in the UK and found that few courses teach alternatives to null hypothesis significance testing (NHST) and that students’ anxiety around mathematics and statistics hold them back (Field, 2014). Note that a British Academy report highlighted this issue in stating that “a co-ordinated and continuous effort at improving quantitative skills across all phases of education and employment, in all four nations of the UK, is therefore now urgently needed” (The British Academy, 2015).
Qualitative research skills have been part of the BPS accreditation standards since 2004. However, there is limited research on how they are taught, and instructors may need additional training and resources to effectively teach qualitative methods (Gibson & Sullivan, 2018; Hugh-Jones et al., 2012; Wiggins et al., 2016). Some people also view qualitative research methods as the alternative and “lesser” approach to quantitative approaches, thus affecting how they are taught (Gibson & Sullivan, 2018; Hugh-Jones et al., 2012). Taken together, the time is ripe to modernize the teaching of quantitative and qualitative research methods in psychology programs.
We conducted the present consensus process with the goal of informing an updated version of the BPS Standards for the Accreditation of Undergraduate, Conversion and Integrated Masters Programmes in Psychology, specifically Section 2.1.4g of this document (BPS, 2019). Similar consensus processes have been used to develop standards in more than 200 medical education programs (Humphrey-Murto et al., 2017). The updated accreditation standards could contain specific actionable items for UK psychology research-methods curricula that reflect the need for data skills in the modern world and are adapted to the evolving educational landscape. Our results also provide a foundation for organizations beyond the BPS who seek to modernize research-methods education in undergraduate psychology programs.
Study Objective
Our objective was to achieve consensus regarding the accreditation standards for research-methods education in undergraduate psychology programs in the UK, specifically, to provide the BPS with information to update their accreditation standards for research-methods education. (For contextual information regarding BPS accreditation and UK undergraduate psychology programs, see Box 1.)
Contextual Information Regarding UK Undergraduate Psychology Programs and British Psychological Society Accreditation
Method
We used a modified Delphi technique to systematically elicit anonymous, asynchronous, and iterative input from a range of stakeholders (Humphrey-Murto et al., 2019). We used a consensus process over other methods 3 of informing updates to the accreditation standards because we wanted to engage the community—which can facilitate the acceptance and uptake of such standards (e.g., Bini & Mahajan, 2016)—and to better understand what the community believes are best teaching practices, because these are not yet established or widely accepted. We used the Delphi method in particular because it can be open to the entire UK psychology community and allows for a large panel of participants from diverse stakeholder groups to provide anonymous responses.
This Delphi study was codeveloped with input from the BPS to ensure that the results could be integrated into their Standards for the Accreditation of Undergraduate, Conversion and Integrated Masters Programmes in Psychology (BPS, 2019). We involved the BPS in the planning and execution of this study to benefit from their detailed knowledge of the accreditation standards and because they have the authority to decide how to implement our findings. By engaging them, we hoped to make our study design highly relevant and increase the use of our findings to inform tangible changes (e.g., in line with utilization-focused evaluation; Patton, 2003). The results from this study are not prescriptive to the BPS. The BPS committees responsible for the accreditation standards will follow their internal processes and evaluate the findings from this study along with other forms of evidence. For a thorough explanation of how the BPS was involved, see Supplementary Material A in the Supplemental Material available online.
The study included a preparatory stage and two Delphi rounds in which participants rated items and provided open-ended feedback (see Fig. 1). In a modification to the traditional Delphi method, our Delphi did not include an idea-generation round. The preparatory stage was conducted by the Delphi steering committee (see next section). They defined consensus, developed a set of questions that would inform updates to the BPS guidelines, and identified stakeholder groups. The steering committee conducted a nonsystematic literature review 4 on research-methods education in psychology programs (see Supplementary Material B). This review served as a basis for formulating the survey questions and was shared with participants at the survey’s outset to support informed responses.
An outline of our modified-Delphi study. A steering committee completed the preparatory stage, participants completed two Delphi rounds, and results were presented to the British Psychological Society Undergraduate Education Committee.
The study was approved by the School of Psychological Science Research Ethics Committee at the University of Bristol (ID: 13394). We preregistered a protocol before advertising the study (https://osf.io/5h7bu). Deviations from the preregistered protocol are outlined in Supplementary Material C. This article is reported in line with the ACCORD Checklist (ACcurate COnsensus Reporting Document; Gattrell et al., 2023), which is provided in Supplementary Material D. We report how we determined our sample size, all data exclusions, all manipulations, and all measures in the study.
Delphi steering committee
We assembled a steering committee to guide this Delphi study. We aimed to include members who met a range of representation criteria. These included representation from the following groups: BPS Undergraduate Education Committee, BPS Partnership and Accreditation Committee, undergraduate psychology program director, quantitative research-methods instructor/expert, qualitative research-methods instructor/expert, psychology researcher, statistician, education expert, UK country other than England, nonacademic psychologist, and relatively recent graduate from an undergraduate psychology program in the UK. With these criteria in mind, the steering committee comprised five members, including the project lead (R. T. Thibault), a BPS representative and undergraduate program lead (R. J. Green), a quantitative psychology research-methods instructor with recent experience as a psychology undergraduate in the UK (J. E. Bartlett), an open-science and pedagogic expert with recent experience as a psychology undergraduate in the UK (M. Pownall), and a qualitative psychology research-methods expert and research-methods instructor (D. Bailey-Rodriguez). We unfortunately did not succeed in achieving representation from a nonacademic person based in psychology. The steering committee communicated with the BPS accreditation operations manager (Patricia Lyons) and chair of the BPS Undergraduate Education Committee (Simon Goodson) to ensure that the study was designed in a way that the results could effectively inform an update to the accreditation standards. Members of the steering committee did not participate in the Delphi.
Participants
We advertised the Delphi survey via mailing lists and social media. We specifically reached out to the BPS, UK Reproducibility Network, ReproducibiliTea, and The Framework for Open and Reproducible Research Training (FORRT) to help advertise. For a template of the invitation text, see Supplementary Material E. We did not impose a limit on the number of participants.
Participants needed to meet two criteria. First, they needed to be a member of at least one of the following stakeholder groups: (a) student in psychology (undergraduate student, graduate student, or nonstudent who completed an undergraduate degree less than 3 years ago), (b) research-methods instructor in psychology, (c) academic based in psychology, or (d) nonacademic person working in psychology. Participants were instructed to select “option a” if they were an undergraduate or graduate student, regardless of whether they taught or did research, and to select “option b” if they teach or coordinate psychology research methods in an undergraduate or master’s conversion program, regardless of whether they are also an academic or nonacademic psychologist (for verbatim instructions, see Supplementary Material B). Second, participants needed to be either (a) based in the UK (or have been a member of one of the four stakeholder groups in the UK within the past 3 years) or (b) based outside of the UK but be associated with a BPS-accredited psychology program (e.g., the BPS accredits some psychology programs outside the UK).
Before presenting the Delphi items, the survey presented registration questions. These asked the participants whether they primarily conduct qualitative or quantitative research, are a research-methods expert, are an undergraduate program director, and live in the UK and the sector in which they are employed (see Supplementary Material F). No incentives were offered for participation.
Definition of consensus
Before launching the survey, the steering committee defined consensus as at least 75% of participants in each nonstudent stakeholder group rating an item as “essential” (i.e., between 7 and 9 on the 9-point scale). A 75% threshold is commonly used in Delphi studies (Diamond et al., 2014), and we felt this was a balanced approach. We excluded students from the definition of consensus because they may not have been exposed to many of the concepts presented in the Delphi. The steering committee further made the a priori decision that if a stakeholder group had fewer than 12 participants, we would not require that group to achieve 75% essential responses; instead, we would require that all nonstudent ratings collapsed together reached 75%. We made this decision to avoid a situation in which a very small number of participants are responsible for consensus not being reached. 5 We performed a sensitivity analysis that required 75% essential ratings from all four stakeholder groups (including students), and it revealed no difference in the items that reached consensus.
Survey
To conduct the survey, we used the DelphiManager software provided by the COMET Initiative. In Round 1 of the Delphi, participants were asked to rate their level of agreement to 72 items on a scale from 1 to 9 in which 1 to 3 is not important, 4 to 6 is important but not essential, and 7 to 9 is essential (for a screenshot of Round 1, see Fig. S1 in the Supplemental Material). These items ranged from specific content to teach (e.g., effect sizes, qualitative data collection, the replication crisis) to ways of teaching (e.g., evaluation methods) and encouraged resources (e.g., freely available software); for a complete list of items, see Supplementary Material H.
Round 1 comprised eight blocks that each presented between five and 15 items covering the following domains: (a) statistical analyses, (b) quantitative data skills, (c) quantitative research-methods concepts, (d) qualitative research methods, (e) research design, (f) reproducibility and open science, (g) accessibility of resources, and (h) miscellaneous. The blocks were presented in a random order. The items within each block were always presented in the same order (as required by the DelphiManager software). The research design block included an attention-check item that asked participants to select “3.”
The motivation for this study came from the lead author’s (R. T. Thibault) and senior author’s (M. R. Munafò) reflections about the quantitative abilities of psychology graduates and shortcomings in the reproducibility of quantitative psychology research. Items were selected based on previous studies on research-methods education in psychology (e.g., TARG Meta-Research Group, 2022) and with the aim of addressing the knowledge and skills gap that leads to irreproducible research. We aimed to word the items in such a way that the BPS could easily integrate the Delphi results into an updated version of the BPS accreditation standards. Moreover, we aimed to make the items specific enough that someone could assess whether that standard is being met. For example, instead of asking if students should learn to “critically evaluate research,” we asked whether students should learn how to “define and explain questionable research practices (QRPs)” or “cognitive biases (e.g., confirmation bias).” Qualitative research-methods skills were subsequently included in this Delphi upon a recommendation from the BPS representatives. R. T. Thibault drafted an initial list of items that the steering committee and BPS representatives modified and added to. 6
After participants rated all items in Round 1, the survey asked them to propose additional items that they felt the survey did not include but that they would deem important. The steering committee added some of these suggested items to Round 2 (many were reworded or combined to better match the Delphi structure). To ensure Round 2 did not take too long to complete, only a few suggested items were added. Suggested items were not added if they did not apply to psychology broadly, overlapped substantially with items in Round 1, or were too vague to be meaningfully integrated into the accreditation standards.
All participants who began Round 1 were invited via email to participate in Round 2. Items that reached consensus in Round 1 were removed from Round 2. No items were modified between Round 1 and Round 2. In Round 2, participants were provided with feedback about other participants’ responses from Round 1 and asked to rerate each item. For each item, they were shown the distribution of ratings from each stakeholder group alongside their own response from Round 1 (for a screenshot of Round 2, see Fig. S2 in the Supplemental Material).
Round 1 was open from February 8, 2023, to March 17, 2023. We extended the length of Round 1 because of delays in sending emails to the BPS mailing lists and because we were receiving fewer responses than expected. The initially low response rate may have been caused in part by the University and College Union strikes in February and March 2023. Round 2 was open from April 6, 2023, to April 28, 2023.
Open-ended questions
Participants were invited to provide open-ended written feedback at several points during the Delphi study. In chronological order, these were as follows: (a) When rating items in Round 1, participants had the option to provide written feedback on each item. (b) After rating all items in Round 1, participants were asked to suggest additional items for Round 2. (c) After completing Round 1, participants were prompted to provide any thoughts they have about the Delphi study. (d) After rating all items in Round 2, participants were asked to give a reason for all their answers that changed rating categories between Round 1 and Round 2. (e) After completing Round 2, participants were prompted to provide any thoughts they have about the Delphi study.
Analyses
For each item in the Delphi, our main results present the percentage of all participants who rated an item as essential, the mean rating across all participants, and whether the item reached consensus. Our open data (https://osf.io/hpsq4/) include six summary statistics data sheets: one for each stakeholder group individually, one for all nonstudent stakeholder groups collapsed together, and one for all four stakeholder groups collapsed together.
To identify common patterns, one team member (D. Bailey-Rodriguez) applied an inductive thematic analysis to the open-ended textual responses; the analysis was data-driven (Braun & Clarke, 2006). The analysis was based on a critical-realist ontological stance, which assumes that participants’ perspectives, beliefs, and thoughts can be accessed through interpretation of the data to understand their underlying structures, which are thought to be “real” (Willig, 2013). The thematic analysis included a careful reading and rereading of the textual data followed by thorough line-by-line coding. Emerging themes were then identified at the semantic level and reviewed. The themes were subsequently grouped together and given labels. A thematic map was produced, which was further refined, and finally, the report of the analysis was produced (Braun & Clarke, 2006).
Upon viewing the data, we became aware that the DelphiManager software outputs data in a format that does not allow the registration information—except for stakeholder group—to be linked to the ratings of the survey items. Thus, we cannot analyze the participant ratings in relation to their self-reported expertise or current employment (as per Table 1). This data structure also leaves us with registration information for only two samples: (a) all participants who at least completed the registration, even if they did not rate any items or failed the attention check, and (b) all participants who at least began Round 2.
Characteristics of Participants Who at Least Began Round 2
Note: Participants could select multiple responses for the “expertise” characteristic. BPS = British Psychological Society.
Results
Participants
Figure 2 provides a flowchart of participant inclusion. One hundred seventy participants began Round 1. Items reaching consensus after Round 1 were based on the 139 participants who passed the attention check and completed Round 1. Our final sample comprised 103 participants who passed the attention check and completed both Round 1 and Round 2. The open data include data sheets for each of the final sample, the participants who completed at least one round, and all participants who at least began Round 1.
Flowchart of participant inclusion in the final sample. The feedback presented during Round 2 came from the participants in the bottom right box (“Sample for items that reached consensus in Round 1”). Blue boxes indicate participants that were excluded from the final sample.
Table 1 outlines participant characteristics. 7 The majority of participants who began Round 2 primarily do quantitative research and are employed by universities. They registered for the Delphi using 64 unique email address domain names, which included 55 distinct UK academic domain names (i.e., ending in “.ac.uk”). For all participants who at least registered for the study, these numbers were 86 and 69, respectively.
Delphi-item ratings
Twenty-six items reached consensus in Round 1 and were not included in Round 2. The steering committee added seven items to Round 2, which thus contained a total of 52 items. 8
Eight items reached consensus in Round 2. Forty-four items did not reach consensus. Two of the items reaching consensus in Round 2 were added after Round 1 was complete, and two others had already reached consensus among the final sample of 103 participants (but not among the 139 who completed Round 1). Thus, among the final sample, only four items shifted from not reaching consensus in Round 1 to reaching consensus in Round 2. Across all 103 participants, regardless of stakeholder group, the median percentage of participants that rated an item as essential was 74% (interquartile range = 60%–86%). The median of the average (mean) rating across the 78 items was 7.4 (interquartile range = 6.8–7.9).
Table 2 presents the results for each of the 78 items. To explore these results, we recommend opening the spreadsheet available at https://osf.io/57mbd. This spreadsheet includes the verbatim items and domains, which could not easily fit in a pdf but contain important keywords. For example, whereas some items asked if students should learn how to apply a technique, others asked if students should learn to define a concept.
Rating and Consensus for the 78 Delphi Items
Note: Items are ordered by the column Consensus and then Rated Essential (%). Many items have been paraphrased so they can fit in this table. The domains have been shortened (stats = statistical analyses; data = quantitative data skills; quant = quantitative research methods concepts; qual = qualitative research methods; design = research design; OS = reproducibility and open science; resources = accessibility of resources; misc = miscellaneous). Full verbatim descriptions of the items and domains are available in the following spreadsheet: https://osf.io/57mbd. The rating scale ranges from 1 to 9, in which 1 to 3 is not important, 4 to 6 is important but not essential, and 7 to 9 is essential. The Consensus column contains a value of 1 if consensus was reached in Round 1, 2 if reached in Round 2, and “no” if consensus was not reached. Items that reached consensus in Round 1 have the columns Rated Essential (%) and Mean Rating taken from Round 1 (because these questions were not included in Round 2).
Sensitivity Analyses
We performed three sets of sensitivity analyses. We assessed differences in ratings between (a) Round 1 and Round 2, (b) the instructor and academic stakeholder groups, and (c) the initial sample of 170 participants and the final sample of 103 participants. We did not compare results from the students and nonacademic stakeholder groups because they had few enough participants that the comparisons would be uninformative or potentially misrepresentative.
Relatively small differences in ratings occurred between rounds. Of the 47 items included in both Round 1 and Round 2, four went from not reaching consensus to reaching consensus, the median absolute change in the percentage of participants that rated an item as essential was 3.8%, and the median absolute difference in participants’ rating of an item was 0.15. The open-ended responses revealed several reasons for why participants changed their ratings between rounds, including viewing the ratings of other participants, reflecting further, discussing with colleagues or students, and gaining further knowledge after Round 1. Between instructors and academic stakeholder groups, these differences were six items reaching consensus, 5.3% rating an item as essential, and a 0.24 median rating difference. Only small differences existed between ratings from the sample of participants who at least began Round 1 and the final sample (one item reaching consensus, 1.3% rating as essential, 0.06 median rating difference). For the data in tabular format, see Table S3 in the Supplemental Material.
Consensus summary results
In this section, we block related items into 13 overarching findings (see Table 3). We conceived these blocks based on whether the items reached consensus and whether they can be interpreted together to help formulate a specific recommendation. 9 In parentheses, we present the percentage of all participants who rated the items as essential.
Recommendation Topics Derived From Consensus Results
Consensus largely reached
Understanding data
Consensus was reached that students should learn how to identify and categorize different types of data (94%), clean data (88%), anonymize data (86%), and represent quantitative data visually (95%).
Research design (general)
Consensus was reached that students should learn how to formulate a research question (99%), design a study to answer a specific research question (97%), explain the difference between a research question and a hypothesis (83%), and identify basic study designs (95%). Consensus was also reached for learning how to create a sampling plan (86%), operationalize all elements of a study (83%), and apply experimental and nonexperimental research designs (82%).
Descriptive statistics
Consensus was reached that students should learn how to calculate descriptive statistics (98%), explain the importance of descriptive statistics and how they differ from inferential statistics (95%), and use descriptive statistics effectively before learning to perform inferential statistical tests (87%).
Inferential statistics
Consensus was reached that students should learn how to calculate significance tests (96%), regressions (85%), effect sizes (86%), and parameter estimations (e.g., confidence intervals; 79%). Consensus was also reached that students should learn to define and explain probability and randomness (81%) and was almost reached that students should learn to explain the existence of different statistical approaches, including parameter estimation and significance testing (72%). 10
Critical assessment
Consensus was reached that students should learn how to identify and assess ethical issues (99%), assess validity and reliability (91%), define and explain the difference between statistical significance and practical significance (89%), and define and explain the value of exploratory research and how it differs from confirmatory research (85%). Defining and explaining systematic reviews and meta-analysis (70%) did not reach consensus.
Consensus reached for some items
Qualitative research
Learning to critically appraise qualitative research reached consensus (83%). Several other qualitative items almost reached consensus, including learning to demonstrate understanding of several methods of qualitative data analysis (74%), perform qualitative analysis (73%), use reflexive practice (72%), demonstrate understanding of various qualitative frameworks (71%), 11 and demonstrate understanding of mixed-methods research (71%). Some qualitative items did not reach consensus, including learning to collect qualitative data (65%), explain the philosophical underpinnings of qualitative research (51%), and apply qualitative frameworks in their own research (50%). Fifteen participants provided open-ended feedback to specific Delphi items on qualitative methods, and 19 participants provided general comments that mentioned qualitative methods. These textual data are analyzed in the article section Theme 1: Important Factors.
Reproducibility and open science
Several items related to reproducibility and open science reached consensus, including that students should learn to define and explain sources of bias (91%), cognitive biases (80%), QRPs (85%), generalizability and robustness (89%), research misconduct (90%), and replication studies and reproducibility (82%).
Other items that almost reached consensus include learning to define and explain the replication crisis (75%) 12 and define and explain data, code, and material sharing (72%). Items more focused on the process of research did not reach consensus, including learning to define and explain the publication process (50%), reward structures in research and academia (44%), preregistration and Registered Reports (59%), and metaresearch (47%).
Consensus largely not reached
Advanced analysis techniques
More-advanced analysis techniques did not reach consensus, including learning to perform equivalence testing (34%), perform factor analysis (33%), simulate data (9%), and explain multiverse analyses/many-analyst approaches (17%).
Research design (specific)
Items on research designs specific to certain study types did not reach consensus, including learning how to design a survey (65%), apply blinding and randomization (63%), and explain psychometrics (65%). Items about sample size and effect sizes also did not reach consensus, including learning to perform a sample-size calculation for quantitative research (62%), determine a smallest effect size of interest (44%), and explain alternative measures of effect sizes (e.g., probability of superiority; 16%). Learning to select a sample size relevant to the qualitative method being used (76%) almost reached consensus.
Approaches to research
Consensus was not reached that students should learn how to consider diverse perspectives when designing a study (69%), define and explain philosophy of science (59%), or define and explain the existence of different statistical frameworks, including frequentist statistics and Bayesian statistics (40%).
Computer skills
Consensus was not reached regarding whether students should learn how to use a programming language to manage and analyze data (23%) or use a statistical analysis package with a graphical user interface (66%). Whether research-methods modules should use only freely available software (28%) did not reach consensus. Consensus was almost reached for learning to demonstrate general computer skills for research (73%) and search and collate published research (77%).
Module format
No items regarding the format of modules reached consensus, including to never entirely grade with closed-book exams (67%), emphasize skills that transfer beyond an academic research context (67%), have a higher staff-to-student ratio than for non-research-methods modules (70%), provide students with syllabi that include a week-by-week outline of the module contents (70%), or make syllabi publicly available (14%). Actively employing teaching and grading methods known to reduce “statistics anxiety” (75%) almost reached consensus.
Final-year projects
Consensus was reached that students should have the option to conduct a qualitative, quantitative, or mixed-methods project in their final-year research (85%). Consensus was not reached on whether students should preregister the quantitative aspects of their final-year project (22%), be allowed to perform a replication as their final-year research project (56%), or be allowed to conduct their final-year project in a team (46%).
Thematic analysis of the open-ended questions
Participants provided written responses to open-ended questions. These included feedback from 41 participants on 222 specific items, 30 participants for 82 suggested items, and 73 participants regarding 337 ratings that crossed a rating boundary (e.g., from important but not essential to essential). A small number of participants were responsible for large portions of these open-ended responses (see Table S1 in the Supplemental Material). Forty-five participants left a general comment after Round 1, and 21 participants left a general comment after Round 2.
The thematic analysis addressed the following research question: What are important issues to consider when designing the research-methods curriculum? The analysis generated two main themes, “important factors” and “constraining factors,” which encompass the key issues in the design of research-methods curricula. Figure 3 illustrates these themes and corresponding subthemes.
Thematic map with two main themes and their subthemes.
Theme 1: important factors
This theme delves into the various factors that were considered to be important to include in the research-methods-curriculum design. The transferability of skills to the workplace, ensuring students understand basic concepts and skills, and the value of qualitative research were prominent issues, and these subthemes are explored below.
Transferability of skills
We identified the transferability of skills to employment contexts as an important factor to be accounted for in the research-methods-curriculum design. For example, a participant said, “A balance is needed to future-proof student skills while focusing on practical understanding of research methods (at UG [undergraduate] level at least).” Another participant wrote, “There should be more consideration given to the . . . skills that might transfer to other fields, outside academia.”
Most graduates do not pursue an academic career after their psychology degree (Palmer et al., 2021). Thus, teaching research-methods skills that apply beyond academia would be important. This issue was highlighted by a participant: It would also be great to see research methods taught in a way that emphasizes real-world applications for the students. The vast majority of psychology undergraduates do not go into Psychological research so, while I appreciate the drive to make University level research methods training more consistent with the current state of psychological research, I am concerned that we might be missing the type of data handling skills that students would use in careers that are data focussed but outside of psychological research.
This issue was further exemplified by other participants, for example, “Most psychology undergraduates are not going to be researchers or undertake research in the future” and It is important for me, as a practicing researcher, to know how to calculate effect sizes, perform a power analysis, navigate publication etc. But is it important for me that students can do these things, given that most of them will not need to in the future?
Participants understood the significance of teaching research methods in ways that are applicable to nonacademic careers.
To this end, participants provided views about the inclusion of transferability skills when teaching research methods that could be applied to nonacademic employment settings, for example, “Teach what is used for research and industry” and “applicability to non-academic settings is important.” Regarding the questions asked in the survey, one participant wrote, I would have liked to see some questions about what research methods skills are important for students who graduate and don’t get employment in psychology, which is what most of our students do. I feel some of the issue[s] are not relevant to someone who is carrying out research in a non-academic context.
These responses illustrate the importance and value of designing a research-methods curriculum that actively includes research-methods training that can be leveraged in nonacademic employment routes. Nevertheless, the value of teaching research skills that apply both to academic and nonacademic pathways was another important factor for the curriculum. For example, a participant wrote, “Asking students to work as a team is more realistic to how research is conducted in the real world (whether that be academia or in the public or private sector).”
Furthermore, for psychology graduates intending to follow academic careers, our analysis suggests that it is also important to teach transferable skills that reflect what happens in real-world research, for example, “we falsely give students the idea research is a fairly solo/small homogenous (student) group activity whereas the truth in many fields related to psychology is that it needs multidisciplinary teams including methodologists and lay advisors.” For further illustration, in relation to the survey question “the publication process,” another participant wrote “very important when it comes to following an academic path.” In addition, in relation to the survey question “students should be allowed to perform a replication as their final year research project,” another participant stated, “This is more representative of actual psychology research, and in line with open science practices.” The comments demonstrate the importance of transferable skills taught in line with academic-employment routes.
However, there was some tension in which specific skills should be taught as important for transferability. For example, a participant stated “SPSS is still very important in government, business, and charities,” and another wrote, “SPSS is in declining use. Psychology departments do students a disservice by continuing to use software that is not used in other disciplines or in industry.” Despite the disparity in the specific transferable skills some participants suggested, it remains important that skills learned in the research-methods curriculum have applicability to employment contexts in both academic and nonacademic pathways.
Basic concepts and skills
We identified that ensuring students understand basic concepts and skills was another factor deemed to be essential in the research-methods-curriculum design. For example, participants stated, “I believe students should learn the basics of all research methods” and “We need to focus on students getting the foundational methods knowledge.”
There were concerns about the increasing number of topics and skills being taught in the research-methods curriculum at this level, to the detriment of students’ understanding and ability to apply these properly. To illustrate, a participant wrote, “There is quite a focus on teaching UG students more and more topics, and using potentially more technical platforms. I would also like to see emphasis on improving students’ understanding and ability to apply foundation concepts,” and another stated, “At the UG level what we need to be doing is covering LESS but with greater rigour and confirmation that students can actually USE and UNDERSTAND what we have taught them.” The responses demonstrate the importance of making sure that the research-methods curriculum includes space and time for the teaching of concepts and skills considered to be foundational and to avoid the overload of an ever-increasing number of topics and skills when basic understanding of concepts and skills seems to be lacking.
The negative impact of the increased taught concepts and skills is illustrated in the following participants’ statements: “I end up essentially doing the major parts of their [final year] project for them” and “We teach research methods in 1st year and 2nd year and they arrive at their 3rd year dissertations knowing practically nothing, despite often doing well on earlier exams.” Not ensuring that students comprehend basic research-methods concepts and skills in the earlier years appears to have a negative knock-on effect on their ability to understand and conduct their final-year projects.
Furthermore, the importance of ensuring a basic understanding of concepts and skills also applies when teaching students to use analysis software. For example, participants wrote, “For all of these [software skills] it’s necessary to explain why we do these tests, what they mean, give real world examples” and “For too long have psychology departments ‘taught SPSS’ instead of teaching statistical analysis.” It is important that students have foundational understandings so that when the time comes to use analysis software, they comprehend what they are doing and why. This issue is further exemplified by the following quote: “Should we address the problematic tendency to emphasise which stats package dialogue boxes to tick at the expense of teaching a deep understanding of what the analysis actually does?” This comment illustrates the importance of designing the research-methods curriculum in a way that ensures students have a solid understanding of basic research concepts and skills.
In addition, our analysis found that ensuring students develop the ability to think critically was another important foundational concept for developing their understanding and appraisal skills. For example, in this context, a participant stated, “I would like the field to be careful about ensuring that students are not simply directed towards a checklist approach,” and another wrote, “We ought to be teaching ways to understand, critique, appraise and undertake research.”
The value of ensuring students learn to evaluate research transcends the particular topic under critical review, as exemplified by some participants: “Not everyone will want to use qualitative research methods; but everyone should be able to evaluate published qualitative research”; “I think they need to be made aware of the limitations of quantitative research, which they tend to think of as sacrosanct”; “It’s essential for students to understand that this [the publication process] is not a neutral process and factors in the process create a biased literature”; and “I’d prefer them to know why replication is relevant and why it’s become a fixation as opposed to other issues.” These responses demonstrate the importance of ensuring students develop critical-thinking skills as foundational in all research-related contexts and integrated into the curriculum as such.
On the whole, the findings illustrate the importance of designing a research-methods curriculum that ensures students understand basic research concepts and skills, including in teaching analysis software and in the development of critical-thinking skills.
Value of qualitative methods
Our analysis identified the value of qualitative research and its inclusion in the research-methods curriculum as another important factor. Illustrating this finding, participants wrote, “Students need more qualitative methods input, this is crucial to ensuring the quality of qualitative research and of future training”; “Qualitative analysis is profoundly important”; and “I have rated most of these [survey items relating to qualitative research] as essential.”
In addition, it was recognized that the survey further problematized the perception that qualitative methods are not as valuable as quantitative methods, for example, “It is interesting that even in this study, a greater emphasis is placed on quantitative over qualitative skills (judged by the number of questions). I believe these should be given equal weight.”
Furthermore, our analysis suggests that qualitative research is currently viewed as having less importance than quantitative methods, but several participants wrote that it should have the same value status in the research-methods curriculum. For example, participants stated, “QUANTitative and QUALitative research methods should be given equal attention by staff and students”; “There is an imbalance here on the quant to qual methods - both are equally important and should be given equal importance in undergraduate study”; and “Qualitative research methodology should be given the same emphasis as quant so that students appreciate the relative value and appropriate applications.” These comments indicate that participants felt that qualitative methods should hold equal value to quantitative research in the curriculum and that this value should be held by research-methods instructors and students.
Another important factor that stems from recognizing the value of qualitative research is that some qualitative concepts are directly applicable to quantitative methods. This was particularly salient when participants were responding to the survey question, “Explain the philosophical underpinnings of qualitative research,” for which some stated, “And of quantitative research!”; “Students need to know about the theoretical and epistemological underpinnings of research full stop, not just qual”; and “Should do this in all research.”
Overall, qualitative methods are considered to be valuable and an important factor to include in the research-methods-curriculum design, alongside the view that they should hold equal status to quantitative research methods.
Theme 2: constraining factors
This theme unpacks several factors that were considered to be constraining in the design of research-methods curricula. Finite teaching capacity and student-capacity limits were significant issues. These subthemes are examined below.
Finite teaching capacity
Our analysis identified that the finite amount of time to teach the many research concepts and skills was considered to be a constraining factor. For example, in response to the survey items, participants stated, “The study doesn’t really take into consideration what is achievable in a given timeframe and whilst a lot of ideas and concepts are really important I am not sure how you would fit them all in”; “A lot of my observations of things being ‘important but not essential’ are really shaped by my understanding of the time constraints in our teaching”; and I feel like it’s a bit too easy to say “everything is really important!” (certainly the RM sample looked like almost everyone was saying “9” for loads of the items), when actually there’s a finite amount we can teach our students and expect them to learn in Y1 and Y2 of an UG degree.
Despite participants wanting to support the value of many of the survey items proposed, it seems responses also took into account the finite amount of capacity available to teach these concepts even though this was not specifically asked as part of the survey. This finding reflects the importance of taking this constraining factor into account when designing the research-methods curriculum.
Concerns regarding finite teaching capacity also extended to the teaching of particular research skills, such as mixed methods, for example, “Mixed methods is often not feasible within the timeframe allowed; it may also require careful staff allocation which could create lack of equality of supervision,” and to learning to use a programming language, for example, “This entails considerable staff time.” These responses illustrate the need for careful consideration as to what concepts and skills should be included in the research-methods curriculum given the finite amount of teaching capacity.
The impact of what can be taught in the given amount of time available for research methods can be a negative one, both on students and on instructors, as exemplified by the following participants. One wrote, “In my experience, most students have absolutely no idea of what they’re doing and have no capacity or time to be trained, either on their part or mine because I am so wildly overwhelmed,” and in response to the survey item, “Have a higher staff to student ratio than for non-research methods modules,” another participant wrote, “Definitely would be good, but hard to implement and also ties closely together with how many taught hours are involved (labs/lectures) and what that means for teaching loads.”
On the whole, the finite capacity for teaching research methods is a constraining factor on the concepts and skills that could be included in a research-methods curriculum, especially in view of the negative impact that overfilling the curriculum can have on students and instructors.
Student-capacity limits
Our analysis also found that restrictions in student capacity was a constraining factor regarding what is feasible to include in a research-methods curriculum. This finding is illustrated by a participant’s comment: “I think at the moment we’re trying to turn all our UG students into PhD level R users and researchers (too much!).”
Participants seemed to feel that there are limits to what students can learn within the degree time frame, for example, “As much as I use and like R, I’m not sure it’s reasonable to make all students learn to code” and “I don’t think it’s necessary for students to apply multiple types of research designs; there is only so much time in a degree. I think it’s more important to know about it.”
Furthermore, participants thought that certain research skills were too advanced for inclusion in the research-methods curriculum. Mixed methods were one such concept, for example, “I worry that mixed methods can be too complex/large scale to be achievable,” “Mixed methods reports are too complicated for final year research,” and “Mixed methods is less important than good qual and good quant and throws up some real issues for final year projects.”
Participants also considered learning to use a programming language to be too advanced for students: “Many students may be overwhelmed by learning a programming language (which may increase students leaving courses),” “This would put off and disadvantage many students,” and “Sadly, we struggle to get our students to get to grips with SPSS; when have tried to teach R, it has been even more difficult.” These responses indicate that designing a research-methods curriculum requires careful consideration regarding the concepts and skills that fit the capacity of students at this level.
Nevertheless, the option for students to specialize in more-advanced research concepts and skills was considered to be a potential way to overcome this constraining factor: “I’d like to see research methods training that starts more broadly and becomes more specific; offering students a choice of where to specialize.” In response to the survey item, “Use a programming language to manage and analyse data,” some participants stated, “We can certainly teach why this would help and encourage those who’re interested/able to explore it. . . . This should be an optional skill” and “This is important but not essential for all students. They should be given an opportunity to learn this as part of an elective module.”
Overall, student-capacity limits are also a constraining factor that should be considered when designing research-methods curricula. There are limits to what students can learn within the degree time frame, and some research concepts and skills were thought to be too advanced. Providing the option for students to learn more-advanced concepts and skills could be a possible solution.
Discussion
Summary of results
Consensus was reached for 34 items. These items spanned topics, including data skills, general research design, descriptive statistics, inferential statistics, critical assessment of research, and to some extent, qualitative methods, reproducibility, and open science. Consensus was not reached for 44 items. These items spanned advanced analysis techniques, specific research designs, approaches to research, computer skills, module formats, and final-year projects. A qualitative analysis of open-ended responses highlighted the importance of understanding basic concepts, valuing qualitative research methods, and learning transferable skills while also acknowledging limits on how much material can fit in an undergraduate program and how much students can absorb in this finite time period. Taken together, these results can provide valuable information for instructors, program directors, and organizations that develop accreditation standards.
Relation to the literature
The consensus results partially overlap with the content that appears on publicly available curricula for quantitative research methods in psychology programs (TARG Meta-Research Group, 2022). For example, items such as descriptive statistics, inferential statistics, and critical evaluation were prevalent across curricula and also rated highly in this Delphi study. On the contrary, items such as effect sizes, confidence intervals, data cleaning, practical significance, and replication were less prevalent in the curricula 13 but were highly rated in this Delphi study. Adding these topics to accreditation standards presents one mechanism to encourage their adoption. Almost all the curricula mention SPSS; however, learning to use a statistical analysis package with a graphical user interface did not reach consensus in our study. Our qualitative analysis further suggests that participants were concerned that students learn how to “point-and-click” in SPSS rather than gain an understanding of the analyses for which they are using the software. Comparing curricula with accreditation standards—and with the results of this Delphi study—can help understand whether the educational content of psychology programs aligns with community expectations.
Several qualitative items had lower ratings than quantitative items, reflecting a previously observed trend. For example, qualitative methods appear to be underrepresented in curricula and perceived as an alternative and “lesser” approach to quantitative methods in UK psychology programs (Gibson & Sullivan, 2018; Hugh-Jones et al., 2012). Interviews with psychology instructors also suggest that some programs would need additional instructor expertise to effectively teach and supervise qualitative research methods (Wiggins et al., 2016). Indeed, almost all participants in our study (96%) agreed that research-methods instructors should be given time and support to improve skills they plan to teach, including qualitative methods. Given the importance the BPS places on qualitative methods (e.g., the Qualitative Methods in Psychology Section of the BPS), the prominence of qualitative methods in the accreditation standards could be raised.
Transferability of research-methods skills emerged as a theme and appears relevant given the diverse career paths that psychology graduates follow. A recent report in which data from the Higher Education Statistics Authority were analyzed found that “there is no common career path for psychology graduates, as they go on to work in a broad array of roles and settings” (Palmer et al., 2021). The report further stated that only about 6% of graduates become registered professionals in psychology and that many go into roles in the health sector, retail, administration, public relations, marketing, and human resources. In these roles, a solid foundation in qualitative and quantitative skills likely trumps the ability to perform inferential statistical tests. Even in careers in which inferential statistics are necessary, such as academic research, foundational data skills are also necessary. For example, a recent article calling for UK psychology education to emphasize data skills demonstrated that to analyze a realistic quantitative data set in psychology, data wrangling (e.g., cleaning and structuring data) accounts for about 80% of the steps, and statistical procedure accounts for only 20% of the steps (McAleer et al., 2022).
Recommendations
As per our study objective, we provided the BPS Undergraduate Education Committee with nine core recommendations for updating their accreditation standards (see Supplementary Material I; also summarized in Box 2). We developed these recommendations by considering the combination of the consensus summary results and the thematic analysis. Although we would recommend that all 34 items that reached consensus be considered for inclusion in a research-methods curriculum, we binned items into nine recommendations that integrate the qualitative data and hopefully facilitate the implementation of the recommendations.
Recommendations for Research-Methods Education in Undergraduate Psychology Programs
Ratings were very high, and consensus was reached for data skills, basic research design, descriptive statistics, and inferential statistics. Almost 90% of participants rated it essential that students learn to use descriptive statistics effectively before learning to perform inferential statistics. Our qualitative analysis also highlighted the need for students to master foundational quantitative and qualitative skills rather than attempt to perform analyses that they understand poorly. These findings challenge the NHST-centric approach taken in many research-methods curricula and suggest that the psychology community should place importance on ensuring students develop a deeper understanding of the research skills they are using and why they are using them.
Open-ended comments raised the point that students should learn how to answer a research question and focus on fewer technical abilities. With this in mind, research-methods education could adopt a problem-solving approach by teaching students how to ask a clear question, design an effective research plan, identify what data are needed to answer their question, and how that data could be collected (e.g., the problem-plan-data-analysis-conclusion model, as suggested by Spiegelhalter, 2019).
Critical assessment is a pillar throughout the 2019 BPS accreditation standards and mentioned in almost all curricula assessed in a previous study (TARG Meta-Research Group, 2022), but specific concepts and tools are generally not outlined. Updated accreditation standards could include specific items, such as learning about replication and sources of bias, to create a more structured approach for critically assessing the psychology literature and other forms of information. This topic is linked to integrating the principles of open science into undergraduate education, which others have encouraged (e.g., Pennington, 2023; Pownall et al., 2023).
Forty-four items did not reach consensus. These items spanned topics including module format, final-year projects, computer skills, approaches to research, and advanced analysis techniques. Many of these items received a high level of agreement but fell short of consensus. There was not consensus against teaching these items.
Limitations
Our study design entails limitations on the claims we can make and how they can be interpreted. First, Delphi studies assess the opinions of a community. They do not establish what educational content is most effective. For example, some participants may simply provide low ratings for items they are unfamiliar with. In our results, for example, consensus was reached for learning about practical significance. However, related concepts that participants may be less familiar with received low ratings (e.g., alternative measures of effect sizes and smallest effect sizes of interest). We hope to have mitigated this limitation by providing the option “unable to rate,” which was used for 3% of ratings.
Second, the format and content of our Delphi was specifically designed in relation to the BPS accreditation standards. This meant that we selected items and phrased them in such a way that our results could be easily integrated into these standards. In this sense, we did not present items that challenged core components of the standards (e.g., the inclusion of a final-year project 14 ) or content that already exists in the standards that we believed participants would be unlikely to disagree with (e.g., “critical evaluation”). Many Delphi studies include an initial idea-generation round in which participants are asked to suggest items before they see or rate any item. Some Delphi studies also include an item-prioritization round, in which participants rank the items that reached consensus. Because of limited resources and time constraints, we did not include these rounds.
Third, we targeted four stakeholder groups but achieved a substantial number of participants in only two of these groups. These two groups overlapped substantially, given that most research-methods instructors are likely also academic psychologists, and their ratings were relatively similar. Very few students and nonacademic psychologists participated even though the BPS sent invitations to relevant mailing lists (e.g., Student Ambassadors, Psychological Professions Network). Thus, in Round 2, most participants saw the ratings from only a few students and nonacademic psychologists (which may have comprised an unrepresentative sample) and from one other group with similar responses to their own group. This combination of factors may have contributed to the limited changes to ratings between Round 1 and Round 2 15 and also resulted in a failure to capitalize on this strength of the Delphi technique. The steering committee also lacked a nonacademic psychologist and a very recent graduate, which may have resulted in the content and wording of Delphi items—and thus the results—being less applicable to nonacademic contexts and student perspectives. The lack of engagement from nonacademic psychologists and students may arise because of a combination of a lack of interest from these stakeholder groups and a further distance from the steering committee who ran this study.
Fourth, our sample was likely biased toward quantitative psychologists and people who are highly interested in or opinionated about research methods. A majority of the participants reported primarily using quantitative methods, and few reported primarily using qualitative methods. This distribution of participants—which may or may not reflect the distribution of the psychology community in the UK—could be responsible for the generally higher ratings for quantitative items compared with qualitative items. By design, our study also reflects only the views and priorities of the UK psychology community.
Fifth, the study was originally conceived to ask only about quantitative issues and was thus weighted toward quantitative methods. Qualitative methods were included upon the suggestion of BPS representatives. We did not include items asking about the proportion or ordering of teaching quantitative versus qualitative methods. Several participants provided feedback expressing concern regarding this quantitative-qualitative imbalance and stated that some qualitative items were poorly worded. This imbalance may have affected the distribution of ratings between the quantitative and qualitative items.
Sixth, participants may have overlooked conditional words that preceded some items. For example, one Delphi item asked if students should learn how to “calculate/perform significance tests,” and another asked if students should learn to “define and explain systematic reviews and meta-analysis.” This oversight could have lowered ratings for some items because participants may have thought that they were being asked if students needed to learn how to “perform” a meta-analysis, for example.
We were aware of the limitations of the modified-Delphi format we used before beginning the study and deemed them acceptable. The shortcomings of sampling bias and limited student and nonacademic engagement limit the generalizability of our results to the psychology community at large. Nonetheless, our data and findings provide a resource that can help inform accreditation bodies, program directors, and module instructors about what the UK academic community believes is essential for undergraduate psychology students to learn.
Our study also had several strengths. The Delphi method is a recommended approach to produce guidelines on topics for which data are scarce and expert opinion or community opinion is the best available evidence. By opening participation in the Delphi to the UK psychology community at large, identifying key stakeholder groups, and sending targeted invitation emails to those groups, we avoided sampling bias that would arise through other methods, such as selecting a panel of experts to complete the Delphi. Our mixed-methods approach also provides a robust understanding of participants’ opinions. We also received responses from more than 100 members of the UK psychology community from more than 50 UK universities, including 20 program directors. We also worked with representatives from the BPS to ensure that we selected items and worded them in a way that facilitates integration into updated accreditation standards. Finally, all the raw data, summary data sheets, and analysis code are publicly available for others to explore.
Conclusion
Our study provides data sets, both quantitative and qualitative, on the research-methods skills that UK-based instructors and academic psychologists deem essential for undergraduate psychology students to learn. Our findings suggest widespread agreement that research-methods education in undergraduate psychology should emphasize foundational skills in research design, data handling, statistics, qualitative methods, and critical assessment while providing students with transferable skills and not overloading them with advanced techniques. Organizations that create educational standards for psychology programs—such as the BPS—can draw on our findings to help develop broadly accepted and clear-cut expectations for research-methods education. Such initiatives could foster cohorts of graduates with an established set of competencies tuned for the contemporary world.
Supplemental Material
sj-docx-1-amp-10.1177_25152459231213808 – Supplemental material for A Delphi Study to Strengthen Research-Methods Training in Undergraduate Psychology Programs
Supplemental material, sj-docx-1-amp-10.1177_25152459231213808 for A Delphi Study to Strengthen Research-Methods Training in Undergraduate Psychology Programs by Robert T. Thibault, Deborah Bailey-Rodriguez, James E. Bartlett, Paul Blazey, Robin J. Green, Madeleine Pownall and Marcus R. Munafò in Advances in Methods and Practices in Psychological Science
Footnotes
Acknowledgements
We thank Patricia Lyons from the British Psychological Society for collaborating in the development of this project. We thank Charlotte Pennington and Patricia Lyons for feedback on drafts of the initial set of Delphi items. This publication is the work of the authors, and R. T. Thibault will serve as guarantor for the contents of this article. The funders have no role in the preparation of this article or the decision to publish.
We asked several organizations if they could advertise this Delphi study by sharing it with their contacts (e.g., UK Reproducibility Network, ReproducibiliTea, Framework for Open and Reproducible Research Training—FORRT). This is the only capacity in which these organizations were involved
Transparency
Action Editor: David A. Sbarra
Editor: David A. Sbarra
Author Contributions
ORCID IDs
Notes
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
