VIRT 2 UE: A European train-the-trainer programme for teaching research integrity

Abstract

Universities and other research institutions are increasingly providing additional training in research integrity to improve the quality and reliability of research. Various training courses have been developed, with diverse learning goals and content. Despite the importance of training that focuses on moral character and professional virtues, there remains a lack of training that adopts a virtue ethics approach. To address this, we, a European Commission-funded consortium, have designed a train-the-trainer programme for research integrity. The programme is based on (1) virtue ethics, (2) the ethos of science, (3) learning by doing and (4) learner-centred teaching. The blended learning programme combines e-learning modules with participatory group sessions. Trainers are taught how to guide researchers through a series of structured exercises for fostering reflection on scientific virtues, and how to promote understanding and application of the European Code of Conduct for Research Integrity. Trainers are provided with adaptable tools and resources that can be used and combined in different ways. The programme implementation began in Spring 2020 and 470 trainers have participated to date. When trainers were asked to grade – between 0 (very bad) and 10 (excellent) – the e-learning modules and the participatory exercises, 60% scored a grade 8 or higher (median = 8, IQR = 2) for the e-learning modules, whereas 80% scored a grade 8 or higher (median = 9, IQR = 1) for the participatory exercises. A majority felt that the training helped them as a trainer to learn about ways to organise and teach a research integrity course (82%) and would recommend the interactive exercises to others (92%). Trainers have educated over 3300 researchers in Europe using our virtue-based approach. The VIRT²UE train-the-trainer programme fosters research integrity by providing trainers with exercises and tools which enable them to stimulate the development of good researchers across Europe.

Keywords

Responsible conduct of research research integrity virtue ethics professional ethics good research practices responsible research and innovation higher education

Introduction

Concerns about the quality and trustworthiness of research have led universities to increasingly provide additional training to ensure their researchers practice research responsibly and with integrity. These courses go by different names – for example, ‘Responsible Conduct of Research’ (RCR) (Kalichman and Plemmons, 2007), ‘Good Scientific Practices’ (Fuerholzer et al., 2020) or ‘Research Integrity’ (RI) (Abdi et al., 2021). However, all aim to promote good practices and prevent questionable research behaviours. The impetus for their development has come from institutional reform following incidents of misconduct (Steneck and Bulger, 2007), normative guidance, such as the European Code of Conduct for Research Integrity (All European Academies (ALLEA), 2017), or in response to the funders’ conditions – for example, the requirement of the National Institutes for Health in the US that grant recipients receive RCR training (Resnik and Dinse, 2012). These courses, referred to hereinafter as RI courses, vary not only in name, but also learning goals, content and target audience (Abdi et al., 2021; Kalichman and Plemmons, 2007). In Europe, normative guidance requires institutions to provide RI courses; however, little direction is given in relation to the specific content of courses (ALLEA, 2017). Abdi et al. (2021) revealed, in a content analysis of European institutions’ RI courses, that courses mostly focus on understanding the principles guiding the scientific method and RI rules. Few included any content on researchers’ moral character and professional habits, such as why these aspects are important or how to strengthen them (Abdi et al., 2021). Pizzolato and Dierickx (2021) found that those involved in research in Europe do, however, consider the development of researchers’ moral behaviour, character, and professional virtues as appropriate goals of RI training. This goal has been neglected in formal RI training, with little attention given to what we consider to be a ‘good researcher’ or what acting with integrity means in specific situations. If we consider acting with ‘integrity’ to reflect desirable professional habits and embodied values or virtues such as trustworthiness, honesty and intellectual humility, then a challenge for RI training is how to foster the development of these virtues.

There are some advantages to using a virtue ethics approach in RI teaching. Here, we are particularly guided by Aristotelian ethics (Aristotle, NE) as well as MacIntyre’s moral philosophy, which is based on Aristotle’s ethics and specifically relates virtues to practice (MacIntyre, 1981). First, virtue ethics focuses on the development of the moral character of the person. According to Aristotle, ethics requires practical wisdom, which is the result of training and exercise, and learning how to deal with the intricacies of moral practice (Aristotle, NE; Ross, 2013). Thus, a virtue ethics approach goes beyond focusing on rules and compliance, and fosters learning how to apply rules in specific situations, taking into account the complexities and dilemmas of real-life research (Mulhearn et al., 2017). Second, a virtue approach helps to develop researchers’ moral sensitivity, teaching them to reflect on the goals and responsibilities of the profession and the researcher qualities that support them (Chen, 2016; Han, 2015; Oakley and Cocking, 2001; Pennock and O’Rourke, 2017). Third, a virtue approach might actually encourage researchers to embody these virtues; by explicitly linking scientific virtues to both the goods that are internal to the practice (such as proficiency) (MacIntyre, 1981) and the ends of the profession (in this case the advancement of knowledge), researchers are motivated to develop these qualities for themselves (Chen, 2016). Fourth, a virtue approach can contribute to a common ethos of science. An ethos refers to the moral standards which are recognised and practised as excellent within a group or society (Lanzerath, 2015; MacIntyre, 1981,). The wider recognition and internalisation of scientific virtues, and awareness of how these relate to the principles and practices of the European Code of Conduct for Research Integrity (ALLEA, 2017), can help foster a shared understanding and commitment to excellent research practices within the European research area. Moreover, a shared ethos of science reflects both the individual virtues of researchers, as well as an institutional culture committed to fostering virtues and creating conditions for good scientific practice, which include adequate training opportunities (Mejlgaard et al., 2020; Moore and Beadle, 2006,).

Although there have been a number of calls for RI training to take a virtue ethics approach (Chen, 2016; Han, 2015; Oakley and Cocking, 2001; Pennock and O’Rourke, 2017), and some initiatives have been developed in the United States (Berling et al., 2019; Pennock, 2018; Pennock and O’Rourke, 2017), such an approach has been distinctly lacking in Europe. In recognition of this, the European Commission developed a funding call for a RI Train-the-Trainer (TtT) programme which would take a virtue ethics approach. The TtT aspect was aimed at building trainer capacity across the continent and reaching as many researchers as possible. In response to this call, we, the VIRT²UE consortium, set out to design a RI course, the characteristics of which are influenced by virtue ethics, relevant pedagogical approaches, and the evidence base for RI education (Evans et al., 2021). An important consideration in designing a European RI programme is that real experiences and concerns related to RI might differ between disciplines and countries. The programme has, therefore, also been developed to allow trainers to adapt their materials and teaching to their target audience and to situational factors.

The aim of this article is to present the VIRT²UE TtT programme. In the following sections, we describe the theoretical and conceptual frameworks guiding the programme, the development process and the course design. We also reflect on the implementation of the programme in practice and provide some first evaluation results.

Theoretical and conceptual background

Virtue ethics

Virtue ethics focuses on developing moral character. A virtuous researcher requires both an awareness of general rules and principles, and an understanding of how to apply them in practical situations. The distinction, therefore, between a ‘rule-based’ approach to teaching RI, and a ‘virtue-based’ approach, is somewhat artificial. In reality, these approaches are complementary (Pellegrino, 1989; Resnik, 2012). Compared to a strictly rule-based approach, however, a virtue-based approach also focuses on participants’ moral development related to upholding scientific values and virtues in ethically complex situations.

Strategies suggested for fostering the development of virtues specifically through teaching (as opposed to role modelling or through supportive environments) include direct instruction on core virtue concepts and terms (Baehr, 2013; Berkowitz and Bier, 2007,) and the application of knowledge through reflection on one’s own character virtues, particularly in relation to specific morally ambiguous situations (Baehr, 2013). The VIRT²UE TtT programme therefore incorporates both direct instruction and reflections related to specific situations. Direct instruction on core virtue concepts and terms and their relevance for RI are especially important for an RI TtT programme, as trainers should understand and be able to communicate the importance of virtues for research practice to the participants of their own training and support participants to reflect on professional virtues in relation to specific dilemmas. Following an Aristotelian approach to moral learning, practical wisdom is developed through experiences, so these reflections should focus on concrete personal experiences (Aristotle, NE; Stolper et al., 2015).

Ethos of science

Scientific methods and disciplines are developed by a community of researchers, reflecting the values and virtues that further the ends of their practice. These can, over time, become characterised as an ‘ethos’ – a set of values and norms which are accepted as fundamental by the community of practitioners (Merton, 1973 [1942]). The ethos of science is not, therefore, an external set of rules imposed on science, but inherent to it. The ethos of science not only safeguards the principles for the generation of knowledge within the scientific community, but also strengthens societal trust in research. The European Code of Conduct for Research Integrity is an initiative to make this ethos explicit by outlining the principles of good scientific conduct (reliability, honesty, respect and accountability), and the practices which follow from these principles (ALLEA, 2017). European trainers should, therefore, be able to make the connection between the European Code of Conduct for Research Integrity (ALLEA, 2017) and the ethics of the research profession, in order to foster an ethos of science in their students.

Learning by doing

The pedagogical approach of ‘learning by doing’ posits that knowledge is best developed and consolidated through actions and reflection (Widdershoven and Molewijk, 2010) and has its roots in an Aristotelian approach to moral learning (Stolper et al., 2015). ‘Learning by doing’ is an experience-based approach that emphasises the role of practical experiences in learning processes and the development of professional knowledge. Through reflection on experiences in practice, professionals develop a ‘practical-knowing good’ (meaning that a professional knows how to act with integrity taking into account the specific context) and professional artistry, rather than solely a ‘technical-cognitive-knowing good’ (meaning one can reproduce rules and principles without taking into account the specific context) derived from theories or concepts (Schön, 1987).

‘Learning by doing’ is particularly important for moral education, due to the importance of social relationships and processes for the internalisation of moral training and the development of moral habits (Dewey, 1897; Inguaggiato et al., 2019). The ‘learning by doing’ approach is reflected at different levels of the VIRT²UE TtT programme. Trainers first experience the programme as participants, before learning how to facilitate the programme themselves. The interactive and participatory core elements of the programme (consisting of e-learning modules and participatory exercises) also reflect a ‘learning by doing’ approach. For example, the programme contains participatory exercises which are dialogical in nature, often use personal RI experiences and dilemmas as a source of reflection, and enable participants to develop knowledge through group-learning processes (Dewey, 1897; Stolper et al., 2015).

Learner-centred teaching

A learner-centred approach to teaching prioritises the learner’s needs, motivations and expectations; the trainer’s role is defined in terms of supporting the learner’s own unique learning process (Biggs and Tang, 2007). A learner-centred approach requires the formulation of learner-centred outcomes covering a range of cognitive levels and activities to engage all learners in developing their knowledge and skills. Learner outcomes, teaching activities and assessments need to reflect each other in ‘constructive alignment’. The learner-centred approach is reflected in the VIRT²UE TtT programme’s learning goals, adaptable modular materials (which can be selected depending on the needs of the target group), and recommendation for trainers to apply a learner-centred approach in their own teaching. This attention to the needs of the learner also allows for easy adaptation of the programme to different contexts (e.g. participant group, discipline and country setting).

Reflecting evidence on effectiveness

Research integrity is a relatively new field, and evidence on the most effective teaching methods, and indeed what ‘effective’ means (e.g. acquisition of knowledge and skills, modification of attitudes/perceptions, or behavioural change) is still developing. There is, however, some evidence about which methods best improve a range of learning outcomes and the VIRT²UE TtT programme incorporates these. A blended learning approach, for instance, has been shown in a systematic review to be more effective for improving students’ knowledge and critical thinking related learning outcomes than either online or in-person teaching approaches alone (Todd et al., 2017). Furthermore, a recent systematic review revealed ‘experiential learning’ to be the strongest predictor of course effectiveness in relation to the course learning outcomes (Katsarov et al., 2022). Experiential learning was described as ‘engaging learners in imagining how they would deal with a situation personally, for example, through role play, or through a discussion of the emotional dimensions of ethical decision-making’. Experiential learning goes beyond theoretical learning or deliberative case-based learning (which focuses on moral reasoning in relation to an appropriate course of action), because ‘experiential’ learning not only departs from the concrete experience of the student/trainee, but also involves a collaborative and affective dimension (Katsarov et al., 2022). Experiential learning is reflected in the ‘virtue as a practice’ and ‘learning by doing’ approach of the VIRT²UE programme.

Development process

Evidence mapping

The first stage of the development process consisted of evidence mapping of stakeholders’ preferences and experiences in relation to RI training, and existing openly available RI training courses and materials. European research stakeholders were consulted about their RI training preferences and experiences (Pizzolato and Dierickx, 2021), as well as their perspectives about the importance of cultivating virtues for research practice (Tomić et al., 2022a). During a series of focus groups on RI education, stakeholders expressed a preference for training that explicitly aims to foster researchers’ moral character and professional virtues (Pizzolato and Dierickx, 2021). Additional focus groups explored the meaning of virtues in scientific practice (Tomić et al., 2022a), and a subsequent Delphi survey (Tomić et al., 2022b) provided a list of scientific virtues prioritised by European stakeholders.

The programme development took into account existing RI educational materials by collecting, reviewing and categorising over 200 openly available educational resources (Pizzolato et al., 2020). Although some of the existing materials mentioned scientific values and virtues, none of the open materials were specifically designed to foster virtues in RI training. To develop a strong and consistent narrative, most of the educational materials used in the VIRT²UE programme were therefore newly developed, although some existing materials were integrated into the programme (e.g. an introductory video on virtue ethics was integrated into the e-learning modules, and the Rotterdam Dilemma Game (Erasmus University Rotterdam, 2022) was adapted for the participatory exercises).

Blended learning programme

To develop the blended learning programme, working groups were formed within the consortium: (i) to develop e-learning modules for individual learning and reflection on the main concepts and principles in RI and virtue ethics; and (ii) to develop participatory exercises to encourage group reflection on dilemmas in research practice and virtues needed to deal with them.

(i) E-learning modules

The working group for the development of the e-learning modules was composed of experts in RI and virtue ethics. The aim of the e-learning modules was to: introduce the main concepts and principles in RI and virtue ethics; provide an introduction to The European Code of Conduct for Research Integrity (ALLEA, 2017); and facilitate individual learning and reflection. The foci and content of the e-learning modules were iteratively developed by the working group, incorporating feedback from the first people trained in the VIRT²UE TtT programme.

The e-learning modules aim to consolidate trainers’ knowledge of the core concepts in RI and virtue ethics (e.g. virtues in research) and specific RI guidance (e.g. the European Code of Conduct for Research Integrity (ALLEA, 2017)). Following Wittrock’s (1992) model of generative learning, learners are invited to align new concepts and guidance with their prior knowledge and experience. Each e-learning module contains at least one open reflection exercise designed to prompt learners’ active engagement with new concepts. This includes asking learners to critically reflect on the relevance of new concepts for their daily research practice, as well as actively contextualising (e.g. by describing the importance for a specific academic discipline) and localising (e.g. by drawing upon national/local rules and guidelines) the information provided in the e-learning modules.

The e-learning modules’ design reflects multimedia learning theory and instructional design principles (Mayer, 2002). For example, to facilitate meaningful learning, cognitive load is minimised (without compromising the instructional message) through the judicious use of different presentation modes (e.g. words, pictures, animation) and sensory modalities (e.g. visual and auditory input) (Mayer, 2002). Narration is also used with animation (cf. modality principle) and only limited on-screen text (cf. redundancy principle). Moreover, the narration has an informal and conversational style (cf. personalisation principle).

(ii) Participatory exercises

The working group for the participatory exercises was composed of experts experienced in developing teaching materials to stimulate ethical reflection. Based on the programme’s theoretical and conceptual background, a number of conditions guided the exercise development process, namely that: (i) the exercises should enable reflection on dilemmas and the virtues needed to deal with them, particularly in the context of specific ethically challenging situations; (ii) the exercises should link to the principles and virtues of the European Code of Conduct (ALLEA, 2017); (iii) the exercises should foster the development of moral knowledge and character by focusing on participants’ personal character virtues; and (iv) there should be value in using the exercises individually, but added value in combining them. Based on these conditions, and the working group’s knowledge of different exercises for group reflection, a series of five exercises were iteratively developed. The exercises provide a structure for reflecting on both hypothetical and personal cases, and the application of knowledge gained from the e-learning modules. By reflecting on specific cases, participants are personally involved, as they are invited to consider the perspective of the researchers in the case. Additionally, trainers are given the chance to experience, reflect upon, and learn about didactical skills which are required to facilitate the participatory exercises in their own work context, such as the ability to recognise and foster a dialogue among the participants of their own RI courses.

One exercise was developed entirely for the purpose of the training (debate and dialogue), whereas the others were based on existing exercises (Erasmus University Rotterdam, 2022; Molewijk et al., 2011; Solbakk, 2015; Stolper et al., 2016,) which were adapted and developed to have a specific RI focus. The development process consisted of a series of workshops and pilots with consortium members and external researchers and experts. After each pilot, feedback was used to improve the exercises and clarify their relevance for the programme as a whole. The development phase culminated with the organisation of a pilot training where VIRT²UE consortium members were trained.

Pilot training

Forty consortium members attended the pilot training. In the pilot training, consortium members were trained by the working group responsible for developing the participatory exercises. Consortium members experienced, facilitated and provided feedback on the exercises both directly after facilitating the exercises verbally, and later in written self-reflection forms. Consortium members were asked to provide feedback particularly on the content and structure of the exercises, the understandability of the written instructions for the exercises, the relevance of the exercises for the programme and their complementarity with the e-learning modules. This feedback was used to refine the instructions for the exercises before the consortium members themselves rolled out the training for every country in Europe.

Programme design

The VIRT²UE TtT programme provides participants with the knowledge, skills, and resources to conduct an RI course from a virtue ethics perspective. The programme follows a blended learning approach, combining online resources with structured participatory exercises. Trainers are taught how to guide researchers through a series of structured dialogical exercises for fostering reflection on scientific virtues, while promoting understanding of the European Code of Conduct for Research Integrity, which is the primary guidance document on RI for research institutions in Europe (ALLEA, 2017). Trainers experience the programme as participants and learn how to deliver the content themselves.

As a TtT programme, VIRT2UE supports trainers and provides them with adaptable resources that can be used and combined in different ways in their own teaching.

Learning objectives

The learning objectives of the VIRT2UE TtT training are to:

Identify and apply the core principles and recommended good practices of the European Code of Conduct for Research Integrity (ALLEA, 2017);

Understand core virtue ethics concepts and terms, and relate virtue ethics to RI;

Facilitate case-based and experiential exercises aimed at fostering reflection on virtues;

Critically reflect on experiences of teaching; and

Adapt teaching approaches based on the target group characteristics.

Participants

The programme is designed for trainers who are able to reflect on experiences of research practice. The programme is therefore primarily targeted at researchers from any discipline and from mid- to senior-career stage, but other stakeholders close to practice, such as research managers or RI officers, are also appropriate potential trainers. Participants do not, therefore, need to have prior training in ethics or philosophy because they learn all the ethics and RI content they need to conduct the programme during the training. Furthermore, trainers can train students from any discipline or career stage because the programme materials are not discipline or career stage specific.

Educational strategy

The programme follows a blended learning educational strategy combining online self-study with participatory sessions. The online and participatory elements are complementary. Online preparation is required for the participatory sessions, and the online learning is consolidated through experiences from the participatory sessions. All materials for the programme are available openly on The Embassy of Good Science (www.embassy.science) in a VIRT²UE Training Guide (https://embassy.science/wiki/Guide:Bbe860a3-56a9-45f7-b787-031689729e52). The Training Guide is additionally available on the Open Science Framework website (see https://osf.io/hg7qc and https://osf.io/vkptx). Trainers are provided with e-learning modules for self-study and instructions of the participatory exercises, including short videos providing an impression of what each exercise looks like in practice. Didactic and practical instructions on how to conduct the participatory exercises (for trainers) and what to do to prepare for the exercise (for participants of the exercise) are provided. In addition, some preparatory reading and viewing on virtues, values, moral dilemmas, and the benefits and characteristics of dialogue is required before the participatory sessions. All preparatory materials are also available on The Embassy of Good Science (www.embassy.science).

Online self-study

The e-learning modules are designed in a way so that they serve as a preparation for the VIRT²UE TtT programme, but they can also be used as standalone material on RI, and the importance of virtue ethics to RI, for any interested researcher, RI trainer, or student from any discipline. The e-learning modules are ‘interactive’ in the sense that they include opportunities for self-reflection and quiz-type exercises on core topics, however, they are undertaken individually. The mini-series, consisting of four e-learning modules each, are:

The Introduction to Research Integrity series, which introduces the concept of RI and describes a range of research practices, from responsible to questionable through to those constituting misconduct. The modules examine the responsibility of the individual researcher, as well as the influence of research culture and the wider scientific system. Principles and recommendations from the European Code of Conduct on Research Integrity (ALLEA, 2017) are referred to throughout.

The Introduction of Virtue Ethics to Research Integrity series, which introduces the relevance of virtue ethics to RI. These e-learning modules provide direct instruction on core virtue concepts and terms.

The Virtue Ethics under Current Research Conditions series, which addresses more systemic issues, like performance pressures in research, and relates these to virtue ethics and the individual experience of the researcher. These modules also pre-emptively address a possible criticism of the virtue ethics approach – that a focus on virtue ethics might be perceived as putting all the responsibility for research culture and conduct on researchers, rather than addressing systemic issues such as pressure to publish or perverse incentives inherent in research (for a discussion of systemic issues that may undermine the integrity of research, see e.g. Bouter, 2015; Lindner et al., 2018).

Group sessions

Trainers attend (online or face-to-face) group sessions in which they learn about, experience, and gain didactic skills needed to facilitate five participatory exercises. The participatory exercises aim to foster reflection on what it means to be a ‘good’ researcher, and provide the format for structured dialogues. These exercises are:

The Debate and Dialogue exercise. This exercise involves identifying and experiencing the features of, and differences between, debate and dialogue. Participants learn the importance of dialogue for group learning processes, and for moral reasoning and ethics education in particular (Stolper et al., 2016; Widdershoven and Molewijk, 2010).

The Self-Declaration Approach. This exercise fosters reflection on different conceptions of ‘good’ and how these relate to research practice (Solbakk, 2015; Von Wright, 1963). The exercise links virtuous behaviour to ‘good’ professional functioning and helps participants become sensitive to what it means to do ‘good’ research and to be a ‘good’ researcher. In preparation for this exercise, trainers write down (‘declare’) situations in which they experienced moral questions concerning RI, which are then connected to the types of goodness during the exercise.

The Modified Dilemma Game. This exercise allows participants to identify principles, virtues and misconduct, and provides a framework to debate possible courses of action, in relation to specific hypothetical cases reflecting diverse disciplines and research contexts. The exercise provides an opportunity for participants to talk about RI issues without having to talk about personal cases (Erasmus University Rotterdam, 2022).

The Virtues and Norms exercise. This exercise allows participants to relate virtues to norms of action in relation to real-life RI related cases (Stolper et al., 2016). Participants reflect on personal RI cases and learn through dialogue by sharing different personal perspectives on the case.

The Middle Position exercise. This exercise develops moral sensitivity and awareness of the moral nuances related to acting in accordance with specific virtues (Molewijk et al., 2011). The exercise enables participants to understand the practical meaning of abstract virtues. In particular, it fosters reflection on how difficult it is to determine the correct course of action from a virtue alone, for example, what does it mean to be honest? Could one be too honest, not honest enough, or honest in the wrong way?

When experienced together, the exercises complement each other. Participants are introduced to, and experience the relevance of, core programme concepts (e.g. in the debate and dialogue exercise), learn how to reflect on dilemmas (e.g. in the modified dilemma game), reflect on the goals of the profession and the qualities that support these goals (e.g. in the self-declaration approach), and identify salient virtues and the actions which would follow from them in relation to dilemmas experienced by the group (e.g. in the virtues and norms and middle position exercises). Together they target different aspects of moral character development, but they can also be used individually depending on the needs of the trainer.

Following the ‘learning by doing’ pedagogical approach, the trainers are expected to practice facilitating the participatory exercises in their own institutional setting during the course of the programme. Because – with the exception of the dilemma game – the exercises take the perspectives and experiences/cases of the student group as the basis for discussion, the trainers are able to teach students from any discipline using content tailored to the RI challenges of the discipline and setting. The trainers facilitate small groups of students, between 7 and 12 people, to ensure that everyone in the group has the possibility to fully participate in the structured exercises.

After practicing, trainers complete a ‘self-reflection form’ on their experiences of facilitating, which subsequently provides material for the group reflection on teaching the exercises. In total, the TtT course is estimated to take 60 hours to complete (Table 1).

Table 1.

Overview of the course structure and duration.

	Content	Tasks for trainers	Time investment
e-learning modules (self-study)	Three mini-series, each composed of four episodes.	Completing the e-learning modules	4 hours
Participatory sessions	1. Introduction to the five participatory exercises and their learning goals. Discussion of the relevance of each exercise within the programme and their use in specific contexts.	• Completing the preparatory assignments.	Preparatory assignments: 5 hours Participation in the sessions: 16 hours
	2. Experiencing the exercises.
	3. Gaining the didactical skills needed to facilitate the exercises
Interim practice work	Practicing the exercises in own institution/context.	• Preparing, organising and practicing the five exercises.	5 hours per exercise, including completing the feedback form = 25 hours
Interim practice work	Practicing the exercises in own institution/context.	• Filling out a reflection form for each exercise.
Participatory session	1. Reflecting on and discussing experiences of practicing the exercises.	Preparing selected exercises	Preparation: 2 hours Participation in the session: 8 hours
	2. Practicing selected exercises within the group.
	3. Group discussion on the implementation of the course in the trainer’s own setting.
Total			60 hours

Implementation

The programme roll-out began across Europe in mid-2020. Different consortium partners were responsible for implementing the training in specific country groups. In the original design of the programme, the participatory sessions were envisaged to be delivered in-person over 3 days, consisting of an initial 2 days to learn about and experience the exercises, and a final day about 1 month later to reflect on experiences of practicing the exercises in the participants’ own settings. The first trainings followed this format. However, the Covid-19 pandemic in Europe prevented in-person sessions and subsequent participatory sessions had to be delivered via video conferencing software. Delivery via video conferencing software required the use of online collaboration tools, such as Jam board and Miro, and extra attention for participant engagement and group processes in the online environment. Trainers also required extra support and materials from the consortium for online delivery, such as template Jam boards and advice regarding online teaching. The online delivery via video conference also precipitated a change in the session structure; to prevent long sessions on the computer, trainers typically arranged shorter participatory sessions over 3–5 days, with breaks in-between to practice individual exercises. The arrangements differed slightly depending on the consortium partner responsible for the training and the needs and preferences of their trainers. For example, in the UK the training involved five online meetings (between 3 and 3.5 hours long) over a 20-week period, whereas in France and Belgium the training involved three online meetings which were 3.5 hours long over a 15-week period.

Evaluation and iterative development

To date, the VIRT²UE TtT programme has been delivered, across Europe, to 470 trainers from over 30 countries. Because the training was paid for by European Commission funding, there were no costs to participants. To evaluate the training, trainers were asked to complete two process evaluation questionnaires, one on the e-learning modules, the other on the participatory exercises. The evaluation and development of the training followed an iterative approach, including a cycle of evaluation and training refinement. This process began during the initial pilot phase of the programme implementation, with feedback from members of the consortium used to adjust and refine various elements of the programme. During the programme roll out, trainers’ evaluation responses were also regularly analysed and findings were fed back to the developers of the e-learning modules and the participatory exercises to improve the programme and its implementation. For example, the e-learning modules were initially three long modules, but trainer feedback about the length and the at times high academic level of the modules caused the developers to split them into series containing shorter modules of increasing difficulty, which can be more easily selected and integrated into teaching.

The questionnaires on the e-learning modules and the participatory exercises contained 35 and 50 items respectively. Trainers were asked to respond to questions about their satisfaction with the VIRT²UE programme, rating the programme as a whole, and specific elements of it. Furthermore, trainers were given the opportunity to describe what they thought of the programme in open questions. The questionnaire on the e-learning modules contained: 25 closed questions, 5 open questions, one grading (1 [very bad] – 10 [excellent]), and 4 questions that evaluated time spent on activities. The questionnaire on the participatory exercises contained: 39 closed questions, 7 open questions, one grading (1 [very bad] – 10 [excellent]), and 3 questions that evaluated time spent on activities. The questionnaires were developed in collaboration with the teams that created the VIRT²UE training and the developers of the platform and are available on OSF (https://osf.io/zk8ae). The evaluation was hosted on Typeform (www.typeform.com, Barcelona, Spain), a web-based platform. No personal information was collected from users. The online database was downloaded every 2 weeks and stored physically offline, and online on local servers of Amsterdam UMC. Data was then deleted from Typeform. Data is saved by Amsterdam UMC for 15 years. Below we present just a few of the items most indicative of overall satisfaction with the programme (full results will be published in follow-up articles).

Evaluation of the e-learning modules

The e-learning modules were evaluated from 3 November 2020 until 13 September 2021. About 130 people participated in the evaluation; however, one person did not check the consent box and was removed, leaving 129 entries. The response rate was 27% (Table 2). More than 60% of participants evaluated the participatory exercises with a grade 8 or higher (median grade 8, interquartile range [IQR] =7–9). The majority of participants felt confident to use the modules in their own teaching and would recommend the modules to others (Table 3).

Table 2.

Response rate of the evaluation survey on the e-learning modules and the participatory exercises (with breakdown by country group for the latter).

	Participants	Responses to evaluation (n)	Response rate (%)
	n	n	%
E-learning modules	470	129	27
Participatory exercises	470	116	25
African countries*	7	4	57
Austria & Switzerland	12	3	25
Belgium	17	4	24
Croatia, Slovenia, Slovakia, Czech Republic & Hungary	37	3	8
Finland & Estonia	21	5	24
France & Luxembourg	30	8	27
Germany	32	9	28
Greece, Bulgaria, Romania, and Cyprus	18	0	0
Italy & Malta	86	24	28
Latvia, Lithuania & Poland	28	11	39
Netherlands	23	5	22
Norway, Sweden, Denmark & Iceland	18	2	11
Spain & Portugal	43	18	42
Turkey	25	12	48
United Kingdom & Ireland	54	8	15

A number of participants from African countries (Ethiopia, Kenya, Nigeria, and South Africa) joined courses that accepted participants from outside Europe. There was, at times, a limited number of participants from these countries in the training and, because knowledge of the country of residence might be identifying, the evaluation survey only asked these participants if they were from African countries.

Table 3.

Willingness of participants to recommend the e-learning modules and the participatory exercises.

	Completely disagree	Disagree	Neutral	Agree	Completely agree
	n (%)	n (%)	n (%)	n (%)	n (%)
E-learning modules, total n = 129
I feel confident that I can use these e-learning modules in my own future teaching.	5 (3.9)	11 (8.5)	30 (23.3)	45 (34.9)	38 (29.5)
I would recommend the e-learning modules to others.	2 (1.6)	9 (7.0)	22 (17.1)	47 (36.4)	49 (38.0)
Participatory exercises, total n = 116
The training helped me as a trainer to learn about ways to organise and teach an RI course.	3 (2.6)	8 (6.9)	10 (8.6)	37 (31.9)	58 (50.0)
I would recommend the participatory exercises to others.	1 (0.9)	3 (2.6)	5 (4.3)	27 (23.3)	80 (69.0)

Evaluation of the participatory exercise

The evaluation of the participatory exercises lasted from 20 January 2021 until 13 September 2021. 117 people participated; however, one person did not check the consent box and was removed, leaving 116 entries. The overall response rate was 25%. See Table 2 for response rates for separate country groups. Most participants experienced the participatory exercises online (online: n = 95 (81.9%); face-to-face: n = 18 (15.5%), mixed: n = 3 (2.6%)). More than 80% of participants evaluated the participatory exercises with a grade 8 or higher (median 9, IQR = 8–9). The majority of participants felt that the training helped them as a trainer to learn about ways to organise and teach an RI course and would recommend the exercises to others (Table 3).

Discussion

The quality and trustworthiness of research stems from researchers’ professional practices and is a concern of professional ethics. Virtue ethics can guide professionals to recognise how to ensure that their actions contribute to the ends to which their profession is dedicated – in this case the advancement of trustworthy knowledge (Oakley and Cocking, 2001; Pellegrino, 1989,). In this article, we have described the theory, development, design and evaluation of the VIRT²UE TtT programme for teaching RI from a virtue ethics perspective.

The theoretical and conceptual frameworks guiding the programme development – virtue ethics, the ethos of science, learning by doing, and learner-centred teaching – are reflected throughout the programme design. The roll out of the programme has enabled a standardised training for 470 trainers across Europe (and a few participants outside of Europe) – who in turn have trained over 3300 researchers – and aims to provide them with the knowledge and skills to be able to, if necessary, adapt their teaching to the specific requirements of their participants. Initial process evaluations of the programme demonstrate high levels of satisfaction amongst the participating trainers.

A number of challenges were however encountered in the development and evaluation of the programme which deserve attention. As with any programme, which aims to cultivate virtues, the assessment of the goals of the training is difficult. In the VIRT²UE programme, trainers are assessed on their active participation in the group processes and the completion of individual reflections as a form of formative assessment, which reflects the importance of learning by doing. The trainer’s ability to foster scientific virtues in others and the development of researchers’ scientific virtues is not (yet) directly assessed. This challenge in assessment of the development of character also proves challenging for the evaluation of the programme as a whole. To date, a process – rather than an outcome – evaluation has been conducted. Whilst the results are promising, the impact of the programme on researchers’ virtues and research practice has yet to be evaluated. The next steps in the programme evaluation include qualitative evaluation of the experiences of the trainers and the researchers they train, as well as the development of an outcome measure appropriate for the content and didactics for this virtue ethics RI training.

A number of authors have previously called for a virtue ethics approach to RI teaching (Chen, 2016; Han, 2015; Pennock and O’Rourke, 2017). The most influential author for using a virtue ethics approach in RI teaching is probably Pennock (2018), whose theoretical work on scientific virtues has informed the ‘Scientific Virtues Toolbox’ (SVT) workshop initiative in the United States (Berling et al., 2019). Although based on similar premises, the design of the SVT workshops, involving participants scoring statements on a Likert scale and then using these in a structured, facilitated discussion, is quite different to the VIRT²UE approach. SVT discussions focus on one preselected scientific virtue (e.g. curiosity, honesty, courage, humility to evidence) and the facilitator, described as a philosophical guide or ‘Socrates in the room’, plays a particularly important role. In our programme, the virtues to be discussed are not preselected, but identified by the participants as those that would guide them in relation to both hypothetical and real-life dilemmas. Also, the participatory exercises follow clearly structured steps, which means that the group reflection process can be facilitated, with some training, by researchers, allowing rapid implementation at scale through the TtT programme. Despite the differences between our and Pennock’s approaches, both prove popular with participants. Participants of the SVT workshop reported high levels of satisfaction with the virtue approach, and preferred it to traditional compliance style RCR training (Berling et al., 2019; McLeskey et al., 2020).

Future directions for the VIRT²UE programme include the development of additional e-learning modules – recent additions include modules on supervision, role modelling and mentoring. We will also enable trainers across Europe, and even globally, to share experiences and materials, and stimulate them to adapt tools and materials. Certified trainers are invited to a closed ‘community’ section on The Embassy of Good Science platform to continue to network and keep the initiative alive (www.embassy.science).

Conclusions

The VIRT²UE TtT programme aims to provide RI trainers with the knowledge and skills to stimulate researchers to not only reflect on the relevance of virtues for RI research practice, but also to act in accordance with RI virtues in concrete situations. This helps develop their moral character and moral sensitivity for RI, whilst at the same time fostering an ethos of science as represented by the European Code of Conduct for Research Integrity (ALLEA, 2017). It combines e-learning modules and participatory exercises, focusing on dialogue about experiences in specific morally ambiguous research situations. The programme is well-structured and supported by materials which enable researchers to become trainers by participating in the programme. Trained trainers can adapt the programme to their own target group, discipline and specific context. The programme has proven popular, as 470 trainers to date have been trained in Europe. The programme’s materials are openly available and creative commons licenced, supporting their continued use and further development. Trainers report high levels of satisfaction with the programme and intend to apply it, in full or in part, in their own teaching practice.

Virtue development comes through practice (Aristotle, NE; MacIntyre, 1981); we contend that an RI course based on a virtue ethics approach can educate and motivate participants to develop themselves, and provide them with the tools and skills to habitually cultivate their own moral learning in their professional practice.

Supplemental Material

sj-docx-1-rea-10.1177_17470161231161267 – Supplemental material for VIRT2UE: A European train-the-trainer programme for teaching research integrity

Supplemental material, sj-docx-1-rea-10.1177_17470161231161267 for VIRT2UE: A European train-the-trainer programme for teaching research integrity by Natalie Evans, Armin Schmolmueller, Margreet Stolper, Giulia Inguaggiato, Astrid Hooghiemstra, Ruzica Tokalic, Daniel Pizzolato, Nicole Foeger, Ana Marušić, Marc van Hoof, Dirk Lanzerath, Bert Molewijk, Kris Dierickx and Guy Widdershoven on in Research Ethics

Supplemental Material

sj-docx-2-rea-10.1177_17470161231161267 – Supplemental material for VIRT2UE: A European train-the-trainer programme for teaching research integrity

Supplemental material, sj-docx-2-rea-10.1177_17470161231161267 for VIRT2UE: A European train-the-trainer programme for teaching research integrity by Natalie Evans, Armin Schmolmueller, Margreet Stolper, Giulia Inguaggiato, Astrid Hooghiemstra, Ruzica Tokalic, Daniel Pizzolato, Nicole Foeger, Ana Marušić, Marc van Hoof, Dirk Lanzerath, Bert Molewijk, Kris Dierickx and Guy Widdershoven on in Research Ethics

Footnotes

Acknowledgements

VIRT²UE aims to develop a sustainable train-the-trainer blended learning programme enabling contextualised research integrity and ethics teaching across Europe focusing on understanding and upholding the principles and practices of the European Code of Conduct for Research Integrity. The VIRT²UE Consortium is composed of the VU Medical Center Amsterdam, KU Leuven, University of Split School of Medicine, Austrian Agency for Research Integrity, University of Oslo, European Network of Research Ethics Committees, Ankara University, National Technical University of Athens, University of Helsinki, University of Latvia, Universidade Catolica Portuguesa, University of Insubria, and Momkai. The authors would like to thank all consortium members, VIRT2UE trainers, and VIRT2UE research integrity course participants.

Author contributions

Evans N: Conceptualisation, Investigation, Methodology, Supervision, Writing – original draft, Writing – review and editing. Schmolmueller A: Investigation, Methodology, Writing – original draft, Writing – review and editing. Stolper M: Investigation, Methodology, Writing – original draft, Writing – review and editing. Inguaggiato G: Conceptualisation, Investigation, Methodology, Writing – review and editing. Hooghiemstra, A: Investigation, Methodology, Writing – original draft, Writing – review and editing. Tokalic R: Investigation, Methodology, Writing – review and editing. Pizzolato D: Investigation, Methodology, Writing – review and editing. Foeger N: Conceptualisation, Investigation, Methodology, Supervision, Writing – review and editing. Marušić A: Conceptualisation, Investigation, Methodology, Supervision, Writing – review and editing. van Hoof M: Conceptualisation, Investigation, Methodology, Supervision, Writing – review and editing. Lanzerath D: Investigation, Methodology, Supervision, Writing – review and editing. Molewijk B: Investigation, Methodology, Supervision, Writing – review and editing. Dierickx K: Conceptualisation, Investigation, Methodology, Supervision, Writing – review and editing. Widdershoven G: Conceptualisation, Investigation, Methodology, Writing – review and editing.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

All articles in Research Ethics are published as open access. There are no submission charges and no Article Processing Charges as these are fully funded by institutions through Knowledge Unlatched, resulting in no direct charge to authors. For more information about Knowledge Unlatched please see here: . The VIRT²UE project (Virtue based ethics and Integrity of Research: Train-the-Trainer programme for Upholding the principles and practices of the European Code of Conduct for Research Integrity) has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement N 787580.

Ethics approval

In the Netherlands, The Medical Ethics Review Committee of VU University Medical Center, Amsterdam, evaluated the study protocol, assessed it as not requiring ethics approval under the Medical Research Involving Human Subjects Act (WMO), and provided a ‘niet-WMO’ declaration (FWA number: FWA00017598).

Supplemental material

Supplemental material for this article is available online.

ORCID iDs

Natalie Evans

Dirk Lanzerath

References

Abdi

Pizzolato

Nemery

, et al. (2021) Educating PhD students in research integrity in Europe. Science and Engineering Ethics 27(1): 5.

All European Academies (ALLEA) (2017) The European code of conduct for research integrity. Available at: https://www.allea.org/wp-content/uploads/2017/05/ALLEA-European-Code-of-Conduct-for-Research-Integrity-2017.pdf (accessed 10 June 2018).

Baehr

(2013) Educating for intellectual virtues: From theory to practice. Journal of Philosophy of Education 47(2): 248–262.

Barak

Green

(2021) Applying a social constructivist approach to an online course on ethics of research. Science and Engineering Ethics 27(1): 1–24.

Berkowitz

Bier

(2007) What works in character education. Journal of Research in Character Education 5(1): 29–48.

Berling

McLeskey

O’Rourke

, et al. (2019) A new method for a virtue-based responsible conduct of research curriculum: Pilot test results. Science and Engineering Ethics 25: 899–910.

Biggs

Tang

(2007) Teaching according to how students learn. In: Biggs

Tang

(eds) Teaching for Quality Learning at University, 3rd edn. Maidenhead: Open University Press, 15–30.

Bouter

(2015) Commentary: Perverse incentives or rotten apples? Accountability in Research 22(3): 148–161.

Chen

(2016) Research as profession and practice: Frameworks for guiding the responsible conduct of research. Accountability in Research 23(6): 351–373.

10.

Dewey

(1897) My pedagogic creed. (No. 25). EL Kellogg & Company. Available at: http://www.infed.org/archives/e-texts/e-dew-pc.htm (accessed 29 July 2021).

11.

Erasmus University Rotterdam (2022) Dilemma Game. Available at: https://www.eur.nl/en/about-eur/policy-and-regulations/integrity/research-integrity/dilemma-game (accessed 1 February 2022).

12.

Evans

Marusic

Foeger

, et al. (2021) Virtue-based ethics and integrity of research: Train-the-trainer programme for upholding the principles and practices of the European Code of Conduct for Research Integrity (VIRT²UE). Research Ideas and Outcomes 7: e68258.

13.

Fuerholzer

Schochow

Steger

(2020) Good Scientific Practice: Developing a curriculum for medical students in Germany. Science and Engineering Ethics 26: 127–139.

14.

Han

(2015) Virtue ethics, positive psychology, and a new model of science and engineering ethics education. Science and Engineering Ethics 21(2): 441–460.

15.

Inguaggiato

Metselaar

Porz

, et al. (2019) A pragmatist approach to clinical ethics support: Overcoming the perils of ethical pluralism. Medicine Health Care and Philosophy 22(3): 427–438.

16.

Kalichman

Plemmons

(2007) Reported goals for responsible conduct of research courses. Academic Medicine 82(9): 846–852.

17.

Katsarov

Andorno

Krom

, et al. (2022) Effective Strategies for Research Integrity Training—a meta-analysis. Educational Psychology Review 34: 935–955.

18.

Lanzerath

(2015) Ethos. In: Sturma

Heinrichs

(eds) Handbuch der Bioethik. Stuttgart: JB Metzler, 35–43.

19.

Lindner

Torralba

Khan

(2018) Scientific productivity: An exploratory study of metrics and incentives. PLoS One 13(4): e0195321.

20.

MacIntyre

(1981) After Virtue a Study in Moral Theory. London: Gerald Duckworth & Co.

21.

Mayer

R E

(2002) Multimedia learning. In: Ross

(ed.) Psychology of Learning and Motivation. Cambirdge, MA: Academic Press, 85–139.

22.

McLeskey

Berling

O’Rourke

, et al. (2020) The evolution of the scientific virtues toolbox approach to responsible conduct of research training. In: Banzhaf

Cheng

BHC

Deb

, et al (eds) Evolution in Action: Past, Present, and Future. Dordrecht: Springer, 535–550.

23.

Mejlgaard

Bouter

Gaskell

, et al. (2020) Research integrity: Nine ways to move from talk to walk. Nature 586: 358–360.

24.

Merton

(1973 [1942]) The normative structure of science. In: Merton

(ed.) The Sociology of Science: Theoretical and Empirical Investigations. Chicago, IL: University of Chicago Press, 267–278.

25.

Molewijk

Kleinlugtenbelt

Widdershoven

(2011) The role of emotions in moral case deliberation: Theory, practice, and methodology. Bioethics 25(7): 383–393.

26.

Moore

Beadle

(2006) In search of organizational virtue in business: Agents, goods, practices, institutions and environments. Organization Studies 27(3): 369–389.

27.

Mulhearn

Steele

Watts

, et al. (2017) Review of instructional approaches in ethics education. Science and Engineering Ethics 23(3): 883–912.

28.

Oakley

Cocking

(2001) Virtue Ethics and Professional Roles. Cambridge: Cambridge University Press.

29.

Pellegrino

(1989) Character, virtue and self-interest in the ethics of the professions. The Journal of Contemporary Health Law and Policy 5: 53–73.

30.

Pennock

(2018) Beyond research ethics: How scientific virtue theory reframes and extends responsible conduct of research. In: Carr

(ed.) Cultivating Moral Character and Virtue in Professional Practice. London: Routledge, 166–177.

31.

Pennock

O’Rourke

(2017) Developing a scientific virtue-based approach to science ethics training. Science and Engineering Ethics 23(1): 243–262.

32.

Pizzolato

Abdi

Dierickx

(2020) Collecting and characterizing existing and freely accessible research integrity educational resources. Accountability in Research 27(4): 195–211.

33.

Pizzolato

Dierickx

(2021) Stakeholders’ perspectives on research integrity training practices: A qualitative study. BMC Medical Ethics 22: 67.

34.

Resnik

(2012) Ethical virtues in scientific research. Accountability in Research 19(6): 329–343.

35.

Resnik

Dinse

(2012) Do U.S. Research institutions meet or exceed federal mandates for instruction in responsible conduct of research? A national survey. Academic Medicine 87(9): 1237–1242.

36.

Ross

(2013) Aristotle: Nicomachean Ethics. Cambridge: Cambridge University Press.

37.

Schön

(1987) The Reflective Practitioner. Surrey: Ashgate Publishing Limited.

38.

Solbakk

(2015) What is it to do good medical ethics? On the concepts of ‘good’ and ‘goodness’ in medical ethics. Journal of Medical Ethics 41(1): 12–16.

39.

Steneck

Bulger

(2007) The history, purpose, and future of instruction in the responsible conduct of research. Academic Medicine 82(9): 829–834.

40.

Stolper

Molewijk

Widdershoven

(2015) Learning by doing. Training health care professionals to become facilitator of moral case deliberation. HEC Forum 27: 47–59.

41.

Stolper

Molewijk

Widdershoven

(2016) Bioethics education in clinical settings: Theory and practice of the dilemma method of moral case deliberation. BMC Medical Ethics 17: 45.

42.

Todd

Watts

Mulhearn

, et al. (2017) A meta-analytic comparison of face-to-face and online delivery in ethics instruction: The case for a hybrid approach. Science and Engineering Ethics 23(6): 1719–1754.

43.

Tomić

Buljan

Marušić

(2022a) Perspectives of key stakeholders on essential virtues for good scientific practice in research areas. Accountability in Research 29(2): 77–108.

44.

Tomić

Buljan

Marušić

(2022b) Development of consensus on essential virtues for ethics and research integrity training using a modified Delphi approach. Accountability in Research 1–24.

45.

Von Wright

(1963) The Varieties of Goodness. London: Routledge & Kegan Paul, 8–12; 114–135.

46.

Widdershoven

Molewijk

(2010) Philosophical foundations of clinical ethics: A hermeneutic perspective. In: Schildmann

Gordon

Vollmann

(eds) Clinical Ethics Consultation. Theories and Methods, Implementation, Evaluation. Farnham: Ashgate, 37–51.

47.

Wittrock

(1992) Generative learning processes of the brain. Educational Psychologist 27: 531–541.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.07 MB

0.03 MB