Evaluating the efficacy of the education and training program of the TCPS 2

Abstract

In 2011, the Secretariat on Responsible Conduct of Research launched a set of educational opportunities to facilitate and enhance the dissemination of TCPS 2, the 2nd edition of the Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans (2010), which guides Canadian research ethics. Three educational modalities were implemented to aid participants in developing or refining their ethical understanding and practice: (i) Regional Workshops, which brought together diverse disciplinary perspectives; (ii) the CORE tutorial, which enabled individuals to discover the various aspects and applications of the Policy; and (iii) Webinars, which provided participants with the opportunity to explore deeper dimensions of research ethics. This article reports on the findings of a national program evaluation team which measured the effectiveness of the educational program and provided recommendations for future opportunities.

Although scholarship on ethics education is still nascent, the literature on research ethics covers a range of topics, from inquiry about the review process (Emanuel et al., 2004; Flory and Emanuel, 2004) to advocacy for a different scale, focus, and format in the approach to ethical research (Eisen and Parker, 2004; Gordon and Parsi, 2002; Schüklenk and Ashcroft, 2002), and to cautionary tales of unawareness, resistance, and transgressions upon research ethics principles (De Vries et al., 2004; DuBois, 2004; Eisen and Berry, 2002; Vallance, 2005).

There is an identified need for research ethics education as evidenced by the Secretariat’s initiative to provide educational opportunities. This need is underscored by a lack of standardization in the application of the TCPS 2 (Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans, 2nd edn: CIHR, SSHRC and NSERC, 2010) policy among individuals and groups, and within institutions (Kon et al., 2011).

There is a growing consensus among program evaluators about the need for a ‘reliable and valid tool to assess knowledge of research ethics’ (Taylor et al., 2011: 241). This tool would encourage researchers, research ethics board members and students to invest in the ‘skills of ethical analysis and problem solving’ and apply their ethical knowledge to the diverse situations that occur in research (Taylor et al., 2011: 241). Antes et al. emphasize the need for program evaluation that is tailored to the various aspects of an educational initiative and therefore capable of evaluating not only its content-based effectiveness, but also its complexity and engagement in terms of its ability to develop participants’ knowledge and skills (Antes et al., 2009: 14).

Canadian research ethics

TCPS 2 is the joint research ethics policy statement of the three Canadian federal research agencies: Canadian Institutes of Health Research (CIHR), Natural Sciences and Engineering Research Council of Canada (NSERC), and Social Sciences and Humanities Research Council of Canada (SSHRC). The Policy expresses the agencies’ continuing commitment to the people of Canada by promoting the ethical conduct of research involving humans. It has been informed, in part, by leading international norms for research ethics, and helps to guide Canadian researchers in Canada and abroad.

TCPS 2, released in 2010, is the first comprehensive revision of the original Policy (CIHR, SSHRC and NSERC, 1998) since its adoption in 1998. The Secretariat on Responsible Conduct of Research (SRCR or Secretariat, formerly Secretariat on Research Ethics) is responsible for the development, interpretation, and implementation of the Policy. As part of its broader implementation strategy to disseminate the Policy, the Secretariat decided to engage in educational activities to familiarize the research community with its concepts and application. The Secretariat provided the research ethics community with three educational activities:

1. Course on Research Ethics (CORE) provides an online self-guided tutorial that presents an introduction to key research ethics issues in relation to the TCPS 2 framework.

2. Webinars provide an in-depth, topical discussion of research ethics issues.

3. Regional Workshops facilitate face-to-face interaction between REB members and researchers in the context of applying TCPS 2 guidance to research ethics issues within the four broad research areas: health science, social science and humanities, natural science, and working with First Nations, Inuit and Metis research participants.

TCPS 2 research education program evaluation

This article highlights a program evaluation conducted by our team that assessed the effectiveness of the TCPS 2 education and training program. It was designed to generate the kinds of information that could be used to identify the strengths and weaknesses of research ethics educational efforts, as well as to provide information that could be useful for future planning purposes. It is important to note that this study was an assessment of the outcomes (i.e. efficacy) of the educational opportunities and not a critique or analysis of the TCPS 2 policy. The findings from this study concern the degree to which the educational efforts have made a difference to participants’ use and understanding of research ethics.

Our program evaluation is guided by three questions:

1. What baseline knowledge do participants have regarding research ethics principles?

2. Does the completion of the tutorial and/or participation in a Regional Workshop or Webinar have any impact on baseline level of knowledge? If so, what knowledge has been acquired?

3. Is there any evidence that participating in the education and training program of the TCPS 2 has had any impact on participants’ application of ethical principles in research practices?

Methodology

The data collection for this program evaluation is based on a mixed methods design (both qualitative and quantitative) as described by Johnson and Onwuegbuzie (2004) in order to triangulate the data. As noted by Cohen et al., a mixed methods design is appropriate when the goal of the research is to ‘attempt to map out, or explain more fully, the richness and complexity of human behavior by studying it from more than one standpoint’ (Cohen et al., 2000: 112). The overall design follows the ‘parallel mixed design’ topology advanced by Teddlie and Tashakkori (2009): each educational opportunity was studied independently of the other. However, as part of triangulation, each data set was compared to the others to provide more detailed information on the efficacy of how individuals and institutions are participating in the educational opportunities. The following provides an overview of each data source.

Pre-/post-questionnaire

Participants were invited to complete both a pre- and post-questionnaire based on key research ethical concepts identified in the TCPS 2. They presented questions which could be answered in an open-ended or a closed-ended fashion and focused on the utility of the education and training program by collecting baseline data about ethical awareness such as criteria of free and informed consent, measures necessary to protect privacy and confidentiality, and successful identification of risks and benefits to participants. Quantitative closed-ended items allowed us to get a feel of the extent of the issue and describe its central tendency. Closed-ended items allowed us to quantify the responses. Qualitative open-ended items allowed us to assess the importance of the responses, clarify them, and elaborate or provide a context for the closed-ended items. For the purpose of this study, we are cautious of quantifying qualitative data.

The pre-questionnaire was electronically distributed online prior to participants beginning Module One of CORE. Participants were encouraged to do the pre- and post-questionnaire for evaluation purposes. Participants were able to move forward in CORE without completing the questionnaire if they did not wish to participate. Those who did complete the pre-questionnaire were contacted approximately 6 months later to complete the post-questionnaire. For the Regional Workshops and Webinars, participants were invited to participate in the data collection and to respond to the questionnaire prior to their attendance (if they were pre-registered for the sessions); a post-questionnaire was sent out 6 months later.

Pre- and post-questionnaire analysis proved difficult to perform for reasons of an overly enthusiastic number of responses to the pre- (over 10,000) and a low response to the post-questionnaire (1666 respondents). Direct comparison of pre- and post-questionnaire data was largely impossible owing to incompatibility between the questionnaire formats.

Focus groups

The second point in our triangulation used the focus group approach. Our focus group participants consisted of individuals who participated in a Regional Workshop, Webinar, or CORE. The invitation to participate in a focus group was included in the options provided on the pre-questionnaire and CORE for future contact for the purposes of evaluation. The number of people in each focus group was determined based on the completion of the pre-questionnaire and consent to be contacted. The recommended numbers of participants for each group is 5–8 participants in non-commercial focus groups (Krueger and Casey, 2008). Five focus groups of 5–8 participants were held: one was composed of students, two of researchers, and two of REB members. A set of thematic questions was developed for the focus groups and each session was facilitated by telephone because of the diverse geographical locations of the participants.

Kidd and Parshall (2000) recommend a process to help ensure reliability and validity that we have followed. Specifically for reliability, ‘[t]he internal consistency of coding is enhanced if one team member has the primary responsibility for conducting the analysis, participates in as many groups and debriefings as possible, and communicates regularly with other team members as the analysis proceeds (e.g. as coding of each transcript is completed)’ (Kidd and Parshall, 2000: 304). In keeping with focus group methodology, we had two facilitators involved in the focus group data collection; however, one of the facilitators had primary responsibility for the data analysis. Kidd and Parshall define validity as ‘a matter of accumulating a pattern of expected convergent and discriminant associations between themes,’ and point out that ‘similar findings derived from multiple sources increase confidence in the validity of constructs and the theoretical generality of relations between them’ (Kidd and Parshall, 2000: 304).

Interviews

The final piece of the triangulation is interview data derived from a sample of participants who responded to the questionnaires. The interviews and focus groups provided an additional means of clarifying the questionnaire data and allowed for an in-depth understanding of how participants felt about their experiences. Interviews were conducted by telephone and analyzed using discourse analysis in search of themes. A pre-determined coding scheme was not created. The topics that were explored within the interviews and focus groups included: (i) what was the efficacy of the education and training program; (ii) what research ethics guidelines were most difficult to apply to one’s research; (iii) what topics should be covered in a research ethics education program; (iv) how should these issues be addressed in a research ethics education program; (v) how well did existing approaches to research ethics education address the application of research ethics to research design and conduct; and (vi) how could they be improved. These questions focused the discussion to facilitate gathering of data that could inform and complement survey data.

We interviewed 120 respondents, targeting participants who had completed one or more of the TCPS 2 learning opportunities (the Regional Workshop, CORE, or the Webinar). Representation was weighted by selecting a proportional percentage of respondents from Eastern, Central, and Western Canada. Within the three regions, we targeted a percentage of respondents from each of the following categories: students, researchers, research services, research ethics board members, and others. The interviews were conducted in English and French, according to the preference of the respondent. Ten of the 120 interviews were completed in French.

Findings

The goal of this study was to evaluate the efficacy of the Secretariat’s education and training program for TCPS 2. Within the program, three educational opportunities aided participants in developing or refining their ethical understanding and practice: Regional Workshops, which brought together diverse disciplinary perspectives to collaboratively work through a research ethics case study; the CORE online tutorial, which enabled individuals to discover various aspects and applications of the Policy; and Webinars, which provided groups with the opportunity to explore deeper dimensions of ethical issues in research with human participants.

The three educational opportunities were available across the country to a range of individuals, including research ethics office staff, research staff, grant administrators, research ethics board members, and researchers whose experience with the Policy ranged from novice (e.g. students who are just entering research) to advanced (e.g. senior researchers with over 20 years of experience). Prior to participation in TCPS 2 educational opportunities, participants were given the opportunity to complete a pre-survey that was used to ascertain individuals’ baseline knowledge of TCPS 2 and research ethics principles. Although participants in the pre-survey sample came from across the country, were employed or studying in various settings, and represented all research-related roles, it was noted that a large proportion of pre-survey respondents were from universities in Ontario and British Columbia. A possible reason for this proportion is that there are an increasing number of institutional mandates (including mandatory ethics education) to provide some form of ethics training to their students in British Columbia and Ontario, thus contributing to an unusually high rate of participation.

Our analysis of the responses indicates that there is a discrepancy between individuals’ satisfaction with their baseline knowledge of research ethics and their baseline knowledge of TCPS 2. In terms of respondents’ satisfaction with their baseline knowledge of research ethics, respondents indicated that they were somewhat to very satisfied with their understanding of research ethics, their ability to complete an ethics application, their preparation/training to complete an ethics application, and the review process provided by the research ethics boards at their institutions. On the other hand, when respondents were asked to indicate their level of satisfaction with their knowledge of TCPS 2, the majority of respondents indicated that they were unsatisfied with their knowledge of TCPS 2. Further evidence for this lack of knowledge of TCPS 2 was found when respondents were asked if they had read either version of the TCPS: 59.2 per cent of respondents indicated not having read either version of the TCPS. Furthermore, when respondents were asked to cite the three core principles of TCPS 2 (i.e. respect for persons, concern for welfare, and justice), only 21.5 per cent of respondents correctly cited all three principles. These findings provide evidence of the need for educational opportunities that are capable of facilitating participants’ enhanced ethical understanding and research practice.

Educational opportunities and their impact on participants

Regional Workshops

Data gathered in the interviews and focus groups speak to the level of participants’ enjoyment of the Regional Workshops. The opportunity to discuss research ethics issues with others, whose roles and experiences differed from theirs, is of value when participants ‘come together as a group.’ As one participant put it, ‘I liked that part because it gave you an insight to what is happening in reality. That was the best part and what I enjoyed the most.’ On the whole, the facilitators generated discussion and engagement with the Policy. As one participant recalls, ‘I found the facilitator extremely open and helpful. People felt very free to express their own opinions and I think that was a really important aspect of it.’ Participants also commented on the fact that they appreciated the case study as an exercise that helped them to become more proficient in referring to TCPS 2.

Although 88.2 per cent of participants agreed or strongly agreed with the statement ‘I benefited from participating in this training’, they did also identify areas for instructional improvement. It was noted that the opening slides were repetitive from one session to another, and that they could be used more effectively. Another area of improvement for the Regional Workshops concerns providing more clarification on what has changed from TCPS and TCPS 2, instead of only a cursory overview.

Participants who completed both the Regional Workshop and CORE commented that both learning opportunities were valuable and offered different, yet complementary, experiences. In the focus groups, it was said that people really liked the fact that they could book a block of time to attend the Workshop. They also said they would like to see more of them taking place around the country, because most participants did not receive travel funding to attend the Regional Workshop.

CORE

To date, over 58,000 users have completed CORE – a free online tutorial that provides information, case stories, and activities surrounding key issues in research ethics (e.g. informed consent). Although most are Canadian, people from over 20 other countries have now taken the course. Participants appreciated the Canadian perspective of ethics and the fact that it ensures that there is a common understanding of research ethics parameters throughout and beyond the country: ‘it cemented the Canadian guidelines’ and ‘consolidates an ethical perspective that will work across borders.’ For REB members and researchers who have supervisory responsibilities, CORE is a useful tool to ensure that students, including those who are international and may be unfamiliar with the Canadian context, receive a fundamental grounding in research ethics.

In the focus groups and interviews, several participants commented on the use of case studies. The majority of participants found them effective, but some found they lacked relevance and were too oriented towards health-oriented professions. In the focus group, respondents said they would like more case studies that are more diversified and contain more complex examples.

Participants also found the asynchronous form of CORE helpful: ‘Given my schedule, anything else would have been impossible.’ Additionally, the format was widely praised for providing a stimulating interactive learning experience: ‘Using case studies as well as animation of sorts was quite helpful to flow between text and engagement.’ However, there were a significant number of complaints about the length of time that it takes to complete the tutorial, as on average it took 4 hours to complete all modules. There were also some technical difficulties with the videos, pop-up windows, and quizzes; participants would like to be able to complete CORE via tablet, iPad, or smartphone. Some students also had difficulty registering for CORE and asked that the process be made more accessible.

CORE was appreciated by students for the content, structure, design, and layout, which make it highly engaging. It was appreciated by the researchers and REB members because it was a good refresher, and helped them to become familiar with specific principles in the TCPS 2 document. It also offers a good tool to enhance the quality of ethics review applications: ‘The submissions are a lot better and things go faster and smoother. It is tougher to identify problems with them. There are also very few full review protocols now.’

Participants have expressed that they would like to go deeper and have more training regarding the following areas that they felt are not explicit enough in the TCPS 2 or CORE, and are listed based on highest priority:

• privacy and confidentiality;

• research integrity;

• vulnerable populations;

• working with partners and stakeholders;

• ethical issues that include dilemmas that one might face in research;

• aboriginal populations;

• informed consent;

• secondary use of data;

• research in difficult conditions; and

• the responsibilities of each member of an REB.

Webinars

Advanced users such as REB members appreciated the Webinars and indicated that they would participate in future Webinars on such topics as: creative practice; clinical trials for non-medical universities; international research (for example, research co-conducted by two countries); and training based on topics that people ask the Interpretation Services. Some advanced users of the Webinars would like to see more difficult cases used. Continuous training opportunities that address difficult or complex questions, such as conferences or teleconferences, would also be welcomed.

Summary of findings

Survey data indicated that respondents to the post-survey were mainly researchers and research staff employed or studying in university or hospital settings. The majority of respondents in the sample had completed the CORE tutorial. Within the sample of respondents who had completed Regional Workshops and Online Webinars, it was noted that a substantial proportion of research ethics office staff and research ethics board members tended to complete Regional Workshops, whereas grant administrators tended to complete Webinars.

In terms of overall satisfaction with educational opportunities, age seemed to influence satisfaction ratings. Whereas the majority of respondents felt that the learning objectives were clear, the information was presented in an engaging manner, and the instructional techniques were used effectively, younger respondents (under 20) tended to be less satisfied with the visual aids/case studies used and were also less likely to recommend TCPS 2 training to others.

Regarding the impact of educational opportunities on respondents, across the three educational opportunities, the majority of respondents indicated that they had learned more about TCPS 2. Additionally, respondents found that educational opportunities had not only addressed important research ethics issues but had also helped them remember ethical principles stated in the TCPS 2. Encouragingly, 92 per cent of respondents indicated that participants’ perspectives were the primary driver of research and, when asked to identify the three core principles of TCPS 2, 70 per cent of the respondents correctly identified these to be respect for persons, concern for welfare, and justice. The impact of educational opportunities on participants was also evidenced in the list of points respondents indicated they remembered from completing an educational opportunity (e.g. core principles behind TCPS 2, assessing the risk and benefits in research, multi-jurisdictional research, and community-based research). Overall, the majority of respondents indicated that, upon completion of educational opportunities, they had gained confidence in their knowledge of TCPS 2 and were very confident in their understanding of research ethics. With regard to predicting future preferences for educational opportunities, in the case of respondents who completed Regional Workshops, it was noted that respondents from hospitals, research institutes and universities were predicted to be less likely to prefer Regional Workshops than respondents who were employed/studying at a college. In the case of respondents who completed the online tutorial, respondents aged 30 and older were predicted to be more likely to indicate a preference for the online tutorial than were respondents aged 20 and under. Finally, for respondents who completed either the online tutorial or online Webinars, males were predicted to be less likely to indicate a preference for either educational approach than were females.

Conclusions and recommendations

Arising from the demographics data are distinct user groups whose differences should be addressed and reflected in the design of the educational opportunities. Demographic differences allow for sufficient tailoring of the educational opportunities to enhance relevance and increase quality of learning. Currently, the Regional Workshop allows users to select a disciplinary stream (health science, social science and humanities, natural science, and working with special populations); however, there are a wide variety of disciplinary combinations and roles within those disciplines, which could be better addressed by means of a more inclusive paradigm that allows for, and caters to, differentiation across educational opportunities.

The REB segment is one learner group for which our findings suggest there are unmet needs for educational opportunities. In particular, there is a need for introductory material for new REB members. A CORE adapted for REB is one of many possibilities. One of the meaningful outcomes for developing material for REB members is creating more sustainable REBs by reducing the time it takes to train a new member, and creating greater consistency in the application of TCPS 2 among institutions. Future research could focus on understanding the particular needs of REB members and the unique challenges they face when leading research ethics within institutional settings.

A novice–expert continuum of research ethics learning would serve as a meaningful heuristics to account for the significant differences between user groups for each of the educational opportunities without sacrificing the richness of interdisciplinary conversation and case-based learning in the Regional Workshop. To adopt the novice–expert continuum for CORE, users would be grouped by their level of experience or understanding with the TCPS 2 (e.g. advanced understanding for REB members or senior researchers who complete CORE as a refresher would be classified as ‘expert’), as distinct from participants who were at a novice level (such as beginning graduate student researchers). This level-based paradigm is most applicable for the delivery of CORE, which serves a diverse demographic. By contrast, it is less important to the Webinar, as this educational opportunity already serves a focused group of advanced users. This is in keeping with the observation that students are a significant group that the Secretariat should focus upon. In their roles as novice researchers or RA Co-PIs who are working under the supervision of a PI, students are often most responsible for administering research and for writing first drafts of ethics proposals.

A useful way of aligning the three educational opportunities is through a tiered model, where CORE forms the first tier of educational strategy that introduces the learner to research ethics principles and the relevant connections to TCPS 2. For those learners who wish to develop deeper expertise, Regional Workshops represent a higher-tier opportunity that provides a learning opportunity conducive to exploring the nuances of research ethics and its application within institutional settings. Finally, for those who wish to develop deep expertise in one particular area of research ethics, Webinars offer the venue to acquire that content expertise and an opportunity to interact with like-minded individuals and members of the Secretariat.