Changes in evidence-based practice beliefs and implementation after a postgraduate educational program: A before-and-after study over one year

Abstract

Healthcare professionals are increasingly expected to use research evidence to improve healthcare outcomes. The aim of this study was to examine change in beliefs and implementation of evidence-based practice (EBP) one year after participation in a 15 ECTS postgraduate educational program in EBP. We conducted a before-and-after study among 158 healthcare professionals in a postgraduate program in EBP. Participants completed the Norwegian version of the EBP Beliefs Scale and the EBP Implementation Scale at baseline and one year after the program. Results indicated increased scores from pre-test to one year follow up on the EBP Beliefs Scale and the EBP Implementation Scale. Participants in evidence-based network groups reported higher scores on the EBP Implementation Scale than those who did not participate in these groups. Further research should examine other factors that may facilitate implementation of evidence-based practice.

Keywords

beliefs evidence-based practice implementation postgraduate education

Introduction

Evidence-based practice (EBP) has been on the international agenda for the past decades, and healthcare professionals are increasingly expected to use research evidence to improve healthcare outcomes. Evidence-based practice is an approach where clinical expertise acquired through clinical practice and experience is integrated with current best available research evidence and patient preferences, within the context of available resources.¹ This involves finding, assessing and interpreting research evidence, implementing evidence findings and evaluating performance. To deliver practice based on evidence, healthcare professionals need to build competence. Today, EBP is taught to all levels of students and practitioners through workshops, journal clubs, courses, seminars, conferences, various educational programs and internet-based interventions. It is no longer a question of whether to teach EBP, but how to teach it.²

Teaching EBP typically involves the five essential steps of EBP: 1) asking an answerable question, 2) searching for the best available evidence, 3) critically appraising the evidence for its validity, clinical relevance and applicability, 4) applying the results in practice and 5) evaluating performance.¹ However, people with differing roles and levels of responsibility require different skills for EBP and different types of evidence.¹ Clinicians may incorporate evidence into their practice by carrying out the steps of EBP, by using pre-appraised evidence summaries or by replicating recommendations and decisions of respected EBP leaders.^2,3 These approaches require different use of time, skills and knowledge. Ideally, educational aims should be matched to the needs and characteristics of the learners.⁴

According to Khan and Coomarasamy's ‘hierarchy of effective teaching and learning’, the best way to learn EBP is through interactive teaching and learning activities clinically integrated in practice.⁵ Using this approach, students will learn not only the principles and skills of EBP, but also how to incorporate these skills with their own life-long learning into patient care.¹ However, other teaching and learning approaches may be suitable for learners who have to acquire a baseline or pre-requisite knowledge and encouragement in EBP.⁵ For these learners, interactive classroom-based teaching or didactic clinically integrated teaching may also be appropriate.

Several systematic reviews have summarized the effect of postgraduate teaching in EBP.^5–11 The interventions reported significant gains in EBP knowledge and skills. Some evidence found changes in EBP attitude and behavior when teaching was integrated into clinical practice.^5,6 However, caution is warranted due to variations in methodological quality, outcome measures, populations and educational interventions. In addition, few of the included studies used validated measurement instruments.^8,9,12

To build competence in EBP, Bergen University College established a postgraduate educational program in EBP in 2004. The purpose of this program was to promote the use of research evidence in teaching and in clinical practice, make healthcare professionals aware of existing access to evidence, and to provide implementation strategies.¹³ The program was organized into one semester, with a total of 42 hours distributed over seven days. Various learning strategies like lectures, group discussions, skills training and assessments were used in the program. The target group was teachers within healthcare professions (nursing, physiotherapy, occupational therapy, radiography and social work), professional development specialists, and healthcare professionals from clinical practice who wanted to practice based on evidence.¹³

Much of the literature on postgraduate teaching in EBP describes educational interventions of shorter duration. Most EBP programs have been variably conducted, with sessions from a few hours to a few days long, as one-hour sessions weekly over a period of weeks or as one-hour sessions monthly for a year.^6,8,9 There is a need to understand how more comprehensive postgraduate educational programs affect healthcare professionals' EBP over time. The aim of this study was to examine changes in EBP beliefs and implementation among healthcare professionals one year after participating in a postgraduate educational program in EBP of 42 hours. A secondary question explored the change in scores with participation in EBP network groups.

Methods and materials

Research design

A pre- and post-test survey design was conducted, using the Norwegian version of the EBP Beliefs Scale and the EBP Implementation Scale. Data were collected before and one year after participation in a postgraduate educational program in EBP.

Sample

The sample consisted of 203 healthcare professionals attending postgraduate education during the period 2009 to 2011. The participants came in six different cohorts going through the program at different periods. Only participants who had passed the final examination were eligible to participate. Participants who followed the postgraduate program without taking the exam and participants who failed the exam were excluded from the study.

Procedure

All participants (203) completed the questionnaires before the teaching started the first day. One year after the program, the 158 participants who had completed the course and passed the examination received a postal questionnaire and a reply envelope. The participants' anonymity was ensured by means of numbered questionnaires, and the researcher was blinded to participants. We sent out to two reminders.

The postgraduate program in EBP

The core content of the postgraduate program was based on the steps of EBP.¹⁴ The students were taught how to reflect on clinical practice and ask answerable clinical questions; search for evidence; critically appraise evidence for its validity, importance, and applicability; apply evidence to practice; and evaluate performance.^1,14 The teaching used principles from adult learning theory¹⁵ and the Critical Appraisal Skills Programme.¹⁶ The program used a combination of various learning strategies, such as interactive and didactic lectures, small-group discussions, problem-based learning, skills training, and oral and written assessments. To ensure a close relationship to clinical practice, clinical issues and research articles relevant to the students' practices were incorporated throughout the course.

The program was organized into one semester, with seven days (six hours per day) of participation distributed over three weeks (Table 1). Various types of clinical questions (intervention, diagnosis, prognosis, qualitative) were covered. Content was also provided about the role of clinical practice guidelines in EBP, how to implement evidence in practice and how to communicate evidence to patients. One day was devoted to finding evidence, by using hands-on experience with a librarian trained in EBP. As part of their assessment, the students had to give two oral presentations to fellow students and write an individual home examination. Most chose to write a paper about a clinical question related to their own practice. In these papers (10 pages), they asked an answerable question, searched relevant databases for evidence, critically appraised the evidence and addressed their findings to their own practice. The examination also included a presentation of this paper to colleagues at work.

Table 1.

Content of the postgraduate educational program in evidence-based practice (EBP).

Week	Day	No. hours	Topic	Teaching approaches
1	1	3 3	Introduction to EBP Asking answerable questions	Lectures, small-group discussion, clinical issues.
1	2	3 3	Critical appraisal: Survey How do we understand statistics?	Lectures, small-group discussion, clinical issues.
2	3	3 3	Critical appraisal: Randomized controlled trials Critical appraisal: Systematic reviews	Lectures, small-group discussion, clinical issues.
	4	3 3	Critical appraisal: Evidence-based guidelines Critical appraisal: Qualitative study	Lectures, small-group discussion, clinical issues.
	5	6	Finding the evidence	Hands-on search with librarian.
3	6	3 3	Shared decision-making Implementing EBP	Lectures, small-group discussion, clinical issues.
3	7	3 3	Oral presentations: 1. Explain a term in EBP 2. Explain how you will use EBP in the workplace	Small-group work. Assignments over 12 weeks.
Home examination: A written paper (10 pages) and oral presentation 1. Describe the steps and process of EBP 2. Choose a clinical question, search for evidence, critically appraise evidence and address findings to your practice 3. Describe a strategy for implementing EBP at your workplace				Assignment over 18 weeks. Tutorial over 14 weeks. Oral presentation at workplace.

To sustain knowledge and skills obtained at the postgraduate program, the students were encouraged to participate in evidence-based network groups while working in clinical practice. In these network groups, which may have been pre-existing or established during follow up, healthcare professionals critically appraised articles in journal clubs, produced clinical guidelines in accordance with the Appraisal of Guidelines Research and Evaluation (AGREE) tool in clinical guideline groups or gained new knowledge by following the steps of EBP in ‘clinical circles in EBP’.

This postgraduate module was worth 15 ECTS credits. Across Europe, The European Credit Transfer and accumulation System (ECTS) is the credit system used in the European Higher Education Area.¹⁷ It is a tool that helps to design, describe, and deliver study programs and award higher education qualifications. In this system, one full-time academic year corresponds to 60 ECTS credits, and one ECTS credit corresponds to 25–30 hours of work.¹⁷

Outcome measures

We used the Norwegian version of the Evidence-Based Practice Beliefs Scale (EBP Beliefs Scale) and the Evidence-Based Practice Implementation Scale (EBP Implementation Scale). For the post measurement, we also included a questionnaire examining demographic data and participation in evidence-based network groups. The EBP Beliefs Scale and the EBP Implementation Scale were translated into Norwegian in accordance with the World Health Organization's principles of forward and backward translation.¹⁸ Nina Rydland Olsen at Bergen University College performed the translation, in collaboration with the authors of the original scales (not published; available from author). For the original scales in English, construct and criterion validity has been established through factor analysis,¹⁹ and good or excellent Cronbach's alpha values have been consistently reported.^19–25 In this study, post-course values of Cronbach's alpha were 0.86 for the EBP Beliefs Scale and 0.88 for the EBP Implementation Scale.

The EBP Beliefs Scale consists of 16 items and measures the respondents’ self-reported beliefs about the value of EBP and their confidence in EBP knowledge and skills. Respondents indicate their agreement with the items on a five-point Likert scale ranging from 1 (‘strongly disagree’) to 5 (‘strongly agree’). The total score for the scale ranges between 16 and 80, with high scores indicating positive beliefs.

The EBP Implementation Scale consists of 18 items and measures how often the respondents engage in behavior relevant to EBP. On a five-point frequency scale, where 0 = ‘0 times’, 1 = ‘1–3 times’, 2 = ‘4–6 times’, 3 = ‘6–8 times’ and 4 = ‘ > 8 times’, the respondents report how often they have performed specific activities related to EBP within the past eight weeks. The total score for the scale ranges between 0 and 72, with high scores reflecting a more frequent use of EBP behaviors and skills.

Analysis

Random checks were performed to ensure accurate data entry. Two negatively phrased items in the EBP Beliefs Scale were reversed before analysis. Mean substitution was used for the scales if the respondent answered more than 80% of the questions.

Descriptive statistics were used to summarize baseline characteristics. We used paired sample t-tests to compare mean scores on the scales before and one year after the postgraduate program. To adjust for age, years in the present job and participation in evidence-based network groups we used generalized estimating equations (GEE) regression analysis.²⁶ An unstructured working correlation structure was applied in the GEE analysis to adjust for within-participant correlation.

The Wilcoxon signed-rank test was used to test changes in pre and post measurements for each item of the scales. Further, we calculated item-specific percentages for study participants who reported higher, equivalent or lower scores, comparing the scores one year after the postgraduate program with the scores at baseline.

Analysis of variance (ANOVA) with adjustment for baseline values was performed to compare mean scores on the scales at one year for those who participated and did not participate in evidence-based network groups after the program. Estimated mean values are presented. P-values less than 0.05 indicated statistical significance. IBM SPSS Statistics version 20 was used for the statistical analysis.

Ethics statement

The survey was voluntary, and consent for participation was given through completion and return of the questionnaires. We treated the data confidentially. The Regional Committee for Medical and Health Research Ethics (REC), Western Norway, exempted this study from review, because the study did not include medical or biomedical aspects. The Norwegian Social Science Data Services (NSD) approved the study (reference number 22883).

Results

Of the 158 eligible participants, 120 (76%) respondents answered the questionnaires before and one year after the postgraduate program. Most of the respondents were nurses (77%), held a postgraduate degree as the highest educational level (64%), worked at a university hospital (70%), and did not participate in evidence-based network groups (60%). The mean age was 42.8 years (range 27–66 years), and the respondents had worked in their present job for a mean time of 9.9 years (range 0–36 years) (Table 2).

Table 2.

Background variables for participants (n = 120).

Background variables	n (%)	Mean (SD)
Sex
Male	15 (13)
Female	105 (88)
Healthcare profession
Nurse	92 (77)
Physiotherapist	5 (4)
Radiographer	6 (5)
Occupational therapist	5 (4)
Social worker	3 (3)
Social educator	3 (3)
Other	6 (5)
Highest educational level
Bachelor's degree	33 (28)
Postgraduate degree	77 (64)
Master's degree	10 (8)
Workplace
University hospital	84 (70)
Local hospital	10 (8)
Nursing home	5 (4)
Community care	3 (3)
University college	6 (5)
Other workplace	12 (10)
Position*
Clinical professional	49 (41)
Professional developer	27 (23)
Specialist	23 (19)
Teacher at university college	5 (4)
Manager	18 (15)
Other position	19 (16)
Participation in evidence-based network groups
Yes	44 (37)
No	72 (60)
Missing	4 (3)
Age		42.8 (9.4)
Years in present job		9.9 (8.1)

Some held more than one position.

Changes in EBP belief scores

The mean beliefs score differed before (mean (M) = 54.5) and one year after (M = 60.6) the postgraduate program (mean difference (MD) = 6.0) (Table 3). This difference persisted after adjustment for age, years in present job and participation in evidence-based network groups.

Table 3.

Mean differences for the EBP Beliefs Scale and the EBP Implementation Scale.

	n	Mean (SD)	Mean diff.	95% CI	p
EBP Beliefs Scale
Before the program	119	54.5 (5.5)
One year after the program	119	60.6 (6.8)	6.0	4.9–7.2	<0.001
EBP Implementation Scale
Before the program	99	8.9 (10.0)
One year after the program	99	11.7 (9.7)	2.8	0.5–5.0	0.02

Note: Analyzed with paired t-test.

For 11 of the 16 items on the EBP Beliefs Scale, the result indicated a statistically significant positive change in scores reported before and one year after the postgraduate program (Figure 1). The largest percentage of respondents with improved scores was for the items ‘I am clear about the steps of EBP’ (88%), ‘I know how to implement EBP sufficiently enough to make practice changes’ (68%) and ‘I am confident about my ability to implement EBP where I work’ (67%). More than 80% of the participants answered ‘agree’ or ‘strongly agree’ before the program started for five items (items 10, 11, 14–16). For the remaining items, less than 50% answered ‘agree’ or ‘strongly agree’ before the program started.

Figure 1.

Changes in scores (%) on the EBP Beliefs Scale one year after the postgraduate education.

Changes in EBP implementation scores

The mean overall implementation score differed before (M = 8.9) and one year after (M = 11.7) the postgraduate program (MD = 2.8) (Table 3). This difference persisted after the same adjustment as for the beliefs scores. For eight of the 18 items on the EBP Implementation Scale, the result indicated a statistically significant positive change in scores reported before and one year after the postgraduate program (Figure 2). The largest percentage of respondents with improved scores was for the item ‘Promoted the use of EBP to my colleagues’ (47%). In addition, 40% reported higher scores for the items ‘Accessed the Cochrane database of systematic reviews’ and ‘Shared an EBP guideline with a colleague’, while 39% improved their scores for critical appraisal of research studies.

Figure 2.

Changes in scores (%) on the EBP Implementation Scale one year after the postgraduate education.

Evidence-based practice implementation and evidence-based network groups

Figure 3 shows mean values for the EBP Beliefs Scale and the EBP Implementation Scale pre and post measurements among those participating or not participating in evidence-based network groups. Healthcare professionals (N = 44) who participated in evidence-based network groups reported higher scores one year after the program than healthcare professionals (N = 72) who did not participate in these groups. This was statistically significant for the EBP Implementation Scale (M_{NP(non participants)} = 9.2 and M_{P(participants)} = 16.2, p = 0.001), but not for the EBP Beliefs Scale (M_NP = 60.1 and M_P = 61.4, p = 0.3).

Figure 3.

Pre and post mean scores for the EBP Beliefs Scale and the EBP Implementation Scale among those participating or not participating in evidence-based network groups.

Discussion

This study showed a positive change in healthcare professionals' EBP beliefs and EBP implementation when scores before and one year after participation in a 15-ECTS-credit (total of 42 hours) postgraduate EBP program were compared. Improvement was strongest in EBP beliefs. Furthermore, healthcare professionals who participated in evidence-based network groups at follow up performed more activities related to EBP compared with healthcare professionals who did not participate in these groups.

Comparing our results with previous follow-up studies using the EBP Beliefs Scale and the EBP Implementation Scale is challenging, as the studies differ in interventions, samples, follow up and context. In previous studies, the education interventions in EBP have been part of a strategic plan.^{21,22,24,25,27} The samples have been nurses targeted for EBP mentor programs,^21,25 nurses chosen to become EBP champions,²⁴ nurses regardless of job roles partaking in an implementation program,^22,27 or health professionals attending a workshop in EBP.²⁸ In addition, most follow ups have been performed in continuation of an organized strategic EBP implementation, rather than one year after as in our study. Only one study measured EBP implementation six months after an intensive educational workshop.²⁸

The result from our study revealed a statistically significant positive change in EBP beliefs one year after the postgraduate program. Previous studies have also reported an increase in mean EBP beliefs score,^{21,22,24,25,27} but only three studies explicitly reported a statistically significant increase in EBP beliefs after an educational intervention.^21,24,27 Our observed mean difference in EBP beliefs at one year follow up was similar to changes reported in studies educating EBP mentors and champions over a similar timeframe.^21,24,25 Furthermore, our post measurement total mean score was similar to the total mean scores reported in studies where the educational program was part of an organized intervention for implementing EBP in clinical practice.^21,24,25,27 Compared to previous studies we found our participants to hold strong beliefs in EBP.

Our result also revealed a statistically significant change in implementation score after the postgraduate program. There are studies reporting larger increase in mean EBP implementation sores,^21,22,24,25 but only two studies explicitly reported a statistically significant increase in EBP implementation.^21,24 Recent studies assessing the EBP Implementation Scale have not obtained significant changes in EBP implementation after an educational intervention.^27,28 We observed a smaller total mean EBP implementation score than the studies on educating EBP mentors.^21,24,25 Only one previous study has reported an equivalent implementation score to ours after a teaching intervention.²²

It is not surprising that the EBP implementation in our study was lower than the scores reported in studies where participants were supported in mentorship programs. Our participants came from different workplaces, and likely their barriers for implementing EBP in clinical practice varied. Barriers frequently reported for implementing EBP are related to organizational issues, such as lack of staff experienced in EBP, supportive leadership, lack of resources and lack of time to read research.^29–31 Context and organizational culture are essential for EBP implementation, and practitioners' use of research will be encouraged in cultures where particular ideas, activities, or events are more highly valued than others.³² Thus, implementing EBP may require contextual system changes, involving changes at the level of the organization, team, and individuals.³³ Our postgraduate program provided individualized training in EBP and was not part of an organized intervention for implementing EBP in clinical practice. As sole representatives, our participants faced a challenging task implementing EBP in practice.

In our study, healthcare professionals who participated in evidence-based network groups implemented EBP more often than the healthcare professionals who did not attend these network groups. These findings are consistent with systematic reviews indicating that postgraduate teaching in EBP that is clinically integrated in practice may improve clinicians’ EBP implementation.^6,34 In evidence-based network groups participants meet regularly to critically appraise articles, produce guidelines or gain new knowledge by following the steps of EBP, as in journal clubs, clinical guideline groups or ‘clinical circles in EBP’. Even though research on journal clubs is limited and mainly based on self-reported questionnaires, Honey and Baker claim there is growing evidence indicating that journal clubs may increase clinicians’ knowledge and confidence such that clinicians include new skills and interventions in clinical practice.³⁵ However, another systematic review could not conclude that journal clubs were effective as decision support in clinical practice.³⁶ Thus, obtaining solid knowledge of how evidence-based network groups may change EBP beliefs and EBP implementation over time requires further studies.

Our postgraduate program assessed various combinations of learning strategies, such as lectures, computer lab sessions, appraisal of various study types, small-group discussions, use of real clinical issues, and assessments over 18 weeks. The educational program was not integrated into clinical practice, as recommended in a recent review of reviews about teaching evidence-based healthcare.³⁴ Nonetheless, the participants were working in clinical practice, paid by their employer to take part in the program and encouraged to carry out discussions, assignments and papers related to their own clinical practice. Although not clinically integrated, our teaching and learning strategies are in accordance with a recent overview of systematic reviews, which found that multifaceted, clinically integrated interventions, with assessment, led to improvements in knowledge, skills and attitudes.³⁴

The postgraduate program aspired to teach all the steps of EBP to a broad range of healthcare professionals. Nevertheless, the curriculum devoted five days to asking clinical questions, finding evidence and critical appraisal, whereas only one day was dedicated to implementation of EBP and shared decision-making. This may explain the smaller change for items on the EBP Implementation Scale related to evaluation of practice. Developing individuals' skills in critical appraisal will not automatically lead to greater evidence use.³³

We used the EBP Beliefs Scale and the EBP Implementation Scale because they were found to be valid and reliable in previous studies.¹⁹ A possible limitation of this study is that the EBP Beliefs Scale and the EBP Implementation Scales have not been validated for healthcare professionals in Norway. In addition to the shortcomings of any self-report scale, we question a possible ceiling effect of some of the items in the EBP Implementation Scale. For example, we question how realistic it is to have participants score eight or more times that they have used an EBP guideline or systematic review to change clinical practice in the past eight weeks.

As for pre- and post-test survey design studies in general, changes may occur as a function of time. In our study, it is possible that the participants over-reported engagement in EBP behavior at baseline. They may have perceived that they were engaged in EBP activities at baseline, but through the postgraduate program gained a better understanding of what is required in EBP implementation. Similar observation has been reported in a previous study.²⁸

We did not have a control group and the effect of the program should be interpreted with caution. We lacked the information to analyze non-responders and dropouts. However, the study had a high response rate. By including six cohorts going through the program at different periods and performing the second measure after one year, we have examined changes in healthcare professionals' EBP beliefs and EBP implementation over a longer period of time than previous studies.

Conclusion

This study examined changes in EBP beliefs and EBP implementation one year after participation in a postgraduate educational program in EBP. The result showed a positive change in EBP beliefs and EBP implementation when scores before and one year after participation in the program were compared. The change was strongest in EBP beliefs. In addition, healthcare professionals who participated in evidence-based network groups at follow up performed more activities related to EBP compared with healthcare professionals who did not participate in these groups. Further research is required to examine factors that may facilitate implementation of EBP among healthcare workers attending postgraduate programs in EBP, and to consider changes in skills and behavior after such educational interventions.

Footnotes

Author contributions

Anne Kristin Snibsøer led the development of the manuscript, including collecting and analyzing data. Birgitte Espehaug participated in analyzing and interpreting data. All authors were involved in drafting the manuscript and critically revising the text. All authors read and approved the final manuscript.

Funding

This research received no specific grant from any agency in the public, commercial, or not-for-profit sectors.

Conflict of interest

The authors declare that there is no conflict of interest.

References

Dawes

Summerskill

Glasziou

et al.

Sicily statement on evidence-based practice. BMC Med Educ 2005; 5: 1.

Straus

Green

Bell

et al.

Evaluating the teaching of evidence based medicine: conceptual framework. BMJ 2004; 329: 1029–1032.

Straus

. Evidence-based medicine: how to practice and teach it, 4th ed. Edinburgh: Churchill Livingstone/Elsevier, 2011, pp. 293, p.XVII.

Tilson

Kaplan

Harris

et al.

Sicily statement on classification and development of evidence-based practice learning assessment tools. BMC Med Educ 2011; 11: 78.

Khan

Coomarasamy

. A hierarchy of effective teaching and learning to acquire competence in evidenced-based medicine. BMC Med Educ 2006; 6: 59.

Coomarasamy

Khan

. What is the evidence that postgraduate teaching in evidence based medicine changes anything? A systematic review. BMJ 2004; 329: 1017.

Coomarasamy

Taylor

Khan

. A systematic review of postgraduate teaching in evidence-based medicine and critical appraisal. Med Teach 2003; 25: 77–81.

Dizon

Grimmer-Somers

Kumar

. Current evidence on evidence-based practice training in allied health: a systematic review of the literature. Int J Evid Based Healthc 2012; 10: 347–360.

Flores-Mateo

Argimon

. Evidence based practice in postgraduate healthcare education: a systematic review. BMC Health Serv Res 2007; 7: 119.

10.

Horsley

Hyde

Santesso

et al.

Teaching critical appraisal skills in healthcare settings. Cochrane Database Syst Rev 2011, pp. CD001270.

11.

Taylor

Reeves

Ewings

. A systematic review of the effectiveness of critical appraisal skills training for clinicians. Med Educ 2000; 34(2): 120–125. .

12.

Shaneyfelt

Baum

Bell

et al.

Instruments for evaluating education in evidence-based practice: a systematic review. JAMA 2006; 296: 1116–1127.

13.

Bergen University College. To teach and work evidence-based: curriculum [Å arbeide og undervise kunnskapsbasert: fagplan], Bergen: Bergen University College, 2012.

14.

Nortvedt

Jamtvedt

Graverholt

et al.

Jobb kunnskapsbasert! En arbeidsbok, 2nd ed. Oslo: Akribe, 2012, pp. 216.

15.

Knowles

. The adult learner: a neglected species 1990, 4th ed. Houston, TX: Gulf Publishingp.x, p.293.

16.

CASP UK. Critical Appraisal Skills Programme: making sense of evidence. http://www.casp-uk.net/ (2012, accessed 9 November 2012).

17.

European Comission. ECTS users' guide, Luxembourg: Office for Official Publications of the European Communities, 2009, pp. 60.

18.

World Health Organization. Process of translation and adaptation of instruments, Geneva: World Health Organization, 2012. .

19.

Melnyk

Fineout-Overholt

Mays

. The evidence-based practice beliefs and implementation scales: psychometric properties of two new instruments. Worldviews Evid Based Nurs 2008; 5: 208–216.

20.

Estrada

. Exploring perceptions of a learning organization by RNs and relationship to EBP beliefs and implementation in the acute care setting. Worldviews Evid Based Nurs 2009; 6: 200–209.

21.

Levin

Fineout-Overholt

Melnyk

et al.

Fostering evidence-based practice to improve nurse and cost outcomes in a community health setting: a pilot test of the advancing research and clinical practice through close collaboration model. Nurs Adm Q 2011; 35: 21–33.

22.

Melnyk

Bullock

McGrath

et al.

Translating the evidence-based NICU COPE Program for parents of premature infants into clinical practice: impact on nurses' evidence-based practice and lessons learned. J Perinat Neonatal Nurs 2010; 24: 74–80.

23.

Melnyk

Fineout-Overholt

Giggleman

et al.

Correlates among cognitive beliefs, EBP implementation, organizational culture, cohesion and job satisfaction in evidence-based practice mentors from a community hospital system. Nurs Outlook 2010; 58: 301–308.

24.

Varnell

Haas

Duke

et al.

Effect of an educational intervention on attitudes toward and implementation of evidence-based practice. Worldviews Evid Based Nurs 2008; 5: 172–181.

25.

Wallen

Mitchell

Melnyk

et al.

Implementing evidence-based practice: effectiveness of a structured multifaceted mentorship programme. J Adv Nurs 2010; 66: 2761–2771.

26.

Liang

K-Y

Zeger

. Longitudinal data analysis using generalized linear models. Biometrika 1986; 73: 13–22.

27.

Hauck

Winsett

Kuric

. Leadership facilitation strategies to establish evidence-based practice in an acute care hospital. J Adv Nurs 2013; 69: 664–674.

28.

Yost

Ciliska

Dobbins

. Evaluating the impact of an intensive education workshop on evidence-informed decision making knowledge, skills, and behaviours: a mixed methods study. BMC Med Educ 2014; 14: 13.

29.

Kajermo

Bostrom

Thompson

et al.

The BARRIERS scale – the barriers to research utilization scale: a systematic review. Implement Sci 2010; 5: 32.

30.

Meijers

Janssen

Cummings

et al.

Assessing the relationships between contextual factors and research utilization in nursing: systematic literature review. J Adv Nurs 2006; 55: 622–635.

31.

Solomons

Spross

. Evidence-based practice barriers and facilitators from a continuous quality improvement perspective: an integrative review. J Nurs Manag 2011; 19: 109–120.

32.

Scott-Findlay

Golden-Biddle

. Understanding how organizational culture shapes research use. J Nurs Adm 2005; 35: 359–365.

33.

Rycroft-Malone

. Evidence-informed practice: from individual to context. J Nurs Manag 2008; 16: 404–408.

34.

Young

Rohwer

Volmink

et al.

What are the effects of teaching evidence-based health care (EBHC)? Overview of systematic reviews. PLoS One 2014; 9: e86706.

35.

Honey

Baker

. Exploring the impact of journal clubs: a systematic review. Nurse Educ Today 2011; 31: 825–831.

36.

Harris

Kearley

Heneghan

et al.

Are journal clubs effective in supporting evidence-based decision making? A systematic review. BEME Guide No. 16. Med Teach 2011; 33: 9–23.