Features and effectiveness of electronic audit and feedback for patient safety and quality of care in hospitals: A systematic review

Abstract

Background: Increasing digitisation in healthcare is flowing through to quality improvement strategies, like audit and feedback. Objectives: To systematically review electronic audit and feedback (e-A&F) interventions in hospital settings, examining contemporary practices and quantitatively assessing the relationship between features and effectiveness. Methods: We performed a systematic review using a structured search strategy from 2011 to July 2022. Searches yielded a total of 5095 unique publications, with 152 included in a descriptive synthesis, reporting publication characteristics and practices, and 63 in the quantitative synthesis, to evaluate the effect size of intervention features. Results: The search returned publications across characteristics, including countries of origin, feedback topics, target health professionals, and study design types. We also identified an association with effectiveness for all but one of the features examined, with a Cohen’s d ranging from above +0.8 (a large positive effect), to −0.67 (a medium negative effect). Socio-technical features related to supportive organisations and the involvement of engaged health professionals were most associated with effective interventions. Conclusion: Key findings have confirmed that a common set of features of e-A&F systems can influence effectiveness. Results provide practitioners with insight into where resources should be focused during the implementation of e-A&F.

Keywords

electronic audit and feedback hospitals safety and quality systematic review

Introduction

Audit and feedback is a widely utilised quality improvement strategy to improve professional practice in healthcare.¹ Fundamentally, audit and feedback involves informing health professionals that their clinical practice is inconsistent with a desired target, to prompt change.¹ Despite audit and feedback being widely utilised, a Cochrane review concluded considerable variability exists in its effectiveness, with sizeable interquartile range of 0.5 to 16.0% reported.¹

These findings suggest that by examining and emulating the features of those more effective interventions, a greater level of clinically significant improvements could be achieved. To date, evidence-based guidelines to support successful implementation of audit and feedback in healthcare have largely been lacking.^2,3 To address this gap, recent feedback theories have been developed from findings of qualitative research.⁴ These theories suggest that effective feedback works in a cycle over time. This cycle is influenced by variables related to the feedback itself, the recipient and the wider context.⁴

At the same time, the digitisation of healthcare is increasing the availability of health data in an electronic format, which in turn has the potential for use for electronic Audit and Feedback (e-A&F).⁵ By extension, there is evidence of this increasing integration of ICT into healthcare quality improvement strategies.⁶ When health Information and Communication Technology (ICT) is used in conjunction with audit and feedback processes within clinical settings, it is commonly referred to as e-A&F.^5,7 In this paper, we use the term e-A&F broadly to encompass all approaches which provide an electronic “summary of clinical performance of health care over a specific period of time to inform quality improvement actions”.¹

The potential benefits of the use of ICT for audit and feedback may mirror the greater adoption of digitisation in healthcare, including improving workflows, real time access and greater efficiency.⁸ However, it has been established that the use of a computer or being online can influence behaviour.⁹ Therefore, despite the potential benefits there could be expected to be differences in practice as compared to traditional audit and feedback.

Some research has commenced examining e-A&F across the healthcare sector. Key previous reviews have considered the healthcare sector more broadly, including aged, primary and hospital care settings.^5,6,10 However, feedback in relation to a hospital setting requires specific considerations. Previous reviews have also identified studies to be highly diverse.^5,6,10 So, efforts to develop evidence-based guidelines from standard systematic review methodology, such as meta-analysis, have been unreliable.^5,6,10 Thus far, the only approach has been via narrative synthesis of the literature.^5,6,10

As more than a decade has passed since the seminal Cochrane findings on audit and feedback, it is appropriate for a quantitative systematic review to be undertaken that focuses on the emergent literature. This includes literature with a particular focus on e-A&F in a hospital setting.¹ To achieve this aim, the following objectives will be pursued:

1. Scoping the existing research of contemporary audit and feedback systems for patient safety and quality in hospitals.

2. Quantitatively examining the features of these systems to determine if there is a link between the nature and pattern of these features and the effectiveness of systems.

Methods

We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement for reporting our systematic review.¹¹ Our approach was published on the International Prospective Register of Systematic Reviews (PROSPERO; registration number CRD42020167008).¹²

Search and selection

To create our search strategy, we examined relevant systematic reviews, original research publications on audit and feedback, and gained input from a librarian.^1,5 We searched MEDLINE (Ovid), EMBASE (Ovid), CINAHL (Ebsco), and the Cochrane Central Register of Controlled Trials (CENTRAL) from 2011 to December 2019 and updated to July 2022. Available tools were used to translate the original Ovid-MEDLINE search syntax (see Appendix 1). Google Scholar was also used as an additional source to support a comprehensive search strategy using the same date restriction and the terms “audit” and “dashboard” and “hospital OR department OR ward”. All publications found were divided into two equal groups and independently screened on title and abstract by two pairs of reviewers using the criteria in Table 1. Agreement was reached by consensus.

Table 1.

Search criteria according to the population, intervention, comparator, outcome and study (PICOS) and typical exclusion examples.

Inclusion criteria	Typical exclusion examples
Population
The intervention primarily targeted hospital health professionals (including clinicians and non-clinicians e.g., managers)	Feedback not for hospital health professionals e.g., patients, public
Hospitals, including multi-setting healthcare services which included hospitals	Non-hospital settings e.g., residential nursing homes
Intervention
Systems that provide a summary of clinical performance of health care over a specific period of time to inform quality improvement actions	Point-of-care alert, decision-support or workflow systems; training or simulation applications; systems that provide feedback on non-clinical performance
Safety and quality performance including prevention and occurrence of harm; clinical standards and pre-defined practice; patient feedback; and quality of care	Other performance including financial efficiency
Feedback is delivered in an electronic format e.g., provided via a web-based platform, email	Feedback provided exclusively on paper or verbally
Comparator
Both comparative and non-comparative designs	Not applicable
Outcome
Qualitative and/or quantitative outcomes	Not applicable
Study
All languages, peer-reviewed and grey literature (including conference abstracts and theses)	Where clinical performance feedback systems are not a core and essential part of the intervention or studies of groups, or, reviews or collections of interventions of which characteristics are not clearly described

Full manuscripts of relevant papers were obtained by one reviewer, double checked by a second reviewer, with the inclusion criteria re-applied. Citation and reference list and related article searching were undertaken for all full publications meeting the inclusion criteria using the Web of Science. Reasons for deeming full publications to be irrelevant were recorded. During this step, to meet the first objective, basic characteristics of publications were extracted, including publication year, country of origin of study, feedback topic, targeted health professional category and publication type / study design. Where these characteristics were not stated, these were recorded as “unknown”.

Data extraction and analysis

Data extraction was undertaken by one reviewer and checked by a second reviewer. Agreement was reached by consensus. For effect size study publications, e-A&F systems were assessed for explicit evidence against 10 features. These features were derived from those identified in the Clinical Performance Feedback Intervention Theory (CP-FIT) published by Brown et al. 2019.⁴ Specific definitions and examples of strong and weak evidence are included in Table 2. Detail on the criteria for features of audit and feedback are reported elsewhere.⁴ Effect size study publications were defined as those that included an estimate of effect size. Where applicable, such as in a head-to-head trial, features of the intervention arm were extracted.⁴

Table 2.

Effect size study publication e-A&F system examples of strong and weak features.

Feature	Definition	Strong example	Weak example
Goals	The focus is on goals that recipients believe to be meaningful, within their control and relevant to their jobs	Indicators were developed in close collaboration with affected staff¹³	Staff provided feedback that certain metrics could not be ‘fixed’ solely by an individual provider¹⁴
Data collection	Data are collected accurately and in a way that reduces burden on recipients (e.g., automatically)	Remote video auditing using remote third-party human auditors¹⁵	Data were entered as an entry checklist, manually¹⁶
Feedback display	Feedback communicates recipient’s current performance in a user-friendly way with context, including trends, benchmarks, and patient lists	A clear comparison of peers’ monthly and overall year-to-date performance, current and historic performance, and ability to view details about each suboptimal VTE prophylaxis order¹⁷	Performance feedback was limited to trends of the number of central venous catheters inserted, compliance with the insertion and maintenance bundle and catheter-related bloodstream infection rate¹⁸
Feedback delivery	Feedback is pushed to groups of recipients to support positive change	Feedback displayed graphically by team and was sent by email to the resident and attending providers on service¹⁹	The “provider dashboard” was available within a tab of the electronic medical record for staff to access²⁰
Health professionals	Health professionals are targeted with positive beliefs about feedback and greater capability for quality improvement for the clinical topic in focus	Practical skills sessions for quality improvement²¹	Concerns of using data for performance management²²
Behaviour	Change is undertaken for system-wide rather than for individual patient and local level change	Feedback topics covered multiple domains (readmissions, length of stay, consumer experience/satisfaction, discharge timeliness, attendance at care coordination rounds) and were hospital-wide¹⁴	Change was undertaken for anaesthetic drug safety within a single clinical department²³
Organisations	Supportive organisations are characterised by resources, leadership, teamwork and networks. Interventions fit alongside existing ways of working	The organisation established an internal working party, published a locally tailored bronchiolitis guideline, joined a national bronchiolitis quality improvement collaborative. Leadership viewed the intervention as instrumental²⁴	Use of the novel electronic hand hygiene monitoring wearable device, an ergonomic wristband made from medical silicone, interfered with clinical activities²⁵
Patient populations	They use a risk adjustment or do not include patients who cannot be optimised further	Data were displayed graphically as risk-adjusted rates and counts of severe hypoglycemia events at the system, hospital, and nursing unit levels²⁵	Complying with the policy was possible for simple cases, but consideration not made for complex clinical scenarios or specialty-specific cases²⁶
Co-interventions	They help recipients to identify and develop solutions for suboptimal performance	An action implementation toolbox was pre-filled with a list of potential barriers and suggested actions with supporting material to overcome these barriers²⁷	Challenges were reported in developing action plans because of constraints of the system²⁸
Implementation	They are implemented with training, tailored to the needs of the organisation and considered inexpensive to deploy	The hand hygiene observation tool is free and can easily be adopted (and adapted) to any health care setting²⁹	Did not engage in educational efforts outside of the dashboard³⁰

To meet the second objective, to examine the association with effectiveness, these features were classified using a 0/1/2 scoring system. This was defined as ‘2 = Strong’ (substantive evidence that it met the definition of the feature), ‘0 = Weak’ (substantive evidence that it did not meet definition of the feature), and ‘1 = Neutral’ (either insufficient evidence to assess, or mixed weak and strong evidence).^5,31 The primary outcomes were also coded as either positive, coded = P, (i.e., the outcome supported the hypothesis) or mixed/negative, coded = N, (i.e., the outcome did not support the hypothesis or there were mixed results). The mean score differences for positive and mixed/negative were compared to produce Cohen’s d effect sizes for each feature. These were interpreted using conventional thresholds – small (0.20), medium (0.50) and large (0.80).^32,33

Risk of bias

Risk of bias for individual studies was assessed using the Mixed Methods Appraisal Tool (MMAT) by one review author and checked by a second reviewer for each criterion.^34–36 Agreement was reached by consensus. In addition to the risk of bias at the individual study level, there exists the potential of a risk of bias across studies. The chosen methodology was expected to reduce the risk of the effect of positive publication bias. The methodology employed by other systematic reviews involves the aggregation of effect sizes across published studies to produce estimated pooled averages. Therefore, any presence of positive publication bias will ultimately bias the estimated pool average.³⁷ Our approach was to group and compare publications by the features of their interventions. We argue there is no clear reason why the features of an e-A&F intervention would systematically bias whether it was published. In addition, we chose to include grey literature and utilise a comprehensive search strategy. This broad scope would help in reducing the bias towards publishing positive outcomes only.

Results

The electronic bibliographic databases and additional searches (Google Scholar, citation and reference list searches) yielded an initial total of 5055 unique publications, of which 151 met the criteria for full text screening. After full review, 39 studies were excluded as a result of not meeting the eligibility criteria. Following the updated search in July 2022, a total of 152 publications were included in the descriptive synthesis, with 63 of these also in the effect size synthesis (Figure 1).

Figure 1.

Flow chart of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) protocol. A total of 6018 were identified from database search and 31 from other sources. After removing duplicates 5055. Title and abstracts of 5055 were screened. Of these, 151 remained and were full-text assessed for eligibility. Of these, 112 were included in the descriptive synthesis. Following a database update search in July 2022 152 were included in the descriptive synthesis and 63 in the effect size synthesis.

Table 3 describes the frequency of key characteristics of all included publications, with the sub-group frequency of effect size study publications also denoted. Full list of publications included are detailed in Appendix 2.

Table 3.

Current state of e-A&F, number (%) of publications by characteristics.

Characteristics		Number (%) of publications (n = 152)	Number (%) of effect size study publications (n = 63)
Publication year	2011	6 (3.9)	2 (3.2)
	2012	6 (3.9)	3 (4.8)
	2013	11 (7.2)	4 (6.3)
	2014	10 (6.6)	1 (1.6)
	2015	12 (7.9)	8 (12.7)
	2016	16 (10.5)	6 (9.5)
	2017	20 (13.2)	9 (14.3)
	2018	11 (7.2)	6 (9.5)
	2019	21 (13.8)	7 (11.1)
	2020	13 (8.6)	3 (4.8)
	2021	17 (11.2)	9 (14.3)
	2022	9 (5.9)	5 (7.9)
Country	USA	76 (50.0)	40 (63.5)
	United Kingdom	19 (12.5)	4 (6.3)
	The Netherlands	18 (11.8)	3 (4.8)
	Canada	11 (7.2)	4 (6.3)
	Australia	10 (6.6)	3 (4.8)
	Europe - other	8 (5.3)	7 (11.1)
	Asia - other	3 (2.0)	0 (0.0)
	Africa - other	2 (1.3)	2 (3.2)
	South America - other	2 (1.3)	0 (0.0)
	Middle east - other	2 (1.3)	0 (0.0)
	Multi-country	1 (0.7)	0 (0.0)
Feedback topic	Anaesthesia and surgery	21 (13.8)	6 (9.5)
	Critical care and pain	20 (13.2)	10 (15.9)
	Infection control	19 (12.5)	14 (22.2)
	Cardiovascular	19 (12.5)	3 (4.8)
	Medications (incl. antimicrobials) and venous thromboembolism	14 (9.2)	10 (15.9)
	More than one	14 (9.2)	1 (1.6)
	Maternity and gynaecology	12 (7.9)	2 (3.2)
	Clinical communication and patient flow	8 (5.3)	7 (11.1)
	Other	7 (4.6)	3 (4.8)
	Cancer, blood and palliative	6 (3.9)	4 (6.3)
	Geriatric, rheumatology and falls	5 (3.3)	2 (3.2)
	Investigations/ results	4 (2.6)	0 (0.0)
	Incidents/ adverse events	3 (2.0)	1 (1.6)
Health professional category	Multi-disciplinary	63 (41.4)	28 (44.4)
	Doctors	45 (29.6)	21 (33.3)
	Nurses	15 (9.9)	4 (6.3)
	Leadership and management	3 (2.0)	1 (1.6)
	Pharmacists	1 (0.7)	1 (1.6)
	Unknown	25 (16.4)	8 (12.7)
Publication type/study design	Descriptive	43 (28.3)	0 (0.0)
	Quantitative descriptive study	15 (9.9)	0 (0.0)
	Protocol	12 (7.9)	0 (0.0)
	Qualitative studies	10 (6.6)	0 (0.0)
	Non-randomised study	48 (31.6)	46 (73.0)
	Mixed methods	13 (8.6)	6 (9.5)
	Randomised-controlled trial	11 (7.2)	11 (17.5)

From 2011 to 2021, the final full year of the study period, the number of publications increased annually, from six to 17, representing 3.9% and 11.2% of the total, respectively. During the same period, effect size study publications also increased annually, from two to 9, representing 3.2% and 14.3% of effect size studies, respectively. Taking into consideration the final year of the study included up to June, the number of publications overall was found to increase year-on-year, equating to an average change of 1.1.

For publications overall, close to half were found to originate from the United States (n = 76, 50.0%), followed by the United Kingdom (n = 19, 12.5%) and the Netherlands (n = 18, 11.8%). Countries of origin with small frequencies overall were grouped by geographic regions. One international study publication was categorised as having a Multi-country (n = 1, 0.7%) origin. Similarly, when effect size study publications were examined, the United States (n = 40, 63.5%) was the largest country of origin however this was followed by Europe - other (n = 7, 11.1%), United Kingdom (n = 4, 6.3%), and Canada (n = 4, 6.3%).

Categories were used to group the feedback topics of interventions into specific clinical processes, such infection control, or broad clinical specialties, such as cardiovascular. The largest feedback topics for publications overall were Anaesthesia and surgery (n = 21, 13.8%), Critical care and pain (n = 20, 13.2%), and Infection control (n = 19, 12.5%). Where an intervention covered more than a single feedback topic, this was categorised as More than one (n = 14, 9.2%). There was some overlap between the breakdown of the feedback categories overall and for effect size studies. For effect size studies, the largest feedback topics were Infection control (n = 14, 22.2%), Critical care and pain (n = 10, 15.9%), and Medications (including antimicrobials) and venous thromboembolism (n = 10, 15.9%).

Trends of health professional group targets of feedback were broadly consistent for publications overall and for effect size studies. Where feedback was specifically provided to a single professional group, this was most often directed to Doctors (n = 45, 29.6%). Single health professional groups associated with fewer interventions included Nurses (n = 15, 9.9%) and Pharmacists (n = 1, 0.7%). The largest cohort of interventions provided feedback to Multi-disciplinary health professional teams (n = 25, 16.4%). Other non-specific health professional groups targeted included Senior leaders and managers (n = 3, 2.0%), including administrative management. A sizeable number of publications, Unknown (n = 25, 16.4%), did not specify the health professionals who were the target recipients of e-A&F feedback.

Following Multi-disciplinary health professional teams, Doctors were the largest targets of interventions. Doctors were found to be disproportionally associated with a number of feedback topics. The proportion of Anaesthesia and surgery interventions which targeted Doctors (n = 13, 61.9%) was substantially higher than those that targeted Multi-disciplinary teams (n = 7, 33.3%). These differences also applied to Medications (including antimicrobials) and venous thromboembolism interventions which targeted Doctors (n = 10, 71.4%), compared to Multi-disciplinary teams (n = 3, 21.4%), and Clinical communication and patient flow interventions which targeted Doctors (n = 4, 50.0%) compared to Multi-disciplinary teams (n = 2, 25.0%). These findings were broadly similar for publications overall and for effect size studies.

Descriptive publications (n = 43, 28.3%) were found to be the largest publication types / study designs overall. These were defined as those that provided an account, such as the history of implementation of an e-A&F intervention. Protocol publication types (n = 12, 7.9%) provided detailed methods of a planned future study. Other publication types/study designs were categorised using the MMAT. Qualitative publications (n = 10, 6.6%) utilised qualitative methods (e.g., interviews) to provide an account of an e-A&F intervention while quantitative descriptive publications (n = 15, 9.9%) utilised quantitative methods (e.g., surveys) to provide an assessment of an e-A&F intervention.

Remaining publication types / study designs provided a comparison between interventions/exposures and provided an assessment of the size of the effect. These used random allocation experimental design, Randomised-controlled trials (n = 11, 7.2%), those without random allocation experimental design, Non-randomised study (n = 48, 31.6%), and those with a combination of designs, Mixed methods (n = 13, 8.6%).

Risk of bias was assessed using the MMAT publication type / study design categories for effect size study publications. The majority met the screening questions, ‘S1. Clear research questions’ (n = 48, 76.2%) and ‘S2. Data address research questions’ (n = 48, 76.2%). Those that met the screening questions were appraised using the MMAT, with variation found across specific methodological quality criteria (Figure 2).

Figure 2.

Risk of bias using the Mixed Methods Assessment Tool for effect size study publications. All effect size studies were screened against the screening questions S1 and S2. Publications that met the screening questions were grouped by type / study design – randomised controlled trials, non-randomised and mixed methods and assessed against five specific methodological quality criteria. Results are displayed as frequency (%) of ratings ‘Yes’ ‘No’ and ‘Can’t Tell’.

Of the 63 total effect size study publications, 48 were coded as having positive primary outcomes and 15 were coded as having negative/mixed primary outcomes. This equated to a positive publication ratio of 3.2:1. When these groups were compared, studies with positive primary outcomes were more likely to be scored as having evidence for the assessed features, a mean score of 1.30, than those with negative/mixed primary outcomes, a mean score of 1.12 (see Table 4).

Table 4.

Comparison of association between positive and negative outcome effect size studies and evidence of intervention features.

Feature	Outcome group	Feature evidence (frequency)			Mean score	Effect size, d
Feature	Outcome group	Strong (=2)	Neutral (=1)	Weak (=0)	Mean score	Effect size, d
Organisations	Positive	29	16	3	1.54	1.01
Organisations	Negative	2	10	3	0.93
Health professionals	Positive	21	27	0	1.44	0.99
Health professionals	Negative	2	9	4	0.87
Data collection	Positive	33	8	7	1.54	0.64
Data collection	Negative	6	3	6	1.00
Implementation	Positive	20	23	5	1.31	0.47
Implementation	Negative	3	9	3	1.00
Feedback display	Positive	22	22	4	1.38	0.36
Feedback display	Negative	5	7	3	1.13
Goals	Positive	6	41	1	1.10	0.35
Goals	Negative	2	10	3	0.93
Co-interventions	Positive	20	27	1	1.40	0.13
Co-interventions	Negative	5	10	0	1.33
Patient populations	Positive	6	41	1	1.10	−0.07
Patient populations	Negative	3	11	1	1.13
Feedback delivery	Positive	24	12	12	1.25	−0.10
Feedback delivery	Negative	8	4	3	1.33
Behaviour	Positive	17	12	19	0.96	−0.67
Behaviour	Negative	11	1	3	1.53
Total	Positive	198	229	53	1.30	n/a
	Negative	47	74	29	1.12
	Total	245	303	82	1.21

Cohen’s d was found to be above 0.8 (a large effect) for ‘organisations’ and ‘health professionals’. A total of 16 of effect size study publications, or 25.4%, were found to have strong evidence for both ‘organisations’ and ‘health professionals’. Cohen’s d was found to be between 0.5 and 0.8 (a medium effect) for ‘data collection’. ‘Implementation’, ‘feedback display’ and ‘goals’ ranged between 0.35 and 0.47 (a small effect). Three features had effect sizes less than 0.2 (a negligible effect) – ‘co-interventions’, ‘feedback delivery’ and ‘patient population’.

Only the intervention feature ‘behaviour’ was found to be negatively associated with effective e-A&F, Cohen’s d between −0.5 and −0.8 (a medium effect).

Mean differences of assessed features used to generate Cohen’s d were found to vary by year of publication. Year-on-year ‘goals’, ‘health professionals’, ‘organisations’, ‘patient population’, ‘behaviour’, and ‘co-interventions’ increased on average. Out of a maximum possible of 2, these increases ranged from ‘patient population’, 0.01, to ‘co-interventions’, 0.05. Year-on-year ‘data collection’, ‘feedback delivery’ and ‘implementation’ were found to decrease on average. These ranged from ‘implementation’, −0.01, to ‘feedback delivery’, −0.03.

Two pairs of publications were found to be related to the same research study.^13,38 A sensitivity analysis was performed by removing one publication from each pair. It was found that the effect sizes of two features changed sufficiently to move them into the adjacent threshold group. These were ‘goals’ and ‘implementation’, which changed from a small to medium effect size (0.35 to 0.55, and 0.47 to 0.61, respectively).

Studies with positive primary outcomes broadly reflected the trends for effect size studies overall (see Table 3). This includes the largest category for each characteristic reflecting the findings for effect size studies generally, including year of publication, 2017 (n = 8, 16.7%), country of origin, USA (n = 40, 70.8%), feedback topic, Infection control (n = 13, 27.1%), target health professional, Multi-disciplinary teams (n = 21, 43.8%), and publication type, Non-randomised studies (n = 39, 81.3%).

Discussion

Principal findings

This review has undertaken a comprehensive search of the literature and by doing so has demonstrated the diversity of publications available on contemporary hospital e-A&F systems. These have been shown to vary across countries of origin, feedback topics, and the target health professional. While broadly consistent with publications overall, there were found to be some key differences when publications included an estimate of effect size, such as feedback topics interventions targeted. Expectedly, we identified differences in the topics of focus amongst specific target health professionals (e.g., doctors disproportionally associated with interventions related to anaesthesia and surgery).

Despite the comprehensive scope used, we found that the breakdown of the characteristics (e.g., feedback topics, target health professionals, publication types) of positive primary outcome studies broadly reflected the breakdown for effect size studies. This supports the generalisability of key results. These being that a common set of features of e-A&F systems (e.g., ‘organisations’, ‘health professionals’) influences effectiveness and ultimately leads to changes to patient care and outcomes.

Two features were found to be most associated with effective interventions. Supportive organisations, such as those who were found have to joined national quality improvement collaboratives,²⁴ were to be most associated with effective interventions. Targeting health professionals with positive beliefs about feedback and greater capability for quality improvement for the clinical topic in focus, such as those that provided practical skills sessions for quality improvement,²¹ were also highly associated with effective interventions. Only one feature was found to be negatively associated with effectiveness. Rather than those targeting system-wide change as predicted by the ‘behaviour’ feature, when an e-A&F intervention targeted individual patient and local level change, such as for anaesthetic drug safety within a single clinical department,²³ it was found to be more likely to be effective.

Far from a stagnant science, the findings also provide insight into strategies for improvement.³⁹ There was also a surprisingly low level of overlap between interventions which scored strongly amongst the features most associated with effectiveness, ‘organisations’ and ‘health professionals’. This indicates that, at the individual level, there are tangible changes which could be made to many interventions to enhance the likelihood of success.

There also appears to be changes occurring overtime, with mean differences of assessed features found to vary by year of publication. This included that, in general, interventions are increasingly being undertaken to target system-wide rather than individual patient and local level change — defined as the ‘behaviour’ feature. Alarmingly, our findings suggest that those interventions with strong evidence of ‘behaviour’, targeting system-wide change, were associated with being less effective.

Strengths and weaknesses

As anticipated, publications were found to be highly diverse. The expected positive publication bias, reported to be in the order of 3.9:1 – similar to our findings of 3.2:1, was also found.³⁷ This would have made aggregation in a typical systematic review approach, such as meta-analysis, likely unreliable.⁵ Instead, we adapted methodology to compare the features of e-A&F, on the basis of their effectiveness.^5,31 While novel, the scoring method for the features of e-A&F was based on a robust theoretical framework.⁴ It was, however, limited by the reporting within the papers and therefore could only consider explicit features, potentially omitting more complex and subjective characteristics that were less evidently reported.^1,3,4,40

To ensure the statistical power would meaningfully quantify the relative effect sizes of contemporary feedback designs, it was necessary to impose minimal inclusion criteria restrictions. This tradeoff involved the inclusion of some publications with risk of bias. While the majority of effect size publications identified (n = 48, 76.2%) met the screening questions and were appraised using the MMAT, a proportion did not subsequently meet the quality criteria including blinding of assessors, accounting for confounders, and adherence to quality criteria.

Interpretation within the context of the wider literature

Evidently this is an emerging area of study, with an average change increase of 1.1 per year. It would appear that healthcare, like other industries, is increasingly shifting to using electronic means to increase its understanding of practice and outcomes. However, to our knowledge, we are the first to quantitatively review the effect of contemporary hospital e-A&F system. It is also to our knowledge the largest systematic review on e-A&F to-date.

We found evidence in the broader literature that supported trends in the characteristics of publications, such as for the countries of origin.⁴¹ These trends of larger research volumes from specific countries may be the result of favourable local policy environments for health ICT. For example, in the United States the Federal Health Information Technology for Economic Clinical Health (HITECH) Act 2009 provided incentives for the meaningful use of electronic health records in the United States.⁴² Similarly, e-A&F in the Netherlands may benefit from access to long standing national electronic data collections.⁴³

The findings provide support for the proposition that the examined features of e-A&F systems (e.g., ‘organisations’, ‘health professionals’) are driving the difference in effectiveness of interventions found. It also supports the hypothesis that wide variability in the designs exist, and there is a relationship with variability in effectiveness.¹ Therefore, the outcomes have been broadly confirmatory of previous work in this area, while providing nuance of the drivers of effective e-A&F.^1,4,44,45

Implications for policy, practice and research

Completing this large and comprehensive systematic review is has been valuable in an environment with increasing interest in digitising healthcare.⁴¹ The findings provide guidance for audit and feedback practitioners in where to focus energy and resources during the setup and implementation of e-A&F. Broadly, the setup and implementation should follow the features of e-A&F examined in this study, in particular the socio-technical features (e.g., ‘organisations’, ‘health professionals’).

There has been a dramatic rise in access to health data and technological advances to support its use, such as business intelligence tools.⁶ Against this backdrop there has been much focus on the delivery of feedback to health professionals. However, our findings suggest that new technology, like business intelligence tools, do not on their own ensure effective e-A&F. Our findings suggest the focus of these tools should be on helping to reduce the burden of reporting and analysis (‘data collection’), or supporting timely reporting of current performance (‘feedback display’).¹⁹

The negative association with the feature of ‘behaviour’ is possibly explained by the fact that action taken by recipients to make wider-level changes are resource-prohibitive and that individuals and small teams lack the requisite capacity to engage with them. In our view, this finding does not imply that system-wide e-A&F has no place. There are instances where system-wide e-A&F translates to benefits for features such as ‘data collection’. For example, for large, high-quality standardised data collections, which minimise the burden on health professionals for reporting and analysis, in which case interventions may still be effective.³⁰

Standardised reporting on e-A&F would help future efforts to confirm the findings of this study. What was often expected to be key background information included in publications was not always reported (e.g., category of health professional being Unknown in 16.5% of publications). To facilitate future comparison, the authors suggest in exploring the adoption of common points for inclusion in e-A&F research publications, analogous to the way SQUIRE 2.0 guidelines are applied to the reporting of quality improvement.⁴⁶ In contrast to existing reporting guidelines⁴⁷ this could include specific reference to basic intervention characteristics and the 10 intervention features examined.

In addition to an informed model for success, the authors have scoped the existing research on contemporary audit and feedback systems. Those embarking on the setup and implementation of e-A&F have available to them a collection of systems to emulate across a wide variety of contexts (see Appendix 2). Within this collection are also further research opportunities. This includes a total of 28 mixed methods and quantitative descriptive studies, some of which included assessing end-user reported usability.⁴⁸ The authors suggest in combination with these existing publications, and the intervention features examined in this review, it may possible to develop a standardised usability assessment tools. This could be used by audit and feedback practitioners to self-assess the likelihood an intervention will be effective.

Conclusion

A large and diverse group of e-A&F interventions were identified, which differed across countries of origin, feedback topics and target health professional recipients. Broadly, we found that published e-A&F interventions were associated with effectiveness where there was explicit evidence that the context and design reflected the features examined. Specifically, we found that interventions benefited from supportive organisations. An association with effectiveness was also found when the health professionals who were targeted had positive beliefs about receiving feedback and greater capability for quality improvement for the clinical topic in focus. Audit and feedback practitioners should focus energy and resources during the setup and implementation to emulate the features as recognised during this review.

Supplemental Material

Supplemental Material - Features and effectiveness of electronic audit and feedback for patient safety and quality of care in hospitals: A systematic review

Supplemental Material for Features and effectiveness of electronic audit and feedback for patient safety and quality of care in hospitals: A systematic review by James Soresi, Christina (Tina) Bertilone, Eileen Banks, Theresa Marshall, Kevin Murray, and David B Preen in Health Informatics Journal

Footnotes

Acknowledgements

No acknowledgments to declare.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Contributorship

JS, DP and KM conceptualised the project, reviewed the literature and drafted the manuscript. CB and EB reviewed the literature and drafted the manuscript. DP, KM and TM reviewed and edited the manuscript. All authors read and approved the final manuscript.

Ethical statement

ORCID iD

James Soresi

Supplemental Material

Supplemental material for this article is available online.

References

Ivers

Jamtvedt

Flottorp

, et al. Audit and feedback: effects on professional practice and healthcare outcomes. Cochrane Database Syst Rev 2012; 2012: CD000259. DOI: 10.1002/14651858.CD000259.pub3.

Ivers

Sales

Colquhoun

, et al. No more 'business as usual' with audit and feedback interventions: towards an agenda for a reinvigorated intervention. Implement Sci 2014; 9: 14. DOI: 10.1186/1748-5908-9-14.

Brehaut

Colquhoun

Eva

, et al. Practice feedback interventions: 15 suggestions for optimizing effectiveness. Ann Intern Med 2016; 164: 435–441.

Brown

Gude

Blakeman

, et al. Clinical Performance Feedback Intervention Theory (CP-FIT): a new theory for designing, implementing, and evaluating feedback in health care based on a systematic review and meta-synthesis of qualitative research. Implement Sci 2019; 14: 40. DOI: 10.1186/s13012-019-0883-5.

Tuti

Nzinga

Njoroge

, et al. A systematic review of electronic audit and feedback: intervention effectiveness and use of behaviour change theory. Implement Sci 2017; 12: 61. DOI: 10.1186/s13012-017-0590-z.

Dowding

Randell

Gardner

, et al. Dashboards for improving patient care: review of the literature. Int J Med Inform 2014; 84: 87–100. DOI: 10.1016/j.ijmedinf.2014.10.001.

Van Den Bulck

Spitaels

Vaes

, et al. The effect of electronic audits and feedback in primary care and factors that contribute to their effectiveness: a systematic review. Int J Qual Health Care 2020; 32: 708–720. DOI: 10.1093/intqhc/mzaa128.

Mihailescu

. The emergence of digitalisation in the context of health care. In: Hawaii International Conference on System Sciences, Hawaii, United States of America, 3 January 2018.

Blumer

Döring

. Are we the same online? The expression of the five factor personality traits on the computer and the Internet. Cyberpsychology 2012; 6: 5. DOI: 10.5817/CP2012-3-5.

10.

Tsang

Peek

Buchan

, et al. Systematic review and narrative synthesis of computerized audit and feedback systems in healthcare. J Am Med Inform Assoc 2022; 29: 1106–1119. DOI: 10.1093/jamia/ocac031.

11.

Moher

Liberati

Tetzlaff

, et al. Preferred reporting Items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 2009; 6: e1000097. DOI: 10.1371/journal.pmed.1000097.

12.

Soresi

Bertilone

Banks

, et al. Electronic audit and feedback for patient safety and quality of care in hospitals: a systematic review. London, United Kingdom: Prospero Publishing, 2020. https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42020167008.

13.

van Engen-Verheul

Gude

van der Veer

, et al. Improving guideline concordance in multidisciplinary teams: preliminary results of a cluster-randomized trial evaluating the effect of a web-based audit and feedback intervention with outreach visits. AMIA Annu Symp Proc 2015; 2015: 2101–2110. Randomized Controlled Trial.

14.

Becker

Nagavally

Wagner

, et al. Creating a culture of quality: our experience with providing feedback to frontline hospitalists. BMJ Open Qual 2021; 10: e001141. DOI: 10.1136/bmjoq-2020-001141.

15.

Armellino

Hussain

Schilling

, et al. Using high-technology to enforce low-technology safety measures: the use of third-party remote video auditing and real-time feedback in healthcare. Clin Infect Dis 2012; 54: 1–7. Evaluation Studies Research Support, Non-U.S. Gov't.

16.

Salazar

Tyroch

Smead

. Electronic trauma patient outcomes assessment tool. J Trauma Nurs 2011; 18: 197–203. DOI: 10.1097/JTN.0b013e31823a49ff.

17.

Aboagye

Haut

Streiff

, et al. Audit and feedback to surgery interns and residents on prescribing risk-appropriate venous thromboembolism prophylaxis. J Surg Educ 2021; 78: 2011–2019. DOI: 10.1016/j.jsurg.2021.03.017.

18.

Hermon

Pain

Beckett

, et al. Improving compliance with central venous catheter care bundles using electronic records. Nurs Crit Care 2015; 20: 196–203.

19.

Patel

Rajkomar

Harrison

, et al. Next-generation audit and feedback for inpatient quality improvement using electronic health record data: a cluster randomised controlled trial. BMJ Qual Saf 2018; 27: 691–699. Randomized Controlled Trial.

20.

Hand

Kerr

Chambers

, et al. Effect of near real-time feedback tool in the electronic medical record on protocol compliance during laparoscopic cholecystectomy: a single-center retrospective analysis. J Clin Monit Comput 2022; 36: 1833–1839. DOI: 10.1007/s10877-022-00833-1.

21.

Boggan

Swaminathan

Thomas

, et al. Improving timely resident follow-up and communication of results in ambulatory clinics utilizing a web-based audit and feedback module. J Grad Med Educ 2017; 9: 195–200.

22.

Redwood

Ngwenya

Hodson

, et al. Effects of a computerized feedback intervention on safety performance by junior doctors: results from a randomized mixed method study. BMC Med Inf Decis Making 2013; 13: 63. DOI: 10.1186/1472-6947-13-63.

23.

Bowdle

Jelacic

Nair

, et al. Electronic audit and feedback with positive rewards improve anesthesia provider compliance with a barcode-based drug safety system. Anesth Analg 2019; 129: 418–425.

24.

Hester

Lang

Madsen

, et al. Timely data for targeted quality improvement interventions: use of a visual analytics dashboard for bronchiolitis. Appl Clin Inf 2019; 10: 168–174. DOI: 10.1055/s-0039-1679868.

25.

Pires

Gayet-Ageron

Guitart

, et al. Effect of wearing a novel electronic wearable device on hand hygiene compliance among health care workers A stepped-wedge cluster randomized clinical trial. JAMA Netw Open 2021; 4: e2035331. DOI: 10.1001/jamanetworkopen.2020.35331.

26.

Baysari

Oliver

Egan

, et al. Audit and feedback of antibiotic use: utilising electronic prescription data. Appl Clin Inf 2013; 4: 583–595. Research Support, Non-U.S. Gov't.

27.

Roos-Blom

M-J

Gude

de Jonge

, et al. Impact of audit and feedback with action implementation toolbox on improving ICU pain management: cluster-randomised controlled trial. BMJ Qual Saf 2019; 28: 1007–1015. DOI: 10.1136/bmjqs-2019-009588.

28.

Conway

Riley

Saiman

, et al. Implementation and impact of an automated group monitoring and feedback system to promote hand hygiene among health care personnel. Joint Comm J Qual Patient Saf 2014; 40: 408–417. Research Support, N.I.H., Extramural.

29.

Lenglet

van Deursen

Viana

, et al. Inclusion of real-time hand hygiene observation and feedback in a multimodal hand hygiene improvement strategy in low-resource settings. JAMA Netw Open 2019; 2: e199118. DOI: 10.1001/jamanetworkopen.2019.9118.

30.

O'Reilly-Shah

Easton

Jabaley

, et al. Variable effectiveness of stepwise implementation of nudge-type interventions to improve provider compliance with intraoperative low tidal volume ventilation. BMJ Qual Saf 2018; 27: 1008–1018. DOI: 10.1136/bmjqs-2017-007684.

31.

Reszel

Dunn

Sprague

, et al. Use of a maternal newborn audit and feedback system in Ontario: a collective case study. BMJ Qual Saf 2019; 28: 635–644. DOI: 10.1136/bmjqs-2018-008354.

32.

Cohen

. Statistical power analysis for the behavioral sciences. 2nd ed. Hillsdale, N.J: L. Erlbaum Associates, 1988.

33.

Cohen

. A power primer. Psychol Bull 1992; 112: 155–159. DOI: 10.1037/0033-2909.112.1.155.

34.

Dufour

Duhoux

Bolduc

. Measured and perceived effects of audit and feedback on nursing performance: a mixed methods systematic review protocol. Syst Rev 2019; 8: 38. DOI: 10.1186/s13643-019-0956-1.

35.

Tsang

Brown

Peek

, et al. A systematic review of computerised audit and feedback. London, United Kingdom: Prospero Publishing, 2016. https://www.crd.york.ac.uk/PROSPERO/display_record.php?RecordID=48695&VersionID=76375.

36.

Hong

Pluye

Fàbregues

, et al. Improving the content validity of the mixed methods appraisal tool: a modified e-Delphi study. J Clin Epidemiol 2019; 111: 49–59.e1. DOI: 10.1016/j.jclinepi.2019.03.008.

37.

Hopewell

Loudon

Clarke

, et al. Publication bias in clinical trials due to statistical significance or direction of trial results. Cochrane Database Syst Rev 2009; 2009: MR000006. DOI: 10.1002/14651858.MR000006.pub3.

38.

Rajkomar

Patel

Valencia

, et al. The data trial: a randomized controled trial of next generation audit and feed back. J Gen Intern Med 2017; 32: S334–S335. Journal: Conference Abstract.

39.

Ivers

Grimshaw

Jamtvedt

, et al. Growing literature, stagnant science? Systematic review, meta-regression and cumulative analysis of audit and feedback interventions in health care. J Gen Intern Med 2014; 29: 1534–1541. DOI: 10.1007/s11606-014-2913-y.

40.

Colquhoun

Carroll

Eva

, et al. Advancing the literature on designing audit and feedback interventions: identifying theory-informed hypotheses. Implement Sci 2017; 12: 117. DOI: 10.1186/s13012-017-0646-0.

41.

Kraus

Schiavone

Pluzhnikova

, et al. Digital transformation in healthcare: analyzing the current state-of-research. J Bus Res 2021; 123: 557–567. DOI: 10.1016/j.jbusres.2020.10.030.

42.

Mathematic Policy Research . Using health information technology to support quality improvement in primary care. In: Agency for Healthcare Research and Quality. Rockville, MD: US Department of Health and Human Services, 2015.

43.

Amsterdam

UMC.

Medical informatics. Patientzorg Amsterdam UMC. Available from: https://www/amc.nl/web/research-75/departments/medical-informatics-1/medical-informatics-1/nice-national-intensive-care-registry.htm

44.

Jamtvedt

Young

Kristoffersen

, et al. Does telling people what they have been doing change what they do? A systematic review of the effects of audit and feedback. Qual Saf Health Care 2006; 15: 433–436. DOI: 10.1136/qshc.2006.018549.

45.

Hysong

. Meta-Analysis: audit and feedback features impact effectiveness on care quality. Med Care 2009; 47: 356–363. DOI: 10.1097/MLR.0b013e3181893f6b.

46.

Ogrinc

Davies

Goodman

, et al. SQUIRE 2.0 (Standards for QUality Improvement Reporting Excellence): revised publication guidelines from a detailed consensus process. BMJ Qual Saf 2016; 25: 986–992. DOI: 10.1136/bmjqs-2015-004411.

47.

Hoffmann

Glasziou

Boutron

, et al. Better reporting of interventions: template for intervention description and replication (TIDieR) checklist and guide. BMJ Br Med J (Clin Res Ed) 2014; 348: g1687. DOI: 10.1136/bmj.g1687.

48.

Barbeito

Segall

. Development and usability testing of an audit and feedback tool for anesthesiologists. JAMA Open 2019; 2: 29–34. DOI: 10.1093/jamiaopen/ooy054.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.22 MB