Abstract
Objectives
To evaluate the utility of Isabel, an online diagnostic decision support system developed by Isabel Healthcare primarily for secondary medical care, in the general practice setting.
Design
Focus groups were conducted with clinicians to understand why and how they used the system. A modified online post-use survey asked practitioners about its impact on their decision-making. Normalization process theory (NPT) was used as a theoretical framework to determine whether the system could be incorporated into routine clinical practice.
Setting
The system was introduced by NHS County Durham and Darlington in the UK in selected general practices as a three-month pilot.
Participants
General practitioners and nurse practitioners who had access to Isabel as part of the Primary Care Trust's pilot.
Main outcome measures
General practitioners’ views, experiences and usage of the system.
Results
Seven general practices agreed to pilot Isabel. Two practices did not subsequently use it. The remaining five practices conducted searches on 16 patients. Post-use surveys (n = 10) indicated that Isabel had little impact on diagnostic decision-making. Focus group participants stated that, although the diagnoses produced by Isabel in general did not have an impact on their decision-making, they would find the tool useful if it were better tailored to the primary care setting. Our analysis concluded that normalization was not likely to occur in its current form.
Conclusions
Isabel was of limited utility in this short pilot study and may need further modification for use in general practice.
DECLARATIONS
None declared
The research was funded through an infrastructure grant from NHS County Durham and Darlington.
The Sub-Ethics Committee of the School of Medicine and Health at Durham University provided ethical approval. The County Durham NHS ethics committee deemed this study to be an evaluation not requiring NHS ethical approval.
EH
EH and GR designed the study and analysed the results. EH conducted the fieldwork. EH and GR prepared, read and approved the final manuscript
Peter Manser
Background
Decision support systems, in the form of software or online programmes, are increasingly being used to assist with problem solving in areas with complex structures and large knowledge bases. Diagnostic decision support systems (DDSSs) in health care are designed to be interactive and aid in decision-making, rather than to provide the ‘right’ answer to a particular problem. Systematic reviews have shown that they can improve practitioner performance, but the evidence for their effectiveness in terms of patient impact and quality of care shows that these effects are limited.1–4
Isabel is an online DDSS that aids clinicians by providing a list of 10 possible diagnoses based on the clinical features of the case in question. The clinician can access the system at any time, and is prompted to enter details about the patient (age, sex, pregnancy status and geographical region) and a minimum of one clinical feature. It is supported by customized components, such as web resources, protocols and guidance notes (www.isabelhealthcare.com). It incorporates a post-use survey, which asks the clinician about his or her experience of the tool and can be used to generate evidence for the clinician's appraisal portfolio.
The effectiveness of Isabel has been the subject of a number of evaluations. In specialist settings Isabel decreased unfavourable outcomes, decreased diagnostic omission errors and increased the number of ‘correct’ diagnoses.5–11 Two studies using the post-use survey in primarily specialist settings found a majority of users reported that it was useful and offered diagnoses not considered before use.12,13 However, a postal survey of primary and secondary care clinicians found that it was not considered useful or convenient by 90% of users. 14
The introduction of the Isabel system within the British National Health Service (NHS) is relatively recent, and its applicability to the primary care setting has yet to be established. Diagnostic decision-making in primary care differs from that in secondary care, in that general practitioners (GPs) develop an initial hypothesis of the diagnosis, which they then refine through further questioning, examination and investigation before arriving at a final diagnosis. This process may also take place over time and through more than one consultation. 15 In 2010, the Isabel system was made available to a limited number of general practices as part of a three-month pilot by NHS County Durham and Darlington in the UK to see whether it might have utility in the general practice setting. We sought to determine its uptake and impact on clinical decision-making and patient management and to elicit users’ views of its utility.
Methods
Pilot design
All 85 practices in NHS County Durham and Darlington were invited to participate in a three-month pilot of the Isabel system. Participating practices were provided with training in its use by Isabel Healthcare and NHS County Durham and Darlington, and were encouraged to run test searches in the system to further their training.
Each practice was encouraged by the Primary Care Trust (PCT) to identify a champion for use of the system during the period of the pilot, however no incentives to use the system were provided by the PCT.
Study design
We obtained anonymized extracts from the clinical records of those patients for whom searches were conducted. We used the patient's unique practice identifier and date of consultation to link data extracted by the Isabel system with practice record data.
We conducted focus groups of participants and non-participants. Focus groups encourage participants to share their experiences with each other, which can stimulate a fuller development of ideas and perspectives. This dialogue can also trigger one's memory, which was particularly important to this evaluation as practitioners were expected to recall past experiences. We offered one-to-one interviews, either over the phone or in person, as an alternative if focus groups could not be arranged with all practice staff involved. Focus groups and interviews were facilitated by EH and audio recorded.
An online post-use questionnaire, designed and pretested by the research team, appeared on the system immediately after a search was conducted in Isabel. It asked four questions on decision-making related to that search (Box 1).
Online post-use survey
How did Isabel PRO help with your differential diagnosis?
Confirmed my differential Broadened my differential No impact How did Isabel PRO influence your decision to run diagnostic tests (blood tests, scans, xrays)?
Initiated a test Confirmed a test Cancelled a test Changed a test No impact How did Isabel PRO influence your decision to prescribe medication?
Initiated a prescription Confirmed a prescription Cancelled a prescription Changed a prescription No impact How did Isabel PRO influence your decision to refer the patient?
Initiated a referral Confirmed a referral Cancelled a referral Changed a referral (e.g. different clinic/consultant/specialty) No impact
Analysis
We determined Isabel's role in patient management and clinical decision-making by comparing the searches conducted with the associated clinical record. The cluster of clinical features that had been entered for each search was assigned to a clinical category by a practising general practitioner, GR, to identify the types of problem for which searches had been conducted.
A conventional qualitative content analysis was applied to the focus group data. 16 Audio recordings were listened to and field notes were read several times, and then coded into categories. Categories were generated inductively in order to develop a theory from the data. 17 Coding was rechecked for consistency.
Data from the post-use survey were imported into Excel 2007 and frequencies for each response were calculated.
Normalization process theory (NPT) was used as a theoretical framework to assess whether and how well the system had been embedded in everyday practice, based on the evidence available from the focus groups and post-use survey. 18 NPT can be used to understand how clinicians adopt new technologies into their working practice. It proposes that practices become routinely embedded within social contexts as a result of individual and collective work to implement them (May and Finch 2005). Implementation is operationalized through the four generative mechanisms of coherence, cognitive participation, collective action and reflexive monitoring, and factors (immediate and organizing components) promote or inhibit routine embedding or normalization.
Handling of data
Informed written consent was obtained from each practice for participation in the evaluation and from individual clinicians who participated in the focus groups. We obtained the necessary PCT and Isabel Healthcare permissions to download responses to the online post-use questionnaire. All data were treated in accordance with the Data Protection Act (1998).
Results
All 85 practices in NHS County Durham and Darlington were invited to take part in the three-month pilot and, through purposive selection by NHS County Durham and Darlington from those practices willing to participate, Isabel was made available to seven general practices across a range of practice size and geography. A reminder notice was sent to all participating practices, encouraging them to use the system, midway through the pilot period. Figure 1 shows the use of the system over the course of the pilot.
Use of Isabel over three-month pilot
Practice characteristics and usage are shown in Table 1. All seven practices accessed the system to run test searches (data not reported here); however, data were only eligible for analysis if searches were made for a specific patient (by using the patient practice number). Four practices made 16 patient specific searches. In two cases, a practitioner conducted multiple searches for the same patient (2 in 1 case, 3 in the other), leaving a total of 21 searches conducted. The remaining three practices did not participate, and did not provide reasons for this.
Practice summary
N/A data not made available by practice
The most frequent searches were for adult males aged 50 years and over, followed by searches for younger children. On average, each search took three minutes and 55 seconds (SD±3:06) to complete (n = 15). Data on duration of searches were not made available by the system in four cases.
Analysis of clinical records
The limited use of Isabel meant that any impact of its use on referral rates and investigations was too small for meaningful quantitative analysis. Instead, we illustrate the diagnostic process by comparing the features entered and corresponding diagnostic output from the system (n = 21) with the features, actions taken and diagnoses noted in the clinical record (Table 2).
Process of diagnostic decision-making in general practice using Isabel
FBC, full blood count; LFT, liver function test; ESR, erythrocyte sedimentation rate; CRP, C-reactive protein; ECG, electrocardiogram; SLE, systemic lupus erythematosus; PSA, prostate-specific antigen; COPD, chronic obstructive pulmonary disease; FOBT, fecal occult blood test
N/S Not stated in clinical record; N/A Clinical record not made available
Clinical records data show that the majority of patients had multiple co-morbidities. The most frequent clinical categories were musculoskeletal and non-specific, followed by abdominal and respiratory. In 11 cases, a diagnosis was noted in the clinical record. Based on the features entered into the system, Isabel only identified four of these, and missed seven. There were 13 cases where a referral or investigation was recorded in the clinical notes; however it was not possible to attribute these to the Isabel output or otherwise.
Post-use survey
Practitioners in all of the four actively participating practices completed at least one online post-use survey, for a total of 10 of the 21 searches conducted. The majority of responses indicated that Isabel did not have an impact on any aspects of diagnostic decision-making, including differential diagnosis, diagnostic tests, prescription of medication or referrals. Five users reported that Isabel helped to either broaden a differential (n = 4) or confirm an intention to refer (n = 1).
Focus groups and individual interviews
We conducted two focus groups, one each at Practice B and Practice G. Because only four practices used the system, we could only invite these to the participant focus groups. Practice E could not arrange a meeting for a focus group with its entire staff, and while one-to-one interviews were offered as an alternative, a suitable time to hold the interview could not be found, and it did not take place. We had no reply from the fourth participating practice.
We sought to arrange group or one-to-one interviews with practitioners in the three practices that did not use the system, but were unable to either obtain a response or agree a suitable time.
There were five GPs present for the focus groups at Practice B. Only one, who was the most senior partner of the practice, had accessed the system. Three were new members of staff and had not received the training. The fifth, a practice partner, had attended the training, but reported that she did not find cause to ever use the system. The practitioners stated that they did not ‘remember’ to use the system, and weekly reminders that Isabel was an option for them to use would have helped. At the same time, this practice also reported that they did not appoint a champion for the pilot as advised by the PCT.
For the focus group with Practice G, there were no GPs present, and only one senior nurse practitioner and the practice manager. The nurse practitioner informed us that the GPs agreed to have her trial the programme on their behalf. She remembered to use the system, but reserved it for specific circumstances, for example in cases that were complex or not responding to treatment. She felt it would have been more beneficial as a training tool.
Practitioners from both practices reported that, although the diagnoses produced by Isabel in general did not have an impact on their decision-making, they might find it more useful if it were tailored to the primary care setting and which they could then use if and when necessary.
Assessment of implementation using NPT
Table 3 summarizes the evidence for each of the factors and components that should be considered in applying NPT. 18 Most of the evidence indicates that normalization was unlikely to occur. The exceptions were for the components involving expertise, where practitioners felt competent using the system, and resources, where practitioners had easy access to the system.
Model exploring reasons for non-use using normalization process theory
In respect of professional (endogenous) factors, practitioners shared a belief in the utility and process of DDSS systems generally, but considered that Isabel itself was not configured adequately for primary care. There was also doubt about the tool itself, indicated by a lack of belief that the system fell within the practitioners’ remit.
At the organizational level (exogenous factors), there was allocational conflict, as agreement to participate had been primarily by practice managers, while its use was to be by clinicians. As a result, roles were not agreed. This was linked to a major issue of no time having been identified for clinicians to use the system during a consultation.
Discussion
In this study of a short-term pilot of the Isabel diagnostic decision support software we found it had limited utility in the UK general practice setting. Not all practices participating in the study used the system and, in those that did, very few clinicians accessed it. There was evidence from the clinical records that outputs from Isabel were associated with clinical decisions in some cases. The main finding of user focus groups was that the current format of Isabel would need modification for use in general practice.
This is one of the first studies of Isabel in a primary care setting in the UK. It was a pragmatic study done in the context of routine general practice. Adoption should be accompanied by interventions to increase uptake; in this instance one reminder was given with short-term effect. The three-month pilot was short in comparison to other published studies of decision support systems. These can take some time to reach ‘steady state’ usage, though we did not see evidence of increasing uptake over the study period.
An important weakness of the study is the small amount of data available for analysis. Low participation in evaluations of poorly designed interventions is a common finding, and in the case of Isabel the low participation observed was probably due to both poor implementation of the tool and its limited utility. 19 The limited funding from the PCT for the initiative meant that only seven general practice sites could be recruited. Coupled with their very low usage rates, this meant the effect on important outcomes of direct relevance to diagnostic decision-making, such as specialist referral and use of diagnostics, was so small as to rule out quantitative analysis. It was also difficult to get health-care professionals to participate in the qualitative interviews, and recruiting non-participants into interview studies is a recognized challenge in qualitative research.
Previous experimental studies of Isabel have been undertaken in controlled clinical contexts, and have used outcome measures such as rates of diagnostic omission errors, unfavourable outcomes and ‘correct’ diagnoses, using more senior clinicians to validate decisions and diagnoses.5–11 It would be methodologically challenging to take the same approach in a pragmatic general practice study and with the case-mix seen by GPs. One observational study in general practice, published in abstract only, reported the usage by 25 GPs over a six-month period. Although 335 queries were entered, only 49 post-use surveys were completed. In 29% of these, Isabel was reported to have modified a referral decision and in 40% it prompted additional test(s). 20 In our study we also found a discrepancy between queries entered and surveys completed. This appeared to be explained by problems with the programme itself or with its output for a given query.
In a previous study, users reported that Isabel offers diagnoses not considered beforehand. 13 However other studies have found that it was not useful or convenient due to the structure of NHS service delivery, specifically the lack of system-wide internet availability and time allocated by practices to clinicians to learn and use the tool. 14 In our study, users had access to the internet, but appeared not to think of using the system. The average search in this study lasted nearly four minutes compared with less than two minutes in other studies.5,6 The longer time required may reflect the range of symptom complexes being searched on, or a lack of familiarity with the system given the three-month duration of the pilot.
In general practice the concept behind Isabel may also presuppose a level of diagnostic certainty that is unrealistic. In up to 40% of patients it is not possible to apply a diagnostic label. 21 Howie 22 described ‘the relative rarity of fully developed hospital illness’ in general practice, and Jones et al 23 have described the danger of spurious and erroneous diagnostic precision. Furthermore, and in distinction to secondary care, diagnostic decision-making in primary care is a complex process that often extends over time, while the causes of misdiagnosis are not limited to cognitive oversight.15,23 Nevertheless, Isabel might have greater utility in general practice if it were to be configured to more closely to mirror the ways that problems present in primary care, and our findings indicate that practitioners would be receptive to diagnostic support of this sort.
NPT is a sociological model which focuses on the dynamic processes that lead to innovations becoming embedded in everyday work. We used the construct of collective action from NPT to structure the interpretation of our findings. 24 Collective action is the operational work that people do to enact a new practice, such as the use of a new technology. We found that most components of this construct were not fulfilled, making it unlikely that Isabel would be integrated into practice in its current form.
Conclusions
Isabel as it is currently configured is unlikely to be incorporated into clinical practice in the primary care setting. There is a need for rigorously designed trials of Isabel in primary care that are adequately powered to determine impact on decision-making. Outcome measures should include measures of patient safety, diagnostic accuracy and health-care resource utilization. Its introduction into primary clinical care by primary care organizations should await or be in the context of such research.
Footnotes
Acknowledgements
We thank Wendy Stevens at NHS County Durham for coordinating with the practices and providing useful feedback. We also thank Natalie Cronje at Isabel Healthcare who provided technical support for the Isabel system. Finally, we are grateful to all practices that participated in the pilot and especially those that participated in our focus groups.