Formalizing clinical practice guideline for clinical decision support systems

Abstract

Clinical practice guidelines are valuable sources of clinical knowledge for healthcare professionals. However, the passive dissemination of clinical practice guidelines like publishing in medical journals is ineffective in changing clinical practice behaviour. In this work, we proposed a framework to help adopting an active clinical practice guideline dissemination approach by automatically extracting clinical knowledge from clinical practice guidelines into a clinical decision support system–friendly format. The proposed framework is intended to help human modellers by automating some of the manual formalization activities in order to minimize their manual effort. We evaluated our framework using all recommendations from two clinical practice guidelines produced by the Scottish Intercollegiate Guidelines Network: the ‘Management of lung cancer’ clinical practice guideline and the ‘Management of chronic pain’ clinical practice guideline. We conclude that the proposed framework can be effectively used to formalize drug and procedure recommendation in clinical contexts.

Keywords

clinical decision support system clinical practice guideline health information systems Unified Medical Language System

Introduction

Clinical practice guidelines (CPGs) as defined by the Institute of Medicine are ‘systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances’.¹ CPGs offer concise instruction on the optimal care for the patient based on the latest clinical findings. The main benefit of CPG is to improve the quality of care and the consistency of care. For a healthcare professional, a CPG can help offer explicit recommendations when a healthcare professional is uncertain about how to proceed.²

It is been shown that passive dissemination of CPGs like publishing in a medical journal is ineffective in changing clinical practice behaviour.³ Many healthcare practitioners are not aware of the existence of the CPG, and even when they are directed to the relevant CPG, they experience difficulties using it in their daily practice.⁴ Nevertheless, integrating CPG knowledge into clinical systems, such as decision support systems, has shown to be more effective.⁵ In order to best benefit from the CPGs’ knowledge by following an active CPG dissemination approach, an interest in formalizing medical knowledge contained in CPGs has grown. The heritage of the narrative text CPGs will remain a source of medical knowledge that awaits its formalized counterpart to be developed and integrated into clinical systems. This integration could be manifested in different technical facades like clinical decision support system (CDSS) or extension to the electronic health record (EHR) system used by healthcare facilities.

There are several formal languages developed to help modelling clinical guidelines into computer-interpretable guidelines (CIGs); the review study⁶ presents some of the CIG modelling languages. Assuredly, the development of the guideline modelling languages is an important step towards facilitating the CPG formalization process, yet the formalization task remains laborious and complex, mainly because it requires human modellers with two different areas of expertise: a medical expertise to correctly interpret the medical knowledge of CPGs and a knowledge engineer expertise to correctly represent the medical knowledge using the syntax of the modelling language.

Nonetheless, CPG formalization approaches have been published.^7–14 These approaches are either based on a set of manual steps to gradually convert narrative text CPGs into CIGs,^7,8 or based on automated information extraction mechanisms frequently using linguistic patterns.^9–14 While the accuracy of the manual approaches is straightforwardly controlled, as the resulted accuracy is as good as the input provided by the human modellers, these approaches are impractical to use in formalizing large numbers of CPGs. On the other hand, the automated and semi-automated information extraction–based approaches are more suited to formalize a relative large number of CPGs, but these approaches have not shown how expandable they are in handling a large number of heterogeneous CPGs, CPGs with different styles, granularity, and so on – a common challenge with all large-scale information extraction systems.¹⁵ Therefore, the motivation of our work in building a CPG formalization framework is to enable the human modeller to control the expressiveness of the extraction rules in the formalization system without rebuilding the system.

The purpose of this work is to minimize the effort required by human modellers to bridge the gap between CPG and CIG by extending the automation of the formalization process of the drug recommendation and procedure recommendation clinical contexts. We followed a two-level clinical context extraction mechanism to assist the human modeller to better control the balance between accuracy and scalability with heterogeneous CPGs.

Methods

The proposed framework follows a multi-step approach, which has been shown to be a good strategy for CPG formalization.¹⁶ We architected the framework to set boundaries around the aspects of the drug recommendation in the CPG formalization, where each aspect is implemented as a separate autonomous component in a CPG formalization pipeline. The framework is based on the Unstructured Information Management Architecture (UIMA).¹⁷ In the following sections, we provide a description of each component in our CPG formalization pipeline as illustrated in Figure 1.

Figure 1.

CPG formalization pipeline.

XML parsing

We used CPGs extracted from the National Guideline Clearinghouse (NGC)¹⁸ in XML format. The XML parsing component extracts the content of the XML CPG documents into a structured object.

Text cleansing

Most of the sections extracted by the XML parsing component contain narrative text mixed with HTML tags. HTML tags are used by Web browsers to render text for visual display, but as we are not interested in composing the text for web browsers, we removed all HTML tags from the text.

Medical concept tagging

This is a component to map CPG text to a medical vocabulary; we used the Unified Medical Language System (UMLS) Metathesaurus as our biomedical vocabulary database. The UMLS Metathesaurus contains more than 2.6 million concepts each assigned to at least one semantic type from the set of 133 semantic types of the UMLS semantic network. We used MetaMap,^19,20 to map CPG text to the UMLS Metathesaurus concepts. For integrating MetaMap with the UIMA framework, we leveraged the MetaMap UIMA Annotator²¹ which is a wrapper that makes the MetaMap tool usable as an UIMA analysis engine.

Medical tags disambiguation

This is the process of finding the correct UMLS concepts, when multiple concepts are assigned by MetaMap with the same score. For example, the word lens could get annotated by MetaMap with three different UMLS concepts that have different meanings as shown in Table 1.

Table 1.

MetaMap UMLS concept mapping for the word lens.

Concept		Semantic type
Unique Id	Name
C0023308	Lens diseases	Disease or syndrome
C0023318	Lens (device)	Medical device
C0023317	Lens, crystalline	Body part, organ, or organ component

UMLS: Unified Medical Language System.

To solve this type of ambiguity, we used a graph-based disambiguation algorithm²² to rank the generated MetaMap UMLS concepts based on their relatedness to the context of co-located text.

Clinical context pattern detection

This is a rule-based extraction component. This component is the first level of our clinical context extraction mechanism; its function is to extract text fragments that contain the minimum necessary features of the clinical context in question. Extracting clinical context based on the minimum necessary features follows the top-down approach¹⁵ where only general rules that cover as many possible instances of clinical context need to be defined, which means rules that have high coverage and poor precision. Because general extraction rules tend to be in small numbers and are simple to define, the rule authoring task is a good fit for the medical experts who usually lack extensive knowledge in rule authoring. To further simplify the rule authoring task for the medical expert, we used UIMA Ruta²³ as it has a defined rule-based language with the ability to build rules against the text as well as against the semantic annotations of the text. We also defined a guideline of four steps to structure the effort required. In the following sections, we describe each of the four steps to author drug recommendation extraction rules with sample Ruta syntax highlighted in Figure 2:

Step 1. Set text analysis boundaries: since analysing the CPG as one big unit is too complex and impractical, we need to break down the CPG document into text chunks small enough for easiness of analysis. The modeller can choose the boundary at the section, paragraph, sentence, or even token level.

Step 2. Cluster UMLS semantic types: each UMLS tag is assigned to one UMLS semantic type. This gives us a wide spectrum of semantic types that is too granular for our analysis. Clustering the UMLS semantic network into smaller set of semantic types will help eliminate duplicate rules across UMLS semantic types. To achieve this goal, we followed the approach presented in Bodenreider and McCray²⁴ and aggregated semantic types. In Table 2, we show two groups of semantic types, the Chemical & Drugs (CHEM) and the Disorder (DISO) used for the drug recommendation clinical context.

Step 3. Structuring clinical data: in this step, the human modeller (1) defines the clinical data structures and (2) provides conditions to assign the newly defined clinical data structure to tokens in the CPG text. Defining clinical data structures could be coarse, for example, the drug prescription data structure composed of the medicine name and the dose, or more granular to include the dose timing and the duration of the treatment. The expressivity of the clinical context extraction rules heavily depends on the granularity of data structures used; the more granular the clinical data structures, the more expressive rules can be authored but also the more complex the rule authoring task becomes. To follow pre-reviewed clinical data structures, we defined our clinical data structures based on the openEHR archetypes.²⁵

Figure 2.

UIMA Ruta patterns for drug recommendation.

Table 2.

UMLS semantic type grouping.

Chemical & Drugs (CHEM)		Disorder (DISO)
Semantic type	Abbreviation	Semantic type	Abbreviation
Amino acid, peptide, or protein	aapp	Acquired abnormality	acab
Antibiotic	antb	Anatomical abnormality	anab
Biologically active substance	bacs	Cell or molecular dysfunction	comd
Biomedical or dental material	bodm	Congenital abnormality	cgab
Carbohydrate	carb	Disease or syndrome	dsyn
Chemical	chem	Experimental model of disease	emod
Chemical viewed functionally	chvf	Finding	fndg
Chemical viewed structurally	chvs	Injury or poisoning	inpo
Clinical drug	clnd	Mental or behavioural dysfunction	mobd
Eicosanoid	eico	Neoplastic process	neop
Element, ion, or isotope	elii	Pathologic function	patf
Enzyme	enzy	Sign or symptom	sosy
Hazardous or poisonous substance	hops
Hormone	horm
Immunologic factor	imft
Indicator, reagent, or diagnostic aid	irda
Inorganic chemical	inch
Lipid	lipd
Neuroreactive substance or biogenic amine	nsba
Nucleic acid, nucleoside, or nucleotide	nnon
Organic chemical	orch
Organophosphorus compound	opco
Pharmacologic substance	phsu
Receptor	rcpt
Steroid	strd
Vitamin	vita

Detecting an instance of the defined clinical data structure in the CPG text is achieved by annotating the CPG text with the clinical data structure types based on predefined conditions. The conditions could be based on specific lexicon, syntax, or previously annotated semantics; Step 3 in Figure 2 shows the Ruta code of our version of the ‘problem diagnosis’ evaluation archetype and the ‘medication order’ instructions archetype. The defined clinical data structures contain one element for simplicity, but each of these data structures can contain multiple elements; we also defined relaxed conditions that capture tokens annotated with the DISO and CHEM semantic groups and tagged the former as an element of the ‘problem diagnosis’ evaluation archetype and the latter as the medicine element of the ‘medication order’ instructions archetype.

Step 4. Clinical context semantic relations: each clinical context could be modelled as an instance of semantic relation between clinical data, for example, drug recommendation could be modelled as a disease-to-drug semantic relation or symptoms-to-drug semantic relation. Annotating CPG text with clinical context semantic relation requires the human modeller to (1) define a semantic relation and (2) define conditions for mapping instances of clinical data structures to a semantic relation.

In Figure 3, we show the ‘Fluoxetine (20–80 mg/day) should be considered for the treatment of patients with fibromyalgia’ sentence from the ‘Management of chronic pain: a national clinical guideline’.²⁶ CPG and the annotations are assigned based on the four clinical context pattern detection steps.

Figure 3.

Annotations for the drug recommendation clinical context.

Clinical context filtering

This component is responsible for removing the clinical context instances wrongly labelled by the clinical context pattern detection component. We used a logistic regression²⁷ classification algorithm to decide on the correctness of the drug recommendation labels. Because this classification algorithm is supervised, which means it requires to be trained using a correctly annotated data set, we generated a training data set composed of 117 recommendation sentences extracted from the Yale Guideline Recommendation Corpus (YGRC).²⁸ The YGRC is composed of 1275 recommendations which cover a broad range of diseases and mental disorders extracted from the NGC. We annotated all YGRC sentences with MetaMap and then selected 117 sentences that have tokens in the DISO semantic group in addition to other tokens in the procedure or CHEM semantic group. We manually tagged each sentence as either drug/procedure recommendation or non-drug/procedure recommendation.

Clinical context mapping

This is a component to map instances of clinical context semantic relations to their target CIG constructs as a set of rules. We used the openEHR Guideline Definition Language (GDL)²⁹ as our target CIG; GDL leverages the designs of openEHR Archetype Model that we used in the pattern detection step.

Although the clinical context mapping is still a manual task to be done by the knowledge engineer, it can be fully automated if the medical expert modeller used a standard naming convention for the clinical data structures used in the clinical context pattern detection component.

Our evaluation was based on measuring the precision, sensitivity/recall, and specificity of the extracted drug recommendation and procedure recommendation rules form the input recommendation sentences. The precision, sensitivity/recall, and specificity are measured based on the correctness of our framework in finding instances for the UIMA Ruta patterns defined by the medical expert. More formally, assume that I is the set of all sentences in a CPG and I_G denotes the subset of I that contains sentences with a medication and a disorder, or sentences with a procedure and a disorder; I_G’ denotes for all sentences in I that are not in I_G ; I_F denotes the set of sentences extracted by our framework; I_F’ denotes set of sentences not extracted by our framework

P r e c i s i o n = \frac{| I_{G} \cap I_{F} |}{| I_{F} |} = \frac{T r u e P o s i t i v e}{T r u e P o s i t i v e + F a l s e P o s i t i v e}

R e c a l l / s e n s i t i v i t y = \frac{| I_{G} \cap I_{F} |}{| I_{G} |} = \frac{T r u e P o s i t i v e}{T r u e P o s i t i v e + F a l s e N e g a t i v e}

S p e c i f i c i t y = \frac{| I_{F^{'}} |}{| I_{F^{'}} \cup {I_{F} \cap I_{G^{'}}} |} = \frac{T r u e N e g a t i v e}{F a l s e P o s i t i v e + T r u e N e g a t i v e}

Results

We implemented our formalization framework in JAVA (version 1.7) and integrated it with the GDL editor.²⁹ Due to the lack of access to independent human modellers, we could not measure the manual effort saving introduced by our framework; nevertheless, we evaluated the accuracy of the knowledge extracted by our framework for the drug recommendation and the procedure recommendation clinical contexts. To build our gold standard for the drug recommendation clinical context to measure against, we used all recommendations from the ‘Management of chronic pain’ CPG²⁶ and the ‘Management of lung cancer’ CPG,³⁰ and then we manually tagged each recommendation as either drug/procedure recommendation or non-drug/procedure recommendation. The resulted test data set is composed of 169 recommendation sentences. In Table 3, we show the accuracy of our framework on classifying the 169 recommendation sentences.

Table 3.

Recommendation sentences classification evaluation.

Recommendation	Precision (%)	Sensitivity/recall (%)	Specificity (%)
Chemical & Drugs	78	71	73
Procedure	70	75	79
Average	74	73	76

We evaluated the correctness of the formalized recommendations by manually checking the extracted rules and we assigned a coefficient of 1 for rules that are correctly coded and complete, 0.5 for rules that are correct but partial (e.g. not all elements of the rule conditions are captured), and 0 for rules that are wrong. In Table 4, we show the accuracy of the extracted rules based on the described metric. There are two main sources of errors for the wrong rules: either wrong MetaMap annotations or wrong classification from our clinical context filtering component using logistic regression.

Table 4.

Extracted rules’ accuracy.

Recommendation	Accuracy (%)
Chemical & Drugs	87
Procedure	81
Average	84

Discussion

Using information extraction, techniques to extract biomedical knowledge have been applied in other non-CPG biomedical documents.^31,32 The proposed framework is tailored to the specific needs of CPGs where the granularity of clinical knowledge differs between different types of CPGs; therefore, we designed the framework to enable the human modeller to control the granularity of the extracted clinical knowledge.

The proposed framework is based on a clinical context pattern detection component using UIMA Ruta, a rule-based information extraction mechanism to extract text fragments that contain the minimum necessary features of the clinical recommendations type in question. We believe that extracting clinical recommendation by following a top-down approach¹⁵ where only general rules that cover as many possible instances of clinical recommendation need to be defined is suitable for CPG formalization. General extraction rules tend to be in small numbers and are simple to define making the rule authoring task a good fit for medical experts who usually lack extensive knowledge in rule authoring. Although general information extraction rules result in high coverage, they have a poor precision; therefore, we added the clinical context filtering component using logistic regression to filter out false-positives introduced by general information extraction rules defined in the clinical context pattern detection component. The framework was tested on extracting only two clinical context types due to the lack of pre-annotated test data sets. In the following discussion, we analyse the achieved results:

The precision is impacted by the size and the quality of our training data set; in the presented example, we used a training data set made of 117 recommendation sentences extracted from YGRC which is small to provide high precision. This issue could be lessened by feeding the outputted rules of the framework back to the training data set, a step that requires a minor manual tagging of which rules are correctly extracted and which ones are wrongly extracted.

The sensitivity/recall is impacted by how we split our CPG into smaller text chunks, for example, in the presented example we split CPG into sentences, but some drug recommendations within the CPG have the drug and the medication located in two separate sentences, and therefore, these ones are missed by our extraction rules. This issue could be lessened by changing the size of our unit of analysis from one sentence to two consecutive sentences or to the whole paragraph, but such a modification would hurt the precision unless we add more rules to handle cross-sentence extraction. Different cross-sentence extraction approaches can be applied. One approach would be to perform cross-sentence extraction when a sentence only contains one part of the clinical context such as a sentence with only a disease, followed by a sentence that only contains the other part of the clinical context such as a sentence with only a medication. This approach is very conservative and would not impact the precision of the in-sentence extraction rule. Incorporating other cross-sentence extraction approaches that have more coverage would likely interfere with other in-sentence extraction rules. Therefore, with every cross-sentence extraction approach, we need to evaluate the cross-sentence extraction precision gain to the in-sentence precision loss.

The specificity is impacted by how strict are our conditions for tagging a sentence with a specific semantic relation. In the presented example, we achieved high specificity because our conditions for tagging drug medication semantic relations are very strict.

The proposed framework can be effectively used to formalize the drug recommendation and procedure recommendation clinical contexts of CPGs into CDSS-friendly format. More significantly, it provides human modellers with a methodology to extend the framework to formalize other clinical context of CPGs. The framework is focused on automating the CPG common formalization steps while allowing the human modeller to stay in control of all the knowledge extraction steps. By configuring the clinical context detection pattern component, the human modeller could define how CPG text would be split into smaller chunks for analysis and control the level of granularity of the clinical knowledge extracted. This control balances generality and specificity in order to maximize usefulness of the extracted knowledge. If the extracted knowledge is too specific/expressive, it unnecessarily complicates the extraction rules. We believe that such configuration capabilities in our framework would help reduce the human modeller’s annoyance and dissatisfaction accompanied with either the lengthy manual CPG formalization steps or the inflexibility of other automated CPG formalization approaches.

Footnotes

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Field

Lohr

. Clinical practice guidelines: directions for a new program. Washington, DC: National Academies Press, 1990, p. 90.

Woolf

Grol

Hutchinson

. Potential benefits, limitations, and harms of clinical guidelines. BMJ 1999; 318: 527–530.

Davis

Thomson

Oxman

. Evidence for the effectiveness of CME: a review of 50 randomized controlled trials. JAMA 1992; 268: 1111–1117.

Kilsdonk

Peute

Riezebos

. From an expert-driven paper guideline to a user-centred decision support system: a usability comparison study. Artif Intell Med 2013; 59: 5–13.

Woolf

. Evidence-based medicine and practice guidelines: an overview. Cancer Control 2000; 7: 362–367.

Peleg

. Computer-interpretable clinical guidelines: a methodological review. J Biomed Inform 2013; 46(4): 744–763.

Medlock

Opondo

Eslami

. LERM (Logical Elements Rule Method): a method for assessing and formalizing clinical rules for decision support. Int J Med Inform 2011; 80: 286–295.

Svátek

Ruzicka

. Step-by-step mark-up of medical guideline documents. Int J Med Inform 2003; 70: 329–335.

Serban

ten Teije

van Harmelen

. Extraction and use of linguistic patterns for modelling medical guidelines. Artif Intell Med 2007; 39: 137–149.

10.

Serban

ten Teije

. Exploiting thesauri knowledge in medical guideline formalization. Method Inform Med 2009; 48: 468.

11.

Kaiser

Akkaya

Miksch

. How can information extraction ease formalizing treatment processes in clinical practice guidelines? A method and its evaluation. Artif Intell Med 2007; 39: 151–163.

12.

Kaiser

Miksch

. Versioning computer-interpretable guidelines: semi-automatic modeling of ‘Living Guidelines’ using an information extraction method. Artif Intell Med 2009; 46: 55–66.

13.

Taboada

Meizoso

Riaño

. From natural language descriptions in clinical guidelines to relationships in an ontology. In: Riaño

ten Teije

Miksch

. (eds) Knowledge representation for health-care: data, processes and guidelines. Berlin, Heidelberg: Springer, 2010, pp. 26–37.

14.

Wenzina

Kaiser

. Identifying condition-action sentences using a heuristic-based information extraction method. In: Riaño

Lenz

Miksch

. (eds) Process support and knowledge representation in health care. Berlin, Heidelberg: Springer, 2013, pp. 26–38.

15.

Sarawagi

. Information extraction. Found Trends Database 2008; 1: 261–377.

16.

Seyfang

Martínez-Salvador

Serban

. Maintaining formal models of living guidelines efficiently. In: Bellazzi

Abu-Hanna

Hunter

(eds) Artificial intelligence in medicine. Berlin, Heidelberg: Springer, 2007, pp. 441–445.

17.

Ferrucci

Lally

. UIMA: an architectural approach to unstructured information processing in the corporate research environment. Nat Lang Eng 2004; 10: 327–348.

18.

National Guideline Clearinghouse, http://www.guideline.gov (accessed 2 February 2015).

19.

Aronson

. Effective mapping of biomedical text to the UMLS Metathesaurus: the MetaMap program. Proceedings of the annual symposium of the American Medical Informatics Association. Washington, DC: Hanley and Belfus, 2001, pp. 17–21.

20.

Rindflesch

Kilicoglu

Fiszman

. Semantic MEDLINE: an advanced information management application for biomedicine. Inform Serv Use 2011; 31: 15–21.

21.

Rogers

. Using the MetaMap UIMA Annotator, 2010, https://metamap.nlm.nih.gov/Docs/README_uima.html

22.

Gad El-Rab

Zaïane

El-Hajj

. Unsupervised graph-based word sense disambiguation of biomedical documents. In: IEEE 15th international conference on e-Health networking, applications & services (Healthcom), Lisbon, 9–12 October 2013, pp. 649–652. New York: IEEE.

23.

Kluegl

Toepfer

Beck

. UIMA Ruta: rapid development of rule-based information extraction applications. Nat Lang Eng 2014; 22: 1–40.

24.

Bodenreider

McCray

. Exploring semantic groups through visual approaches. J Biomed Inform 2003; 36: 414–432.

25.

Beale

. Archetypes: constraint-based domain models for future-proof information systems. OOPSLA 2002 workshop on behavioural semantics, Seattle, WA, 4 November 2002.

26.

Management of chronic pain: a national clinical guideline, http://www.sign.ac.uk/pdf/SIGN136.pdf (accessed 2 February 2015).

27.

Hosmer

Jr Beale

Lemeshow

. Applied logistic regression. Hoboken, NJ: John Wiley & Sons, 2004.

28.

Hussain

Michel

Shiffman

. The Yale Guideline Recommendation Corpus: a representative sample of the knowledge content of guidelines. Int J Med Inform 2009; 78(5): 354–363.

29.

Chen

. Guideline Definition Language (GDL), http://www.openehr.org/news_events/releases/releases.php?id=79 (accessed 2 February 2015).

30.

Management of lung cancer, http://www.sign.ac.uk/pdf/SIGN137.pdf (accessed 10 October 2015).

31.

Bratsas

Koutkias

Kaimakamis

. KnowBaSICS-M: an ontology-based system for semantic management of medical problems and computerised algorithmic solutions. Comput Methods Programs Biomed 2007; 88(1): 39–51.

32.

Uramoto

Matsuzawa

Nagano

. A text-mining system for knowledge discovery from biomedical documents. IBM Syst J 2004; 43(3): 516–533.