SuperOrder: Provider order recommendation system for outpatient clinics

Abstract

This study aims at developing SuperOrder, an order recommendation system for outpatient clinics. Using the electronic health record data available at midnight, SuperOrder predicts the order contents for each upcoming appointment on a daily basis. A two-level prediction framework is proposed. At the base-level, the predictions are produced by aggregating three machine learning methods. The meta-level predictions are generated by integrating the base-level predictions with the order co-occurrence network. We used the retrospective data between 1 April 2014 and 31 March 2015 in pulmonary clinics from five hospital sites within a large rural health care facility in Pennsylvania to test the feasibility. With a decrease of 6 per cent in the precision, the improvement of the recall at the meta-level is approximately 20 per cent from the base-level. This demonstrates that the proposed order co-occurrence network helps in increasing the performance of order predictions. The implementation will bring a more effective and efficient way to place outpatient orders.

Keywords

clinical decision model machine learning network analytics order recommendation system outpatient clinics

Introduction

Orders placed by healthcare providers for care, treatments and diagnostic tests to patients are termed as clinical orders. Providers make these orders with their clinical knowledge considering patients’ demographics, medical history and current symptoms and diagnoses. To place a single order, it requires providers to manually type the order name or keywords to find the order from the search results. If providers look for multiple orders, this process needs to be repeated until all the desired orders are filled.^1,2 This is a time-consuming process, and the resulting cognitive burden may lead to wrong, missed and duplicated orders. A number of studies have dedicated to stopping order errors by setting up alarms at the order entry to avoid drug–drug interactions and overdose.^3–7 Other researchers have identified the areas of improvement for clinical ordering through investigating how the provider and patient characteristics affect their order patterns.^8,9

In order to enhance clinical order efficiency and consistency, previous studies have focused on designing order sets, which are the collections of orders grouped by specific clinical purposes such as a certain type of diagnoses or procedures.^1,2,10,11 Literature shows that well-developed order sets can save time in searching necessary and sufficient orders for patients and lower the variability in medical care.^1,11,12 However, order sets require high cost of development, maintenance and updating.^2,12–14 In addition, most of the order sets lack diverse patient conditions and consistency with the current practice, and therefore some studies show the low usage frequency of order sets.^13–15 An alternative approach to reinforce the order effectiveness is to build an automated recommendation system for clinical orders, which is analogous to Amazon.com’s recommendation system where customers who bought A also bought B.^13,15–20 Frequent itemset mining and association rule mining were used to identify which clinical orders are often placed together.^15,17,19 However, only the associations between order themselves were considered and other factors were ignored such as patients’ medical history and current symptoms. To address this issue, some order recommendation systems were designed to incorporate other non-order clinical items such as diagnoses and lab results using Bayesian network.^16,18,21 These schemes were limited to inpatient settings, in which a suggested list was generated after the initial diagnoses and orders were made. This concept is not applicable to outpatient clinics where a suggested order is demanded before the provider sees the patient since there is not enough time during the appointments.

This study aims at establishing SuperOrder, an automated order recommendation system in outpatient clinics, to predict the order contents of each appointment before a provider sees a patient. The predictions are generated based on the data from Electronic Health Records (EHR) available at midnight before the corresponding appointments occur. By giving a list of potential orders, SuperOrder will help to ease order documentation and increase order efficacy. With SuperOrder, the providers will have more time for patients, which will improve care quality and patient satisfaction. To test the system feasibility, we use 1-year EHR data in pulmonary-related outpatient clinics from five hospital sites at Geisinger Health System (Geisinger) which serves rural areas across northeastern and central Pennsylvania. The reason we choose data from pulmonary-related clinics is that order contents regarding pulmonary appointments are relatively simple compared to those in other specialties. Once the performance is validated, SuperOrder will be generalized to other specialties and benefit hospital-wide providers.

Methods

Study design

SuperOrder was developed using retrospective data in outpatient pulmonary clinics from 1 April 2014 to 31 March 2015, in which a total of 9815 patients and 18,348 appointments were involved. Pulmonary-related outpatient clinics consisted of three specialties, including pulmonary, thoracic medicine and thoracic surgery. In all, 42 providers, including 25 physicians, 3 physician assistants, 8 certified registered nurse practitioners, 5 fellows and 1 resident, were involved. There were six different order types, including laboratory tests, imaging tests, pulmonary function tests (PFTS), other tests, referrals and medications. The contents of medications involved drug dose and drug–drug interaction, which alone were complicated and should be a separate problem from the other types of orders, so in this article, we excluded medication orders. The total number of orders excluding medication orders was 34,816. The recommendation was made based on the data available at midnight before appointments, so we only considered predicting the orders made at the end of or after appointments. Orders might be cancelled or incomplete after placed due to variety of reasons. Since these cancelled and incomplete orders were truly under consideration of providers, in this study, these orders were considered the target orders as well. Same orders might be repeated in one appointment. Our objective was to predict whether an order has been placed or not, not the number of times it was ordered. During the study period, 724 different kinds of orders were collected. Most orders had only small volume and overlooking them had only small effect on information loss but improved computational efficiency. In consultation with physicians, we focused on the top 34 (4.5%) common orders (Table 1). Below is the process how these 34 orders were selected:

Table 1.

Top 34 common orders in pulmonary clinics.

Order type	Order name	Abbreviation
Lab test	25-hydroxy vitamin D test	25OHD
	Acid-fast bacilli test	AFB
	Activated partial thromboplastin time test	APTT
	Allergen-specific immunoglobulin E	IgE
	Alpha-1 antitrypsin test	AAT
	Antinuclear antibody test	ANA
	Arterial blood gas test	ABG
	Basic metabolic panel	BMP
	Blood group and type test	ABO
	Complete blood count test	CBC
	Comprehensive metabolic panel	COMPR
	Gram stain test	GRAM
	Hepatic function panel	HEPFA
	Northeast regional allergy panel	NERAST
	Prothrombin time and international normalized ratio test	PT/INR
	Rheumatoid factor test	RF
Imaging test	Chest computerized axial tomography scan	CHEST CT
	Chest X-ray	CXR
	Echocardiography	ECHO
	Positron emission tomography–computed tomography (whole body)	PET CT FULL
PFTS	Pulmonary function tests	PFTS
Other test	Electrocardiogram	EKG
	Nocturnal home oximetry	NHO
	Plethysmography–airway resistance	RAW
	Pulse oximeter test	PULSE OX
	Respiratory test	RESP
	Six minute walk test	SMW
	Stress test	ST
	Test of diffusing capacity of the lungs for carbon monoxide	DLCO
	Tobacco cessation counsel	TCC
Referral	Cardiology referral	CARD REF
	Occupational therapy referral	OT REF
	Pulmonary rehabilitation referral	PULM REHAB REF
	Sleep medicine referral	SLEEP MED REF

We included all the orders with the order volume higher than 1 per cent.

Except sleep medicine referral, the order volume of all other referrals was less than 1 per cent (Table 2). To consider more referral orders, the first four most common referrals were included.

Electrocardiogram (EKG) was one of the common orders for pulmonary patients and was included in this study although the order volume was less than 1 per cent.

Table 2.

Appointments in pulmonary-related clinics.

	Number of appointments
Total	18,348
Specialties
Pulmonary	5339 (29.10%)
Thoracic Medicine	10,684 (58.23%)
Thoracic Surgery	2325 (12.67%)
Appointments with order or not
Appointments with orders	12,627 (68.82%)
Appointments without orders	5712 (31.18%)
Order type
Lab test	3388 (18.47%)
CBC	1134 (6.18%)
BMP	894 (4.87%)
AFB	644 (3.51%)
IgE	615 (3.35%)
PT/INR	567 (3.09%)
NERAST	403 (2.20%)
APTT	383 (2.09%)
ABG	251 (1.37%)
ABO	248 (1.35%)
ANA	236 (1.29%)
25OHD	232 (1.26%)
RF	229 (1.25%)
COMPR	223 (1.22%)
GRAM	213 (1.16%)
AAT	209 (1.14%)
HEPFA	206 (1.12%)
Imaging test	6635 (36.16%)
CHEST CT	3512 (19.14%)
CXR	2204 (12.01%)
ECHO	784 (4.27%)
PET CT FULL	230 (1.25%)
PFTS	5577 (30.40%)
Other test	5030 (27.41%)
SMW	1017 (5.54%)
NHO	833 (4.54%)
PULSE OX	746 (4.07%)
DLCO	633 (3.45%)
ST	432 (2.35%)
RAW	377 (2.05%)
TCC	320 (1.74%)
RESP	232 (1.26%)
EKG	160 (0.87%)
Referral	1824 (9.94%)
SLEEP MED REF	800 (4.36%)
OT REF	168 (0.92%)
CARD REF	141 (0.77%)
PULM REHAB REF	112 (0.61%)

CBC: complete blood count test; BMP: basic metabolic panel; AFB: acid-fast bacilli test; IgE: allergen-specific immunoglobulin E; PT/INR: prothrombin time and international normalized ratio test; NERAST: northeast regional allergy panel; APTT: activated partial thromboplastin time test; ABG: arterial blood gas test; ABO: blood group and type test; ANA: antinuclear antibody test; 25OHD: 25-hydroxy vitamin D test; RF: rheumatoid factor test; COMPR: comprehensive metabolic panel; GRAM: Gram stain test; AAT: Alpha-1 antitrypsin test; HEPFA; hepatic function panel; CHEST CT: chest computerized axial tomography scan; CXR: chest X-ray; ECHO: echocardiography; PET CT FULL: positron emission tomography – computed tomography (whole body); PFTS: pulmonary function tests; SMW: Six minute walk test; NHO: nocturnal home oximetry; PULSE OX: pulse oximeter test; DLCO: test of diffusing capacity of the lungs for carbon monoxide; ST: Stress test; RAW: plethysmography – airway resistance; TCC: tobacco cessation counsel; RESP: respiratory test; EKG: electrocardiogram; SLEEP MED REF: sleep medicine referral; OT REF: occupational therapy referral; CARD REF: cardiology referral; PULM REHAB REF: pulmonary rehabilitation referral. Bold-faced signifies names and numbers for higher-level categories.

The order volume of these 34 orders is 25,128 (72.2%) during the study period. Other detailed information regarding appointments and orders is shown in Table 2.

Model framework

Figure 1 shows the framework of SuperOrder, in which the order predictions are divided into two levels. At the base-level, the occurrences of different orders are assumed to be independent of each other, so that the predictions can be achieved by aggregating the outputs of a pool of individual classifiers. Each classifier predicts whether a given appointment contains a specific order. At the meta-level, the predictions are produced by integrating an order co-occurrence network with the base-level predictions. In the co-occurrence network, nodes relate to orders, and if two orders co-occur in one appointment, the nodes are connected by a link. Repeated co-occurrences strengthen the link between the particular nodes. The predicted orders at the base-level are matched with the nodes in the co-occurrence network, and the links are followed to extract the sub-graph. In the sub-graph, we can identify the new nodes (orders) that are missing from the base-level. We hypothesize that the meta-level predictions we have formulated can obtain a more complete and accurate order recommendation list than the base-level predictions.

Figure 1.

Framework of SuperOrder.

The whole data set was split into two parts. The first 11-month data (01 April 2014–28 February 2015) were utilized for training each binary classifier, testing the classifier performance and deciding a cut-off threshold for each classifier. The first 11-month data were also used for constructing the order co-occurrence network. At the base-level, for each classifier, 80 per cent of the first 11-month data were randomly selected for training, and 20 per cent of them for testing the classifier performance and deciding the cut-off threshold. To avoid that some small-volume orders have too few samples in a test set, for each classifier, we forced 80 per cent of positive outcomes to be in the training set and 20 per cent of them to be in the test set under the random selection. This random sampling caused that different orders might have different training and testing sets. The last-month data (01 March 2015–31 March 2015) were used to test the performance of the aggregated predictions at the base-level and the meta-level.

Base-level predictions

At the base-level, each binary classifier was built by three methods including logistic regression (LR), random forest (RF) and gradient boosting method (GBM).^22–24 For each order, we chose the method which produced the best performance on the corresponding test set in terms of the area under precision–recall (PRC) curve (AUPRC).²⁵ This method was denoted as Mixed method to distinguish this from purely using LR, RF and GBM in the remainder of this article. The next step was to determine a cut-off point of each classifier. SuperOrder was expected to have a precision no less than 0.3, so that for each classifier, we selected the cut-off point which had the maximum recall value given a precision of 0.3 on the test set. Recall and F-measure were the metrics to evaluate model’s performance.²⁶ Each binary classifier used the same pool of predictors (Table 3). The total number of the predictors was 152.

Table 3.

Predictors of SuperOrder.

Predictor category	Predictor descriptions
Patient demographics	• Age (0–20, 20–49, 50–59, 60–69, 70–79, 80+) • Gender
Visit type, reason and expected time length	• Visit type (new patients, return patients, post op, …) • Reason for visit (sick visit, follow-up, short of breath, …) • Time length (15, 30, 45, 60 min)
Pulmonary chronic disease	• Asthma, COPD, etc.
Patient’s past history within 1 year at Geisinger	• Number of distinct specialties • Number of ED visits • Time to the last ED visits • Number of inpatient visits • Time to the last inpatient visits
Patient previous orders	• Whether a lab/imaging/PFTS/other test/referral order has been placed within 1 month/3 months/6 months/1 year/3 years/5 years before the current appointment. • Whether the current appointment had an order placed before the appointment occurred
Provider	• Skill level • Specialty • Secondary provider or not

PFTS: pulmonary function tests, COPD: Chronic Obstructive Pulmonary Disease.

Meta-level predictions

At the meta-level predictions, an order co-occurrence network was built to be directed and weighted. To derive edge weights in this network, we traced back to the bipartite network of orders and appointments. The order-appointment bipartite network was denoted as G = (U, V, E), where U = {u₁, u₂, …, u_n} was a set of orders and V = {v₁, v₂, …, v_m} was a set of appointments. E was a set of undirected and unweighted edges, which connected nodes in U with nodes in V. a (u_i, v_j) was an adjacency matrix of network G, where

a (u_{i}, v_{j}) = {\begin{cases} 1, (u_{i}, v_{j}) \in E \\ 0, otherwise \end{cases}

(1)

The order co-occurrence network, $G^{'} = (U, E^{'})$ , was a one-mode projection of $G = (U, V, E)$ , where $E^{'}$ was a set of directed and weighted edges connecting nodes in $U$ . The adjacency matrix of $G^{'}$ , $a^{'} (u_{i}, u_{j})$ , was designed as follows

a^{'} (u_{i}, u_{j}) = \frac{\sum_{k = 1}^{m} a (u_{i}, v_{k}) \cdot a (u_{j}, v_{k})}{\sum_{k = 1}^{m} a (u_{i}, v_{k})}

(1)

In the weighting system, $a^{'} (u_{i}, u_{j})$ considered not only the number of appointments containing both order $u_{i}$ and order $u_{j}$ but also the number of appointments containing $u_{i}$ . For example, about one-third of pulmonary appointments had order PFTS, but only about 4 per cent of pulmonary appointments had order BMP. Suppose that the number of appointments containing both order PFTS and another order $u_{i}$ was the same as the number of appointments that contains both order BMP and order $u_{i}$ . The contribution of PFTS to order $u_{i}$ should be much less than the contribution of BMP to order $u_{i}$ since the co-occurrences of PFTS and order $u_{i}$ might just result from the prevalence of PFTS. We defined $a^{'} (u_{i}, u_{j})$ as the number of appointments that contain both order $u_{i}$ and order $u_{j}$ divided by the number of appointments containing order $u_{i}$ . Figure 2 shows a synthetic example of order-appointment network and the corresponding one-mode projection network on orders. Both order $u_{2}$ and $u_{4}$ co-occur once with order $u_{5}$ . Since four appointments contain order $u_{2}$ and only two appointments contain order $u_{4}$ , the edge weight from order $u_{2}$ to order $u_{5}$ is only one-fourth, but the weight from $u_{4}$ to order $u_{5}$ increases to one-half.

Figure 2.

Development of order co-occurrence network from bipartite order-appointment network.

This order co-occurrence network was used for identifying the potential orders missing in the base-level predictions due to the assumption of order independence. Suppose that appointment x was predicted to contain a set of predicted orders $B U^{x}$ at the base-level, where $B U^{x} = {b u_{1}^{x}, \dots, b u_{p_{x}}^{x}}$ and $p_{x}$ was the number of predicted orders at the base-level for appointment x. The predicted score of order $u_{i}$ for appointment x, $s^{x} (u_{i})$ , is given below

s^{x} (u_{i}) = \sum_{j = 1}^{p_{x}} a^{'} (b u_{j}^{x}, u_{i}), \forall u_{i} \notin B U^{x}

(2)

By deciding a proper threshold for $s^{x} (u_{i})$ , $T_{s}$ , we could identify potential orders which are missing at the base-level. Figure 3 demonstrates how to use the order co-occurrence network to find the potential orders missing at the base-level. Appointment x is predicted to have order $u_{2}$ and $u_{3}$ at the base-level, which are marked as red in the figure. The predicted scores of order $u_{1}$ , $u_{5}$ and $u_{6}$ , marked as green, are computed since three of them are the neighbours of either order $u_{2}$ and $u_{3}$ . Suppose that we use the threshold $T_{s} = 1$ to identify potential orders at the meta-level for appointment x. In addition to order $u_{2}$ and $u_{3}$ , order $u_{1}$ is another potential order since the corresponding predicted score is 1.25, which is higher than threshold $T_{s} = 1$ .

Figure 3.

Illustration of predicted score calculation at the meta-level.

Results

SuperOrder modelling was developed using R 2.3.2. The performances between LR, RF and GBM were compared across the 34 orders in terms of AUPRC. The best method for each order and the corresponding performance on the test set and last-month data set are shown in Table 4. In general, RF performs the best. For each classifier, although the performance may vary from the test set to the last-month data set, the average of F-measure over all the classifiers on the test sets is 0.35, which is very close to an average F-measure of 0.36 on the last-month data set.

Table 4.

Cut-off point of each order and the corresponding prediction performance.

Order name	Method	AUPRC on test set	Cut-off point	Test set			Last-month data set
Order name	Method	AUPRC on test set	Cut-off point	Precision	Recall	F-measure	Precision	Recall	F-measure
25OHD	RF	0.7873	0.1220	0.3	0.7895	0.4348	0.6522	0.8824	0.7500
AFB	RF	0.7441	0.0630	0.3	0.7600	0.4302	0.4182	0.8214	0.5542
APTT	RF	0.7249	0.0530	0.3	0.8361	0.4416	0.2069	0.7200	0.3214
IgE	LR	0.4887	0.1109	0.3	0.5728	0.3938	0.2589	0.4677	0.3333
AAT	LR	0.0710	0.8411	0.3	0.0278	0.0508	0.0000	0.0000	0.0000
ANA	GBM	0.0838	0.3386	0.3	0.0476	0.0822	0.0000	0.0000	0.0000
ABG	RF	0.1638	0.2990	0.3	0.1053	0.1558	0.0000	0.0000	0.0000
BMP	GBM	0.8936	0.0213	0.3	0.9384	0.4546	0.4591	0.9481	0.6186
ABO	RF	0.6659	0.0710	0.3	0.8824	0.4478	0.2286	0.8000	0.3556
CBC	GBM	0.8954	0.0151	0.3	0.9375	0.4545	0.3012	0.9286	0.4548
COMPR	RF	0.8410	0.0650	0.3	0.8462	0.4430	0.3514	0.8667	0.5000
GRAM	RF	0.6611	0.1190	0.3	0.6389	0.4083	0.5556	0.7143	0.6250
HEPFA	RF	0.8179	0.1000	0.3	0.8182	0.4390	0.6316	0.5714	0.6000
NERAST	RF	0.1523	0.2410	0.3	0.0952	0.1446	0.4706	0.1739	0.2540
PT/INR	RF	0.8484	0.0500	0.3	0.9111	0.4514	0.2909	0.8421	0.4324
RF	RF	0.1548	0.1830	0.3	0.1500	0.2000	0.1250	0.0500	0.0714
CHEST CT	GBM	0.8197	0.0444	0.3	0.9511	0.4561	0.2785	0.9502	0.4307
CXR	LR	0.9042	0.0151	0.3	0.9719	0.4585	0.2592	0.9389	0.4063
ECHO	GBM	0.5597	0.0799	0.3	0.6107	0.4023	0.2703	0.4301	0.3320
PET CT FULL	RF	0.0825	0.2360	0.3	0.1081	0.1589	0.1667	0.0500	0.0769
PFTS	GBM	0.9005	0.0169	0.3	1.0000	0.4615	0.3027	1.0000	0.4647
EKG	RF	0.4202	0.1200	0.3	0.5000	0.3750	0.3158	0.6667	0.4286
NHO	LR	0.5446	0.0971	0.3	0.5621	0.3912	0.2358	0.4464	0.3086
RAW	RF	0.2904	0.1290	0.3	0.4375	0.3559	0.6022	0.2593	0.3625
PULSE OX	RF	0.7483	0.0620	0.3	0.7778	0.4330	0.5026	0.6621	0.5714
RESP	RF	0.5423	0.0870	0.3	0.5385	0.3853	0.4333	0.6842	0.5306
SMW	GBM	0.5872	0.0842	0.3	0.8125	0.4382	0.1608	0.5287	0.2466
ST	RF	0.4843	0.1180	0.3	0.5636	0.3916	0.6296	0.3469	0.4474
DLCO	RF	0.4300	0.1230	0.3	0.3929	0.3402	0.9076	0.3112	0.4635
TCC	RF	0.6615	0.0910	0.3	0.8814	0.4476	0.3220	0.8636	0.4691
CARD REF	RF	0.0991	0.1570	0.3	0.1250	0.1765	0.0000	0.0000	0.0000
OT REF	GBM	0.0745	0.2145	0.3	0.0606	0.1008	0.2000	0.0476	0.0769
PULM REHAB REF	LR	0.0494	0.1360	0.3	0.2222	0.2553	0.1429	0.2778	0.1887
SLEEP MED REF	GBM	0.5276	0.0925	0.3	0.6000	0.4000	0.3263	0.6139	0.4261
		Average		0.3	0.5727	0.3488	0.3237	0.5254	0.3559

AUPRC: area under precision–recall (PRC) curve; 25OHD: 25-hydroxy vitamin D test; RF: random forest; CBC: complete blood count test; BMP: basic metabolic panel; AFB: acid-fast bacilli test; IgE: allergen-specific immunoglobulin E; LR: logistic regression; PT/INR: prothrombin time and international normalized ratio test; NERAST: northeast regional allergy panel; APTT: activated partial thromboplastin time test; ABG: arterial blood gas test; ABO: blood group and type test; ANA: antinuclear antibody test; RF: rheumatoid factor test; COMPR: comprehensive metabolic panel; GRAM: Gram stain test; AAT: Alpha-1 antitrypsin test; HEPFA; hepatic function panel; CHEST CT: chest computerized axial tomography scan; CXR: chest X-ray; ECHO: echocardiography; PET CT FULL: positron emission tomography – computed tomography (whole body); PFTS: pulmonary function tests; SMW: Six minute walk test; GBM: gradient boosting method; RESP: respiratory test; EKG: electrocardiogram; NHO: nocturnal home oximetry; PULSE OX: pulse oximeter test; DLCO: test of diffusing capacity of the lungs for carbon monoxide; ST: Stress test; RAW: plethysmography – airway resistance; TCC: tobacco cessation counsel; CARD REF: cardiology referral; OT REF: occupational therapy referral; PULM REHAB REF: pulmonary rehabilitation referral; SLEEP MED REF: sleep medicine referral.

Table 5 illustrates the performance of aggregated predictions over all the classifiers in terms of micro-average precision, recall and F-measure on the last-month data set. It compares the predictions using Mixed method against those purely using LR, RF and GBM. The results show that all the methods have the precisions higher than 0.3, and Mixed method outperforms the other three methods in terms of F-measure. Other than the 34 orders, there are 690 types of orders excluded in the recommendation system. By including them as false negatives in the system and recalculating the recall and F-measure, Mixed method achieves a recall of 0.5232. It means that on average, a provider can find more than half of the items he or she wants to place for the patients on the list.

Table 5.

Performance at the base-level by different methods.

Method	For 34 orders				For all orders
Method	Precision	Recall	F-measure	Specificity	Precision	Recall	F-measure	Specificity
Mixed method	0.3191	0.6515	0.4284	0.9274	0.3191	0.5232	0.3965	0.9968
LR	0.3224	0.6111	0.4221	0.9349	0.3224	0.4907	0.3892	0.9971
RF	0.3180	0.6448	0.4259	0.9278	0.3180	0.5178	0.3940	0.9968
GBM	0.3070	0.5809	0.4017	0.9305	0.3070	0.4665	0.3703	0.9969

LR: logistic regression; RF: random forest; GBM: gradient boosting method.

To take the order dependence into account, the order co-occurrence network is built as shown in Figure 4, which contains 34 nodes and 1000 edges. The network was visualized using Gephi 9.2.0. The size of a node corresponds to its in-degree. Nodes with high in-degree, such as order ‘PFTS’ and ‘CHEST CT’, represent the orders that are usually placed with other orders by providers. The arrow size represents the edge weight from one node to another. The heaviest weight is 0.38, which occurs on the edge from node ‘ABG’ to ‘PFTS’, followed by the edge from node ‘SLEEP MED REF’ to ‘PFTS’ with a weight of 0.36.

Figure 4.

Pulmonary order co-occurrence network.

Using the order co-occurrence network with a desired weighting threshold $T_{s}$ , we enabled SuperOrder to consider the order dependency and supplement the base-level predictions. Figure 5 compares the precision and recall of the meta-level predictions with different weighting thresholds against those of the base-level predictions. Note that the performance metrics computed consider all 724 orders. It states that for Mixed method, LR and RF, the order co-occurrence network does help in increasing the value of recall when $T_{s} = 0.6$ without sacrificing the value of precision. For Mixed method, the recall and precision of the meta-level predictions with $T_{s} = 0.6$ are 0.5506 and 0.3202, respectively, which are the 5.24 per cent and 0.34 per cent improvement from the base-level predictions. In addition, when $T_{s} = 0.5$ and $T_{s} = 0.4$ , although the precisions of the meta-level predictions by Mixed method drop a little bit to 0.3142 and 0.2992, respectively, the recalls of them have much more improvement compared to that of the base-level predictions. The improvement of the recall of the meta-level predictions is 8.77 per cent when $T_{s} = 0.5$ and is 19.67 per cent when $T_{s} = 0.4$ . On the contrary, when GBM is used, the order co-occurrence network seems to not help in enhancing the performance of the predictions. When $T_{s} = 0.6$ , the performance of the meta-level predictions by GBM is the same as the corresponding base-level predictions. When $T_{s} = 0.5$ , the precision and the recall of the meta-level predictions are 0.3058 and 0.5816, respectively. And when $T_{s} = 0.4$ , the precision and the recall of the meta-level predictions are 0.3026 and 0.5837, respectively. They are also very similar to those of the base-level predictions.

Figure 5.

Comparison of the meta-level against the base-level predictions in terms of precision and recall: (a) Mixed method, (b) logistic regression, (c) random forest and (d) gradient boosting method.

Figure 6 compares the F-measure of meta-level predictions against that of base-level predictions across all the four methods. Note that the results consider all 724 medical orders in pulmonary clinics. It shows that when Mixed method, LR and RF are adopted, the order co-occurrence network helps in improving the prediction performance. However, when we use GBM, the order co-occurrence network cannot enhance the prediction power.

Figure 6.

F-measures for meta-level predictions and base-level predictions.

Discussion

SuperOrder, analogous to commercial recommendation systems, has been developed to predict the provider orders for upcoming outpatient appointments by statistical and graph mining techniques on the data from EHR. By embedding this prediction model in the EHR system, we aimed at easing provider order documentation and enhancing order effectiveness to improve care quality and patient satisfaction. Among these 724 different orders in pulmonary clinics, we focused on predicting 34 common orders which occupied 72 per cent of the total order volume in pulmonary-related clinics.

For these 34 common orders in pulmonary-related clinics, SuperOrder was designed as a two-level prediction model. At the base-level, the predictions were produced by aggregating the outputs of a pool of individual classifiers, where each classifier was to predict whether a given appointment contains a specific order. For each classifier, LR, RF and GBM were used to train the model, and the one producing the best performance on the corresponding test set is selected to be the method for that classifier. This framework was under the hypothesis that different methods used on different orders could provide the best prediction power. Table 4 shows this aggregated mechanism (Mixed method) did outperform the model with only one single prediction method. Under the circumstances that SuperOrder has a precision of no less than 0.3, the base-level predictions could achieve a recall of 0.5232, which meant that the base-level predictions correctly pointed out more than half of the orders that providers wanted to give to patients. This demonstrated that for half of the orders, instead of searching the keywords and finding the exact names of the orders one by one, providers can simply hit the orders shown on the recommendation list and submit the orders. This can reduce approximately half of the time previously required by providers for order placement.

After the base-level predictions generated, the order co-occurrence network was developed to incorporate order dependence and produce the meta-level predictions. The results from Figure 5 show that when Mixed method, LR and RF are used for the base-level predictions, the order co-occurrence network does increase the recall considerably with only little sacrifice in precision. This improvement is reaffirmed by comparing the F-measure of the meta-level predictions to that of the base-level predictions as shown in Figure 6. It proves the hypothesis that the meta-level predictions generate a more complete and accurate order recommendation list than the base-level predictions when Mixed method, LR and RF are used. However, when GBM is applied to produce the base-level predictions, the utilization of the order co-occurrence network cannot produce more accurate meta-level predictions, which violates our hypothesis. We speculate that this is caused by the insufficient accuracy of the base-level predictions by GBM. Of note, the meta-level predictions are generated based on the corresponding base-level predictions integrating with the order co-occurrence network. Therefore, the poor base-level predictions are very likely to lead the poor meta-level predictions even if an order co-occurrence network is well-developed. We conclude that the accurate enough base-level predictions are required for producing the better meta-level predictions with our order co-occurrence network.

As displayed in Figure 6, by using Mixed method, the meta-level predictions with $T_{s} = 0.4$ can achieve a recall of 0.6261 with a precision of 0.2992, which is very close to the expected value of 0.3. This method has the best performance in terms of recall and F-measure among all the methods we test in the study. It demonstrates that by using Mixed method with the order co-occurrence network, SuperOrder is expected to capture more than 60 per cent of the orders that providers will place for the upcoming patients. The use of the meta-level predictions will save more than 60 per cent of the total order documentation time, which is about 20 per cent of the improvement from the base-level predictions. This indicates that providers will reduce 60 per cent of the documentation time that is required without a prediction method and will have more time to treat and consult with patients. This we expect will improve the quality of care and patient satisfaction.

Limitations

This study has some limitations. First, the order volume of some procedures is highly fluctuating over time. This changing trend sometimes cannot be captured by the training set and test set. Figure 7 shows the four procedures that have the steep increase in order volume after January 2015. Since our study period for training and testing is the first 11 months, it is insufficient to detect the sudden increase in order volume in the next month. Therefore, when we test our prediction model on the last-month data (12th month), these four procedures have higher precision than expected but much lower recall. In addition, only 1-year patient data cannot distinguish if this increase is due to seasonality or new treatment introduction. In an attempt to solve this problem, we suggest that longer study period is needed (at least 2–3 years) to improve the prediction performance. Furthermore, SuperOrder has to be updated frequently in response to new treatment introduction or policy changes.

Figure 7.

Order volumes of the four common procedures by month.

Although there are 724 different orders in pulmonary-related clinics during the study period, we only consider 34 common orders. Even if our predictions are all correct, the maximum recall we can achieve is 72 per cent. In order to improve the recall of SuperOrder without the loss of the precision, predicting the occurrence of other uncommon orders is necessary. How to predict these uncommon orders will be a challenge in the future. In our collected data, there was a data field named ‘order status’ indicating if an order was cancelled, sent, resulted or completed. However, many of the records had missing value in this field or showed that the orders were sent or resulted, which indicated that the order status was not updated after completion or cancellation. Therefore, we were unable to perform further research on cancelled orders. In future, if the information regarding order status can be more reliable, SuperOrder will incorporate the model to predict the likelihood of the orders being cancelled as well.

Conclusion

The abundant information in EHR data provides an opportunity to develop a better clinical decision support system by utilizing these data. This study has shown the viability of using EHR data to develop the order recommendation system in outpatient clinics that can automatically predict the orders that providers would like to place for patients. This order recommendation system, developed by integrating machine learning methods including LR, RF and GBM with a well-developed order co-occurrence network, is successful in predicting non-medication orders in pulmonary-related clinics. The implementation of this system will bring a more effective and efficient way to place medical orders for outpatients, so that providers will have more time to see patients, which can improve the care quality and patient satisfaction. This work implements one more step towards real-time clinical decision support tools on the basis of EHR data by using machine learning methods as well as graph-based techniques, which will help in providing better care for patients.

Footnotes

Acknowledgements

We would like to acknowledge Christopher Stromblad, Eric Reich and Rebecca Maff, who gave valuable feedback on the study.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship and/or publication of this article: Soundar Kumara is funded by Geisinger Health System to undertake Health Analytics Research. Yi-Shan Sung was employed by Geisinger Health System as a graduate student intern to conduct the study at Geisinger Health System. Ronald Dravenstott, Jonathan Darer and Priyantha Devapriya were also employed by Geisinger Health System when the study was being conducted.

ORCID iD

Yi-Shan Sung

References

Zhang

Padman

Levin

JE.

Paving the COWpath: data-driven design of pediatric order sets. J Am Med Inform Assoc 2014; 21(e2): e304–e311.

Zhang

Levin

Padman

Data-driven order set generation and evaluation in the pediatric environment. AMIA Annu Symp Proc 2012; 2012: 1469–1478.

Paterno

Maviglia

Gorman

, et al. Tiering drug-drug interaction alerts by severity increases compliance rates. J Am Med Inform Assoc 2009; 16(1): 40–46.

Abarca

Malone

, et al. Practitioners’ views on computerized drug-drug interaction alerts in the VA system. J Am Med Inform Assoc 2007; 14(1): 56–64.

van der Sijs

Lammers

vandenTweel

, et al. Time-dependent drug-drug interaction alerts in care provider order entry: software may inhibit medication error reductions. J Am Med Inform Assoc 2009; 16(6): 864–868.

Seidling

Schmitt

Bruckner

, et al. Patient-specific electronic decision support reduces prescription of excessive doses. Qual Saf Health Care 2010; 19(5): e15–e17.

Scharnweber

Lau

Mollenkopf

, et al. Evaluation of medication dose alerts in pediatric inpatients. Int J Med Inform 2013; 82(8): 676–683.

Lee

Ponce

Ettner

, et al. Ordering of CT by emergency department provider type: analysis of a nationally representative sample. AJR Am J Roentgenol 2012; 199(5): 1054–1059.

Fischer

Tjia

Reed

, et al. Factors associated with ordering laboratory monitoring of high-risk medications. J Gen Intern Med 2014; 29(12): 1589–1598.

10.

Hasan

Duncan

Neill

, et al. Automatic detection of omissions in medication lists. J Am Med Inform Assoc 2011; 18(4): 449–458.

11.

McGreevey

3rd . Order sets in electronic health records: principles of good practice. Chest 2013; 143(1): 228–235.

12.

Wright

Feblowitz

Pang

, et al. Use of order sets in inpatient computerized provider order entry systems: a comparative analysis of usage patterns at seven sites. Int J Med Inform 2012; 81(11): 733–745.

13.

Chen

Altman

RB.

Mining for clinical expertise in (undocumented) order sets to power an order suggestion system. AMIA Jt Summits Transl Sci Proc 2013; 2013: 34–38.

14.

Zhang

Padman

. Data driven order set development using metaheuristic optimization. In: Proceedings of the conference on artificial intelligence in medicine in Europe, Pavia, 20 June 2015, pp. 47–56. Cham: Springer.

15.

Wright

Sittig

DF.

Automated development of order sets and corollary orders by data mining in an ambulatory computerized physician order entry system. AMIA Annu Symp Proc 2006; 2006: 819–823.

16.

Chen

Altman

RB.

Automated physician order recommendations and outcome predictions by data-mining electronic medical records. AMIA Jt Summits Transl Sci Proc 2014; 2014: 206–210.

17.

Chen

Podchiyska

Altman

RB.

OrderRex: clinical order decision support and outcome predictions by data-mining electronic medical records. J Am Med Inform Assoc 2016; 23(2): 339–348.

18.

Klann

Schadow

Downs

SM.

A method to compute treatment suggestions from local order entry data. AMIA Annu Symp Proc 2010; 2010: 387–391.

19.

Klann

Schadow

McCoy

JM.

A recommendation algorithm for automating corollary order generation. AMIA Annu Symp Proc 2009; 2009: 333–337.

20.

Wang

Tian

, et al. An electronic medical record system with treatment recommendations based on patient similarity. J Med Syst 2015; 39(5): 55.

21.

Klann

Szolovits

Downs

, et al. Decision support from local data: creating adaptive order menus from past clinician behavior. J Biomed Inform 2014; 48: 84–93.

22.

Gortmaker

Hosmer

Lemeshow

Applied logistic regression. Contemp Sociol 1994; 23(1): 159.

23.

Liaw

Wiener

Classification and regression by randomforest. R News, December 2002, pp. 18–22.

24.

Friedman

JH.

Greedy function approximation: a gradient boosting machine. Ann Stat 2001; 29(5): 1189–1232.

25.

Davis

Goadrich

. The relationship between precision-recall and ROC curves. In: Proceedings of the 23rd international conference on machine learning, Pittsburgh, PA, 25–29 June 2006, pp. 233–240. New York: ACM.

26.

Powers

DM.

Evaluation: from precision, recall and F-measure to ROC, informedness, markedness & correlation. J Mach Learn Technol 2011; 2(1): 37–63.