Sage Journals: Discover world-class research

Abstract

Background:

The recent availability of high-quality data from clinical trials, together with machine learning (ML) techniques, presents exciting opportunities for developing prediction models for clinical outcomes.

Methods:

As a proof-of-concept, we translated a hypoglycemia risk model derived from the Action to Control Cardiovascular Risk in Diabetes (ACCORD) study into the HypoHazardScore, a risk assessment tool applicable to electronic health record (EHR) data. To assess its performance, we conducted a 16-week clinical study at the University of Minnesota where participants (N = 40) with type 2 diabetes mellitus (T2DM) had hypoglycemia assessed prospectively by continuous glucose monitoring (CGM).

Results:

The HypoHazardScore combines 16 risk factors commonly found within the EHR. The HypoHazardScore successfully predicted (area under the curve [AUC] = 0.723) whether participants experienced least one CGM-assessed hypoglycemic event (glucose <54 mg/dL for ≥15 minutes from two CGMs) while significantly correlating with frequency of CGM-assessed hypoglycemic events (r = 0.38) and percent time experiencing CGM-assessed hypoglycemia (r = 0.39). Compared to participants with a low HypoHazardScore (N = 19, score <4, median score of 4), participants with a high HypoHazardScore (N = 21, score ≥4) had more frequent CGM-assessed hypoglycemic events (high: 1.6 ± 2.2 events/week; low: 0.3 ± 0.5 events/week) and experienced more CGM-assessed hypoglycemia (high: 1.4% ± 2.0%; low: 0.2% ± 0.4% time) during the 16-week follow-up.

Conclusions:

We demonstrated that a hypoglycemia risk model can be successfully adapted from the ACCORD data to the EHR, with validation by a prospective study using CGM-assessed hypoglycemia. The HypoHazardScore represents a significant advancement toward implementing an EHR-based decision support system that can help reduce hypoglycemia in patients with T2DM.

Keywords

hypoglycemia continuous glucose monitoring T2DM risk score validation

Introduction

Clinical trials present a wealth of adjudicated data for knowledge discovery and secondary outcomes analysis.^1,2 Machine learning (ML) methods applied to clinical trial data have identified risk prediction models and treatment selection algorithms.^3,4 However, only a small percentage of these models have been incorporated clinically.^5,6 Since data collection during clinical trials focus on scientific rigor rather than clinical translation, clinical trial data can differ from electronic health record (EHR) data with regard to data frequency and data elements. As a result, conversion of clinical trial derived prediction models needs to reconcile EHR data elements before translating into an EHR-based model. Furthermore, once the EHR-based model is created, efficacy evaluation within the target population remains critical.

We present a “proof-of-concept” study demonstrating the translation of our hypoglycemia risk prediction⁷ model in type 2 diabetes mellitus (T2DM) patients participating in the Action to Control Cardiovascular Risk in Diabetes (ACCORD) study into an EHR-based risk assessment tool for hypoglycemia. Hypoglycemia is a serious complication of diabetes. In patients with T2DM, self-reported hypoglycemia occurred in 63% within the preceding six months where severity and frequency of hypoglycemia were associated with lower quality of life⁸ and increased emergency room visits.⁹

Multiple models predicting hypoglycemia either from EHR data^10,11 or clinical studies data exist.^12-15 Yet, important limitations exist. Although adopting an EHR-based risk model is theoretically easier than translating a clinical trial–derived model, EHR-based risk models are limited by under-documentation of hypoglycemia within the EHR.¹⁶ In the EHR, hypoglycemia is often documented in unstructured fields, such as clinical notes. Although natural language processing methods have improved identification of hypoglycemic events, their performance remains far from perfect.¹⁷ Likewise, clinical studies which establish risk models for hypoglycemia have traditionally depended on patient cooperation for assessing hypoglycemia, either through patient self-report or self-monitoring of blood glucose (SMBG).^12-14 In these cases, hypoglycemic events may be missed for multiple reasons, such as impaired awareness of hypoglycemia, poor patient engagement, or inconsistent documentation. It is estimated that reliance on patient-reported symptoms fails to capture 95% of continuous glucose monitoring systems (CGM)–assessed hypoglycemic events.¹⁸ In addition, many previously reported risk models were derived based on a mixed population that also includes patients with type 1 diabetes mellitus (T1DM).^12,13,15 In response, we translated a model which our group had established in 2018 from the ACCORD clinical trial.⁷ This model was constructed in patients with T2DM where hypoglycemic events were regularly assessed and adjudicated by the investigative team. This model estimated the risk for severe hypoglycemia (SH: hypoglycemia requiring assistance and either blood glucose <54 mg/dL or requiring the administration of glucose) over a long-term horizon (five years) in patients with T2DM and showed good risk prediction performance (c-index: 0.782).⁷

To successfully translate the ACCORD-based hypoglycemia prediction model to an EHR-based hypoglycemia prediction model, we faced two critical challenges. The first involved variable mapping of clinical trial data to EHR data elements. The original 2018 risk model⁷ was modified to include variables reliably measured in the EHR. The second challenge involved model validation. As validation using the EHR was not feasible due to the under-documentation of hypoglycemia within the EHR, we performed a prospective clinical study to objectively validate hypoglycemic events using CGM. Commercially available to patients in the United States via prescription, CGM technology measures interstitial glucose levels with high frequency and fidelity (glucose sampling every 1-5 minutes, glucose values reported in 10- to 15-minute intervals for 10-14 days, manufacturer-dependent) with minimal patient engagement.^19-21 By addressing the aforementioned challenges, we developed the HypoHazardScore, an EHR-based risk assessment tool for hypoglycemia and demonstrated its effectiveness in assessing CGM-documented hypoglycemia risk in T2DM over a short-term (~16 weeks) observation period.

Methods

Figure 1 diagrams the study design and procedures.

Figure 1.

Diagram summarizing the study. Process in gray represents the previous study, and process in black represents the current study. (Upper panel) (1) In our previous study, a risk model for hypoglycemia was developed using ACCORD clinical trial data.⁷ (2) To translate this model to an EHR-based risk model for hypoglycemia, experts reviewed the ACCORD variables and mapped them to corresponding data elements within the EHR data when applicable. (3) The Chow 2018 model was modified and translated to the HypoHazardScore, EHR-based risk model for hypoglycemia. (4) The HypoHazardScore was implemented within the Fairview’s EHR data to automatically assess risk for hypoglycemia based on EHR data. (Lower panel) A prospective clinical study was conducted to assess the effectiveness of HypoHazardScore.

The presented data were collected from patients participating in the “Role of CGM Usage in Predicting Risk of Hypoglycemia” study at the University of Minnesota (NCT03481530). The University of Minnesota’s Institutional Review Board approved the protocol. All participants provided written informed consent before participation. Part of the raw CGM data were pooled, analyzed, and previously published to examine CGM functionality.^22,23 However, the primary outcome of the study, CGM-assessed hypoglycemia, has not been published.

Development of the HypoHazardScore

Our previous risk model developed from the ACCORD clinical trial data contains 17 risk factors, of which three factors (intensive treatment, education, and waist-circumference) are not consistently captured in the EHR. We revisited the ACCORD data, eliminated these three factors and developed a new risk model (Supplemental Table 1) to create the HypoHazardScore (Table 1), using the same methods previously reported.⁷ The HypoHazardScore comprises of 16 risk factors pulled from the ACCORD data which can be easily obtained from EHR data.

Table 1.

EHR Components and Procedure to Calculate the HypoHazardScore.

Risk factor	Categories	Points
Race	Black	+4
Race	Other	+0
Sex	Female	+3
Sex	Male	+0
Baseline age	40-49 years old	+0
	50-59 years old	+2
	60-69 years old	+3
	70+	+5
SBP	<130 mm Hg	+0
	130-139.9 mm Hg	+2
	≥140 mm Hg	+3
DBP	<70 mm Hg	+0
	70-79.9 mm Hg	−2
	≥80 mm Hg	−4
Serum creatinine (median)	<0.0796 mmol/L	+0
Serum creatinine (median)	≥0.0796 mmol/L	+2
HbA_1c	<7%	+0
	7-8%	+1
	>8%	+2
Sulfonylurea use	Yes	−5
Sulfonylurea use	No	+0
Biguanide use	Yes	−3
Biguanide use	No	+0
Meglitinide use	Yes	+4
Meglitinide use	No	+0
Insulin use	Yes	+7
Insulin use	No	+0
BMI (kg/m²)	<25	+0
	25-29.9	−1
	30-34.9	−2
	35+	−3
HMG-CoA reductase inhibitors use	Yes	−2
HMG-CoA reductase inhibitors use	No	+0
Antihypertensive medication use	Yes	+5
Antihypertensive medication use	No	+0
Years with diabetes	0-4.9	+0
	5-14.9	+1
	15-19.9	+3
	≥20	+4
Hypoglycemic in past week	Yes	+4
Hypoglycemic in past week	No	+0

Abbreviations: EHR electronic health record; SBP, systolic blood pressure; DBP, diastolic blood pressure; BMI, body mass index; HMG-CoA, β -hydroxy β -methylglutaryl-CoA.

Range of the score [−17, 43].

Implementation of the HypoHazardScore Within the Fairview Electronic Health Record

We worked with the University of Minnesota Best Practices Integrated Informatics Core at the Clinical and Translational Science Institute to implement the HypoHazardScore within the Fairview EHR. To identify the EHR elements that correspond to the risk factors in HypoHazardScore, a team of clinicians and clinical informatics experts manually reviewed the risk factors and mapped them to their corresponding EHR-based measures. Institutional-specific result codes were used to identify laboratory results. International Classification of Diseases, Tenth Revision (ICD-10) codes were used to identify diagnoses. Pharmaceutical subclasses and/or drug names were used to identify medication.

After completing the EHR element mapping, we implemented an automated risk assessment tool based on EHR data elements for assessing hypoglycemia risks using Structured Query Language for the Fairview EHR with the scoring rule (Table 1). Based on the date when score computation was requested, we used the most recent relevant lab value/measurement, any relevant medications ordered in the past one year, and diabetes duration by considering the first date of documented diabetes diagnosis.

Validation of the HypoHazardScore in a Prospective Clinical Study

We recruited participants from the Fairview EHR who had a least one encounter between 2018 and 2019 and compared their hypoglycemia risk predicted by the HypoHazardScore to CGM-assessed hypoglycemia.

Subject recruitment

We identified patients through the EHR that met inclusion/exclusion criterion. The inclusion criteria included adult ≥18 years and patients with T2DM. Exclusion criteria included having T1DM, current/anticipated pregnancy, significant hypertension (study-visit measured systolic blood pressure >160 or diastolic blood pressure >100), or resting tachycardia (study-visit measured heart rate >130), and current CGM usage or self-reported anticipated CGM usage within the next four months. A HypoHazardScore was computed for each eligible patient who met study criteria. As the HypoHazardScore for this population ranged from −16 to 26 (median of 4), we defined a high HypoHazardScore as a score ≥4 and a low HypoHazardScore as a score<4. Due to the high number (n = 36 350) of potentially eligible patients, the recruitment pool was limited to patients living within the Minneapolis-St. Paul metropolitan area with available HbA_1c values within the three months of a mailing. A total of eight mailings were performed for recruitment. One hundred patients were randomly selected for each mailing as the positive response rate is ~5% to 10% for letter-based research recruitment. Any subject responding to the recruitment letter underwent a telephone screen and if eligible, then scheduled for an in-person study visit. We had 47 potential participants contact the study of which 40 consented and completed the study (Figure 1).

Study design

During the screening visit, participants had their vital signs (height, weight, blood pressure, heart rate) measured and provided blood samples after fasting for at least 8 hours for glucose, HbA_1c, and fructosamine. Afterward, participants wore a CGM sensor for 16 weeks. Every two weeks (eight visits per participant), study staff replaced the CGM sensor, collected self-reported data on adverse events, hypoglycemic events, medication changes, and SMBG logs for the past 14 days. Participants reported any hypoglycemic event requiring assistance from another person, regardless of whether they checked their blood sugar. At the study conclusion, study staff re-measured the participant’s vital signs (height, weight, blood pressure, heart rate) and blood tests (HbA_1c and fructosamine).

Continuous glucose monitoring

Two types of FreeStyle Libre sensors were used in the study (FreeStyle Libre Pro and FreeStyle Libre 14-day Sensor). Each participant wore two CGM sensors simultaneously. One of these was a FreeStyle Libre Pro (blinded sensor), while the other was either a FreeStyle Libre Pro (blinded sensor) or FreeStyle Libre 14-day Sensor (unblinded sensor—first generation of the FreeStyle Libre sensor for personal use), assigned randomly to investigate whether having access to an unblinded sensor could reduce the risk of hypoglycemia as a secondary study outcome. Participants used the same type of sensor throughout the study period, and the randomization was based on the HypoHazardScore. Both sensors sample interstitial glucose every 1 minute while reporting glucose measurements every 15 minutes for up to 14 days. Study staff downloaded results from the blinded sensor every two weeks, without sharing them with the participant. For the FreeStyle Libre 14-day Sensor, we required participants to scan the sensor using the study-provided reader, rather than the FreeStyle LibreLink app; scanning was per participant discretion. Notably the FreeStyle Libre 14-day Sensor does not have any alarm settings and is not a “real-time” streaming device as it requires scanning for the participant to see the glucose results. If the participant did not scan the FreeStyle Libre 14-day Sensor every 8 hours, a data gap would occur because the sensor can only store up to 8 hours of glucose data at a time.

Statistical Analysis for the Validation Study

Identifying continuous glucose monitoring–assessed hypoglycemia

Given that the FreeStyle Libre system has a tendency to over-estimate hypoglycemia,²⁴ we increased the specificity our findings by defining a CGM-assessed hypoglycemic event as a glucose level <54 mg/dL²⁵ for ≥15 minutes²⁶ as measured simultaneously from both CGM sensors. Custom Python scripts were used to identify hypoglycemic events from the CGM records. After identifying each CGM-assessed hypoglycemic event from individual participants, we computed the frequency (number of events per week) of hypoglycemia, duration of each hypoglycemic event, and percent time spent experiencing CGM-assessed hypoglycemia over the observation period (duration of CGM-assessed hypoglycemic events over total CGM monitoring time).

Evaluating the relationship between HypoHazardScore vs continuous glucose monitoring–assessed hypoglycemia

We assessed the predictive performance of the HypoHazardScore for CGM-assessed hypoglycemic events over a 16-week period. We constructed a receiver operating characteristic curve (ROC) and calculated the area under the curve (AUC) to evaluate its performance. The AUC’s confidence interval was estimated using Delong’s method.²⁷ In addition, we measured the correlation between the HypoHazardScore and two variables: frequency of CGM-assessed hypoglycemia and time spent in CGM-assessed hypoglycemia. Spearman’s correlation was used for these calculations.

Comparing continuous glucose monitoring–assessed hypoglycemia between participants with high vs low HypoHazardScores

Participants were divided into two groups based on their HypoHazardScore: high (≥4) and low (<4). The two groups were compared regarding CGM-assessed hypoglycemia using multiple measures, such as event frequency, duration, percentage of time in hypoglycemia, and self-reported instances requiring treatment assistance. Chi-square tests were used for binary variables (eg, occurrence of hypoglycemic events), while the Student’s t-test was employed for continuous variables.

Ancillary analysis of the impact of continuous glucose monitoring sensor blind/unblind status

As a secondary analysis, we investigated the impact of access to unblinded CGM data, available for participants using the FreeStyle Libre 14-day Sensor with reader, on CGM-assessed hypoglycemia. This analysis, although not the primary focus of the study, is presented in the Supplemental Material (section “Ancillary Analysis”) since it did not have a significant impact on the rates of CGM-assessed hypoglycemia.

Results

HypoHazardScore

The HypoHazardScore contains 16 risk factors (Table 1) which were variables available from the ACCORD data and the EHR data. The HypoHazardScore achieved comparable risk assessment performance in the ACCORD data compared to the original 2018 score; the HypoHazardScore had a c-index = 0.752 (95% confidence interval [CI]: [0.732, 0.771]) vs the original score’s c-index of 0.782 [0.764, 0.800]. The relationship between the score points and estimated risk in the ACCORD population remained similar to the original 2018 model (Supplemental Figure 1).

Validation Study

For the validation study, Figure 1 (lower panel) depicts the main recruitment processes. Among 47 screened participants, 40 completed the screening visit (high risk: n = 21, low risk: n = 19). Baseline characteristics are described in Table 2, and distribution of their HypoHazardScore is described in Supplemental Figure 2. The high HypoHazardScore group (HypoHazardScore ≥4) comprised older individuals with higher HbA_1c levels. Fructosamine level did not differ between groups. The percentage of participants using insulin was significantly higher in the high HypoHazardScore group (95.2%) than the low HypoHazardScore group (0.0%) (P < .01).

Table 2.

Descriptive Statistics of Participants’ Baseline Measures for the Entire Cohort, High, and Low HypoHazardScore Groups, Respectively.

	High HypoHazardScore (score ≥4)(N = 21)	Low HypoHazardScore (score <4)(N = 19)	P value	Overall (N = 40)
Age at enrollment
Mean (± SD)	66.9 (± 10.7)	50.7 (± 8.2)	<.001	59.2(± 12.5)
Median [min, max]	70.5 [43.9, 79.0]	50.6 [37.0, 68.5]		56.8 [37.0, 79.0]
Sex
Male	12 (57.1%)	8 (42.1%)	.527	20 (50.0%)
Female	9 (42.9%)	11 (57.9%)		20 (50.0%)
Medication use
% of participants having insulin or sulfonylureas usage as part of their medication program (not mutually exclusive)	Insulin (95.2%)Sulfonylurea (4.8%)	Insulin (0.0%)Sulfonylurea (68.4 %)	<.01	Insulin (50 %)Sulfonylurea (35 %)
HypoHazardScore
Mean (± SD)Range of −16 to 53	13.3 (± 4.0)	−6.8 (± 4.3)	<.001	3.7 (± 11.0)
Median [min, max]	14.0 [6.0, 21.0]	−8.0 [−12.0, 3.0]		6.0 [-12.0, 21.0]
Fructosamine (µmol/L)Normal range 170-285 µmol/L
Mean (± SD)	319 (± 46.9)	304 (± 48.8)	.324	312 (± 47.8)
Median [min, max]	320 [253, 425]	307 [225, 398]		309 [225, 425]
HbA_1c (%) normal <5.7%
Mean (± SD)	8.0 (± 1.3)	7.2 (± 1.1)	.0319	7.6 (± 1.2)
Median [min, max]	7.9 [5.5, 12.0]	7.2 [5.8, 10.2]		7.4 [5.5, 12.0]

Abbreviation: SD, standard deviation.

For continuous variables, statistical comparisons were conducted using the t-test. For discrete variables, statistical comparisons were conducted using the chi-square test.

Bolded text indicates p values < 0.05.

Number of Continuous Glucose Monitoring–Assessed Hypoglycemia Events

By defining CGM-assessed hypoglycemia events as both sensors measuring <54 mg/dL for at least 15 minutes, we found 287 events in the 16-week cohort of 40 patients. However, if we only required one sensor to measure <54 mg/dL for at least 15 minutes, the number of CGM-assessed hypoglycemia events significantly rose. The left arm sensor recorded 623 events, while the right arm sensor detected 749 events. Consequently, to enhance the specificity of the HypoHazardScore in measuring CGM-assessed hypoglycemia events, we mandated that both sensors measure <54 mg/dL for at least 15 minutes.

Association Between HypoHazardScore and Continuous Glucose Monitoring–Assessed Hypoglycemia

Supplemental Figure 3 illustrates the ROC curve for using the HypoHazardScore to predict if a participant would experience any CGM-assessed hypoglycemic event during their 16-week monitoring period. The HypoHazardScore showed good predictivity, demonstrated by AUC = 0.723 with 95% CI of [0.559, 0.887].

We also examined the relationship between the HypoHazardScore, frequency of CGM-assessed hypoglycemic events, and percent time experiencing hypoglycemia. We found significant correlation between the HypoHazardScore and frequency of CGM-assessed hypoglycemia with correlation coefficient of 0.38 (P = .0154), and significant correlation between the HypoHazardScore and percent time experiencing CGM-assessed hypoglycemia with a correlation coefficient of 0.39 (P = .0118).

Participants with High HypoHazardScore had Higher Incidence, Greater Frequency, and Longer Duration of Continuous Glucose Monitoring–Assessed Hypoglycemic Events

The high HypoHazardScore group showed a significantly greater proportion of participants experiencing CGM-assessed hypoglycemia, higher frequency of hypoglycemic events, and more time spent in CGM-assessed hypoglycemia compared to the low HypoHazardScore group (Table 3).

Table 3.

Participants’ Hypoglycemia Outcomes for Entire Cohort, High, and Low HypoHazardScore Groups, Respectively.

Outcomes for participants with high vs low HypoHazardScore
	High HypoHazardScore (score ≥4) (N = 21)	Low HypoHazardScore (score <4) (N = 19)	P value	Overall (N = 40)
Observation period (weeks)
Mean (± SD)	14.6 (± 3.6)	14.9 (± 2.1)	.705	14.8 (± 2.9)
Median [min, max]	15.7 [1.6, 17.7]	15.4 [8.7, 18.0]		15.5 [1.6, 18.0]
Number of CGM-assessed hypoglycemic events over entire period of observation
Mean (± SD)	10.9 (± 15.8)	2.6 (± 5.3)	.0329	7.0 (± 12.6)
Median [min, max]	4.0 [0, 59.0]	0 [0, 20.0]		1.0 [0, 59.0]
Number of participants without CGM-assessed hypoglycemic events over the observation period
Count (%)	6 (28.6%)	13 (68.4%)	.354	19 (47.5%)
Frequency of CGM-assessed hypoglycemic events (number of events/week)
Mean (± SD)	1.6 (± 2.2)	0.3 (± 0.5)	.0153	1.0 (± 1.7)
Median [min, max]	0.4 [0, 6.5]	0 [0, 1.7]		0.1 [0, 6.5]
Average length of CGM-assessed hypoglycemic event (minutes)
Mean (± SD)	79.3 (± 37.2)	61.5 (± 38.2)	.354	74.2 (± 37.4)
Median [min, max]	86.3 [15.0, 143.0]	51.9 [30.0, 135.0]		65.0 [15.0, 143]
Total time (hours) experiencing CGM-assessed hypoglycemia over the observation period
Mean (± SD)	16.3 (± 25.4)	2.7 (± 5.5)	.0255	9.9 (± 19.8)
Median [min, max]	4.8 [0, 97.8]	0 [0, 17.5]		0.4 [0, 97.8]
% Time experiencing CGM-assessed hypoglycemia over the observation period
Mean (± SD)	1.4 (± 2.0)	0.2 (± 0.4)	.0114	0.8 (± 1.6)
Median [min, max]	0.4 [0, 5.7]	0 [0, 1.3]		0.03 [0, 5.7]

Abbreviation: SD, standard deviation.

For continuous variable, statistical comparisons were conducted using the t-test. For discrete variables, statistical comparisons were conducted using the chi-square test.

Bolded text indicates p values <0.05.

The high HypoHazardScore group had significantly higher rates of self-reported hypoglycemia requiring treatment compared to the low HypoHazardScore group (P < .04). Throughout the observation period, the high HypoHazardScore group reported an average of 1.81 ± 3.87 hypoglycemic events requiring treatment, whereas none of the participants in the low HypoHazardScore group reported any such events.

Discussion

We conducted a proof-of-concept study to develop the HypoHazardScore, an EHR-based risk model for hypoglycemia in T2DM. We validated the HypoHazardScore in a prospective 16-week clinical study, where it successfully predicted the occurrence of CGM-assessed hypoglycemia. We found that a higher HypoHazardScore was associated with increased frequency and duration of CGM-assessed hypoglycemia. These findings are particularly relevant due to recent Medicare guideline changes that now cover CGM for T2DM patients using any type of insulin as well as patients not using with a history of hypoglycemia.²⁸

This study identifies the difficulties encountered when translating a risk model into an EHR-based decision support tool. The first challenge was selecting the appropriate data set for model development. We opted for the ACCORD clinical trial data because of its hypoglycemia documentation and adjudication. Moreover, the ACCORD data set encompasses diverse data elements that can be aligned with EHR data elements. The second challenge focused on validating the risk assessment tool. While EHR-based risk models are typically evaluated through prospective studies using EHR data, we deemed this approach unsuitable due to under-documentation of hypoglycemia within the EHR. Therefore, we validated our HypoHazardScore against a prospective, 16-week study using CGM-assessed hypoglycemia, providing a more objective assessment of hypoglycemia than prior studies.^12-15,29

This proof-of-concept study highlights the possibility of leveraging EHR-derived data to create personalized prediction models. The HypoHazardScore holds promise for various applications, such as assessing a participant’s personalized hypoglycemia risk in research studies or developing clinical decision support tools to customize CGM prescription for patients with T2DM. This is particularly relevant considering that CGM usage is predominantly initiated in patients with T1DM (91%) rather than those with T2DM (9%).²⁹ Finally, continuous prediction of hypoglycemia risk could be possible by integrating the HypoHazardScore with real-time CGM data.

The study has several strengths. First, the validation study gathered a detailed data set, encompassing participants with diverse predicted hypoglycemic risk who wore CGM continuously for 16 weeks. This data set offers valuable insights into the frequency of short-term hypoglycemia (measured over weeks), an understudied topic in patients with T2DM. Second, the study rigorously documented CGM-assessed hypoglycemia, defining a hypoglycemic event as glucose levels <54 mg/dL sustained for at least 15 minutes as measured by both CGM sensors.

Several limitations exist. The HypoHazardScore is derived from the ACCORD study, which does not incorporate newer treatments for T2DM, such as sodium-glucose transport protein 2 (SGLT2) inhibitors, glucagon-like peptide-1 (GLP-1) agonists, or gastric inhibitory polypeptide (GIP)/GLP-1 agonists. These treatments have shown diverse benefits beyond glycemic control, potentially altering the relative contribution of the HypoHazardScore components (eg, sulfonylureas). The validation cohort was relatively small and restricted to a single health system, possibly affecting the generalizability of the findings to other populations. In addition, the performance of the HypoHazardScore relies on EHR data quality, which can vary across different health systems. While we mitigated potential issues by requiring CGM data from both sensors, the use of the first-generation FreeStyle Libre system may still be influenced by sensor-related overestimation of hypoglycemia. Finally, it is important to recognize that the HypoHazardScore represents a “point in time,” and ongoing monitoring of its performance and relevance to the current patient population is necessary. Model updates should be considered as performance shifts occur, clinical practices evolve, and new discoveries become pertinent.^30,31

Conclusions

In a proof-of-concept study, we developed the HypoHazardScore, an EHR-based risk model for hypoglycemia by translating a clinical trial prediction model. We validated the HypoHazardScore’s effectiveness in a prospective study, using CGM-assessed hypoglycemia as the outcome. Our goal is to demonstrate potential of the HypoHazardScore as an EHR-based decision support system, with the ultimate goal of identifying and reducing hypoglycemia frequency in patients with T2DM.

Supplemental Material

sj-docx-1-dst-10.1177_19322968231184497 – Supplemental material for Development and Validation of an Electronic Health Record–Based Risk Assessment Tool for Hypoglycemia in Patients With Type 2 Diabetes Mellitus

Supplemental material, sj-docx-1-dst-10.1177_19322968231184497 for Development and Validation of an Electronic Health Record–Based Risk Assessment Tool for Hypoglycemia in Patients With Type 2 Diabetes Mellitus by Sisi Ma, Alison Alvear, Pamela J. Schreiner, Elizabeth R. Seaquist, Thomas Kirsh and Lisa S. Chow in Journal of Diabetes Science and Technology

Footnotes

Acknowledgements

The authors thank Dr Niki Oldenburg for her critical review and manuscript edits.

Abbreviations

ACCORD, Action to Control Cardiovascular Risk in Diabetes; CGM, continuous glucose monitoring; EHR, electronic health record; SMBG, self-monitored blood glucose; SH, severe hypoglycemia; T1DM, type 1 diabetes mellitus; T2DM, type 2 diabetes mellitus.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: For potential conflict of interest disclosure, LSC acknowledges several investigator initiated grants from DEXCOM for product only. The remaining authors have no relevant conflicts of interest.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The study was supported by the University of Minnesota, Academic Health Center (AHC-FRD-17-08 to LSC) and the National Institutes of Health (NIH National Center for Advancing Translational Sciences, UL1TR002494 and UL1TR000114, National Institute of Diabetes and Digestive and Kidney Diseases, R01DK095360 to ERS). UL1TR000114 also supported SM’s time on this project.

ORCID iD

Lisa S. Chow

Supplemental Material

Supplemental material for this article is available online.

References

Doolan

Winters

Nouredini

. Answering research questions using an existing data set. Med Res Arch. 2017;5. https://esmed.org/MRA/mra/article/view/1543. Accessed June 19, 2023.

Heaton

. Secondary analysis of qualitative data: an overview. Hist Soc Res. 2008;33:33-45.

Weissler

Naumann

Andersson

, et al. The role of machine learning in clinical research: transforming the future of evidence generation. Trials. 2021;22:537. doi:10.1186/s13063-021-05489-x.

Réda

Kaufmann

Delahaye-Duriez

. Machine learning applications in drug development. Comput Struct Biotechnol J. 2020;18:241-252. doi:10.1016/j.csbj.2019.12.006.

Maier-Hein

Eisenmann

Sarikaya

, et al. Surgical data science—from concepts toward clinical translation. Med Image Anal. 2022;76:102306. doi:10.1016/j.media.2021.102306.

Vayena

Blasimme

. A systemic approach to the oversight of machine learning clinical translation. Am J Bioeth. 2022;22:23-25. doi:10.1080/15265161.2022.2055216.

Chow

Zmora

Seaquist

Schreiner

. Development of a model to predict 5-year risk of severe hypoglycemia in patients with type 2 diabetes. BMJ Open Diabetes Res Care. 2018;6:e000527.

Marrett

Radican

Davies

Zhang

. Assessment of severity and frequency of self-reported hypoglycemia on quality of life in patients with type 2 diabetes treated with oral antihyperglycemic agents: a survey study. BMC Res Notes. 2011;4:251.

Leese

Wang

Broomhall

, et al. Frequency of severe hypoglycemia requiring emergency treatment in type 1 and type 2 diabetes: a population-based study of health service resource use. Diabetes Care. 2003;26:1176-1180.

10.

Misra-Hebert

Pantalone

, et al. Risk prediction for severe hypoglycemia in a type 2 diabetes population with previous non-severe hypoglycemia. J Diabetes Complications. 2020;34. doi:10.1016/j.jdiacomp.2019.107490.

11.

Lin

Zhang

Ling

Chen

. Development and validation of a hypoglycemia risk model for intensive insulin therapy in patients with type 2 diabetes. J Diabetes Res. 2020;2020:7292108. doi:10.1155/2020/7292108.

12.

Kovatchev

Clarke

Breton

Brayman

McCall

. Quantifying temporal glucose variability in diabetes via continuous glucose monitoring: mathematical methods and clinical application. Diabetes Technol Ther. 2005;7:849-862. doi:10.1089/dia.2005.7.849.

13.

Jacober

Zhang

Wolka

DeVries

. Rate of hypoglycemia in insulin-treated patients with type 2 diabetes can be predicted from glycemic variability data. Diabetes Technol Ther. 2012;14:1008-1012. doi:10.1089/dia.2012.0099.

14.

Murata

Hoffman

Shah

Wendel

Duckworth

. A probabilistic model for predicting hypoglycemia in type 2 diabetes mellitus—the diabetes outcomes in veterans study (DOVES). Arch Intern Med. 2004;164:1445-1450. doi:10.1001/archinte.164.13.1445.

15.

Schroeder

Goodrich

Nichols

O’Connor

Steiner

. Predicting the 6-month risk of severe hypoglycemia among adults with diabetes: development and external validation of a prediction model. J Diabetes Complications. 2017;31:1158-1163. doi:10.1016/j.jdiacomp.2017.04.004.

16.

Rauh

Rutters

Thorsted

, et al. Self-reported hypoglycemia in patients with type 2 diabetes treated with insulin in the Hoorn Diabetes Care System Cohort, the Netherlands: a prospective cohort study. BMJ Open. 2016;6:e012793. doi:10.1136/bmjopen-2016-012793.

17.

Zheng

Dickson

Blecker

, et al. Identifying Patients with hypoglycemia using natural language processing: systematic literature review. JMIR Diabetes. 2022;7:e34681. doi:10.2196/34681.

18.

Chow

Bernjak

Williams

, et al. Risk of cardiac arrhythmias during hypoglycemia in patients with type 2 diabetes and cardiovascular risk. Diabetes. 2014;63:1738-1747. doi:10.2337/db13-0468.

19.

Lin

Groat

Chan

, et al. Associations between the time in hypoglycemia and hypoglycemia awareness status in type 1 diabetes patients using continuous glucose monitoring systems. Diabetes Technol Ther. 2020;22:787-793. doi:10.1089/dia.2020.0016.

20.

Ahmad

Zelnick

Batacchi

, et al. Hypoglycemia in people with type 2 diabetes and CKD. Clin J Am Soc Nephrol. 2019;14:844-853. doi:10.2215/CJN.11650918.

21.

Lin

Hung

Sharma

, et al. Impaired awareness of hypoglycemia continues to be a risk factor for severe hypoglycemia despite the use of continuous glucose monitoring system in type 1 diabetes. Endocr Pract. 2019;25:517-525. doi:10.4158/EP-2018-0527.

22.

Tucker

Erdman

Schreiner

Chow

. Examining sensor agreement in neural network blood glucose prediction. J Diabetes Sci Technol. 2022;16:1473-1482. doi:10.1177/19322968211018246.

23.

Olawsky

Zhang

Eberly

Helgeson

Chow

. A new analysis tool for continuous glucose monitor data. J Diabetes Sci Technol. 2021;15:19322968211028909. doi:10.1177/19322968211028909.

24.

The FreeStyle Libre Pro system: a clinical overview of accuracy, 2017. https://usermanual.wiki/m/3760e987c028307feb3719d185ff7ae4797c3ecfa9e5bf7a9767a1c1ee4d0c95.pdf

25.

International Hypoglycaemia Study Group. Glucose concentrations of less than 3.0 mmol/L (54 mg/dl) should be reported in clinical trials: a joint position statement of the American Diabetes Association and the European Association for the Study of Diabetes. Diabetes Care. 2017;40:155-157. doi:10.2337/dc16-2215.

26.

Danne

Nimri

Battelino

, et al. International consensus on use of continuous glucose monitoring. Diabetes Care. 2017;40:1631-1640. doi:10.2337/dc17-1600.

27.

DeLong

Clarke-Pearson

. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44:837-845. doi:10.2307/2531595.

28.

Medicare coverage database. Glucose Monitor—Policy Article. Published 2015. https://www.cms.gov/medicare-coverage-database/view/article.aspx?articleid=52464&ver=49&contractorName=all&sortBy=updated&bc=13. Accessed June 19, 2023.

29.

Karter

Warton

Lipska

. Development and validation of a tool to identify patients with type 2 diabetes at high risk of hypoglycemia-related emergency department or hospital use. JAMA Intern Med. 2017;177:1461-1470. doi:10.1001/jamainternmed.2017.3844.

30.

Magrabi

Ammenwerth

McNair

, et al. Artificial intelligence in clinical decision support: challenges for evaluating AI and practical implications. Yearb Med Inform. 2019;28:128-134. doi:10.1055/s-0039-1677903.

31.

Sutton

Pincock

Baumgart

Sadowski

Fedorak

Kroeker

. An overview of clinical decision support systems: benefits, risks, and strategies for success. npj Digit Med. 2020;3:17. doi:10.1038/s41746-020-0221-y.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.15 MB