Sage Journals: Discover world-class research

Abstract

Background:

Advanced decision support systems for type 1 diabetes (T1D) management often embed prediction modules, which allow T1D people to take preventive actions to avoid critical episodes like hypoglycemia. Real-time prediction of blood glucose (BG) concentration relies on a subject-specific model of glucose-insulin dynamics. Model parameter identification is usually based on the mean square error (MSE) cost function, and the model is usually used to predict BG at a single prediction horizon (PH). Finally, a hypo-alarm is raised if the predicted BG crosses a threshold. This work aims to show that real-time hypoglycemia forecasting can be improved by leveraging: a glucose-specific mean square error (gMSE) cost function in model’s parameters identification, and a “prediction-funnel,” that is, confidence intervals (CIs) for multiple PHs, within the hypo-alarm-raising strategy.

Methods:

Autoregressive integrated moving average with exogenous input (ARIMAX) models are selected to illustrate the proposed solution (use of gMSE and prediction-funnel) and its assessment against the conventional approach (MSE and single PH). The gMSE penalizes the model misfit in unsafe BG ranges (e.g., hypoglycemia), and the prediction-funnel allows raising an alarm by monitoring if the CIs cross a suitable threshold. The algorithms were evaluated by measuring precision (P), recall (R), F1-score (F1), false positive per day (FP/day), and time gain (TG) on a real dataset collected in 11 T1D individuals.

Results:

The best performance is achieved exploiting both the gMSE and the prediction-funnel: P = 65%, R = 88%, F1 = 75%, FP/day = 0.29, and mean TG = 15 minutes.

Conclusions:

The combined use of a glucose-specific metric and an alarm-raising strategy based on the prediction-funnel allows achieving a more effective and reliable hypoglycemia prediction algorithm.

Keywords

hypoglycemia forecasting black-box identification data-driven glucose prediction

Introduction

The technology advancements provided by continuous glucose monitoring (CGM) devices¹ and portable pumps for continuous subcutaneous insulin infusion (CSII)² have considerably improved the quality of life for subjects with type 1 diabetes (T1D). As pointed out in a recent report on artificial intelligence (AI) applications for diabetes management,³ the combined use of CGM devices, insulin pumps, and dedicated mobile applications⁴ brought the possibility of recording different type of information, for instance, CGM data, insulin, meal, physical activity, and self-reported life events. This enables the development of advanced AI-enabled decision support systems (DSSs), which are composite tools that implement multiple software modules to support the patient in the decision-making process.

One of the key elements that can be embedded in an advanced DSS is the prediction module. In fact, knowing ahead in timeif blood glucose (BG) is approaching dangerous levels allows patients to take proactive actions to mitigate or avoid critical episodes considerably improving T1D management.^5
-10

Several research efforts investigated BG prediction,¹¹ and a number of literature studies focused on the challenge of forecasting hypoglycemic episodes (HEs in the short term).¹² More specifically, hypoglycemia prediction was addressed by either classification-based or regression-based approaches. Classification-based approaches consist in developing a binary classifier,¹³ that is, an algorithm producing only two types of possible output, “impending hypoglycemia” or “no hypoglycemia predicted.” On the contrary, regression-based approaches are two-step procedures that (as a first step) predict the future glucose concentration after a certain prediction horizon (PH; usually chosen in the range 30-60 minutes) and then (as a second step) raise an alarm if the predicted value falls below a suitable threshold (usually, but not necessarily, 70 mg/dL). Predicted glucose levels in the first step can be obtained by using either linear^14
-17 or nonlinear methodologies.^18
-24

The purpose of this work is to improve the real-time forecasting of impending hypoglycemic events in T1D when CGM data, injected insulin, and meal intake information are available. For such a purpose, this work exploits autoregressive integrated moving average with exogenous input (ARIMAX) models which were proved to be effective in BG prediction^25,26 and able to capture the essential dynamics of glucose-insulin regulation.^14
-17,^27,28 Moreover, they present two important characteristics: (1) the model parameter individualization (a key aspect to deal with the large intersubject and intrasubject variability of glucose-insulin dynamics) has been deeply studied, and powerful convergence results as well as statistical properties analysis are available in the literature²⁹; and (2) BG prediction can be computed using well-established and computationally convenient algorithms, such as those tracing back to Kalman filtering.³⁰

Specifically, this work demonstrates that there are two margins of improvement within conventional approaches employed in the literature for the real-time forecasting of hypoglycemic events. The first one is about the model identification procedure. In particular, instead of using the classical mean square error (MSE) cost function, we consider a glucose-specific mean square error (gMSE) cost function proposed in Del Favero et al.³¹ This cost function, that applies an extra penalty to overestimation of low BG and to underestimation of high BG, enables the identification of more effective subject-tailored models for predicting HEs. The second margin of improvement regards the alarm-raising strategy. Instead of focusing on a single PH (as conventionally done in the literature), we consider simultaneously multiple PHs, accounting also for the expected decrease of accuracy in the prediction as the PH increases.

The novel personalized hypoglycemia prediction algorithm, based on the two innovations, is tested on real data collected in a population of 11 T1D individuals³² and compared with the conventional approaches.

Overall, the new approach can predict hypoglycemic events with an average time anticipation of 15 minutes and, if compared with the conventional approach, it improves both F1-score (F1) and false positive per day (FP/day) achieving 75% versus 59% and 0.29 versus 0.77, respectively.

Methods

Database and Preprocessing

Data were taken from a previously published paper,³² to which we refer the reader for any protocol details. Briefly, a group of 14 T1D individuals participated in the five-month test of the 24/7 use of a hybrid closed-loop insulin delivery system. Among the collected data, those of interest for the purpose of this work are as follows: CGM readings, recorded using a DexCom G4 Sensor (DexCom, Inc., San Diego, CA, USA) with a sampling period time of five minutes; time course of the insulin infusion delivered by the Roche Accu-Check Spirit Combo Device (Roche Diabetes Care, Inc., Indianapolis, IN, USA); amounts of carbohydrates (CHO) ingested at meals, as estimated and manually annotated by the participants.

Dealing with experimental data of different nature (CGM, insulin information, and CHO intake) poses some technical issues related to (1) device synchronization and (2) completeness of stored data. These problems are faced as follows:

All signals were aligned to the same time grid uniformly sampled at $T_{S}$ = 5 minutes.

If one or more signals presented gaps longer than 30 minutes, the entire data portion was discarded, as data off-line inference is deemed unreliable.

Then, for each patient, two consecutive portions of seven days without discarded data were selected. . The so-selected 14-day long dataset was then split in training set (first 7 days) and test set (last 7 days). Finally, the remaining short data gaps were reconstructed. The reconstruction strategy depends on whether the gap occurs in the training or in the test set. As the training set is entirely available during the model training phase, noncausal techniques can be used to reconstruct missing training data. In this case, we adopted a third-order spline interpolation. On the contrary, on the test set, glucose prediction has to be performed in real time, so missing data (about 1% on average per subject) were filled by exploiting the last few available CGM samples. In particular, a first-order polynomial is fitted by exploiting the last 15 minutes of recorded CGM data and then such a model is used to infer missing data.

Notably, 3 out of 14 subjects lacked a sufficiently long portion of consecutive data.

In summary, the analysis is conducted on data of 11 subjects. For each subject, seven days of training data and seven days of test data are available. In this population, 39 HEs were observed in the training set and 42 episodes occurred in the test set. This amounts to an average of one HE in every two days per patient.

Baseline: Conventional Approach to the Prediction of Impending Hypoglycemic Events

In this work, we considered as baseline for performance assessment a regression-based hypoglycemia prediction algorithm which consists of two steps: (1) identification of a model suited to predict future glucose levels in real time using CGM, insulin, and CHO information; and (2) threshold-crossing alarm-raising strategy.

For step 1, based on previous findings,^25,26 we employ an ARIMAX model that can be described by the following equation:

\begin{array}{l} A (q) y (t) = B_{ins} (q) ins (t - {nk}_{ins}) + \\ B_{cho} (q) cho (t - {nk}_{cho}) + \frac{C (q)}{{(1 - q)}^{- 1}} e (t), \end{array}

(1)

where $y (t)$ is the current CGM reading; $ins (t)$ and $cho (t)$ are the model inputs: insulin and CHO intake, respectively; and $e (t)$ is the noise term assumed to be identically, independently distributed (i.i.d). Finally, $q$ is the backward shift operator such that $q^{- 1} u (t) = u (t - 1)$ :

\begin{array}{l} A (q) = 1 + a_{1} q^{- 1} + \dots + a_{n_{a}} q^{- n_{a}}, \\ B_{ins} (q) = b_{1 ins} + b_{2 ins} q^{- 1} + \dots + b_{n_{b_ins}} q^{- n_{b_ins} + 1}, \\ B_{cho} (q) = b_{1 cho} + b_{2 cho} q^{- 1} + \dots + b_{n_{b_cho}} q^{- n_{b_cho} + 1}, \\ C (q) = 1 + c_{1} q^{- 1} + \dots + c_{n_{c}} q^{- n_{c}} . \end{array}

The order of the autoregressive part (i.e., the number of past CGM readings required by the model) is defined by na, and $n_{b_i n s}$ and $n_{b_c h o}$ indicate the order of the inputs (insulin and CHO, respectively). nc is the order of the moving average part of the model. The inputs’ delays are indicated by ${nk}_{ins}$ and ${nk}_{cho}$ .

$n_{a}$ , $n_{b_ins}$ , $n_{b_cho}$ , and nc represent the degrees of freedom of the model, and they are selected by using the Bayesian information criterion (BIC).

To estimate the unknown model parameters

$θ = [a_{1}, \dots, a_{n_{a}} | b_{1 i n s}, \dots, b_{n_{b_i n s}} | b_{1 c h o}, \dots, b_{n_{b_c h o}} | c_{1}, \dots, c_{n_{c}}]$ , the classic prediction error method (PEM) approach of minimizing the standard MSE cost function:

\hat{θ} = \underset{θ}{argmin} MSE (g (k), \hat{g} (k | k - 1, θ)),

(2)

is considered, with

\begin{matrix} MSE (g (k), \hat{g} (k | k - 1, θ)) = \frac{1}{N} \sum_{1}^{N} {(\begin{array}{l} g (k) - \\ \hat{g} (k | k - 1, θ) \end{array})}^{2} \end{matrix},

(3)

where N is the number of available data samples, $g (k)$ is the CGM-measured glucose at time k, and $\hat{g} (k | k - 1, θ)$ is the one-step ahead prediction, that is, the glucose value at k predicted at time $k - 1$ by the model (and thus dependent also on the model parameter $θ$ ). Once the model parameters are identified, the model can be used to predict the future glucose level ahead in time at a certain PH, a quantity denoted as $\hat{g} (k + P H | k, θ)$ .

For step 2, a hypoglycemic alarm is raised if $\hat{g} (k + P H | k, θ)$ is below the hypoglycemic threshold of 70 mg/dL.

Note that using this method, from now on called single-PH approach, at each step $k$ only one prediction is considered. Hence, PH is a key tuning parameter of this strategy. A typical PH value is 30 minutes.

A schematic representation of the conventional approach is reported in Figure 1 (upper panel).

Figure 1.

Schematic representation of the conventional approach (upper panel: MSE + single-PH alarm strategy) and the proposed approach (bottom panel: gMSE + prediction-funnel alarm strategy). Abbreviations: MSE, mean square error; PH, prediction horizon; gMSE, glucose-specific mean square error; ARIMAX, autoregressive integrated moving average with exogenous input; CGM, continuous glucose monitoring; CHO, carbohydrates.

Proposed Novel Approach to the Prediction of Impending Hypoglycemic Events

In this work, an innovation is introduced for each of the two steps of the conventional approach described in the previous paragraph. In step 1, the ARIMAX model is identified, at the individual subject level, using a glucose-specific cost function which takes into account the clinical impact of the prediction error. In step 2, the alarm strategy is based on a prediction-funnel approach, which exploits the possibility of handling the ARIMAX model within a Kalman filtering framework. A schematic representation of the proposed approach is reported in Figure 1 (bottom panel). Details of the innovations are described below.

In step 1, the parameter estimation strategy relies on the minimization of an ad hoc cost function, called glucose-specific mean square error.³¹ This metric, inspired by the well-known Clarke error grid (CEG),³³ modifies MSE to account for the clinical impact of the prediction error. This is done by increasing MSE values up to 250% in case of glucose overestimation during hypoglycemia and up to 200% in case of glucose underestimation in hyperglycemia. By doing so, overestimation in hypoglycemia is penalized more than underestimation in the same region. In fact, the first situation is clinically more dangerous: it could prevent the detection of a HE or induce an optimistic evaluation of its severity, leading to inadequate treatment. A symmetric reasoning holds for the case of hyperglycemia but, because hypoglycemia is deemed more dangerous than hyperglycemia, in the first case MSE is increased more (up to 250%) than in the second case (only up to 200%). The MSE is increased so that gMSE retains two fundamental mathematical properties of the original metric, smoothness and convexity, as they simplify the optimization involved in the parameter estimation. The estimated model parameter $\hat{θ}$ was obtained as follows:

\hat{θ} = \underset{θ}{argmin} gMSE (g (k), \hat{g} (k | k - 1, θ)) .

(4)

By doing so, the identified model provides more accurate and clinically relevant prediction in the hyperglycemic and hypoglycemic range than in the normal range, thus permitting more effective hypoglycemia prediction.

In step 2, the novelty consists in the use of the prediction-funnel within the alarm-raising strategy. In devising the new strategy, the starting point was noting that several approaches proposed in the literature focused on one single prediction and they seldom account for prediction accuracy in detecting the crossing of the hypoglycemic threshold.^14,15 So, the proposed alarm strategy considers simultaneously multiple predictions at different PHs, accounting also for their uncertainty.

In particular, the model identified in step 1 is then used to build a Kalman predictor, as summarized in Supplemental Appendix A. Then, it is possible to obtain multiple predictions in a computationally efficient manner:

\hat{g} (k + 1 | k), \hat{g} (k + 2 | k), \dots, \hat{g} (k + {PH}_{\max} | k),

and the corresponding variance of the prediction error, for each PH:

σ^{2} (k + 1 | k), σ^{2} (k + 2 | k), \dots, σ^{2} (k + {PH}_{\max} | k) .

This information is used to equip each prediction with a confidence interval (CI):

\hat{g} (k + i | k) \pm m σ (k + i | k), for i = 1, \dots, {PH}_{\max},

(5)

thus obtaining a prediction-funnel, shown in Figure 2.

The parameter $m$ is a tuning parameter that increases or decreases the funnel width.

Once the prediction-funnel is obtained, the upper bound of the prediction’s CIs is monitored to detect whether it goes below the hypoglycemic threshold ${tr}_{h}$ = 70 mg/dL for some of the considered PHs. In other words, it is checked whether

\hat{g} (k + PH | k) + m σ (k + PH | k) \leq {tr}_{h} = 70 mg/dL,

(6)

for any $PH = 1, \dots, {PH}_{\max}$

Figure 2.

Illustration of the prediction-funnel and the role of $α (m)$ . The blue dot is the current blood glucose values $(g (k))$ , the gray dots are the predicted glucose levels $\hat{g} (k + 1 | k), \hat{g} (k + 2 | k), \dots, \hat{g} (k + {PH}_{\max} | k)$ , and the dashed lines are the confidence intervals. In this illustrative example, the orange area, $α (m)$ , is the probability that $\hat{g} (k + 4 | k)$ (green dot) is below the hypoglycemic threshold.

An alarm is then raised if this happens for at least $N_{pred}$ of the considered PHs.

For instance, if $N_{pred} = 1$ , to raise an alarm it is required that one sample of the prediction-funnel falls below the hypoglycemic threshold. On the contrary, if $N_{pred} = {PH}_{\max}$ , an alarm is raised if the entire prediction-funnel is found to be below the hypoglycemic threshold. A preliminary investigation to tune $N_{pred}$ , reported in Supplemental Appendix A, shows that the best performances are achieved with $N_{pred} = 1$ and ${PH}_{\max} = 60$ minutes.

The tuning of the remaining degrees of freedom, the parameter $m$ , will be briefly described in the following and in Supplemental Appendix B, proposing two different strategies. In fact, as described in equation (6), increasing $m$ will make the alarm more conservative.

To gain a more intuitive feeling on the role of this parameter, it might be useful to interpret $m$ as the tuning knob associated with the probability $α (m)$ that the predicted BG level falls below 70 mg/dL (see Figure 2). While the exact relation between $m$ and the probability $α (m)$ depends on the true distribution of the prediction error, a commonly adopted approximation is to assume it as a normal distribution. The accuracy of this approximation is not of critical importance because the only purpose of $α (m)$ is interpretability and the actual value of this probability is not used with the algorithm that is based only on m.

Algorithm Assessment Criteria

Following the definition proposed in the consensus paper,³⁴ we say that an HE has occurred at time $k^{*}$ if CGM is below 70 mg/dL for a period of at least 15 minutes. Then, we define the following:

True positive (TP) if an HE occurred at time $k^{*}$ and an alarm is activated before the event, precisely in a window from 60 and 5 minutes before $k^{*}$ .

Remark: According to this definition, only the alarms which are relatively close to the HE are considered correct, while alarms too far apart in the past are not counted as TP;

False positive (FP) if an alarm is raised at time $\bar{k}$ but no hypoglycemia occurred in the following 60 minutes;

False negative (FN) if an HE occurred at time $k^{*}$ but no alarms are triggered by the algorithm in the previous 60 minutes.

Finally, special attention is given to late alarms, defined as alarms at time $k^{*}$ or up to 15 minutes after. Clearly, these episodes are not counted as TP, as they were not timely triggered. On the contrary, they increase the count of FN. Nonetheless, they cannot be considered as erroneous, so they do not increase the FP count.

Once the events were labeled as TP, FP, and FN, the following metrics were used to evaluate the state-of-art and the proposed approaches:

Precision: P = 100 \times \frac{TP}{TP+FP},

(7)

Recall: R = 100 \times \frac{TP}{TP + FN},

(8)

F 1-score: F 1 = 2 \times \frac{P \times R}{P + R} .

(9)

Precision is the fraction of the correct alarms over the total number of raised alarms. Recall, also known as sensitivity, is the fraction of correctly detected hypoglycemic events over the total number of events. F1 is the harmonic mean of the two previous metrics.

As the dataset is strongly unbalanced, we also evaluated the average number of FPs generated by the algorithm in one day (FP/day). Finally, we calculated the Time Gain (TG) of the hypoglycemic alarms as the time between when the alarm was raised by the algorithm and the start of the HE. According to the definition of TP, the maximum achievable TG is 60 minutes, while the lowest is 5 minutes.

We reported only the overall recall on the population (fraction of detected HEs with respect to all the events occurred in the population) and the overall precision, as well as the overall F1. To assess the statistical significance of the differences observed in recall and precision, a chi-square test of independence with 1 degree of freedom was applied, as suggested in Yeh.³⁵ Moreover, the TG is reported as mean and standard deviation (SD) over all the successfull detections (regardless of the patient in which the detection occurred).

While the overall test set (seven monitoring days for each subject) includes 42 HEs, single-patient data present only a small number of HEs (on average, one every two days for each subject), and thus single-patient-level analysis is omitted. In fact, single-patient recall would be strongly quantized: for instance, in a patient with two HEs, recall can take only three values: 100%, 50%, or 0%.

Results

Table 1 shows the hypoglycemia prediction performances of several configurations of the described prediction algorithms. In particular, the first row reports the baseline performance achieved by the conventional algorithm using individualized models, identified minimizing MSE, and considering only one PH (PH = 30 minutes). The last row of the table reports the performance achieved by the new algorithm proposed in this work that includes both the improvements induced (the use of gMSE for model identification and the prediction-funnel-based strategy). To elucidate the contribution of each proposed innovations to the final performance, Table 1 reports also the performance achieved with inclusion of one modification at the time (gMSE + single PH and MSE + prediction-funnel). Moreover, different values of the hyper-parameters are investigated.

Table 1.

Comparison of Hypoglycemia Prediction Performance.

Alarm strategy	PH	Cost function	P (%)	R (%)	F1 (%)	FP/day	TG (minutes)	Notes
Single PH	30 minutes	MSE	43	95	59	0.77	15 (10)	Conventional approach
	30 minutes	gMSE	44	100	61	0.77	15 (10)
	45 minutes	MSE	37	83	51	0.86	15 (10)
	45 minutes	gMSE	45	98	61	0.73	15 (10)
	60 minutes	MSE	36	76	49	0.82	20 (15)
	60 minutes	gMSE	42	90	58	0.75	15 (15)
Prediction-funnel (tuning 1)	from 5 to 60 minutes	MSE	51	91	65	0.59	15 (15)
Prediction-funnel (tuning 1)	from 5 to 60 minutes	gMSE	51	96	66	0.59	15 (15)
Prediction-funnel (tuning 2)	from 5 to 60 minutes	MSE	62	76	68	0.33	15 (15)
Prediction-funnel (tuning 2)	from 5 to 60 minutes	gMSE	65	88	75	0.29	15 (10)	Best configuration of the proposed approach

The individualized ARIMAX models, identified using different cost functions, are exploited both by applying the single-PH alarm strategy for different PH and the prediction-funnel. Results are reported in terms of P, R, F1, FP/day, and TG, reported as mean (standard deviation).

Abbreviations: PH, prediction horizon; P, precision; R, recall; F1, F1-score; FP/day, false positive per day; TG, time gain; MSE, mean square error; gMSE, glucose-specific mean square error; ARIMAX, autoregressive integrated moving average with exogenous input.

Focusing first on the single-PH strategy, we investigated the impact of PH on the prediction performance of the state-of-art approach, by evaluating three possible PHs: PH = 30, 45, and 60 minutes. The best results are achieved with the PH = 30 minutes, that is, in fact, commonly adopted in literature. In particular, the larger the PH, the higher the TG, but at the expense of a worse P, R, and FP/day. For instance, comparing the state-of-art approach with PH = 30 minutes and with PH = 60 minutes, we can see that TG increases from 15 to 20 minutes (mean values), but P falls from 43% to 36% and R from 95% to 76%, while FP/day increases from 0.77 to 0.82.

The introduction of the gMSE in place of the MSE improves both the precision and the recall with respect to the state-of-art approach. This holds true for all considered values of PH. Considering, for instance, PH = 30 minutes, with the use of gMSE, P increases from 43% to 44%, R from 95% to 100%, while FP/day and TG are almost the same.

We then investigated the impact of the improved alarm strategy (prediction-funnel-based instead of using a single PH). In particular, two different approaches to the tuning of the parameter $m$ were considered. In both cases, we considered a patient-specific $m$ , but in the first approach $m$ was set to get a similar recall as the one achieved by the state of art (MSE + single PH, PH = 30 minutes). As a second alternative approach, $m$ was chosen in each patient to maximize the F1 in the training set of that patient. The first approach is presented in Table 1 as tuning 1, whereas the second is denoted tuning 2.

The improvement granted by the prediction-funnel is clearly visible with tuning 1 that offers higher precision, higher F1, and less FP/day with respect to the state of art: P increases from 43% to 51%, F1 from 59% to 65%, and FP/day decreased from 0.77 to 0.59. This improvement is achieved while retaining similar recall (R from 95% to 91%). The performances of tuning 2 are more difficult to interpret, because it selects a different trade-off between precision and recall. Specifically, it renounces to some recall in favor of a better precision and less FP/day, leading to an overall improvement in the F1.

Finally, combining the two improvements, the algorithm proposed in this work outperforms the state of art. Once again, this is clearly visible with tuning 1 that grants similar recall and TG of the state of art (above 95%) but with higher precision, F1, and less FPs: P from 43% to 51%, F1 from 59% to 69%, and FP/day decreased from 0.77 to 0.59. By adopting a slightly more conservative approach, proposed in tuning 2, precision and FPs can be further improved (P = 65%, FP/day = 0.29, i.e., about one every three days), at the expenses of a deterioration of the recall (R = 88%). This new trade-off offers a better F1 that reaches 75%. The chi-square test proposed in Yeh³⁵ shows that the proposed approach has statistically significant better precision and FP/day, with respect to the conventional approach (P < .01 and P < .0001, respectively). No statstistically significant difference is found in the recall of the two approaches (P = .22).

Remark

According to HE definition formulated in Danne et al,³⁴ an event in which CGM data fall below the hypoglycemic threshold only for one-time or two-time samples should not be considered an hypoglycemia. However, if an alarm is raised in these situations (that we will call in the following quasi-hypoglycemic episodes, qHE), such an alarm will count as a FP. FP error caused by a qHE could be considered less clinically relevant than other FPs, therefore we also report how many FPs are of this kind. For the state-of-art method (MSE + single-PH approach, PH = 30 minutes), 26% of the recorded FPs were associated to qHE, while this percentage raises to 34% with the newly proposed algorithm (gMSE + prediction-funnel), further supporting the superiority of the proposed algorithm. Discarding these events from the FP count, as frequently done in the literature, would reduce the FP/day from 0.77 to 0.62 and increase the precision from 43% to 54% for the state-of-art approach. Similarly it would decrease FP/day from 0.29 to 0.21 and increase P from 65% to 73% for the proposed approach. Another consequence of the HE definition adopted is that even a nonpredictive hypoglycemia detection algorithm, simply based on the CGM trace crossing the 70 mg/dL threshold, may raise FP alarms (FP/day = 0.2). In fact, according to our definition of TP, the CGM reading produces only late hypo-alarms (events that will be detected exactly whenever they started: TG = 0 minutes). As a consequence, TP count is bound to be 0, and both recall and precision are necessarily 0 (i.e., P = R = 0).

Discussion

In this work, we explored a new approach to predict hypoglycemic events that is based on an individualized ARIMAX models, identified by using a cost function specifically designed to account for the clinical impact of prediction error, and on a novel alarm strategy that considers the entire prediction-funnel. The results show that models identified via gMSE minimization provide better hypoglycemia prediction performances than models based on MSE. Furthermore, results show that the new alarm-raising strategy based on the prediction-funnel improves hypoglycemia forecasting, thanks to the possibility of exploiting multiple PHs. The adoption of both the proposed improvements grants the best performances.

While the presented results showed the benefit of including the two proposed novelties in the conventional, regression-based, linear hypoglycemia prediction algorithm, it might be of interest to investigate how the prediction performance of the improved algorithm positions with respect to other contributions. To this aim, we propose a tentative comparison based on the results reported in the previously mentioned works.^{10,14,16,19,20},^22
-24 However, it is difficult to define a fair comparison: different definition of the events might significantly impact on the final metrics, as shown by the previous section, where a seemingly minor modification in the definition of HE has nonnegligible impact on FP/day and precision. Moreover, different dataset might be collected in very different conditions (highly controlled clinical trials vs real life) introducing a further confounding factor. Yang et al,¹⁴ Dave et al,¹⁹ and Gadaleta et al²⁰ reached a recall, respectively, of about 93%, 93%, and 86%, comparable or slightly superior to the recall of our algorithm, with R = 88%, at the expense of lower precision: about 24%, 38%, and 62%, while the proposed approach in this work achieved P = 65%. Similarly, Yang et al¹⁴ and Daskalaki et al²⁴ achieved the remarkable recall of R = 100%, but at the expense of a very high number of FPs (more than one FP/day). Eren-Oruklu et al,¹⁶ Toffanin et al,¹⁷ and Georga et al,²³ showed a similar recall to the one obtained in this work (89%, 94%, and 94%) with a better precision (78%, 90%, and 77%). However, the authors adopted a more permissive HE definition. For instance, in Georga et al²³ the alarms raised after the CGM has crossed the hypoglycemic threshold were considered TP whereas in this work we consider these late alarms as FNs. In Eren-Oruklu et al,¹⁶ performance was assessed using controlled inpatient data. Eljil et al²² showed a slightly inferior recall (86%) and did not report any metrics related to false alarms. To overcome the above-mentioned limitations in comparing literature contributions with the proposed algorithm, we provided the performance granted by a baseline based on astate-of-art approach. Remarkably, such a baseline is found to be a challenging competitor: its performance is in line —or even outperform— some of the performances reported in litterature, for instance.^14,19,20 However, the proposed solution largely improves the conventional approach in terms of precision (65% vs 43%, P < .01), FP/day (0.29 vs 0.77, P < .0001), and F1 (75% vs 59%) at the expenses of a slightly but not statistically significant (P = .22) deterioration of the recall (88% vs 95%).

In this work, we focused on personalized models to deal with the large interindividual variability characterizing T1D population. The slow changes in patient physiology occurring over the weeks (intrapatient variability) are not evaluated as this contribution focuses on one week of data. A natural option to deal with intrapatient variability is to resort to recursive model identification techniques, well-established methods for tracking the changes in patient dynamic by updating the model every time a new measurement becomes available.^36
-38 Nevertheless, given the relatively slow time scale of intrapatient changes, daily or weekly updates of the model, simply obtained by periodically repeating the proposed model identification procedure on the most recent seven days of data, are expected to be sufficient.

As a final comment, we report that real-time data-gap filling method we adopted—namely, first-order polynomial extrapolation of the last 15 minutes of CGM data—was compared with the simpler zero-order hold. No practical difference in the final performance is observable, suggesting that the simpler method (zero-order hold) can be used in place of the more elaborated strategy proposed. This is not surprising, in fact gaps are relatively rare events (1% of the expected data in the test set) and the gaps affecting hypoglycemia prediction are only those occurring near hypoglycemic events (also rare events). Moreover, visual inspection of the dataset confirms that the simultaneous occurrence of hypoglycemia and data gaps is extremely rare. This makes the choice of the interpolation method of little practical impact.

Conclusions and Future Developments

In this work, we proposed a new algorithm to predict in real-time upcoming hypoglycemic events that can be embedded in an advanced DSS to support the T1D patient during the decision-making process. To tackle the challenging task of hypoglycemia prediction, the novel approach combines, on one hand, models which are identified via minimization of a glucose-specific cost function to improve the model’s prediction capability in the hypoglycemic region and, on the other hand, an alarm-raising strategy that is based on the prediction-funnel. Notably, due to the large inter-intra subject variability characterizing T1D patients, the proposed algorithm requires individualized models. Hypoglycemic prediction performance is evaluated on 11 T1D individuals, and the best results, P = 65%, R = 88%, F1 = 75%, FP/day = 0.29, and TG = 15 (10) minutes, are obtained by the proposed approach. Therefore, the use of a specific metric to improve prediction in the hypoglycemic region, as well as the prediction-funnel, allows achieving a more effective and reliable hypoglycemia prediction algorithm. While the proposed hypoglycemia prediction algorithm is based on linear regression-based techniques, the two proposed improvements can be exploited in other machine-learning or nonlinear techniques (with adequate adjustments). Future works will investigate this option.

Supplemental Material

sj-docx-1-dst-10.1177_19322968221093665 – Supplemental material for Combined Use of Glucose-Specific Model Identification and Alarm Strategy Based on Prediction-Funnel to Improve Online Forecasting of Hypoglycemic Events

Supplemental material, sj-docx-1-dst-10.1177_19322968221093665 for Combined Use of Glucose-Specific Model Identification and Alarm Strategy Based on Prediction-Funnel to Improve Online Forecasting of Hypoglycemic Events by Simone Faccioli, Francesco Prendin, Andrea Facchinetti, Giovanni Sparacino and Simone Del Favero in Journal of Diabetes Science and Technology

Footnotes

Abbreviations

AI, artificial intelligence; ARIMAX, autoregressive integrated moving average with exogenous input; BG, blood glucose; CEG, Clarke error grid; CGM, continuous glucose monitoring; CHO, carbohydrates; CI, confidence interval; CSII, continuous subcutaneous glucose infusion; DSS, decision support system; F1, F1-score; FN, false negative; FP, false positive; FP/day, false positive per day; gMSE, glucose-specific mean square error; HE, hypoglycemic episode; MSE, mean square error; P, precision; PH, prediction horizon; R, recall; T1D, type 1 diabetes; TG, time gain; TP, true positive.

Authors’ Note

DexCom, Inc. (San Diego, CA, USA) did not edit or influenced the material presented. None of the data, or data or analysis derived from the data, are derived using DexCom, Inc., proprietary algorithms in any way. Nor does anything in this paper, including descriptions of algorithms and the state of art, purport to describe or use such proprietary algorithms. However, the algorithms disclosed in this paper may be covered by one or more US and international pending patents and applications. The source of the data is the Juvenile Diabetes Research Foundation (JDRF) Artificial Pancreas Consortium Coordinating Center (JDRFAPPCC), but the analyses, content, and conclusions presented herein are solely the responsibility of the authors and have not been reviewed or approved by the JDRFAPPCC. The public CTR3 dataset was financed by JDRF through the grants JDRF 22-2011-649 and JDRF 17-2013-509.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: DexCom, Inc. (San Diego, CA, USA) provided financial support to the research presented in this work. This work was also supported by MIUR (Italian Ministry of Education, Universities and Research) through the initiatives “SIR: Scientific Independence of Young Researchers” (project “Learn4AP: Patient-Specific Models for an Adaptive, Fault-Tolerant Artificial Pancreas,” ID: RBSI14JYM2) and “Departments of Excellence” (Law 232/2016).

ORCID iDs

Giovanni Sparacino

Simone Del Favero

Supplemental Material

Supplemental material for this article is available online.

References

Vettoretti

Cappon

Acciaroli

Facchinetti

Sparacino

Continuous glucose monitoring: current use in diabetes management and possible future applications. J Diabetes Sci Technol. 2018;12(5):1064-1071.

McAdams

Rizvi

An overview of insulin pumps and glucose sensors for the generalist. J Clin Med. 2016;5(1):5.

Contreras

Vehi

Artificial intelligence for diabetes management and decision support: literature review. J Med Internet Res. 2018;20(5):e10775.

Cossu

Cappon

Sparacino

Facchinetti

. A new integrated platform for gathering and managing multivariable and multisensor data in diabetes clinical studies. In: 2020 International Conference on e-Health and Bioengineering (EHB). 2020:1-4. doi:10.1109/EHB50910.2020.9280100.

Turksoy

Kilkus

Hajizadeh

, et al. Hypoglycemia detection and carbohydrate suggestion in an artificial pancreas. J Diabetes Sci Technol. 2016;10(6):1236-1244.

Camerlingo

Vettoretti

del Favero

Cappon

Sparacino

Facchinetti

A real-time continuous glucose monitoring-based algorithm to trigger hypotreatments to prevent/mitigate hypoglycemic events. Diabetes Technol Ther. 2019;21(11):644-655.

Shivers

Mackowiak

Anhalt

Zisser

“Turn it off!” diabetes device alarm fatigue considerations for the present and the future. J Diabetes Sci Technol. 2013;7(3):789-794.

McGarraugh

Alarm characterization for continuous glucose monitors used as adjuncts to self-monitoring of blood glucose. J Diabetes Sci Technol. 2010;4(1):41-48.

Wang

Hypoglycemia prediction using extreme learning machine (ELM) and regularized ELM. In: 2013 25th Chinese Control and Decision Conference. 2013:4405-4409. doi:10.1109/CCDC.2013.6561727.

10.

Turksoy

Bayrak

Quinn

Littlejohn

Rollins

Cinar

Hypoglycemia early alarm systems based on multivariable models. Ind Eng Chem Res. 2013;52(35):12329-12336.

11.

Oviedo

Vehí

Calm

Armengol

A review of personalized blood glucose prediction strategies for T1DM patients. Int J Numer Method Biomed Eng. 2017;33(6):1-21.

12.

Mujahid

Contreras

Vehi

Machine learning techniques for hypoglycemia prediction: trends and challenges. Sensors (Switzerland). 2021;21(2):1-21.

13.

Efendic

Kirchsteiger

Freckmann

del Re

. Short-term prediction of blood glucose concentration using interval probabilistic models. In: 2014 22nd Mediterranean Conference on Control and Automation. 2014:1494-1499. doi:10.1109/MED.2014.6961587.

14.

Yang

Shi

Xie

An ARIMA model with adaptive orders for predicting blood glucose concentrations and hypoglycemia. IEEE J Biomed Health Inform. 2019;23(3):1251-1260.

15.

Eren-Oruklu

Cinar

Rollins

Quinn

Adaptive system identification for estimating future glucose concentrations and hypoglycemia alarms. Automatica. 2012;48(8):1892-1897. doi:10.1016/j.automatica.2012.05.076.

16.

Eren-Oruklu

Cinar

Quinn

Hypoglycemia prediction with subject-specific recursive time-series models. J Diabetes Sci Technol. 2010;4(1):25-33.

17.

Toffanin

Del Favero

Aiello

Messori

Cobelli

Magni

Glucose-insulin model identified in free-living conditions for hypoglycaemia prevention. J Process Control. 2018;64:27-36. doi:10.1016/j.jprocont.2018.02.003.

18.

Vehí

Contreras

Oviedo

Biagi

Bertachi

Prediction and prevention of hypoglycaemic events in type-1 diabetic patients using machine learning. Health Informatics J. 2020;26(1):703-718.

19.

Dave

DeSalvo

Haridas

, et al. Feature-based machine learning model for real-time hypoglycemia prediction. J Diabetes Sci Technol. 2021;15:842-855.

20.

Gadaleta

Facchinetti

Grisan

Rossi

Prediction of adverse glycemic events from continuous glucose monitoring signal. IEEE J Biomed Health Inform. 2019;23(2):650-659.

21.

Frandes

Timar

Lungeanu

A risk based neural network approach for predictive modeling of blood glucose dynamics. Stud Health Technol Inform. 2017;228:577-581.

22.

Eljil

Qadah

Pasquier

. Predicting hypoglycemia in diabetic patients using data mining techniques. In: 2013 9th International Conference on Innovations in Information Technology. 2013:130-135. doi:10.1109/Innovations.2013.6544406.

23.

Georga

Protopappas

Ardigò

Polyzos

Fotiadis

DI.

A glucose model based on support vector regression for the prediction of hypoglycemic events under free-living conditions. Diabetes Technol Ther. 2013;15(8):634-643.

24.

Daskalaki

Nørgaard

Züger

Prountzou

Diem

Mougiakakou

An early warning system for hypoglycemic/hyperglycemic events based on fusion of adaptive prediction models. J Diabetes Sci Technol. 2013;7(3):689-698.

25.

Prendin

Del Favero

Vettoretti

Sparacino

Facchinetti

Forecasting of glucose levels and hypoglycemic events: head-to-head comparison of linear and nonlinear data-driven algorithms based on continuous glucose monitoring data only. Sensors. 2021;21(5):1647. https://www.mdpi.com/1424-8220/21/5/1647. Accessed April 9, 2022.

26.

Faccioli

Facchinetti

Sparacino

Pillonetto

Del Favero

Linear model identification for personalized prediction and control in diabetes. IEEE Trans Biomed Eng. 2022;69:558-568.

27.

Finan

Zisser

Jovanovic

Bevier

Seborg

. Identification of linear dynamic models for type 1 diabetes: a simulation study. IFAC Proceed Vol. 2006;39(2):503-508.

28.

Messori

Ellis

Cobelli

Christofides

Magni

. Improved postprandial glucose control with a customized Model Predictive Controller. In: Proceedings of the American Control Conference. 2015:5108-5115. doi:10.1109/ACC.2015.7172136.

29.

Ljung

System Identification-Theory for the User. 2nd ed. Upper Saddle River, NJ: Prentice Hall; 1999.

30.

Franklin

Powell

Workman

ML.

Digital Control of Dynamic Systems (Vol. 3). Reading, MA: Addison-Wesley; 1998.

31.

Del Favero

Facchinetti

Cobelli

. A glucose-specific metric to assess predictors and identify models. IEEE Trans Biomed Eng. 2012;59(5):1281-1290.

32.

Kovatchev

Cheng

Anderson

, et al. Feasibility of long-term closed-loop control: a multicenter 6-month trial of 24/7 automated insulin delivery. Diabetes Technol Ther. 2017;19(1):18-24.

33.

Clarke

Cox

Gonder-Frederick

Carter

Pohl

SL.

Evaluating clinical accuracy of systems for self-monitoring of blood glucose. Diabetes Care. 1987;10(5):622-628.

34.

Danne

Nimri

Battelino

, et al. International consensus on use of continuous glucose monitoring. Diabetes Care. 2017;40(12):1631-1640.

35.

Yeh

. More accurate tests for the statistical significance of result differences. In: Proceedings of the 18th Conference on Computational linguistics-Volume 2. 2000:947-953. doi:10.3115/992730.992783.

36.

Hajizadeh

Rashid

Turksoy

, et al. Multivariable recursive subspace identification with application to artificial pancreas systems. IFAC-PapersOnLine. 2017;50:886-891.

37.

Eren-Oruklu

Cinar

Quinn

Smith

Estimation of future glucose concentrations with subject-specific recursive linear models. Diabetes Technol Ther. 2009;11(4):243-253.

38.

Finan

Doyle

III Palerm

, et al. Experimental evaluation of a recursive model identification technique for type 1 diabetes. J Diabetes Sci Technol. 2009;3(5):1192-1202.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.13 MB