Published on in Vol 6 , No 2 (2021) :Apr-Jun

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/26909, first published .
Improved Low-Glucose Predictive Alerts Based on Sustained Hypoglycemia: Model Development and Validation Study

Improved Low-Glucose Predictive Alerts Based on Sustained Hypoglycemia: Model Development and Validation Study

Improved Low-Glucose Predictive Alerts Based on Sustained Hypoglycemia: Model Development and Validation Study

Original Paper

1Department of Industrial and Systems Engineering, Texas A&M University, College Station, TX, United States

2Department of Pediatrics, Baylor College of Medicine / Texas Children's Hospital, Houston, TX, United States

3Department of Biomedical Engineering, Texas A&M University, College Station, TX, United States

4Division of Pediatric Urology, Texas Children's Hospital, Houston, TX, United States

5Scott Department of Urology, Baylor College of Medicine, Houston, TX, United States

Corresponding Author:

Madhav Erraguntla, PhD

Department of Industrial and Systems Engineering

Texas A&M University

4021 Emerging Technologies Building

College Station, TX, 77843

United States

Phone: 1 979 458 2336

Email: merraguntla@tamu.edu


Background: Predictive alerts for impending hypoglycemic events enable persons with type 1 diabetes to take preventive actions and avoid serious consequences.

Objective: This study aimed to develop a prediction model for hypoglycemic events with a low false alert rate, high sensitivity and specificity, and good generalizability to new patients and time periods.

Methods: Performance improvement by focusing on sustained hypoglycemic events, defined as glucose values less than 70 mg/dL for at least 15 minutes, was explored. Two different modeling approaches were considered: (1) a classification-based method to directly predict sustained hypoglycemic events, and (2) a regression-based prediction of glucose at multiple time points in the prediction horizon and subsequent inference of sustained hypoglycemia. To address the generalizability and robustness of the model, two different validation mechanisms were considered: (1) patient-based validation (model performance was evaluated on new patients), and (2) time-based validation (model performance was evaluated on new time periods).

Results: This study utilized data from 110 patients over 30-90 days comprising 1.6 million continuous glucose monitoring values under normal living conditions. The model accurately predicted sustained events with >97% sensitivity and specificity for both 30- and 60-minute prediction horizons. The false alert rate was kept to <25%. The results were consistent across patient- and time-based validation strategies.

Conclusions: Providing alerts focused on sustained events instead of all hypoglycemic events reduces the false alert rate and improves sensitivity and specificity. It also results in models that have better generalizability to new patients and time periods.

JMIR Diabetes 2021;6(2):e26909

doi:10.2196/26909

Keywords



Glucose measurements are critical for effective diabetes management. Real-time continuous glucose monitoring (CGM) devices allow for frequent, automated glucose readings from interstitial fluid in the subcutaneous tissue space. CGM has been shown to improve glycemic control and reduce glycemic excursions—decreasing both hypoglycemia and hyperglycemia [1]. An important feature of CGM devices is their ability to provide real-time auditory alerts for trending glucose excursions above or below customized threshold levels. The CGM data can also be used to develop models to predict future hypoglycemia [1,2].

Since the first attempt at predicting future glucose values based on CGM data in 1999 [3], numerous studies have been performed to provide alerts for hypoglycemia based on predictive models. Some of the initial works relied on classical time-series based forecasting methods such as autoregressive integrated moving average (ARIMA)–ARIMAX [4-9] and state-space models [10-19]. Later, different machine learning methods were explored to predict future glucose values [20-25]. More recently, deep learning techniques have been proposed toward solving the problem [26-33]. Some works have also explored incorporating contextual data such as meal information, insulin levels, heart rate, and physical activity as features in the predictive model [34-40]. However, currently integrated information on the data sources is not readily available for real-time predictions.

Predictive hypoglycemia alerts have the potential to be extremely helpful in reducing hypoglycemia risk; however, false alerts have been a major hindrance to the acceptance of predictive hypoglycemia alerts among users [41-43]. In our earlier work, we developed a machine learning–based hypoglycemia predictive model with a sensitivity and specificity of >95%, comparable with the best predictive models in the literature [44]. Typically, the number of hypoglycemic events is very small compared with that of nonhypoglycemic events. For example, only 2.13% (35,075/1,644,875) of readings in the data set used in this study were in the hypoglycemic range (ie, <70 mg/dL). The high-class imbalance resulted in a false alert rate (FAR) of around 85% (40,502/47,683), even with an impressive specificity of 95%. Improvement in specificity in such highly imbalanced class cases will reduce the FAR and therefore improve user experience and trust in the alerts, facilitating persuasive adoption of alerts.

Previous studies have found that hypoglycemia prediction model performance is reduced when applied to new patients and different time periods [45]. Improvement in model generalizability to new patients and time periods will facilitate ease of deployment and retention of performance postdeployment.

Thus, despite the many advances made in terms of hypoglycemia prediction models, the shortcoming of a high FAR makes the alerts ill-suited for real-world application [42,46,47]. These results indicate a need for the development of approaches to reduce the FAR, maintain high sensitivity, and improve the generalizability of the prediction model.


Data Description

The CGM data sets were obtained from 110 pediatric patients with type 1 diabetes over 30 to 90 days. The data comprised over 1.6 million CGM values under normal living conditions. Dexcom G6 CGM devices were used to collect the CGM readings. The cohort-level profile of patients in this study can be found in Table 1. Summary statistics of a patient hypoglycemia profile and a patient pump profile are presented in Multimedia Appendices 1 and 2, respectively. Of note, data were obtained from a mix of patients using multiple daily injections, sensor-augmented pump therapy without automated basal rate modulation, and sensor-augmented pump therapy with a predictive low-glucose suspend feature (ie, t:slim X2 insulin pump with Basal-IQ technology; Tandem Diabetes Care, Inc [48]). The t:slim X2 insulin pump uses a simple linear regression algorithm to predict glucose levels 30 minutes ahead and suspend basal insulin in the pump if glucose values are predicted to drop below 80 mg/dL in the next 30 minutes or if a CGM value falls below 70 mg/dL. Insulin delivery can remain suspended for a minimum of 5 minutes to a maximum of 2 hours and will then resume as soon as glucose values begin to rise.

Table 1. Demographic and diabetes profile of patients enrolled in the study.
CharacteristicMean (SD)MinimumMaximum
Baseline demographics



Age (years)12.67 (4.84)121

Glycated hemoglobin A1c (%)7.70 (1.63)5.0012.50

Duration of diabetes (years)4.93 (4.09)0.2519.18
Continuous glucose monitoring metrics



Number of hypoglycemic values per day per patient6.20 (5.98)0.1023.73

Percentage of hypoglycemic values below 70 mg/dL2.13 (2.10)0.5012.20

Hypoglycemic Events

A glucose threshold of 70 mg/dL is used to identify the hypoglycemic range [49,50]. Of the 1.6 million CGM readings in the study data, approximately 35,000 values representing 6010 events were in the hypoglycemic range. The period of time between a first CGM reading below the threshold and the point where the CGM value rises to ≥70 mg/dL is considered a single “hypoglycemic event.”

Table 2 shows the frequency distribution of the duration of the hypoglycemic events. The hypoglycemic events in this analysis were further classified as transient (lasting less than 15 minutes; 77.24% [4642/6010] of the hypoglycemic events) or sustained (lasting 15 minutes or longer; 22.76% [1368/6010] of the events) based on the recommendations in previous studies [51,52]. Nonhypoglycemic events are CGM observations of ≥70 mg/dL. Detailed information regarding the distribution of hypoglycemic events and a breakdown of sustained events by day and night are presented in Multimedia Appendices 3-5.

Table 2. Frequency distribution of the duration of hypoglycemic events (n=6010).
Hypoglycemic eventsDuration of consecutive CGM values falling below 70 mg/dL (minutes)

51015202530354045>45
Frequency, n (%)572 (9.51)796 (13.24)885 (14.73)842 (14.01)676 (11.25)562 (9.35)391 (6.51)283 (4.71)209 (3.48)794 (13.21)

Evaluation Metrics

We define the following metrics for evaluating model performance: sensitivity, specificity, and FAR.

Sensitivity measures the proportion of true positives that are correctly identified. It is also known as the true-positive rate.

where TP are the true positives and FN are the false negatives.

Specificity measures the proportion of true negatives that are correctly identified. It is also known as the true-negative rate.

where TN are the true negatives and FP are the false positives.

FAR was defined based on the definition provided by Mosquera-Lopez et al [53] and measures the proportion of alerts that are not truly indicative of predicted hypoglycemic events.

Machine Learning Methodologies

Random Forest

Random forest (RF) is a nonparametric approach that builds on an ensemble prediction of a “forest” of regression trees grown via bootstrap sampling. Model predictions are obtained from the mean of the predictions of the individual trees. RF performs well when dealing with nonlinear relationships among variables and makes no assumptions about data distributions. Owing to these characteristics, utilizing RF-based machine learning modeling resulted in good performance in our previous work [44] compared with other machine learning methods for hypoglycemia prediction. In this study, an RF-based model was used to classify events as sustained hypoglycemic events (positive class) or transient and nonhypoglycemic events (negative class).

Quantile Regression Forest

For the multistep prediction approach, future CGM values were predicted using quantile regression forests (QRFs). The concept of quantile regression was introduced by Koenker and Hallock [54] and is advantageous when quantile functions are of interest. Quantile functions provide information about the spread of the response variable beyond the conditional mean by estimating the full conditional distribution. This is particularly useful for predicting values other than the mean (eg, median or 90th quantile). QRFs are a generalization of RFs and provides an accurate way of estimating the conditional quantiles [55]. Since it is more important to accurately predict CGM values near the hypoglycemic range rather than within the euglycemic or hyperglycemic range, QRFs were used to predict future CGM values using the regression approach.

QRFs were used as a multistep forecasting method to predict the glucose values for every 5-minute interval in the prediction horizon (PH). This resulted in 6 predictions for the 30-minute PH and 12 predictions for the 60-minute PH. Based on these predictions, a sustained hypoglycemic event was detected if 3 or more consecutive predicted CGM values were <70 mg/dL.

Validation Mechanism

An appropriate validation mechanism is critical to assess the performance of a machine learning model [56,57]. This validation strategy helps to ascertain the generalizability of the model and ensures that the model performs well in real-world scenarios. This can be performed by sampling a subset of the data for model development and sampling a different sample of data for model validation [58]. Two validation strategies—patient-based and time-based—were used to evaluate model performance.

Patient-Based Approach

In this approach, the prediction model was developed on a subset of patients and validated on a different set of patients. Of the 110 patients, 70 patients (approximately 65% of the data) were randomly selected for training and the remaining 40 patients were used for performance evaluation. The final model performance reported is the mean of 5 replications of this procedure of 65%/35% split of training and validation data.

Time-Based Approach

In this approach, for each of the 110 patients, the first 70% of the data was used for model training and the last 30% of the data was used for validation. The average performance using validation data on all 110 patients was reported.

Features Extracted for the Prediction

A rich combination of demographic, dynamic, snowball, interaction, and contextual features were extracted from the data. An optimal set of features for hypoglycemia prediction was identified in our previous work [44] and these features were used for the model development in this study (Multimedia Appendix 6). Records associated with missing data were eliminated from the analysis; that is, if a feature was dependent on a missing CGM value, that record—as well as all dependent time-lagged records—were eliminated from the analysis.


Model Performance

Table 3 summarizes the performance of the model based on patient-based and time-based validation strategies. The total number of false alerts when considering transient and nonhypoglycemic events as false alerts and when considering nonhypoglycemic events as false alerts are provided along with sensitivity and specificity metrics.

In the patient-based validation approach, for both 30-minute and 60-minute PHs, the QRF method provided a significant advantage over the RF method with high sensitivity, high specificity, and low FAR. The patient-based validation approach indicated that the sustained hypoglycemic model developed using QRFs is generic and can be applied to new patients without performance degradation.

In a time-based validation setting, the RF method performed well for both 30-minute and 60-minute predictions with high sensitivity, high specificity, and low FAR, but the QRF method still outperformed it. The time-based validation methodology indicated that both models retain performance when applied to new time periods and in postdeployment.

Table 3. Comparison of model performance based on sensitivity, specificity, and false alerts with patient-based and time-based validation for 30-minute and 60-minute prediction horizons (PHs).
MetricsPatient-based validationTime-based validation

30-minute PH60-minute PH30-minute PH60-minute PH

Method 1: RFaMethod 2: QRFbMethod 1: RFMethod 2: QRFMethod 1: RFMethod 2: QRFMethod 1: RFMethod 2: QRF
Sensitivity, % (SD)39.11 (2.25)99.09 (0.16)49.27 (3.03)97.61 (0.41)96.1798.9495.3497.91
Specificity, % (SD)98.65 (0.09)98.19 (0.10)98.63 (0.12)98.09 (0.11)98.398.2997.9598.20
False alerts, n (SD)








Considering transient and nonhypoglycemic events as false6936 (356)9339 (459)7043 (317)9672 (431)6476821173468465

Considering only nonhypoglycemic events as false3907 (200)5368 (162)4109 (156)5677 (201)3324453143344799
False alert rate, % (SD)26.32 (2.56)26.50 (2.41)26.44 (2.37)26.36 (2.57)22.7923.8626.4123.79

aRF: random forest.

bQRF: quantile regression forest.

Comparison of Sustained Versus All Hypoglycemic Events Prediction Models

Table 4 provides a comparison of model performance between sustained hypoglycemia and all-hypoglycemia prediction models. Even though the all-hypoglycemia model had high sensitivity, a specificity of 93% on a large number of nonhypoglycemic events resulted in an FAR of 85%. Focusing the alerts on sustained hypoglycemic events resulted in an increase in specificity to 98% and reduced the FAR to approximately 20% to 30%. Also, the performance of the all hypoglycemic events prediction model was adversely affected when evaluated on new patients and new time periods (drop in sensitivity of 5%). On the other hand, prediction models based on sustained hypoglycemic events retained their performance for new patients and new time periods, indicating better generalizability of the sustained hypoglycemic events model.

Table 4. Comparison of model performance based on sensitivity, specificity, and false alert rate with different characterizations of hypoglycemic events and different validation strategies (patient-based and time-based) for giving predictive alerts.
Model30-minute prediction horizon60-minute prediction horizon

Sensitivity (%)Specificity (%)False alert rate (%)Sensitivity (%)Specificity (%)False alert rate (%)
All hypoglycemic events prediction (5-fold validation)93.6193.5084.9491.0189.8277.20
All hypoglycemic events prediction (new time periods)87.1092.6685.1673.8787.2979.81
All hypoglycemic events prediction (new patients)87.6092.4775.2073.7987.0671.50
Sustained hypoglycemic events prediction (QRFa—new patients)99.0897.7930.0098.1397.5830.19
Sustained hypoglycemic events prediction (QRF—new time periods)98.5498.5722.3697.7298.4922.44

aQRF: quantile regression forest.

A graphical comparison between the classifiers at different threshold values using receiver operating characteristic (ROC) curves can be found in Multimedia Appendix 7. The ROC plots show that the QRF model outperformed the RF models over the entire range of sensitivity and specificity levels. Table 5 shows the QRF model's performance metrics at different threshold levels. The table also presents the average time required to predict a hypoglycemic event at different threshold levels.

Table 5. Performance of the quantile regression forest model at different thresholds and the average time to predict a hypoglycemic event.
Metric 30-minute prediction horizon60-minute prediction horizon

Threshold 1Threshold 2Threshold 3Threshold 1Threshold 2Threshold 3
Sensitivity (%)98.5499.2799.5197.7298.2998.99
Specificity (%)98.5797.5696.6898.4997.0695.53
False alerts (n)693211,96016,049721514,02721,297
False alerts with transient events as positives (n)3736814912,0073974995616,775
False alert rate (%)22.3635.3643.3422.4437.1946.96
Average time to predict an event (minutes)18.7822.9526.5125.2435.0848.35

Principal Findings

We present a robust prediction model for providing high-quality alerts for sustained hypoglycemic risk in patients with type 1 diabetes. The final model (QRF model) was demonstrated to be robust to different validation approaches that best represent real-world application scenarios (new patients and new time periods). The primary research contributions of this work are (1) the development of a prediction model that focused on sustained hypoglycemic events and resulted in high sensitivity, high specificity, and a low FAR; and (2) improved generalizability of the model to new patients and new time periods. The model makes use of only CGM data in the past 4 hours and contextual information about the current hour of the day and day of the week to make predictions. A methodology contribution is the use of glucose predictions at multiple time points to facilitate inference of sustained hypoglycemia. The model was built using data collected from 110 patients over a range of 30 to 90 days under normal living conditions, ensuring validity of the results. The QRF model proposed in this work had sensitivity and specificity >97% for both 30- and 60-minute PHs. The FAR was also kept low at 22% and 29% for 30-minute and 60-minute PHs, respectively, which will lead to improved user trust in and adoption of CGM-based alerts.

Comparison with the Literature

A comparative analysis of different hypoglycemia prediction methodologies can be found in the literature [11,20,59,60]. A straightforward comparison between different hypoglycemia prediction studies is complicated due to differences in CGM sensors used, sampling intervals, and data collection (synthetically generated, controlled study, free-living conditions). In addition, different studies have used different definitions of hypoglycemia, which makes their findings difficult to compare [21,61-63]. When using a regression approach, the majority of the works present an overall root-mean-square error (RMSE) value, but accuracy pertaining to the observations in the hypoglycemic range might be more relevant. Thus, providing an overall RMSE for the entire data set could misrepresent the model's performance. On the classification side, sensitivity and specificity provide accurate information on the TPs and FPs, respectively. However, due to high-class imbalance, even a moderately high specificity can lead to a high FAR. It becomes important to consider the FAR, in addition to sensitivity and specificity, in such class-imbalanced applications.

In machine learning, a standard approach to validate prediction models is to split the data into a training set (to train the model) and a validation set (to evaluate model performance) [64]. This random partitioning of the data into training and validation subsets and repeating the process across multiple folds is called cross-validation. Studies across the literature have used different validation strategies such as random sampling [20,65-67], time-based splitting [4,5,67-69], patient-specific splitting [6,32,53,70,71], or a combination of these methods to estimate predictive model performance. Simple random sampling–based cross-validation [72,73] may not fully address the generalizability aspect of the model to new patients and new time periods. Some studies [6] using a patient-based validation strategy used a part of their test data for tuning model parameters, which affected the validity of the performance estimation. The model presented in this paper had high performance in both patient- and time-based validation methods.

Mosquera-Lopez et al [28] used a patient-specific validation approach in which patient data in the test set were exclusive from the training data. Performance was reported on metrics such as sensitivity, RMSE, and FARs. However, leveraging some of the preprocessing and postprediction error-correction steps to improve performance made it difficult to achieve similar results in a real-world setting. Also, the test performance was evaluated on a small sample size of 10 patients (in a 4-week study). This might affect the generalizability of the presented results.

Dave et al [44] recently showed good results with respect to sensitivity and specificity using a random sampling–based validation approach and a threshold of 70 mg/dL for hypoglycemia. However, it was observed that performance of this model was reduced when applied to new patients and new time periods (Table 3). In addition, even with sensitivity and specificity of >95%, the model resulted in an FAR of 80% due to a large number of nonhypoglycemic events relative to the number of hypoglycemic events. From a user experience perspective, this will lead to false alert fatigue. The model presented in this paper reduced the FAR to 22%.

Having an accurate and actionable hypoglycemia prediction model with low FARs is essential to the durability of CGM in diabetes management. Furthermore, a patient-facing hypoglycemia prediction algorithm may give patients the confidence to aim for in-range glucose values without fear of hypoglycemia, potentially leading to lower glycated hemoglobin A1c (HbA1c) values and increased time in range. Of note, 22.3% of patients analyzed were using sensor-augmented pump therapy with a predictive low-glucose suspend feature (ie, Basal-IQ technology). Patients using this system are still at risk for hypoglycemia because of insulin on board, exercise, overdosing on carbohydrates, and/or hyperglycemia, so a notification for predicted hypoglycemia using advanced machine learning models with good performance could still be clinically useful.

Limitations

A limitation of our approach is that transient hypoglycemic events were ignored in generating alerts. Ignoring the transient events helped the machine learning model better learn the more stable patterns of sustained events. Even though the alerts were focused on detecting sustained events, 61% of the transient events were still classified as FPs. This resulted in just 39% of the transient events (representing 13% of the total hypoglycemic events) not being detected. This trade-off was justified because transient events are not as serious as sustained hypoglycemic events. Transient events may occur because of random variations in glycemic levels (ie, noise) or temporal lag in the effect of an intervention taken by the patient (eg, consuming fast-acting carbohydrates). In either case, ignoring transient events will help in learning the stable patterns of sustained hypoglycemia. The improved FAR, sensitivity, specificity, and generalizability of the sustained hypoglycemia model presented in this paper justify this trade-off.

This study was based on patients with pediatric type 1 diabetes in the age range of 0 to 20 years using Dexcom G6 CGM devices. As such, the results are directly applicable to this population. The model may need to be recalibrated to other CGM devices such as the Guardian (Medtronic) or FreeStyle Libre (Abbott Laboratories Co.); however, the performance measures should be generalizable to other platforms provided the accuracy and frequency of incoming glucose readings remain the same. Similarly, while no specific activity profile of pediatric patients was explicitly used in the model development, the model may need to be calibrated to an adult cohort by retraining on adult CGM data [74-76]. Pediatric patients were selected as the focus of the study because of our collaboration in the United States Food and Drug Administration (FDA)–funded Southwest National Pediatric Device Innovation Consortium [77]. Additionally, there is a need for a paradigm shift in diabetes management in pediatrics to avoid risk of hypoglycemia to ameliorate parental and patient fear and move toward optimizing time in range and lowering HbA1c.

Conclusions

Providing predictive alerts for hypoglycemia focused on sustained events instead of all hypoglycemic events reduces FARs and improves sensitivity and specificity. It also results in models that have better generalizability to new patients and time periods. This has important implications for sustaining CGM use and optimizing glycemic control with fewer hypoglycemic events, improved confidence, and potentially lower HbA1c. To that end, the predictive model presented in this paper will be implemented in a smartphone app in an upcoming clinical pilot study at Texas Children’s Hospital.

Acknowledgments

This study was supported by FDA P50 Pediatric Device Consortia Grant #5P50FD006428 (SWPDC) (Dr Koh—Contact Principal Investigator). The authors would like to thank the contributions of Achu Byju, Department of Biomedical Engineering, Texas A&M University, for designing the overall architecture for the implementation of the data collection system.

This study involves the use of secondary analysis of deidentified data that were not collected specifically for this project and is not human subject research (Texas A&M IRB number 2019-0710).

Authors' Contributions

DDave, ME, and ML conceived the idea for the study. DDave implemented the model and analyzed the data. DDave and ME wrote the manuscript. All authors provided input and helped in revising the manuscript.

Conflicts of Interest

DDeSalvo serves as an independent consultant for Dexcom.

Multimedia Appendix 1

Patient hypoglycemia profile.

PDF File (Adobe PDF File), 57 KB

Multimedia Appendix 2

Patient pump profile.

PDF File (Adobe PDF File), 68 KB

Multimedia Appendix 3

Breakdown of sustained events by daytime and nighttime.

PDF File (Adobe PDF File), 858 KB

Multimedia Appendix 4

Breakdown of sustained and transient events.

PDF File (Adobe PDF File), 46 KB

Multimedia Appendix 5

Breakdown of sustained events.

PDF File (Adobe PDF File), 48 KB

Multimedia Appendix 6

Features extracted for prediction.

PDF File (Adobe PDF File), 59 KB

Multimedia Appendix 7

Receiver operating characteristic plot showing a comparison between different classifiers for giving out predictive alerts for 30-minute (top) and 60-minute (bottom) prediction horizons.

PDF File (Adobe PDF File), 782 KB

  1. Rodbard D. Continuous glucose monitoring: A review of recent studies demonstrating improved glycemic outcomes. Diabetes Technol Ther 2017 Jun;19(S3):S25-S37 [FREE Full text] [CrossRef] [Medline]
  2. Pettus J, Edelman SV. Recommendations for using real-time continuous glucose monitoring (rtCGM) data for insulin adjustments in type 1 diabetes. J Diabetes Sci Technol 2017 Jan;11(1):138-147 [FREE Full text] [CrossRef] [Medline]
  3. Bremer T, Gough DA. Is blood glucose predictable from previous values? A solicitation for data. Diabetes 1999 Mar;48(3):445-451. [CrossRef] [Medline]
  4. ElMoaqet H, Tilbury DM, Ramachandran SK. Multi-step ahead predictions for critical levels in physiological time series. IEEE Trans Cybern 2016 Jul;46(7):1704-1714. [CrossRef] [Medline]
  5. Eren-Oruklu M, Cinar A, Quinn L, Smith D. Estimation of future glucose concentrations with subject-specific recursive linear models. Diabetes Technol Ther 2009 Apr;11(4):243-253 [FREE Full text] [CrossRef] [Medline]
  6. Gani A, Gribok A, Lu Y, Ward W, Vigersky R, Reifman J. Universal glucose models for predicting subcutaneous glucose concentration in humans. IEEE Trans Inf Technol Biomed 2010 Jan;14(1):157-165. [CrossRef] [Medline]
  7. Eren-Oruklu M, Cinar A, Quinn L. Hypoglycemia prediction with subject-specific recursive time-series models. J Diabetes Sci Technol 2010 Jan 01;4(1):25-33 [FREE Full text] [CrossRef] [Medline]
  8. Sparacino G, Zanderigo F, Corazza S, Maran A, Facchinetti A, Cobelli C. Glucose concentration can be predicted ahead in time from continuous glucose monitoring sensor time-series. IEEE Trans Biomed Eng 2007 May;54(5):931-937. [CrossRef] [Medline]
  9. Wang Y, Wu X, Mo X. A novel adaptive-weighted-average framework for blood glucose prediction. Diabetes Technol Ther 2013 Oct;15(10):792-801 [FREE Full text] [CrossRef] [Medline]
  10. Wayne Bequette B, Faye C, Bruce Buckingham A, David Maahs M, John L. Overnight Hypoglycemia and Hyperglycemia Mitigation for Individuals with Type 1 Diabetes: How Risks Can Be Reduced. IEEE Control Syst 2018 Feb;38(1):125-134. [CrossRef]
  11. Howsmon D, Bequette BW. Hypo- and hyperglycemic alarms: Devices and algorithms. J Diabetes Sci Technol 2015 Apr 30;9(5):1126-1137 [FREE Full text] [CrossRef] [Medline]
  12. Messer LH, Calhoun P, Buckingham B, Wilson DM, Hramiak I, Ly TT, In Home Closed Loop Study Group. In-home nighttime predictive low glucose suspend experience in children and adults with type 1 diabetes. Pediatr Diabetes 2017 Aug;18(5):332-339 [FREE Full text] [CrossRef] [Medline]
  13. Bequette BW. Continuous glucose monitoring: real-time algorithms for calibration, filtering, and alarms. J Diabetes Sci Technol 2010 Mar 01;4(2):404-418 [FREE Full text] [CrossRef] [Medline]
  14. Cameron F, Wilson DM, Buckingham BA, Arzumanyan H, Clinton P, Chase HP, et al. Inpatient studies of a Kalman-filter-based predictive pump shutoff algorithm. J Diabetes Sci Technol 2012 Sep 01;6(5):1142-1147 [FREE Full text] [CrossRef] [Medline]
  15. Hughes CS, Patek SD, Breton MD, Kovatchev BP. Hypoglycemia prevention via pump attenuation and red-yellow-green "traffic" lights using continuous glucose monitoring and insulin pump data. J Diabetes Sci Technol 2010 Sep 01;4(5):1146-1155 [FREE Full text] [CrossRef] [Medline]
  16. Palerm CC, Bequette BW. Hypoglycemia detection and prediction using continuous glucose monitoring-a study on hypoglycemic clamp data. J Diabetes Sci Technol 2007 Sep;1(5):624-629 [FREE Full text] [CrossRef] [Medline]
  17. Palerm CC, Willis JP, Desemone J, Bequette BW. Hypoglycemia prediction and detection using optimal estimation. Diabetes Technol Ther 2005 Feb;7(1):3-14. [CrossRef] [Medline]
  18. Sankaranarayanan S, Kumar SA, Cameron F, Bequette BW, Fainekos G, Maahs DM. Model-based falsification of an artificial pancreas control system. SIGBED Rev 2017 Mar 31;14(2):24-33. [CrossRef]
  19. Turksoy K, Bayrak ES, Quinn L, Littlejohn E, Rollins D, Cinar A. Hypoglycemia early alarm systems based on multivariable models. Ind Eng Chem Res 2013 Sep 04;52(35):12329-12336 [FREE Full text] [CrossRef] [Medline]
  20. Gadaleta M, Facchinetti A, Grisan E, Rossi M. Prediction of adverse glycemic events from continuous glucose monitoring signal. IEEE J Biomed Health Inform 2019 Mar;23(2):650-659. [CrossRef] [Medline]
  21. Georga EI, Protopappas VC, Ardigò D, Polyzos D, Fotiadis DI. A glucose model based on support vector regression for the prediction of hypoglycemic events under free-living conditions. Diabetes Technol Ther 2013 Aug;15(8):634-643. [CrossRef] [Medline]
  22. Hamdi T, Ben Ali J, Di Costanzo V, Fnaiech F, Moreau E, Ginoux J. Accurate prediction of continuous blood glucose based on support vector regression and differential evolution algorithm. Biocybern Biomed Eng 2018;38(2):362-372. [CrossRef]
  23. Gadaleta M, Facchinetti A, Grisan E, Rossi M. Prediction of adverse glycemic events from continuous glucose monitoring signal. IEEE J Biomed Health Inform 2019 Mar;23(2):650-659. [CrossRef] [Medline]
  24. Georga EI, Protopappas VC, Ardigo D, Marina M, Zavaroni I, Polyzos D, et al. Multivariate prediction of subcutaneous glucose concentration in type 1 diabetes patients based on support vector regression. IEEE J Biomed Health Inform 2013 Jan;17(1):71-81. [CrossRef] [Medline]
  25. Abbas H, Alic L, Erraguntla M, Ji JX, Abdul-Ghani M, Abbasi QH, et al. Predicting long-term type 2 diabetes with support vector machine using oral glucose tolerance test. PLoS One 2019;14(12):e0219636 [FREE Full text] [CrossRef] [Medline]
  26. Cappon G, Vettoretti M, Marturano F, Facchinetti A, Sparacino G. A neural-network-based approach to personalize insulin bolus calculation using continuous glucose monitoring. J Diabetes Sci Technol 2018 Mar;12(2):265-272 [FREE Full text] [CrossRef] [Medline]
  27. Li K, Liu C, Zhu T, Herrero P, Georgiou P. GluNet: A deep learning framework for accurate glucose forecasting. IEEE J Biomed Health Inform 2020 Feb;24(2):414-423. [CrossRef] [Medline]
  28. Mosquera-Lopez C, Dodier R, Tyler N, Resalat N, Jacobs P. Leveraging a Big Dataset to Develop a Recurrent Neural Network to Predict Adverse Glycemic Events in Type 1 Diabetes. IEEE J. Biomed. Health Inform 202019:1-1. [CrossRef]
  29. Erraguntla M, Zapletal J, Lawley M. Framework for Infectious Disease Analysis: A comprehensive and integrative multi-modeling approach to disease prediction and management. Health Informatics J 2019 Dec;25(4):1170-1187 [FREE Full text] [CrossRef] [Medline]
  30. Mo X, Wang Y, Wu X. Hypoglycemia prediction using extreme learning machine (ELM) and regularized ELM. 2013 Presented at: Proceedings of The 25th Chinese Control and Decision Conference (CCDC): IEEE; 25-27 May 2013; Guiyang, China p. 4405-4409. [CrossRef]
  31. Pérez-Gandía C, Facchinetti A, Sparacino G, Cobelli C, Gómez EJ, Rigla M, et al. Artificial neural network algorithm for online glucose prediction from continuous glucose monitoring. Diabetes Technol Ther 2010 Jan;12(1):81-88. [CrossRef] [Medline]
  32. Zecchin C, Facchinetti A, Sparacino G, Cobelli C. Jump neural network for online short-time prediction of blood glucose from continuous monitoring sensors and meal information. Comput Methods Programs Biomed 2014;113(1):144-152. [CrossRef] [Medline]
  33. Zecchin C, Facchinetti A, Sparacino G, De Nicolao G, Cobelli C. Neural network incorporating meal information improves accuracy of short-time prediction of glucose concentration. IEEE Trans Biomed Eng 2012 Jun;59(6):1550-1560. [CrossRef] [Medline]
  34. Herrero P, Bondia J, Giménez M, Oliver N, Georgiou P. Automatic adaptation of basal insulin using sensor-augmented pump therapy. J Diabetes Sci Technol 2018 Mar;12(2):282-294 [FREE Full text] [CrossRef] [Medline]
  35. Walsh J, Roberts R, Bailey TS, Heinemann L. Bolus advisors: Sources of error, targets for improvement. J Diabetes Sci Technol 2018 Jan;12(1):190-198 [FREE Full text] [CrossRef] [Medline]
  36. Zecchin C, Facchinetti A, Sparacino G, Cobelli C. How much is short-term glucose prediction in type 1 diabetes improved by adding insulin delivery and meal content information to CGM data? A proof-of-concept study. J Diabetes Sci Technol 2016 Sep;10(5):1149-1160. [CrossRef] [Medline]
  37. Contreras I, Oviedo S, Vettoretti M, Visentin R, Vehí J. Personalized blood glucose prediction: A hybrid approach using grammatical evolution and physiological models. PLoS One 2017;12(11):e0187754 [FREE Full text] [CrossRef] [Medline]
  38. Hovorka R, Canonico V, Chassin LJ, Haueter U, Massi-Benedetti M, Orsini Federici M, et al. Nonlinear model predictive control of glucose concentration in subjects with type 1 diabetes. Physiol Meas 2004 Aug;25(4):905-920. [CrossRef] [Medline]
  39. Oviedo S, Vehí J, Calm R, Armengol J. A review of personalized blood glucose prediction strategies for T1DM patients. Int. J. Numer. Meth. Biomed. Engng 2016 Oct 28;33(6):e2833. [CrossRef]
  40. Yang J, Li L, Shi Y, Xie X. An ARIMA model with adaptive orders for predicting blood glucose concentrations and hypoglycemia. IEEE J. Biomed. Health Inform 2019 May;23(3):1251-1260. [CrossRef] [Medline]
  41. Phillips J, Barnsteiner JH. Clinical alarms: improving efficiency and effectiveness. Crit Care Nurs Q 2005;28(4):317-323. [CrossRef] [Medline]
  42. Shivers JP, Mackowiak L, Anhalt H, Zisser H. "Turn it off!": diabetes device alarm fatigue considerations for the present and the future. J Diabetes Sci Technol 2013 May 01;7(3):789-794 [FREE Full text] [CrossRef] [Medline]
  43. Kowalczyk L. The Boston Globe. Patient alarms often unheard, unheeded. 2011 Dec 15.   URL: https://www.bostonglobe.com/2011/02/13/bgcompatientalarms/S6iu8XpdJHghSQvTXxCQbI/story.html [accessed 2021-04-14]
  44. Dave D, DeSalvo DJ, Haridas B, McKay S, Shenoy A, Koh CJ, et al. Feature-based machine learning model for real-time hypoglycemia prediction. J Diabetes Sci Technol 2020 Jun 01:1932296820922622. [CrossRef] [Medline]
  45. Shipe ME, Deppen SA, Farjah F, Grogan EL. Developing prediction models for clinical use using logistic regression: an overview. J Thorac Dis 2019 Mar;11(Suppl 4):S574-S584 [FREE Full text] [CrossRef] [Medline]
  46. Freeze J, Erraguntla M, Verma A. Data integration and predictive analysis system for disease prophylaxis: Incorporating dengue fever forecasts. In: Proceedings of the 51st Hawaii International Conference on System Sciences. 2018 Presented at: Hawaii International Conference on System Sciences 2018; January 3 - 6; Hawaii. [CrossRef]
  47. Wiklund M. Medical device alarm systems can serve as trusted advisors. In: Advancing Safety in Medical Technology, AAMI Medical Device Alarm Summit; Henrdon, VA. 2011 Presented at: AAMI Medical Device Alarm Summit; October 4-5, 2011; Henrdon, VA.
  48. Forlenza GP, Li Z, Buckingham BA, Pinsker JE, Cengiz E, Wadwa RP, et al. Predictive low-glucose suspend reduces hypoglycemia in adults, adolescents, and children with type 1 diabetes in an at-home randomized crossover study: Results of the PROLOG trial. Diabetes Care 2018 Oct;41(10):2155-2161. [CrossRef] [Medline]
  49. Leiter LA, Yoon K, Arias P, Langslet G, Xie J, Balis DA, et al. Canagliflozin provides durable glycemic improvements and body weight reduction over 104 weeks versus glimepiride in patients with type 2 diabetes on metformin: a randomized, double-blind, phase 3 study. Diabetes Care 2015 Mar;38(3):355-364. [CrossRef] [Medline]
  50. Ratner RE. Hypoglycemia: New definitions and regulatory implications. Diabetes Technol Ther 2018 Jun;20(S2):S250-S253. [CrossRef] [Medline]
  51. Battelino T, Danne T, Bergenstal R, Amiel SA, Beck R, Biester T, et al. Clinical targets for continuous glucose monitoring data interpretation: Recommendations from the International Consensus on Time in Range. Diabetes Care 2019 Aug;42(8):1593-1603 [FREE Full text] [CrossRef] [Medline]
  52. Danne T, Nimri R, Battelino T, Bergenstal RM, Close KL, DeVries JH, et al. International Consensus on Use of Continuous Glucose Monitoring. Diabetes Care 2017 Dec;40(12):1631-1640 [FREE Full text] [CrossRef] [Medline]
  53. Mosquera-Lopez C, Dodier R, Tyler N, Resalat N, Jacobs P. Leveraging a Big Dataset to Develop a Recurrent Neural Network to Predict Adverse Glycemic Events in Type 1 Diabetes. IEEE J Biomed Health Inform 2019 Apr 17. [CrossRef] [Medline]
  54. Koenker R, Hallock KF. Quantile regression. J Econ Perspect 2001 Nov 01;15(4):143-156. [CrossRef]
  55. Meinshausen N. Quantile regression forest. J Mach Learn Res 2006;7:983-999.
  56. James G, Witten D, Hastie T, Tibshirani R. Resampling methods. In: An Introduction to Statistical Learning: With Applications in R. Germany: Springer New York; 2013:175-186.
  57. Mathai N, Chen Y, Kirchmair J. Validation strategies for target prediction methods. Brief Bioinform 2020 May 21;21(3):791-802 [FREE Full text] [CrossRef] [Medline]
  58. Hastie T, Tibshirani R, Friedman J. Model assessment and selection. In: The Elements of Statistical Learning: Data Mining, Inference, and Prediction. New York: Springer; 2009:241-247.
  59. Contreras I, Vehi J. Artificial intelligence for diabetes management and decision support: Literature review. J Med Internet Res 2018 May 30;20(5):e10775 [FREE Full text] [CrossRef] [Medline]
  60. Woldaregay AZ, Årsand E, Botsis T, Albers D, Mamykina L, Hartvigsen G. Data-driven blood glucose pattern classification and anomalies detection: Machine-learning applications in type 1 diabetes. J Med Internet Res 2019 May 01;21(5):e11030 [FREE Full text] [CrossRef] [Medline]
  61. Cameron F, Niemeyer G, Gundy-Burlet K, Buckingham B. Statistical hypoglycemia prediction. J Diabetes Sci Technol 2008 Jul;2(4):612-621 [FREE Full text] [CrossRef] [Medline]
  62. Jensen MH, Christensen TF, Tarnow L, Seto E, Dencker Johansen M, Hejlesen OK. Real-time hypoglycemia detection from continuous glucose monitoring data of subjects with type 1 diabetes. Diabetes Technol Ther 2013 Jul;15(7):538-543. [CrossRef] [Medline]
  63. Jensen MH, Mahmoudi Z, Christensen TF, Tarnow L, Seto E, Johansen MD, et al. Evaluation of an algorithm for retrospective hypoglycemia detection using professional continuous glucose monitoring data. J Diabetes Sci Technol 2014 Jan;8(1):117-122 [FREE Full text] [CrossRef] [Medline]
  64. Zapletal J, Erraguntla M, Adelman ZN, Myles KM, Lawley MA. Impacts of diurnal temperature and larval density on aquatic development of Aedes aegypti. PLoS One 2018;13(3):e0194025 [FREE Full text] [CrossRef] [Medline]
  65. Bertachi A, Viñals C, Biagi L, Contreras I, Vehí J, Conget I, et al. Prediction of nocturnal hypoglycemia in adults with type 1 diabetes under multiple daily injections using continuous glucose monitoring and physical activity monitor. Sensors (Basel) 2020 Mar 19;20(6):1705 [FREE Full text] [CrossRef] [Medline]
  66. Kushner T, Breton MD, Sankaranarayanan S. Multi-hour blood glucose prediction in type 1 diabetes: A patient-specific approach using shallow neural network models. Diabetes Technol Ther 2020 Dec;22(12):883-891. [CrossRef] [Medline]
  67. Munoz-Organero M. Deep physiological model for blood glucose prediction in T1DM patients. Sensors (Basel) 2020 Jul 13;20(14):3896. [CrossRef] [Medline]
  68. Zhu T, Li K, Chen J, Herrero P, Georgiou P. Dilated recurrent neural networks for glucose forecasting in type 1 diabetes. J Healthc Inform Res 2020 Apr 12;4(3):308-324. [CrossRef]
  69. Meijner C, Persson S. Blood Glucose Prediction for Type 1 Diabetes using Machine Learning Long Short-term Memory based models for blood glucose prediction [Master's Thesis].: Chalmers University of Technology and University of Gothenburg; 2017.   URL: https://publications.lib.chalmers.se/records/fulltext/251317/251317.pdf [accessed 2020-06-11] [WebCite Cache]
  70. Cichosz SL, Frystyk J, Hejlesen OK, Tarnow L, Fleischer J. A novel algorithm for prediction and detection of hypoglycemia based on continuous glucose monitoring and heart rate variability in patients with type 1 diabetes. J Diabetes Sci Technol 2014 Jul;8(4):731-737 [FREE Full text] [CrossRef] [Medline]
  71. Reddy R, Resalat N, Wilson LM, Castle JR, El Youssef J, Jacobs PG. Prediction of hypoglycemia during aerobic exercise in adults with type 1 diabetes. J Diabetes Sci Technol 2019 Sep;13(5):919-927 [FREE Full text] [CrossRef] [Medline]
  72. Zapletal J, Gupta H, Erraguntla M, Adelman ZN, Myles KM, Lawley MA. Predicting aquatic development and mortality rates of Aedes aegypti. PLoS One 2019;14(5):e0217199 [FREE Full text] [CrossRef] [Medline]
  73. Zhu Y, Jayagopal JK, Mehta RK, Erraguntla M, Nuamah J, McDonald AD, et al. Classifying major depressive disorder using fNIRS during motor rehabilitation. IEEE Trans Neural Syst Rehabil Eng 2020 Apr;28(4):961-969. [CrossRef] [Medline]
  74. Lin YK, Groat D, Chan O, Hung M, Sharma A, Varner MW, et al. Associations between the time in hypoglycemia and hypoglycemia awareness status in type 1 diabetes patients using continuous glucose monitoring systems. Diabetes Technol Ther 2020 Nov;22(11):787-793. [CrossRef] [Medline]
  75. Maahs DM, Lal R, Shalitin S. Diabetes technology and therapy in the pediatric age group. Diabetes Technol Ther 2019 Feb;21(S1):S123-S137 [FREE Full text] [CrossRef] [Medline]
  76. Muñoz Fabra E, Díez JL, Bondia J, Laguna Sanz AJ. A comprehensive review of continuous glucose monitoring accuracy during exercise periods. Sensors (Basel) 2021 Jan 12;21(2):479 [FREE Full text] [CrossRef] [Medline]
  77. Southwest National Pediatric Device Innovation Consortium.   URL: https://swpdc.org/ [accessed 2021-04-13]


ARIMA: autoregressive integrated moving average
CGM: continuous glucose monitoring
FAR: false alert rate
FDA: Food and Drug Administration
FN: false negatives
FP: false positives
HbA1c: glycated hemoglobin A1c
PH: prediction horizon
QRF: quantile regression forest
RF: random forest
RMSE: root-mean-square error
ROC: receiver operating characteristic
TN: true negatives
TP: true positives


Edited by D Griauzde, K Mizokami-Stout; submitted 03.01.21; peer-reviewed by P Prahalad, YK Lin; comments to author 25.02.21; revised version received 09.03.21; accepted 17.03.21; published 29.04.21

Copyright

©Darpit Dave, Madhav Erraguntla, Mark Lawley, Daniel DeSalvo, Balakrishna Haridas, Siripoom McKay, Chester Koh. Originally published in JMIR Diabetes (https://diabetes.jmir.org), 29.04.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Diabetes, is properly cited. The complete bibliographic information, a link to the original publication on https://diabetes.jmir.org/, as well as this copyright and license information must be included.