Prediction of Glucose Metabolism Disorder Risk Using a Machine Learning Algorithm: Pilot Study

Introduction

The incidence of diabetes has been increasing for the last decade and is expected to continue to increase in the future [1-3]. At present, diabetes is diagnosed and predicted based on fasting plasma glucose (FPG), glycated hemoglobin (HbA_1c), and plasma glucose levels 2 hours after a 75-g oral glucose tolerance test (OGTT) [4]. In an OGTT, a patient is asked to ingest a glucose drink, and their plasma glucose (PG) levels and immunoreactive insulin (IRI) levels are measured before and at intervals after the glucose drink is consumed. Although OGTT provides important information regarding pathological conditions of glucose metabolism, many diabetes survey tools predict the risk of diabetes development based only on noninvasive information, such as self-administered questionnaires [5]. The combination of parameters used to diagnose diabetes helps to identify individuals with a high risk of developing diabetes in the future. Heianza et al [6] showed that the combination of HbA_1c and FPG is useful for finding patients with a high risk of developing diabetes. Fujibayashi et al [7] used HbA_1c values, FPG levels, and 2-hour PG to predict instances of high future risk of developing diabetes. Complete data, including 1-hour and 2-hour PG and IRI values obtained by OGTT, may improve the prediction accuracy for diabetes risk.

Previously, logistic regression (LR) analyses were used as initial screening tests [5,8-10]. Recently, studies have demonstrated new methods, including machine learning algorithms, big data mining approaches, and genomic information, for the improved screening and prediction of diabetes [11,12]. Machine learning methods using all relevant information from OGTTs may be able to more accurately predict the risk of developing diabetes and prediabetes. The goal of this study was to verify this hypothesis. To our knowledge, no previous study has predicted the development of diabetes using all of the information from OGTTs combined with machine learning.

We used XGBoost [13,14] for machine learning, an advanced algorithm known for obtaining the winning solutions in data competitions such as Kaggle. In addition, XGBoost has been applied to other medical fields [15-17]. Gao et al [15] compared model-based approaches (such as LRs) and model-free approaches (including using XGBoost) for the task of forecasting falls by patients with Parkinson disease. The authors reported that the model-free approach provided more reliable forecasting. Nishio et al [16] applied XGBoost and support vector machine methods to the computer-aided diagnosis of lung nodules. The authors reported that XGBoost was generally superior to support vector machine methods. Qiao et al [17] applied XGBoost and recurrent neural networks to a task of emergency room visit prediction. The authors reported that the nonlinear models had better performance than linear models.

Methods

Ethics Statement

This study was conducted using data from comprehensive periodic medical examinations at the Center for Preventive Medicine, NTT Medical Center Tokyo, from May 2006 to April 2017. In Japan, employers are required by the Industrial Safety and Health Law to commission medical examinations once a year to ensure the health of their employees. The Center for Preventive Medicine has been contracted by a telecommunications company, Nippon Telegraph and Telephone Corporation (NTT), to provide periodic medical examinations to their employees to comply with this law. This program involves comprehensive periodic medical examinations as well as many services beyond those mandated by law. The data used in this study were collected as part of this general health check-up program at the center. We retrieved subject clinical data from an institutional database, although the examinations were not specifically intended to collect new data for our study. Our research plan was announced on the websites of both our facility and the Center for Preventive Medicine. All subjects were informed that the clinical data obtained by the program would be retrospectively analyzed and published. In addition, it was announced that subjects could withdraw from our research study at any time. The study protocol was approved by the ethical review board of Juntendo University (No. 2017114) and the institutional ethics committee at the Center for Preventive Medicine (No. 17-664).

Study Population

Most of the study subjects were volunteers from among the employees of NTT and their families. They were primarily healthy office workers ranging in age from 40 to 60 years, with more male subjects than females. Our investigation focused on subjects who underwent a 75-g OGTT at the center between May 2006 to April 2017. Subjects without serious diabetes or advanced renal failure were assessed regarding the status of their glucose metabolism using the OGTT.

A total of 20,458 OGTT trials were collected from 9906 subjects during the period at the center. Table 1 shows the distribution of subjects with the number of OGTT trials obtained for each during the period. Overall, 6437 subjects underwent OGTT only once, while 1 subject had 12 OGTTs.

Data Collection

The examinations were performed on 2 consecutive days. On the first day, each patient’s weight and height were measured after the removal of shoes and heavy clothing, and blood pressure was measured with an automatic monitor with the person in the sitting position. In addition, serum samples were collected from each participant after overnight fasting and immediately subjected to biochemical analysis. The blood samples were also used to determine each subject’s HbA_1c level, which was measured using high-performance liquid chromatography with an automatic analyzer. On the second day, the subjects underwent an OGTT. We obtained the subjects’ FPG levels along with 1-hour and 2-hour postloading PG IRI levels during the OGTT. The Japan Diabetes Society (JDS) HbA_1c values were converted to National Glycohemoglobin Standardization Program values using the formula developed by the JDS [18]: HbA_1c=[HbA_1c(JDS)(%)×1.02+0.25(%)]. Insulin sensitivity was calculated with the insulin sensitivity index (ISI; composite) [19,20]: ISI (composite) =[10,000/ sqrt(FPG level(mg/dL)×fasting IRI level(μU/mL)×2-hour PG level(mg/dl)×2-hour IRI level(μU/mL))]. The sum of plasma glucose (SPG) is defined as SPG=FPG levels+1-hour PG level+2-hour PG level. The sum of immunoreactive insulin (S-IRI) is defined as S-IRI=fasting IRI level+1-hour IRI level+2-hour IRI level.

We defined diabetes, normal glucose tolerance (NGT), and prediabetes according to the American Diabetes Association guidelines [4]. Diabetes is defined as subjects with an FPG level ≥126 mg/dL, a 2-hour postloading PG level ≥200 mg/dL, or an HbA_1c concentration ≥6.5%. NGT is defined as subjects with an FPG level <100 mg/dL, a 2-hour postloading PG level <140 mg/dL, and an HbA_1c level <5.7%. Prediabetes is defined as subjects without diabetes who failed to have NGT. In our study, we defined glucose metabolism disorders (GMD) as either diabetes or prediabetes.

Data Handling

Inclusion and Exclusion Flow

Initially, a total of 20,458 OGTT trials across all subjects were included. Data were removed based on the inclusion and exclusion criteria as shown in Figure 1. First, 6437 subjects who underwent OGTT only once during the period were excluded to increase the reliability of the data. Second, missing data were excluded, and the remaining number of trials was 14,020. Then, 2 studies were conducted: predicting the future risk of developing diabetes (study 1) or GMD (study 2).

Study 1. Prediction of Future Risk of Developing Diabetes

Study 1 was aimed at predicting the future risk of developing diabetes. To apply supervised machine learning to the data, data labels (at risk: y=1, not at risk: y=0) were required for each OGTT trial. It is widely known that diabetes can recur even after remission. In addition, women with a history of gestational diabetes have a high risk of developing diabetes in the future [21]. We considered that subjects with a diagnosis of diabetes in the past had a high risk of developing diabetes in the future. Because of this hypothesis, we defined the risk group as follows: a subject was in the risk group (y=1) for diabetes if he or she was diagnosed with diabetes at least once during the period. We defined a subject to be in the nonrisk group (y=0) if he or she did not belong to the risk group. From 14,020 trials, 439 data points from patients diagnosed with diabetes were excluded to focus only on nondiabetic subjects.

‎View this figure

‎View this figure

Examples of the risk group and nonrisk group for diabetes are shown in Figure 2 (left). Subjects A and B underwent OGTT 3 times. Subject B was diagnosed with prediabetes, NGT, and diabetes in the first, second, and third OGTTs, respectively. Thus, Subject B was classified into the risk group, since he or she was diagnosed with diabetes at least once during the period. The third OGTT data point, which occurred after the diagnosis (marked with superscript a), is removed to focus only on nondiabetic data. Subject A was diagnosed with NGT, prediabetes, and NGT in the first, second, and third OGTTs, respectively. Subject A was classified into the nonrisk group, since he or she was never diagnosed with diabetes during the period.

At the end, we had a total of 13,581 OGTT trials of patients who were diagnosed with NGT or prediabetes, each of which was labeled with future risk information (y=0 or y=1). We randomly selected 10,869 (80%) for the training data and used the remaining 2712 (20%) for test data. A classification model was trained using the training data, and the prediction accuracy was evaluated with the test data. Nine classification models were trained, as described below. Table 2 shows a summary of the analyzed OGTT data. No significant differences were observed between the training and test data.

Study 2. Prediction of Future Risk of Glucose Metabolism Disorders

Study 2 was aimed at predicting the future risk of developing GMD, which includes either diabetes or prediabetes. Similar to study 1, we defined a subject as being in the risk group (y=1) for GMD if he or she was diagnosed with GMD (prediabetes or diabetes) at least once during the period. We defined a subject to be in the nonrisk group (y=0) if he or she did not belong to the risk group for GMD. From 14,020 trials, 7260 data points from patients diagnosed with GMD were excluded to focus only on NGT subjects.

Figure 2 (right) shows examples of the risk group and nonrisk group for GMD. Subjects C and D underwent OGTT 3 times. Subject D was diagnosed with prediabetes, NGT, and diabetes in the first, second, and third OGTTs, respectively. Subject D was classified into the risk group of GMD, since he or she was diagnosed with prediabetes at least once during the period. The first and third OGTT, which were diagnosed with prediabetes and diabetes, respectively, were removed to focus only on NGT data. Subject C was diagnosed with NGT all 3 times. Therefore, subject C was classified into the nonrisk group, since he or she was never diagnosed with prediabetes or diabetes during the period.

Finally, we had 6760 OGTT trials of patients who were diagnosed with NGT, each of which was labeled with future risk information (y=0 or y=1). We randomly selected 5408 (80%) for training data, and used the remaining 1352 (20%) for test data. A classification model was trained using the training data, and the prediction accuracy was evaluated using the test data. Nine models were trained, as detailed in the Classification Models section. Table 3 shows a summary of the analyzed data. No significant differences were observed between the training and test data.

Statistical Analysis

XGBoost

XGBoost [13] is open-source software [14] that provides a machine learning method of regression and classification using ensemble learning with gradient tree boosting (GTB) [22]. XGBoost is well known for obtaining the winning solutions in data competitions. Chen and Guestrin [13] reported that “Among the 29 challenge-winning solutions published on Kaggle’s blog during 2015, 17 winning solutions used XGBoost.” Applications of XGBoost include practical tasks such as “store sales prediction, high energy physics event classification, Web text classification, customer behavior prediction, motion detection, ad click-through rate prediction, malware classification, product categorization, hazard risk prediction, and massive on-line course dropout rate prediction.” See Chen and Guestrin [13] for details of the applications. In addition, XGBoost has been applied to medical fields [15-17].

XGBoost (or GTB) learns a regression and classification function in the data space by sequentially optimizing weak learners, called regression trees. The parameters of a regression tree consist of the tree structures and the weights of the leaf nodes. They are sequentially optimized to minimize an objective function, consisting of a fitting loss term plus a regularization term, using gradient methods. XGBoost software is designed to increase the scalability and acceleration of optimized computation for practical use. See Chen and Guestrin [13] for technical details. The underlying GTB algorithm is briefly discussed in Multimedia Appendix 1. XGBoost includes several hyperparameters—including the maximum depth of regression trees, number of weak learners, learning rate, and regularization parameters—which need to be tuned.

Classification Models

To predict the future risk of diabetes (study 1) or GMD (study 2), we developed 9 classification models (A to I) with different input variables, shown in Table 4.

Models A, B, and C used XGBoost. For comparison, models D to I used LR, which is traditionally used in medical research studies. For each classification model, the input variables were set as follows. Model A inputs some basic variables relevant to diabetes or GMD, without OGTT variables. Model B inputs OGTT variables (1-hour PG, 1-hour IRI, 2-hour PG, and 2-hour IRI), as well as the variables of model A. Model C inputs all the measured variables. Blood pressure, lipid parameters, uric acid values, markers of liver function, and markers of kidney function are parameters related with metabolic syndrome, fatty liver, and chronic kidney disease. These conditions are associated with diabetes and were included as variables [23-25]. Models D to F served as baselines using the well-known biomarkers FPG and HbA_1c. Models G to I used the same variables as models A to C to directly compare the performances of XGBoost and LR.

Evaluation

To evaluate the 9 trained classification models, we used the receiver operating characteristic (ROC) curves and their area under the curve (AUC) values computed from the test data [26,27]. ROC curves have commonly been used in diabetes prediction research. In addition, we report the sensitivity and specificity at the cutoff values determined by the Youden index.

Hyperparameter Tuning of XGBoost

As mentioned, XGBoost includes several hyperparameters such as maximum depth of the regression trees, number of weak learners, learning rate, and regularization parameters that need to be tuned.

We tuned the parameters using a grid search to maximize the mean AUC value computed from 5-fold cross validation on the training data. Specifically, the training data were divided into 5 subsets at random: 4 subsets were used for training XGBoost and the other subset was used for validation. The ROC curve and AUC value can be evaluated from the validation subset. This procedure was repeated 5 times with different validation subsets. The mean AUC value can be computed by averaging the 5 AUC values. We tuned the hyperparameters, including the regularization parameters, with a grid search method to maximize the mean AUC value. After finding the optimal values of the hyperparameters, XGBoost was trained using the entire training data set. The final ROC and AUC values were then evaluated with the test data.

Given a ROC curve, a cutoff value is required to compute the sensitivity and specificity. The cutoff value was determined by averaging 5 cutoff values computed from the Youden index from 5-fold cross validation.

Results

Study 1. Prediction of Future Risk of Developing Diabetes

Figure 3 shows the 6 ROC curves for models A to I. Similarly, Multimedia Appendix 2 shows the corresponding curves for models A to F. The horizontal axis represents the false positive rate, and the vertical axis represents the true positive rate. The 3 solid lines (models A, B, and C) show the ROC curves obtained from XGBoost. The dashed lines (models G, H, and I) show the ROC curves obtained from LR.

The AUC values for the 9 classification models are shown in Table 5. For each model, we also show the sensitivity and specificity as determined by the Youden index. We observed that XGBoost had superior performance compared with LR. The AUC value increased with the number of input variables. Models B and C, which exploit XGBoost and complete OGTT information for input variables, showed the best AUC values, 0.90 and 0.93, respectively.

In addition, XGBoost provides an importance score for each input variable. The importance value for each input variable in models A to C are shown in Multimedia Appendix 3 (left), Multimedia Appendix 4 (left), and Multimedia Appendix 5 (left), respectively. In model B, we observed that the OGTT variables (1-hour PG, 1-hour IRI, 2-hour PG, and 2-hour IRI) were more important than FPG or HbA_1c. In model C, we observed that SPG and 2-hour PG were more important variables than FPG or HbA_1c, although multicollinearity needs to be considered.

Study 2. Prediction of Future Risk of Glucose Metabolism Disorders

Figure 4 shows the 6 ROC curves for models A to I. Similarly, Multimedia Appendix 6 shows the corresponding 6 ROC curves for models A to F. The 3 solid lines (models A, B, and C) show the ROC curves obtained from XGBoost. The dashed lines (models G, H, and I) show the ROC curves obtained from LR.

‎View this figure

The AUC values for the 9 models are shown in Table 6. The sensitivity and specificity, as determined by the Youden index, are also shown in Table 6. Similar to study 1, we observed that XGBoost had better performance than LR. The AUC values also increased with the number of input variables. Models B and C, which exploit XGBoost and complete OGTT information as input variables, displayed the highest AUC values, 0.75 and 0.78, respectively.

The importance score of each input variable for models A, B, and C are shown in Multimedia Appendix 3 (right), Multimedia Appendix 4 (right), and Multimedia Appendix 5 (right), respectively. In model B, we observed that the OGTT variables (1-hour PG, 1-hour IRI, 2-hour PG, and 1-hour IRI) were more important than FPG or HbA_1c. In model C, we observed that the OGTT variables (1-hour PG, 1-hour IRI, 2-hour PG, 2-hour IRI, and SPG) were more important variables than FPG or HbA_1c, although multicollinearity needs to be considered.

‎View this figure

Discussion

Principal Findings

In this study, we reported on 2 results for predicting the future risk of diabetes or GMD using complete OGTT information and machine learning.

A feature of our study is that we used a large-scale dataset that included thousands of Japanese OGTT trials, even though OGTT is expensive and invasive. The amount of data enabled us to use a machine learning approach. It is known that one of the earliest detectable abnormalities in the development of diabetes is the deterioration of the early insulin response after glucose loading [28], and the aggravation of insulin sensitivity affects the development of diabetes [7,29-32]. Data obtained from OGTT provide important pathological glucose metabolism information. We believe that the data obtained from OGTT contributed to the improvement in prediction of future risk of diabetes and GMD.

Another feature of our study is that we used an advanced and powerful machine learning method, XGBoost, which resulted in better performance compared with LR. Many previous diabetes risk assessment tools used LR analyses. Recently, various machine learning algorithms have been used for the screening and prediction of diabetes [12,33-35]. Linear approaches are generally unsuited for prediction models with complex correlations. We believe that XGBoost plays an important role in improving the prediction of the future risk of developing diabetes or GMD. To our knowledge, no previous study combined large-scale Japanese OGTT data and XGBoost.

By observing the importance scores (Multimedia Appendix 3-Multimedia Appendix 5) of the input variables computed from XGboost, the OGTT variables (1-hour PG, 1-hour IRI, 2-hour PG, 2-hour IRI) were found to be more important predictors than FPG or HbA_1c for the future risk of diabetes or GMD, although multicollinearity needs to be taken into account. The progress of the PG level after loading can reflect abnormalities in the insulin response. Simultaneous measurement of PG and IRI levels can help evaluate the insulin sensitivity. Although multiple collinearity affects the results, PG and IRI levels after loading appeared to be more important than FPG or HbA_1c.

Limitations

Our research had several limitations. First, we did not use any information obtained from questionnaires in our research. This was because we were concerned that subject recall bias may impact the accuracy of the predictions [36]. As far as we know, previous diabetes risk assessment tools were based on information obtained from questionnaires such as family history and lifestyle habits [5,8-10,37]. Because of this, our results could not be easily compared with these reports. A previous study showed that combining the results of blood tests and questionnaire information improved the prediction accuracy of diabetes risk assessment [37]. We will attempt to improve the accuracy of diabetes risk prediction by integrating information obtained from blood tests and questionnaires. Second, we merged data from subjects who underwent OGTT different numbers of times. That is, we handled data from subjects who had 2 OGTT trials in the same manner as subjects with 10 OGTT trials. Data from subjects who had frequent OGTT trials may have impacted the calculation. Finally, our subjects were affected by selection bias, specifically, the “healthy worker” effect. More than 70% of our participants were healthy male office workers who ranged in age from 40 to 60 years. Thus, the limited sample might not accurately represent the entire population. In addition, we believe that our method is not suitable for predicting rapidly progressing diabetes, as with type 1 diabetes. Also, validation is still required using other data sets.

Comparison With Prior Work

Thoopputra et al [5] considered many diabetes risk assessment tools developed worldwide. In the review, although there a few that used OGTT data or decision tree algorithms, many diabetes risk assessment tools used only noninvasive information and LR analyses. Values for the AUC ranged from 62% to 87%. In Japan, Nanri et al [37] reported a risk score showing an AUC value of 0.882 for predicting type 2 diabetes based on noninvasive information, FPG level and HbA_1c, using an LR analysis. Recently, studies have demonstrated new methods, including machine learning algorithms, big data mining approaches, and genomic information, for the screening and prediction of diabetes [11,12]. Habibi et al [34] developed a model with an AUC value of 0.875 when screening for type 2 diabetes that used a decision tree method and did not require any laboratory tests. López et al [12] reported a model for diabetes prediction having an AUC value of 0.89 that used genetic information and a random forest algorithm.

Conclusion

Our predictions for the future risk of developing diabetes or GMD, using data from thousands of OGTT trials and the machine learning program XGBoost, resulted in higher accuracy compared with traditional LR analysis. Combining complete OGTT information with advanced machine learning algorithms may be useful for detecting the future risk of diabetes or GMD more accurately.