This paper is in the following e-collection/theme issue:

Glucose Tracking and Self-Monitoring of Blood Glucose (162) Telehealth and Telemonitoring (1671) Diabetes Self-Management (484) Diabetes Health Services and System Innovations (117)

Published on in Vol 7, No 2 (2022): Apr-Jun

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/27284, first published .
Algorithm-Enabled, Personalized Glucose Management for Type 1 Diabetes at the Population Scale: Prospective Evaluation in Clinical Practice

Algorithm-Enabled, Personalized Glucose Management for Type 1 Diabetes at the Population Scale: Prospective Evaluation in Clinical Practice

Algorithm-Enabled, Personalized Glucose Management for Type 1 Diabetes at the Population Scale: Prospective Evaluation in Clinical Practice

Authors of this article:

David Scheinker1, 2, 3, 4 Author Orcid Image ;   Angela Gu3 Author Orcid Image ;   Joshua Grossman3 Author Orcid Image ;   Andrew Ward3 Author Orcid Image ;   Oseas Ayerdi3 Author Orcid Image ;   Daniel Miller3 Author Orcid Image ;   Jeannine Leverenz2 Author Orcid Image ;   Korey Hood1, 4 Author Orcid Image ;   Ming Yeh Lee2 Author Orcid Image ;   David M Maahs1, 4, 5 Author Orcid Image ;   Priya Prahalad1, 4 Author Orcid Image
Article Authors Cited by (12) Tweetations (3) Metrics

    Original Paper

    1Department of Pediatrics, Division of Pediatric Endocrinology, Stanford University, Stanford, CA, United States

    2Lucile Packard Children's Hospital, Stanford University, Stanford, CA, United States

    3Department of Management Science and Engineering, Stanford University School of Engineering, Stanford, CA, United States

    4Stanford Diabetes Research Center, Stanford University, Stanford, CA, United States

    5Department of Health Research and Policy, Stanford University, Stanford, CA, United States

    Corresponding Author:

    Priya Prahalad, MD, PhD

    Department of Pediatrics, Division of Pediatric Endocrinology

    Stanford University

    300 Pasteur Dr.

    Room G313

    Stanford, CA, 94025

    United States

    Phone: 1 6507211811

    Fax:1 6507258375

    Email: prahalad@stanford.edu


    Background: The use of continuous glucose monitors (CGMs) is recommended as the standard of care by the American Diabetes Association for individuals with type 1 diabetes (T1D). Few hardware-agnostic, open-source, whole-population tools are available to facilitate the use of CGM data by clinicians such as physicians and certified diabetes educators.

    Objective: This study aimed to develop a tool that identifies patients appropriate for contact using an asynchronous message through electronic medical records while minimizing the number of patients reviewed by a certified diabetes educator or physician using the tool.

    Methods: We used consensus guidelines to develop timely interventions for diabetes excellence (TIDE), an open-source hardware-agnostic tool to analyze CGM data to identify patients with deteriorating glucose control by generating generic flags (eg, mean glucose [MG] >170 mg/dL) and personalized flags (eg, MG increased by >10 mg/dL). In a prospective 7-week study in a pediatric T1D clinic, we measured the sensitivity of TIDE in identifying patients appropriate for contact and the number of patients reviewed. We simulated measures of the workload generated by TIDE, including the average number of time in range (TIR) flags per patient per review period, on a convenience sample of eight external data sets, 6 from clinical trials and 2 donated by research foundations.

    Results: Over the 7 weeks of evaluation, the clinical population increased from 56 to 64 patients. The mean sensitivity was 99% (242/245; SD 2.5%), and the mean reduction in the number of patients reviewed was 42.6% (182/427; SD 10.9%). The 8 external data sets contained 1365 patients with 30,017 weeks of data collected by 7 types of CGMs. The rates of generic and personalized TIR flags per patient per review period were, respectively, 0.15 and 0.12 in the data set with the lowest average MG (141 mg/dL) and 0.95 and 0.22 in the data set with the highest average MG (207 mg/dL).

    Conclusions: TIDE is an open-source hardware-agnostic tool for personalized analysis of CGM data at the clinical population scale. In a pediatric T1D clinic, TIDE identified 99% of patients appropriate for contact using an asynchronous message through electronic medical records while reducing the number of patients reviewed by certified diabetes care and education specialists by 43%. For each of the 8 external data sets, simulation of the use of TIDE produced fewer than 0.25 personalized TIR flags per patient per review period. The use of TIDE to support telemedicine-based T1D care may facilitate sensitive and efficient guideline-based population health management.

    JMIR Diabetes 2022;7(2):e27284

    doi:10.2196/27284

    Keywords

    telehealth; diabetes; population health; continuous glucose monitor; personalized medicine 


    Background

    For patients with type 1 diabetes (T1D) receiving insulin therapy, the American Diabetes Association (ADA) recommends the use of continuous glucose monitors (CGMs) as the standard of care along with quarterly clinic visits with hemoglobin A1c (HbA1c) laboratory testing [1]. However, most people with T1D remain on self-monitored blood glucose because of patient, clinician, or insurance preference and do not meet the current HbA1c targets [2]. The long feedback cycle and the use of relatively little data when self-monitoring are barriers to timely detection and personalized response to deteriorating glucose control. An individual self-monitoring their glucose levels in line with the 2018 ADA standard of care recommendations generates glucose readings 6-10 times per day and receives feedback from their care team every 1-4 months based on a clinic visit or an HbA1c test [3]. In contrast, CGMs record glucose levels once every 5-15 minutes (96-288 times per day). The initiation and continued use of CGMs have increased and are associated with improved clinical outcomes and patient-reported quality of life measures [1-9]. In a US pediatric T1D registry, the use of CGMs increased from 4% in 2013 to 33% in 2017 [6].

    Numerous commercial and open-source platforms provide individual-level visualizations and analyses of CGM data [10-13]. Recent studies by the Advanced Technologies and Treatments for Diabetes consensus on CGM and the ADA and European Association for the Study of Diabetes consensus on precision medicine in diabetes found that although the use of CGMs offers an opportunity to use high-frequency data to identify deteriorating glucose control and tailor personalized management strategies, no standardized, validated methods currently exist outside of automated insulin delivery systems [4,14]. Patient-level tools, such as manufacturer or data aggregator platforms, require physicians or certified diabetes care and education specialists (CDCESs) to examine the data of each individual to identify those people whose glucose management may need improvement. Population-level tools that analyze and present data for the entire population to facilitate prioritizing patients are less common. LibreView (Abbott Laboratories) enables whole-population data review but is proprietary, works only with Libre sensors, and provides access primarily to prespecified metrics [12]. A hardware-agnostic tool would be more appropriate to support care for a population in which patients use a variety of CGMs. An open-source tool would facilitate external evaluation and the development and comparison of alternative models. A tool that calculates personalized metrics for each patient based on their historical data would facilitate the tracking of temporal changes in glucose management. To the best of our knowledge, no validated, hardware-agnostic, open-source tool is available to facilitate the delivery of timely, population-level, personalized care through telehealth.

    Numerous efforts have been made to improve T1D management using remote monitoring, the most successful of which relied on asynchronous messages sent to patients [15]. However, not all studies demonstrated significant and sustained improvement [15]. The implementation of clinical decision support (CDS) has faced a variety of challenges and has led to structured recommendations for their successful design and deployment [16]. Two of the primary areas of focus are that the CDS should improve, rather than disrupt, the appropriate workflows and that it should be designed with an iterative approach [16]. A recent multi-institution, cluster randomized clinical trial on the use of a CDS to improve the management of heart disease showed no significant improvements [17]. There was insufficient evaluation and redesign of the system based on feedback from its intended users [18]. The lack of iterative design and the resulting challenges to the workflow are common in the design of clinical software following the waterfall approach, a structured top-down approach in which the intent is to test and finalize the tool before deployment [19]. The agile approach, a more iterative approach based on rapid deployment and iterative redesign, is a popular alternative [19].

    Objectives

    We sought to design timely interventions for diabetes excellence (TIDE), a decision support tool to identify patients appropriate for asynchronous contact using a secure message through electronic medical record (EMR). To facilitate successful deployment and sustained use, we sought to fit into and improve current workflows by reducing the number of patients requiring review by a physician or the CDCES. We followed the agile approach to deploy the earliest viable version of TIDE in clinical practice and update it based on feedback from physicians and CDCESs.


    Study Design

    This study followed the Guidelines for Developing and Reporting Machine Learning Predictive Models in Biomedical Research [20]. The first phase was hardware-agnostic algorithm design based on the data collected using a variety of CGM hardware. Data were obtained from a convenience sample of eight external data sets: 6 from clinical trials and 2 donated by research foundations [21-28]. The second phase was the design of TIDE, an interactive visual interface presenting CGM data for the entire clinical population, based on iterative feedback from physicians and CDCESs at an academic pediatric T1D clinic. One physician and CDCES used TIDE for 4 weeks and provided feedback based on which TIDE was comprehensively redesigned. Following this, they used the redesigned version for 5 weeks, during which minor improvements were made and errors were fixed. The third phase was a prospective evaluation over the course of 7 weeks of the sensitivity of TIDE for identifying patients who are appropriate for asynchronous contact using a secure message through the EMR and the difference in the number of patients requiring review with and without the use of TIDE. In the final phase, we simulated several measures of the workload generated by TIDE on the same convenience sample of 8 external data sets used to design the algorithms.

    Setting

    TIDE was developed at an academic pediatric T1D clinic caring for youth newly diagnosed with T1D who initiated CGM use within 1 month of onset and enrolled in a weekly remote monitoring program. All patients used Dexcom G6 (Dexcom) monitors, from which data were uploaded and made available to physicians and CDCESs through the Dexcom Clarity Clinic Portal [10]. Participants consented to participate in a longitudinal study evaluating the initiation of CGM early in the course of diabetes and the effects of weekly CGM data review as part of a larger ongoing study, for which the details of the consent process, eligibility criteria, screening, and enrollment process have been reported [8]. Each week, a CDCES used the Dexcom Clarity Clinic Portal to review each patient’s data and send an asynchronous message through the EMRs to those patients who they determined required glucose management guidance. This study was approved by the Stanford University institutional review board. The leadership of the clinic, study authors PP and DMM, who serve on relevant national organizations governing diabetes technology, approved the use of TIDE in clinical care.

    Generic and Personalized Metrics

    A metric is generic if it is calculated the same way for each patient, for example, mean glucose (MG), and personalized if it is calculated for each patient based on their historical data; for example, the month-to-month change in MG. Consensus guidelines were used to generate a large set of generic CGM-based metrics from which clinicians could select the metrics to be tracked in TIDE [29]. The metrics included the number of days the CGM was active and collected more than a minimum percentage of valid readings (ACT), MG, percentage of time in range (TIR) defined as readings 70 to 180 mg/dL, percentage of time extremely hypoglycemic (eHyp) defined as readings <54 mg/dL, and percentage of time hypoglycemic (Hyp) defined as readings <70 mg/dL. Following the same consensus guidelines, each metric was calculated for each day for the entire day from 0:00 to 24:00, daytime 6:00 AM to midnight, and nighttime midnight to 6:00 AM. A complete list of generic metrics is provided in Table S1 in Multimedia Appendix 1. For each generic metric, a personalized metric was defined as the change from a baseline period to a review period (eg, the month-to-month change in the MG). The review period is the timeframe over which the metrics are calculated. It is defined relative to the day on which the data are being reviewed (eg, the last full week). The baseline period is the timeframe over which the baseline value for each personalized metric is calculated (eg, the last full month before the review period). For each metric, “a flag is triggered” when the metric exceeds a prespecified target value.

    Algorithms to calculate generic and personalized metrics and to generate flags as a function of the review period, baseline period, and target value were developed and tested using data from 8 external data sets. The data sets were identified based on an internet search and the professional contacts of the authors: 6 previously published clinical trials and observational data donated by Tidepool and OpenAPS (Table S2 in Multimedia Appendix 1). Data use agreements were signed with Tidepool and OpenAPS, each stipulating that the data may be used for this research project and that those donating the data would not take part in the study design or the reporting of results and would be acknowledged in writing. The data sets had 168,723 patient days of included CGM readings collected with seven types of CGMs: Freestyle Navigator, Dexcom STS, Medtronic Paradigm or Guardian, iPro2, iPro2 Professional CGM, Freestyle Libre Pro Flash, and Dexcom G4. The algorithms with annotated codes and an overview of their design, TIDE, and synthetic CGM data for use with TIDE are available on GitHub [30,31].

    Iterative Design of an Interactive Tool

    The design of an initial version of TIDE was based on a convenience sample of informal interviews and observations used to establish the current state, achieve buy-in from stakeholders, and solicit suggestions for and perceived problems with the proposed workflow. The initial version of TIDE was designed to require a one-time setup followed by repeated use. During the one-time setup, based on their clinical practice and population, clinicians select the metrics to be displayed, the review period over which the generic metrics are calculated, and a baseline period based on which the personalized metrics are calculated. Two pediatric endocrinologists, study authors DMM and PP, and a CDCES, study author JL, identified the consensus glucose metrics currently being used in the clinic to evaluate patient glucose management: ACT was measured as the number of valid readings as a percentage of the maximum number of readings possible (the number of 5-minute intervals during the review period), MG, TIR, eHyp, and Hyp. The review period was set to 1 week ending on the last Sunday before the data review. The targets were initially set as follows: ACT >75%, TIR >70%, eHyp <1%, and Hyp <4%. No target was set for the MG. The valid wear threshold was used by the CDCES to determine whether to reach out to the patient to discuss their use of the CGM and to assist with challenges in obtaining additional sensors. When the valid wear threshold was not met, TIDE presented the metrics and flags as usual, and the CDCES used their judgment to determine whether patient data required further review.

    During each of the first 4 weeks of the use of TIDE, the physician and study author PP logged into the Dexcom Clarity Clinic Portal and downloaded patient ID numbers, CGM readings, and CGM timestamps for all patients in the study. The physician used TIDE to identify patients with flags, reviewed the output of TIDE for appropriateness and patient safety, and forwarded the list of patients flagged by TIDE to CDCES (study author JL). For each flagged patient, the CDCES opened Dexcom Clarity and reviewed detailed patient data, opened the EMR and sent a secure message to the patient, and, if a dose adjustment was made, the dose was updated in the patient’s chart. After these 4 weeks, PP and JL suggested changes that were incorporated into TIDE between weeks 4 and 5. During weeks 5 to 9, the tool was used by the study authors PP and JL to provide clinical care and identify minor adjustments required to improve usability or correct errors. Minor adjustments and corrections were made immediately, usually within 24 hours of identification.

    Prospective Evaluation

    The primary measures were the sensitivity of TIDE for identifying patients appropriate for asynchronous contact using a secure message through EMRs and the reduction in the number of patients reviewed by the CDCES with the use of TIDE. For 7 weeks, a CDCES reviewed the CGM data of each patient in the population and determined those appropriate for asynchronous contact. Sensitivity was defined as the number of appropriate patients flagged by TIDE divided by the number of appropriate patients. The reduction in the number of patients reviewed using TIDE was the number of patients not flagged by TIDE as a percentage of the total population.

    The secondary outcomes calculated were as follows: the average amount of time required for per-patient review and contact as measured by the CDCES using TIDE, specificity (the number of patients not flagged by TIDE divided by the number of patients not appropriate for asynchronous contact), positive predictive power (the number of patients appropriately flagged by TIDE divided the number of patients flagged by TIDE), and negative predictive power (the number of patients not appropriate for asynchronous contact not flagged by TIDE divided by the number of patients not flagged by TIDE).

    Validation on External Data Sets

    The primary determinant of the workload associated with the use of TIDE is the number of patients for whom flags are generated in each review period, equivalent to the rate at which flags are generated per patient per review period. To evaluate the workload associated with the use of TIDE in other settings, we simulated the rate at which generic and personalized flags would be generated for populations with varying levels of glucose management. Patient ID numbers, CGM readings, and CGM timestamps were extracted from each of the 8 external data sets for all patient days that met the inclusion criteria and at least 70% valid CGM readings. Metrics MG, TIR, eHyp, and Hyp Flag and the thresholds for these metrics were chosen based on consensus guidelines [29]. The thresholds for generic metrics were MG >170 mg/dL, TIR <60%, eHyp >1%, and Hyp >3%. Flag thresholds for personalized metrics were chosen based on the clinical experience of the study authors DMM and PP, as TIR less than baseline TIR minus 10 percentage points and MG greater than baseline MG plus 10 mg/dL. The duration of each measurement period was 1 week starting on Sunday. Personalized flags were calculated for patients with at least 4 weeks of data. For each personalized metric, the baseline period was the last week in the patient data that preceded the week being analyzed. The rate at which flags were generated was measured for each type of flag, each patient, and each dataset. For each data set, the rates at which flags were generated and the percentage of patients for whom at least one MG or TIR flag was generated every week were calculated. The rates of generic and personalized flags for MG and TIR were compared using a 2-tailed paired t test. The number of patients for whom an MG or TIR flag was generated every week was recorded.


    Iterative Design of an Interactive Tool

    The first version of TIDE displayed the metrics that triggered a flag in red and those that did not trigger a flag in green (Figure 1). After the feedback from the first 4 weeks of its use, the primary changes to the interface were as follows: the columns displaying MG, the number of readings, and the number of 5-minute intervals during the review period were removed to minimize the number of patients who received flags that did not require dose adjustments while not missing those who required dose adjustments; the criterion for TIR target was changed from 70% to 60%; a personalized metric was added to compare each patient’s TIR in the previous week to their TIR in the previous 4 weeks with a target of an increase in TIR or a drop in TIR of no more than 10% points; the color-coding was revised so only metrics that triggered a flag were highlighted; the wording of the display names of the metrics was changed to be more interpretable; a feature was added to allow the person using the tool to specify whether to use data from the most recent 7 days or from the default review period of 7 days ending on the previous Sunday; the patient data, previously presented on a single tab, were split tab into four tabs that displayed all patients, patients with alerts, patients with no data, and patients with data but no alerts; and the visual presentation was made more compact to display more patients per page (Figure 1).

    The primary change in the workflow was that the step of downloading each patient’s data from Dexcom Clarity was replaced with a Python script that downloaded all patient data (Figure 2). The participation of the physician in the review process was no longer necessary as part of the workflow but was continued to ensure patient safety and quality of care. The ultimate intended workflow, initiated months after the completion of this study, is for the CDCES to use TIDE without the participation of physicians.

    ‎
    Figure 1. Initial and revised timely interventions for diabetes excellence (TIDE) interface. TBR: time below range; TIR: time in range.
    View this figure
    ‎
    Figure 2. Workflow illustrated with examples of weekly and monthly cadence of data collection, data transfer, and provider review. CDCES: certified diabetes care and education specialists; CGM: continuous glucose monitor; EMR: electronic medical record; TIDE: timely interventions for diabetes excellence.
    View this figure

    Prospective Evaluation

    Over the last 7 weeks of the study, the number of patients increased from 56 to 64, totaling 427 patient weeks. The sensitivity of TIDE for identifying patients appropriate for contact using an asynchronous message through the EMR was 94% in the first week, 96% in the last week, and 100% in all other weeks (mean 99%, SD 2.5%; Table 1). The average reduction in the number of patients reviewed by the CDCES was 42.8% (182/427; SD 10.9%), that is, the fraction of patients not flagged for review by TIDE (Table 1).

    For patients identified by TIDE as requiring review, the mean duration of the data review process averaged 4.5 minutes per patient (1.5 minutes to access the data in Dexcom Clarity and review it for patterns, 1 minute to log into the patient’s record and document changes, 2 minutes to send the patient a message using a secure EMR-based messaging platform). The weekly specificity, positive predictive power, and negative predictive power of TIDE are shown in Table 1.

    Table 1. Outcomes including sensitivity identifying patients appropriate for asynchronous contact through the medical record and the reduction in the number of patients reviewed.
    Week of reviewFebruary 7, 2020February 14, 2020February 21, 2020February 28, 2020March 6, 2020March 13, 2020March 20, 2020
    Patients in study, N56585962646464
    Patients flagged by TIDEa, n29293639393736
    Patients reviewed, %52506163615856
    Reduction in patients reviewedb, %48503937396263
    True positive flagsc, %29213127303434
    True negative flagsd, %2317191591619
    False positive flagse, %23333640393331
    False negative flagsf, %2000002
    Insufficient CGMg data, %910106899
    Sensitivityb, %9410010010010010096
    Specificity, %50343426193238
    Positive predictive value, %55394640435152
    Negative predictive value, %9310010010010010092

    aTIDE: timely interventions for diabetes excellence.

    bPrimary objective (italicized).

    cFlagged by TIDE and appropriate for asynchronous contact.

    dNot flagged by TIDE and not appropriate for asynchronous contact.

    eFlagged by TIDE and not appropriate for asynchronous contact.

    fNot flagged by TIDE and appropriate for asynchronous contact.

    gCGM: continuous glucose monitor.

    Validation on External Data Sets

    There were 1424 patients with at least 1 day of CGM data that met the inclusion criteria in the 8 external data sets. There were 168,723 patient days with CGM readings across 30,076 weeks. The patient with the most included days had 1028 days over 154 weeks, whereas the patient with the fewest days had 1 day. The mean weekly ACT was 5.1 (IQR 4.0-6.33), MG was 170.7 (IQR 148.8-189.2), mean percentage of TIR 70 to 180 mg/dL was 56.6% (IQR 45.1%-68.4%), eHyp <54 mg/dL was 1.9% (IQR 0.25%-2.32%), and Hyp <70 mg/dL was 3.3% (IQR 1.50%-4.60%). Across data sets, the minimum and maximum number of patients included were 12 and 450, respectively; the number of CGM days per patient was 9.8 and 256.8, respectively; MG was 141.3 (SD 21.3) and 207.0 (SD 35.3), respectively; mean percentage TIR was 38.2% (SD 14.2%) and 74.2% (SD 13.3%), respectively; Hyp was 2.1% (SD 2.3%) and 4.6% (SD 3.8%), respectively; and eHyp was 0.5% (SD 0.9%) and 3.7% (SD 4.9%), respectively (Table 2).

    Data sets were numbered by increasing mean MG. The analysis of personalized MG and TIR flags included 1100 patients with at least 4 weeks of data. The median frequency was significantly higher for generic TIR flags than for personalized TIR flags, 0.47 (IQR 0.12-0.83) versus 0.19 (IQR 0.12-0.26) flags per patient per week (P<.001), respectively, as was the SD of the frequency of flags (0.36 vs 0.11, respectively; P<.001; Figure 3). The median frequency of flags was significantly higher for generic MG than for personalized MG, 0.31 (IQR 0.05-0.71) versus 0.30 (IQR 0.22-0.37) flags per patient per week (P<.001), as was the SD of the frequency of flags 0.35 versus 0.12 (P<.001; Figure 3). In the two data sets with the highest mean MG, data sets 7 and 8, respectively, 49.7% (97/212) and 81% (26/32) of patients had a generic MG or TIR flag every week. Across all data sets, 15.64% (172/1100) of patients had a generic MG or TIR flag every week, and 0% (1/1100) of patients had a personalized MG or TIR flag every week (Table 3).

    Table 2. Continuous glucose monitor (CGM) data in external data sets.
    Data setPatients, nCGM days, n (number per patient)CGM weeks, n (number per patient), nDays active, mean (SD)Glucose (mg/dL), mean (SD)Time in rangea (%), mean (SD)Percentage of time hypoglycemicb, mean (SD)Percentage of time extremely hypoglycemicc, mean (SD)
    18521,236 (249.8)3372 (39.7)6.3141.3 (21.3)74.2 (13.3)3.1 (2.4)2 (3.1)
    212315 (26.2)63 (5.2)5152.9 (23.3)66.4 (13.2)4 (3)1.7 (2.4)
    312030,815 (256.8)4546 (37.9)6.8158.3 (26.7)65.9 (16)2.1 (2.3)0.5 (0.9)
    422548,805 (216.9)7670 (34.1)6.4162.3 (24.1)62.1 (14.1)3 (2.6)0.9 (1.4)
    545054,535 (121.2)11,936 (26.5)4.6162.6 (28)62.4 (16.3)2.9 (3.1)1 (2.3)
    61801682 (9.3)378 (2.1)4.4176.6 (33.3)50.4 (15.6)3.6 (3.2)3.4 (4.3)
    721910,025 (45.8)1727 (7.9)5.8184.9 (44.4)45.1 (16.6)4.6 (3.8)3.7 (4.9)
    81331310 (9.8)384 (2.9)3.4207 (35.3)38.2 (14.2)2.9 (3.7)1.6 (3.6)

    aPercentage of readings that were 70 to 180 mg/dL.

    bPercentage of readings <70 mg/dL.

    cPercentage of readings <54 mg/dL.

    ‎
    Figure 3. Frequency of generic and personalized flags in external cohorts.
    View this figure
    Table 3. Frequency of generic and personalized flags based on continuous glucose monitor data in external data sets.
    Data setPatients, n Patients with at least 4 weeks of data, n (%)Glucose flags, n (flags per patient per week)Time in range flagsa, n (flags per patient per week)Patients with mean glucose or time in range flag every week, n (%)



    		GenericbPersonalizedaGenericcPersonalizeddGenericPersonalized
    18578 (92)329 (0.1)725 (0.22)493 (0.15)493 (0.12)1 (1)0 (0)
    21210 (83)13 (0.22)14 (0.24)19 (0.33)19 (0.16)1 (10)0 (0)
    3120120 (100)1417 (0.31)1128 (0.25)1535 (0.34)1535 (0.15)4 (3.3)0 (0)
    4225225 (100)2595 (0.34)2002 (0.26)3204 (0.42)3204 (0.16)9 (4)0 (0)
    5450436 (97)4137 (0.35)3498 (0.29)4895 (0.41)4895 (0.21)31 (7.1)0 (0)
    61804 (2)6 (0.38)4 (0.25)9 (0.56)9 (0.19)1 (25)0 (0)
    7219195 (89)967 (0.58)525 (0.31)1360 (0.81)1360 (0.18)97 (49.7)0 (0)
    813332 (24)116 (0.84)47 (0.34)131 (0.95)131 (0.22)26 (81.3)1 (3.1)

    aPersonalized mean glucose (MG) flag triggered when MG>MG+10 mg/dL in baseline period.

    bGeneric mean glucose flag triggered when mean bigeneric glucose >170 mg/dL.

    cGeneric time in range flag triggered when the percentage of readings of 70-180 mg/dL was <60%.

    dPersonalized time in range (TIR) flag triggered when TIR<TIR−10% points in baseline period.


    Principal Findings

    We designed TIDE as an open-source hardware-agnostic tool for the personalized analysis of CGM data at the clinic population scale. In a pediatric T1D clinic, TIDE identified 99% of patients appropriate for contact using an asynchronous message through the EMR while reducing the number of patients reviewed by certified diabetes educators by 43%. For each of the 8 external data sets, simulation of the use of TIDE produced fewer than 0.25 personalized TIR flags per patient per review period. TIDE and the underlying algorithms are free open-source software available on GitHub [30,31]. Upon request, we will help clinicians customize the tool to their setting and deploy it in practice.

    TIDE was developed with an iterative agile approach to support asynchronous contact with patients, the form of contact found to be most effective by a systematic review of T1D telemetry systems [15]. An initial version of TIDE was produced quickly, and few significant design decisions were made before physicians and CDCESs used TIDE to provide feedback. Physicians and CDCESs identified their preferences for a rule-based approach over less-interpretable approaches such as machine learning, time-series analysis, or alternative statistical smoothing techniques. On the basis of physician and CDCES feedback, TIDE was designed to identify how a patient’s glucose management differs from validated recommendations and to produce interpretable flags to facilitate recommendations. As TIDE uses consensus guidelines, it may be more broadly applicable than a model trained on a small or nonrepresentative subset of the population would be. The use of TIDE to identify patients for asynchronous messaging fits well with the CDCES workflow. The CDCES could send a message to each patient identified by TIDE and move immediately to the next patient, rather than spending time scheduling an appointment or trying to contact the patient or family. The primary challenges of the agile approach and asynchronous messaging are that additional resources may be required to ensure that patient care is not adversely affected during the deployment of an early stage tool or because of patients ignoring messages. In this study, during the initial testing period, a physician reviewed the output of TIDE to ensure that the quality of care was not compromised, and the CDCES tracked whether patients read their messages and followed up accordingly.

    The use of generic and personalized flags has complementary benefits. For patients with average glucose management, the generic flags provide a standardized approach to care based on the most recent consensus guidelines. For patients with very well or very poorly managed glucose levels, personalized flags based on patient progress may be more informative. If a person with MG 208 (the mean in one of the external data sets) consistently reduced their glucose by 5 mg/dL per review period, a generic metric may trigger a flag for numerous consecutive review periods, whereas a personalized metric would indicate improvement. During the first major revision of TIDE, a personalized metric was added to track the TIR to help CDCESs identify changes in patient management that did not cross the threshold of a personalized metric. With any flags, particularly in pediatric and young adult populations, it is important to further test the optimal timing and frequency (ie, the dose) to strike the correct balance of receiving action-oriented guidance while not further burdening the person.

    Nonendocrinologists care for numerous people with T1D [32]. Clinicians who are not aware of the most recent consensus guidelines or who are not comfortable with diabetes technology, such as CGM data, may not have the resources to provide patients with appropriate care recommendations. Programs to provide telemedicine-based care or train nonspecialists are associated with better outcomes but require resources and time investment that limit participation and scope [33,34]. The use of a relatively simple tool with metrics and targets based on the consensus guidelines may be useful for nonspecialist clinicians to inform the care of such patients.

    Strengths and Limitations

    A strength of this study is the evaluation across 8 external datasets of the potential applicability of TIDE. Most T1D clinics in the United States see patients 3 to 12 times per year primarily in person [3]. In this study, reviewing a patient’s data with TIDE and sending the patient a message required an average of 4.5 minutes. In a clinic in which patients use CGM and 15-minute-long in-person visits, eliminating an average of 1 in-person visit per patient per year would provide sufficient time for an average of 3 message-based contacts. Using TIDE to support such a workflow requires that TIDE flags an average of 0.25 patients per month for review, equivalent to 3 reviews per patient per year. In a simulation of the use of TIDE for populations with differing average levels of glucose management, the personalized TIR metric flagged no more than 0.22 patients per review period, even for the population with an average MG level of 207 mg/dL. As TIDE flags patients based on deteriorating glucose control, patient contacts would be targeted to address the need rather than per a fixed schedule. Each patient would be more likely to receive care when their control deteriorates. On an average, patients with worse control may receive more contacts than those with better control. Such deployments of TIDE are ongoing with two partner clinics, one in the United States and another in Australia, each caring for ≥1000 patients with T1D and using CGMs.

    This study has several limitations. The workflow presented requires downloading data and toggling between the tool and Dexcom Clarity and is not integrated with the EMR. Integration of CGM data with the EMR will facilitate integration with current telehealth workflows and allow the tool to incorporate data on the timing of each patient’s previous and upcoming visits into the recommendations. The specificity of this tool was significantly lower than its sensitivity. However, specificity is less relevant than the direct measure of the primary outcome and time savings associated with the use of TIDE. Specificity may be improved by using the data on which patients did and did not require a review to tune the algorithm for which patients should be flagged. Improvements are ongoing to streamline, standardize, and scale the data review; to improve the sensitivity and specificity of the tool; and to incorporate the tool into the EMR [35]. There was a 43% reduction in the number of patients reviewed each week; however, reviewing patients weekly is not currently the standard of care and may represent increased time spent for most diabetes clinicians. The reduction in the number of patients requiring review in a different setting may change with the cadence of the review and the criteria for review. This study was conducted as a novel proof-of-concept intervention to create new knowledge and generate data on the performance of an automated tool. The metrics and thresholds were not derived from a systematic hypothesis-based approach or a survey of patients, families, and clinicians. The thresholds used to create the tool were based on consensus guidelines with the input of a group of experts in diabetes technology, and the revised version of the tool used feedback from 4 weeks of use in clinical care.

    Future Studies

    Subsequent efforts to deploy TIDE in other settings may benefit from a formal quality improvement framework with an established aim, predefined targets for metrics, implementation criteria, and well-defined iteration cycles. Subsequent research is ongoing to explicitly incorporate measurements of the additional time necessary for clinical decision-making based on the review of the data, examine the operational requirements to expand the number of individuals monitored with the help of this tool, and identify if the use of such a tool may allow a clinic with fixed resources to provide care for more patients through a more efficient use of clinician time [35].

    Conclusions

    We developed and deployed TIDE, a tool that uses metrics based on consensus guidelines, to identify 99% of patients appropriate for contact using an asynchronous message while reducing the number of patients requiring review by a physician or certified diabetes educator by 43%. Further investigation is necessary to understand the potential of automated analyses of CGM data to support broader access to personalized and timely glucose management.

    Acknowledgments

    The authors thank the staff of the type 1 diabetes (T1D) clinic in which the study was conducted. The authors thank Tidepool (Howard Look and Brandon Arbiter) and OpenAPS (Dana Lewis) for sharing data for these analyses. For the data used from the T1D Exchange, the analyses, content, and conclusions presented herein are the sole responsibility of the authors and have not been reviewed or approved by the T1D Exchange.

    We thank the 4T Study team including Ananta Addala, Dessi Zaharieva, Nora Arrizon-Ruiz, Franziska Bishop, Annette Chmielewski, Barry Conrad, Ana Cortes, Manisha Desai, Victoria Ding, Rebecca Gardner, Simrat Ghuman, Carolyn Herrera, KH, Julie Hooper, Brianna Leverenz, JL, Erica Pang, Natalie Pageler, Piper Sagan, Julie Senaldi, Anjoli Martinez-Singh, and Michelle Wiedemann.

    We thank the numerous engineering students and physicians that contributed to the technical design of TIDE, the design of the operational model, and its iterative improvement: Johannes O Ferstad, Jacqueline J Vallon, Daniel Jun, Angela Gu, Anastasiya Vitko, Dianelys P Morales, MYL, Christos Vasilakis, Andrew Shin, and Esli Osmanlliu.

    This study was supported by R18DK122422, and PP was supported by P30DK116074.

    Conflicts of Interest

    DM has research support from the National Institutes of Health, JDRF, National Science Foundation, and the Helmsley Charitable Trust, and his institution has research support from Medtronic, Dexcom, Insulet, Big Foot Biomedical, Tandem, and Roche. DM consulted Abbott, Helmsley Charitable Trust, Sanofi, Novo Nordisk, Eli Lilly, and Insulet. KH has research support from Dexcom, Inc, for investigator-initiated research and consultant fees from the Lilly Innovation Center; Lifescan Diabetes Institute; Insulet, Inc; and Roche Diagnostics. AW receives compensation from HealthPals, Inc. in the form of salary. AW was supported by the US Department of Defense through a National Defense Science and Engineering Graduate (NDSEG) Fellowship.

    ‎
    Multimedia Appendix 1

    Full list of generic metrics, sources of data for retrospective analysis, and timely interventions for diabetes excellence screenshots.

    PDF File (Adobe PDF File), 222 KB
    1. American Diabetes Association Professional Practice Committee, Draznin B, Aroda VR, Bakris G, Benson G, Brown FM, et al. 7. Diabetes technology: standards of medical care in diabetes-2022. Diabetes Care 2022 Jan 01;45(Suppl 1):S97-112. [CrossRef] [Medline]
    2. Foster NC, Beck RW, Miller KM, Clements MA, Rickels MR, DiMeglio LA, et al. State of type 1 diabetes management and outcomes from the T1D exchange in 2016-2018. Diabetes Technol Ther 2019 Feb;21(2):66-72 [FREE Full text] [CrossRef] [Medline]
    3. Chiang JL, Maahs DM, Garvey KC, Hood KK, Laffel LM, Weinzimer SA, et al. Type 1 diabetes in children and adolescents: a position statement by the American Diabetes Association. Diabetes Care 2018 Sep;41(9):2026-2044 [FREE Full text] [CrossRef] [Medline]
    4. Danne T, Nimri R, Battelino T, Bergenstal RM, Close KL, DeVries JH, et al. International consensus on use of continuous glucose monitoring. Diabetes Care 2017 Dec;40(12):1631-1640 [FREE Full text] [CrossRef] [Medline]
    5. DeSalvo DJ, Miller KM, Hermann JM, Maahs DM, Hofer SE, Clements MA, T1D Exchange and DPV Registries. Continuous glucose monitoring and glycemic control among youth with type 1 diabetes: international comparison from the T1D exchange and DPV initiative. Pediatr Diabetes 2018 Nov;19(7):1271-1275 [FREE Full text] [CrossRef] [Medline]
    6. Miller KM, Hermann J, Foster N, Hofer SE, Rickels MR, Danne T, T1D Exchange and DPV Registries. Longitudinal changes in continuous glucose monitoring use among individuals with type 1 diabetes: international comparison in the German and Austrian DPV and U.S. T1D exchange registries. Diabetes Care 2020 Jan;43(1):e1-e2 [FREE Full text] [CrossRef] [Medline]
    7. Prahalad P, Yang J, Scheinker D, Desai M, Hood K, Maahs DM. Hemoglobin A1c trajectory in pediatric patients with newly diagnosed type 1 diabetes. Diabetes Technol Ther 2019 Aug;21(8):456-461 [FREE Full text] [CrossRef] [Medline]
    8. Prahalad P, Zaharieva DP, Addala A, New C, Scheinker D, Desai M, et al. Improving clinical outcomes in newly diagnosed pediatric type 1 diabetes: teamwork, targets, technology, and tight control-the 4T study. Front Endocrinol (Lausanne) 2020;11:360 [FREE Full text] [CrossRef] [Medline]
    9. Prahalad P, Addala A, Scheinker D, Hood KK, Maahs DM. CGM initiation soon after type 1 diabetes diagnosis results in sustained CGM use and wear time. Diabetes Care 2020 Jan;43(1):e3-e4 [FREE Full text] [CrossRef] [Medline]
    10. Dexcom Clarity.   URL: https://clarity.dexcom.com [accessed 2021-05-01]
    11. Kumar RB, Goren ND, Stark DE, Wall DP, Longhurst CA. Automated integration of continuous glucose monitor data in the electronic health record using consumer technology. J Am Med Inform Assoc 2016 May;23(3):532-537 [FREE Full text] [CrossRef] [Medline]
    12. Abbot LibreView.   URL: https://www.libreview.com/ [accessed 2021-05-01]
    13. Nimri R, Muller I, Atlas E, Miller S, Fogel A, Bratina N, et al. MD-Logic overnight control for 6 weeks of home use in patients with type 1 diabetes: randomized crossover trial. Diabetes Care 2014 Nov;37(11):3025-3032. [CrossRef] [Medline]
    14. Chung WK, Erion K, Florez JC, Hattersley AT, Hivert MF, Lee CG, et al. Precision medicine in diabetes: a consensus report from the American Diabetes Association (ADA) and the European Association for the Study of Diabetes (EASD). Diabetologia 2020 Sep;63(9):1671-1693 [FREE Full text] [CrossRef] [Medline]
    15. Eberle C, Stichling S. Telemetric interventions offer new opportunities for managing type 1 diabetes mellitus: systematic meta-review. JMIR Diabetes 2021 Mar 16;6(1):e20270 [FREE Full text] [CrossRef] [Medline]
    16. Greenes RA, Bates DW, Kawamoto K, Middleton B, Osheroff J, Shahar Y. Clinical decision support models and frameworks: seeking to address research issues underlying implementation successes and failures. J Biomed Inform 2018 Feb;78:134-143 [FREE Full text] [CrossRef] [Medline]
    17. Adusumalli S, Westover JE, Jacoby DS, Small DS, VanZandbergen C, Chen J, et al. Effect of passive choice and active choice interventions in the electronic health record to cardiologists on statin prescribing: a cluster randomized clinical trial. JAMA Cardiol 2021 Jan 01;6(1):40-48 [FREE Full text] [CrossRef] [Medline]
    18. Maddox TM. Clinical decision support in statin prescription-what we can learn from a negative outcome. JAMA Cardiol 2021 Jan 01;6(1):48-49. [CrossRef] [Medline]
    19. Jackson S, Yaqub M, Li C. The agile deployment of machine learning models in healthcare. Front Big Data 2019 Jan 8;1:7 [FREE Full text] [CrossRef] [Medline]
    20. Luo W, Phung D, Tran T, Gupta S, Rana S, Karmakar C, et al. Guidelines for developing and reporting machine learning predictive models in biomedical research: a multidisciplinary view. J Med Internet Res 2016 Dec 16;18(12):e323 [FREE Full text] [CrossRef] [Medline]
    21. OpenAPS.   URL: https://openaps.org [accessed 2022-03-16]
    22. Haymond MW, DuBose SN, Rickels MR, Wolpert H, Shah VN, Sherr JL, T1D Exchange Mini-dose Glucagon Study Group. Efficacy and safety of mini-dose glucagon for treatment of nonsevere hypoglycemia in adults with type 1 diabetes. J Clin Endocrinol Metab 2017 Aug 01;102(8):2994-3001. [CrossRef] [Medline]
    23. Tidepool.   URL: https://tidepool.org [accessed 2022-03-16]
    24. Aleppo G, Ruedy KJ, Riddlesworth TD, Kruger DF, Peters AL, Hirsch I, REPLACE-BG Study Group. REPLACE-BG: a randomized trial comparing continuous glucose monitoring with and without routine blood glucose monitoring in adults with well-controlled type 1 diabetes. Diabetes Care 2017 Apr;40(4):538-545 [FREE Full text] [CrossRef] [Medline]
    25. JDRF CGM Study Group. JDRF randomized clinical trial to assess the efficacy of real-time continuous glucose monitoring in the management of type 1 diabetes: research design and methods. Diabetes Technol Ther 2008 Aug;10(4):310-321. [CrossRef] [Medline]
    26. Weinstock RS, DuBose SN, Bergenstal RM, Chaytor NS, Peterson C, Olson BA, T1D Exchange Severe Hypoglycemia in Older Adults With Type 1 Diabetes Study Group. Risk factors associated with severe hypoglycemia in older adults with type 1 diabetes. Diabetes Care 2016 Apr;39(4):603-610. [CrossRef] [Medline]
    27. Bergenstal RM, Gal RL, Connor CG, Gubitosi-Klug R, Kruger D, Olson BA, T1D Exchange Racial Differences Study Group. Racial differences in the relationship of glucose concentrations and hemoglobin A1c levels. Ann Intern Med 2017 Jul 18;167(2):95-102. [CrossRef] [Medline]
    28. Nwosu BU, Maranda L, Cullen K, Greenman L, Fleshman J, McShea N, et al. A randomized, double-blind, placebo-controlled trial of adjunctive metformin therapy in overweight/obese youth with type 1 diabetes. PLoS One 2015;10(9):e0137525 [FREE Full text] [CrossRef] [Medline]
    29. Maahs DM, Buckingham BA, Castle JR, Cinar A, Damiano ER, Dassau E, et al. Outcome measures for artificial pancreas clinical trials: a consensus report. Diabetes Care 2016 Jul;39(7):1175-1179 [FREE Full text] [CrossRef] [Medline]
    30. CGM Algorithms. SURF. 2020.   URL: https://github.com/joshuagrossman/a1c [accessed 2022-03-16]
    31. SURF-TIDE. GitHub. 2021 Sep 15.   URL: https://github.com/jferstad/SURF-TIDE [accessed 2022-03-16]
    32. Lipman TH, Hawkes CP. Racial and socioeconomic disparities in pediatric type 1 diabetes: time for a paradigm shift in approach. Diabetes Care 2021 Jan;44(1):14-16. [CrossRef] [Medline]
    33. Malasanos TH, Burlingame JB, Youngblade L, Patel BD, Muir AB. Improved access to subspecialist diabetes care by telemedicine: cost savings and care measures in the first two years of the FITE diabetes project. J Telemed Telecare 2005;11 Suppl 1:74-76. [CrossRef] [Medline]
    34. Zhou C, Crawford A, Serhal E, Kurdyak P, Sockalingam S. The impact of project ECHO on participant and patient outcomes: a systematic review. Acad Med 2016 Oct;91(10):1439-1461. [CrossRef] [Medline]
    35. Ferstad JO, Vallon JJ, Jun D, Gu A, Vitko A, Morales DP, et al. Population-level management of type 1 diabetes via continuous glucose monitoring and algorithm-enabled patient prioritization: Precision health meets population health. Pediatr Diabetes 2021 Nov;22(7):982-991. [CrossRef] [Medline]

    ‎
    ACT: number of days continuous glucose monitor active
    ADA: American Diabetes Association
    CDCES: certified diabetes care and education specialist
    CDS: clinical decision support
    CGM: continuous glucose monitor
    eHyp: extreme hypoglycemic
    EMR: electronic medical record
    Hyp: hypoglycemic
    MG: mean glucose
    T1D: type 1 diabetes
    TIDE: timely interventions for diabetes excellence
    TIR: time in range

    Edited by T Leung; submitted 20.01.21; peer-reviewed by S Hussain, S Schembre; comments to author 13.04.21; revised version received 19.05.21; accepted 22.02.22; published 06.06.22

    Copyright

    ©David Scheinker, Angela Gu, Joshua Grossman, Andrew Ward, Oseas Ayerdi, Daniel Miller, Jeannine Leverenz, Korey Hood, Ming Yeh Lee, David M Maahs, Priya Prahalad. Originally published in JMIR Diabetes (https://diabetes.jmir.org), 06.06.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Diabetes, is properly cited. The complete bibliographic information, a link to the original publication on https://diabetes.jmir.org/, as well as this copyright and license information must be included.