This work was conducted within the 4T Study. The 4T Study aims to improve care for newly diagnosed youth with T1D in Stanford’s Pediatric Endocrinology clinics. A summary of the 4T program is provided below, and details of the program have been published [9].

The 4T Study includes the 4T Pilot Study and 4T Study 1, as well as long-term follow-ups for each study. The long-term follow-up programs assess the ongoing impact of the 4T Pilot and Study 1 over an extended period. The 4T Pilot Study (n=135) enrolled participants from July 2018 to June 2020 [7,18]. 4T Pilot participants diagnosed before March 2019 were offered CGM, but RPM was not yet available. Those diagnosed between March 2019 and January 2020 were offered non–algorithm-enabled RPM (eg, Dexcom Clarity), which provided individual T1D insights rather than population-wide overviews such as TIDE. Pilot participants diagnosed during and after January 2020 were enrolled in the algorithm-enabled care platform for RPM. Participants diagnosed before January 2020 could transition to algorithm-enabled care when it launched. 4T Study 1 (n=133) enrolled participants from June 2020 to March 2022 and focused on initiating CGM use within the first 30 days after T1D diagnosis.

The quantitative RPM framework presented here is designed to support an algorithm-enabled care model. The 4T Study uses TIDE to support asynchronous review of CGM data for enrolled participants. All 4T participants had the choice to opt into TIDE when it launched.

Participants uploaded their CGM data via a personal smart device or study-provided iPod Touch (Apple Inc) to Dexcom Clarity. TIDE automatically pulled CGM data from the 4T Study’s Dexcom Clarity clinic account using a Python script, analyzed the data, and displayed results in an interactive data visualization dashboard created in Tableau (Salesforce). Guided by CGM clinical consensus metrics, the TIDE algorithm ranked participants based on the urgency of data review and contact by a CDCES [2,19,20].

Each week, the TIDE platform tracked whether participants met the following clinical categories: spending less than 65% of CGM time in range (TIR; 70‐180 mg/dL), more than 4% time below range (TBR) level 1 (<70 mg/dL), more than 1% TBR level 2 (<55 mg/dL), a drop in TIR of more than 15% points from the previous week, more than 50% missing CGM data, or meeting clinical targets [20]. Meeting targets refers to all participants who do not meet any of the other criteria (ie, they have >65% TIR, <4% TBR level 1, <1% TBR level 2, <15% point drop in TIR, and <50% missing CGM data). These clinical categories are based on clinical consensus guidelines [20]. Some metrics were updated between the Pilot Study and Study 1; for example, the TIR target was changed from 70% to 65%, changing the criteria flagged.

The CDCES team members used TIDE to view a list of participants, ranked by their clinical risk likelihood as defined by the CGM metrics listed above. CDCES team members then took appropriate follow-up steps if needed, such as sending secure messages with care recommendations to participants through the EHR. In both the 4T Pilot and Study 1, participant data were initially reviewed weekly for the first year after T1D diagnosis and monthly thereafter [9,12]. After the conclusion of each study, participants had the option to participate in an ongoing long-term follow-up program with monthly data reviews conducted by a CDCES.

A metric is a quantitative measure of an aspect of a care process or the patient population. A KPI combines metrics into a high-level measure or visualization that reflects strategic goals and provides insights into the overall success and performance of an organization [14]. Commonly reported KPIs for health care organizations include patient waiting time, patient health status, patient quality of life, patient safety, and clinical efficiency [14].

Potential KPIs were eligible for inclusion if they aligned with the aims of the 4T Study and could be operationalized using patient CGM data within the existing 4T and TIDE infrastructure.

Using the above evidence-based list of recommended KPIs, inclusion criteria, and input from 4T clinicians, we selected clinical workload, glucose management, and timeliness of care for inclusion in the KPI framework.

We did not include safety or quality-of-life KPIs, as these are the focus of dedicated retrospective analyses (rather than ongoing KPIs). The 4T Study improved HbA_1c, TIR, and patient-reported outcomes (PROs) with minimal TBR (<70 mg/dL) as published [7,9,17,21,22]. The algorithm does not recommend any specific insulin dose adjustment (that is done by the CDCES team) but rather alerts the CDCES that the CGM glucose data require attention.

In 2022, the engineering team began developing a standardized approach to monitor the clinical workload, patient glucose management, and timeliness of care associated with the use of TIDE. Numerous ad hoc metrics were developed and measured based on the experiences of the 4T Study team, CDCES team, and administrative team [2].

The metrics were calculated from data continuously pulled from servers on each participant’s data completeness, the study cohort they were enrolled in, the most recent week they were shown in the TIDE dashboard, their CGM data, and the CDCES responsible for reviewing their data.

We developed an instantiation of the quantitative framework with clinic-specific metrics deployed in an interactive dashboard engineered to streamline the monitoring and visualization of the entire program. All visualizations developed in Tableau were double-checked with manual calculations. We note that participants were enrolled in and departed from the 4T Study continuously; thus, the number of participants shown at any time in the visualizations is less than the total study population.

Feedback from all care team members highlighted 7 metrics as essential for monitoring clinical workload, patient glucose management, and timeliness of care (Table 2). Each metric is associated with a KPI displayed in an interactive figure with various filters and categorizations (Table 2).

At regular TIDE team meetings with clinical and administrative 4T team members, the dashboard was presented, and the metrics were discussed to facilitate an overview of the program. The KPIs were developed and used in parallel with the algorithm-enabled clinical care model and any related clinical studies. While the clinic provides patient care using the algorithm, the KPIs are based on the data gathered, are reviewed at a cadence set by the clinic, and may be used to adjust the care model or workflows (Figure 1). For key insights informed by each metric, refer to Section 3 in Multimedia Appendix 1.

Figure 1 shows how clinical care processes and the KPI framework interact. As noted in Step 1, the RPM algorithm ranks patients based on how urgently they need care. A CDCES then reviews patient CGM data and messages them to provide recommendations and education. As discussed in Step 3, the KPI dashboard pulls information from the RPM algorithm. The KPI framework’s metrics and associated visualizations help inform clinical and operational decisions and actions that modify the clinical care process.

The study was approved by Stanford’s Institutional Review Board (ClinicalTrials.gov: NCT03968055 and NCT04336969; IRB Protocol 52812). Patients and families who consented were enrolled in the 4T Study. Those who did not consent received standard clinical care, which allowed early CGM initiation but did not include RPM. Participants aged 18 years or older or legal guardians of minors provided informed consent, while youth aged 7 years or older provided assent prior to study initiation. Participants in the 4T Study were between the ages of 6 months and 21 years and started on CGM (Dexcom G6; Dexcom Inc) within the first month of T1D diagnosis. Youth and families were not reimbursed for their participation in TIDE, but they could receive compensation for completing PRO surveys, doing home HbA_1c fingerstick tests, participating in focus groups, and participating in the 4T exercise substudy. 4T participants also received a small monetary gift (US $10) for their birthdays as a thank you for their continued participation.

Of the 268 participants in the 4T Pilot and Study 1, a total of 222 were eligible for RPM and were included in the KPI framework (Table 3). All 4T Study 1 participants (n=133) were enrolled in TIDE and included in the KPI framework. In the Pilot Study, participants diagnosed between July 2018 and February 2019 (n=46) were not enrolled in TIDE or included in the KPI framework, and the other 89 were included [7]. The 4T Pilot Study and 4T Study 1 enrolled 92% and 84%, respectively, of all newly diagnosed patients seen in our clinic.

Unlike in traditional care models, there is no fixed number of participant counts. Participant counts change with glucose management, patients enrolling into the program, and patients transferring to other clinics, as well as the variable availability of the care providers, a format that has become more common in microrandomized trials [27]. Rigorous mathematical modeling to understand this is underway [28]. In visualizations for metrics 1-6, the gray area represents the duration of the 4T Pilot, and the white area represents the ongoing duration of the 4T Pilot long-term follow-up program.

The first metric was the total number of participants shown in the TIDE dashboard. By monitoring the number of participants shown in TIDE, this metric allowed for the tracking of population statistics over time. The visualization of the metric selected by the CDCES team included the total number of participants in the program over time, segmented by those who did and did not require review (Figure 2). Based on this metric and visualization, the 4T Study team could investigate recruitment efforts or participant withdrawal rates if the metric was unexpectedly low or high. This metric was also used to understand the stringency of the criteria for participant review if the number of participants being reviewed was too high or too low.

The second metric was the number of participants shown in TIDE, categorized by each CDCES in the program. CDCES names are numbered to protect anonymity. This metric revealed significant differences in the number of participants assigned to each CDCES, as well as stability in the number of participants reviewed by each CDCES over time (Figure 3). Differences in the number of patients assigned to each CDCES can be attributed to the level of their patients’ complexity, their patients’ primary languages, and each CDCES’s time available to review participant data. This metric allowed for the monitoring of CDCES workload and identifying which CDCES team members may experience heavier participant workloads and require additional support, enabling workload balancing.

The third metric was the weekly number of participants shown in TIDE per study. This metric may help clinics with several studies view the number of participants in the RPM program for each study (Figure 4). Alternatively, this metric may be used for tracking the number of patients within various populations, such as the number of patients who are within 1 year of T1D diagnosis, at high risk for hypoglycemia, pregnant, non-English speaking, recently hospitalized, diagnosed with certain medical conditions or with certain family histories, using specific tools to manage their health, or pediatric versus adult patients.

The fourth metric was the number of participants shown in TIDE per clinical category. This metric identified the level of variability in glucose management over time (Figure 5). This metric may help monitor participants’ data and identify areas of improvement or concern, allowing CDCES team members to adjust care plans and interventions. For example, an increase in the number of participants missing glucose data may prompt CDCES team members to educate patients on how to improve connectivity with their CGM and how to upload glucose data.

The fifth metric was the number of participants shown in TIDE per clinical category and reviewing CDCES. This metric identified the similarities and differences in glucose management for the participants reviewed by each CDCES (Figure 6). In our clinic, the metric revealed that all CDCES team members had similar proportions of participants in each clinical category (Figure 6). This metric can track how well each CDCES’s participants were meeting clinical consensus guidelines for CGM targets.

The sixth metric showed the weekly participant counts per study and clinical category. This metric may allow clinics to compare glucose management across studies or populations. This metric allowed the 4T team to identify variation in the numbers of participants in different studies and how often the groups were reviewed by CDCES team members (Figure 7). This provided insight into how well participants in different studies met clinical consensus glycemic targets. This metric facilitated easy comparison of health differences between different groups, study protocols (ie, Pilot vs Study 1), and clinical contact frequency (ie, weekly vs monthly contact).

The seventh metric was originally the number of days since each participant’s most recent date of being displayed in TIDE. The patients shown were sorted from least to most recent display in TIDE and presented along with each participant’s patient ID, the CDCES that reviewed their data, and the study they were enrolled in. After review of this metric at a 4T leadership meeting, several alternative metrics and visualizations were proposed. The chosen visualization was a table comprising participants who had not been shown for more than 20 days and ranked in the top 25% of patients with the longest time since being shown. These restrictions focused on the patients most likely to benefit from review and limited the number of patients to be reviewed by the CDCES team. In addition to showcasing the days since their last appearance in TIDE, the table included each participant’s patient ID, the CDCES responsible for them, and the study they were enrolled in. If a participant had not been flagged in the RPM program in the last 20 days, this figure allowed the CDCES team to message or schedule a visit with participants, making sure no participants were inadvertently ignored. Similarly, this metric allowed clinical research coordinators to remove certain participants from studies that they were no longer enrolled in. For example, based on this metric, research coordinators checked if the participants who had not been shown in the TIDE dashboard for more than 50 days were still enrolled.

The engineering team developed and implemented 7 operational metrics mapped to 3 KPI domains and embedded them in a dashboard used in regular leadership meetings. Administrative, clinical, and research feedback indicated the framework helped monitor clinical workload, patient outcomes, and timeliness of care.

We developed a quantitative framework for monitoring clinical workload, patient glucose management, and timeliness of care for a whole-population T1D care model in which an algorithm analyzes CGM data to help direct care providers to patients with CGM metrics not meeting targets. Through a multiyear process of data analysis, data visualization, and feedback from different team members, we identified metrics that informed oversight and decision-making for the algorithm-enabled care model. This framework may be helpful for other pediatric T1D clinics seeking to optimize their RPM-based care strategies around algorithms that help direct clinician efforts. This framework enables clinics to transition from fixed-interval models to data-driven, responsive care models. Although this work focuses on RPM in a pediatric T1D population, the concepts may translate to adult T1D populations and other chronic health conditions in which algorithms support and guide clinical care [29-31].

The current framework complements, rather than duplicates or substitutes, the way clinical trials report measures of workload, efficiency, and PROs and family-reported outcomes. It complements those measures by aggregating them at the clinic and population level for standardized review on an ongoing basis. For example, in the 4T Study, explicit measures of efficiency measured as per-patient time have been reported in studies describing years of work, while the current KPIs are reviewed on a regular basis [2,32].

The metrics developed yielded valuable insights into the RPM program and may provide insights into health disparities. First, our glucose management metrics—the fourth, fifth, and sixth metrics—reinforce the safety of our RPM platform and the larger 4T Study. As shown in Figure 5, flags related to time below range were minimal, while a significantly larger share resulted from patients meeting all glycemic targets. This aligns with our published findings, which demonstrate improved HbA_1c and TIR, minimal TBR, and positive PROs [9,17,22,33].

Second, despite efforts to evenly distribute CDCES workloads, the number of individuals meeting criteria for review varied during certain review periods, which may have resulted in less time dedicated to those individuals. This finding prompted the care team to consider redistributing subsequent enrollees. Third, the number of participants shown in TIDE per clinical category and reviewing CDCES (the fifth metric) may be used as a starting point to compare the performance of CDCES team members. Meaningful comparisons will require significant additional controls and considerations, such as patient complexity, engagement, and level of access to health care. Fourth, this tool has the potential to identify positive outliers (CDCES team members whose patients achieve clinical targets) or to monitor the results of pilot interventions deployed by a subset of the CDCESs.

Finally, this metric may be used as a tool to investigate health disparities and the impact of patient-provider identity congruence on the relationship between CDCESs and their participants. By stratifying metrics by study group and individual CDCES, we can identify disparities in care delivery and patient outcomes. Previous work in the 4T Pilot Study has already reported similar improvement in HbA_1c for Hispanic and non-Hispanic youth, as well as for publicly and privately insured youth, but it may be valuable to continually monitor these comparisons (and possibly patient primary language) as part of the KPI framework [34]. The algorithmic prioritization of patients may also limit clinician bias and subjective decision-making, as well as prioritize high-risk patients who may otherwise be overlooked. In the future, we may investigate the disparity in usage of diabetes technologies, incongruence in clinician and patient perceptions of diabetes technologies, as well as how interest in and feasibility of diabetes technology programs for patients with public insurance change over time.

The ability to track clinical categories can help inform decisions related to clinical efficiency and care delivery. Guided by insights from the fifth metric, CDCES team members could stage interventions to improve glucose management or CGM wear time, compare our clinic’s TIR to other clinics, or observe how seasonal variations impact TIR.

The proposed framework is the first quantitative approach, of which we are aware, to monitor a CGM-based RPM program’s outcomes, process metrics, and workloads. Developing metrics that drive decision-making is critical, particularly in an era where algorithms can significantly influence patient care and clinical workload. In our clinic, the review of these algorithm-enabled KPIs is becoming standard practice, integral to monitoring and adjusting a care model in which algorithms reduce clinical workload.

KPI frameworks may be helpful for clinics that remotely access patient data, provide RPM-based care, and use algorithms to direct care delivery. Although our work focuses on RPM in a pediatric T1D population, these concepts may translate to an adult population, as well as other chronic health populations in which algorithms support clinical care. Using KPIs to monitor and adjust aspects of a care model may be particularly important for long-term interventions and large clinics or hospital systems [17].

To implement a KPI framework, the clinic or hospital might begin developing metrics by collaborating across multiple departments, potentially including internet technology specialists, data scientists, clinical research coordinators, clinicians, engineers, and quality improvement specialists. Clinics or hospitals may protect the time of or hire dedicated staff to implement the KPI framework and help with data analysis and visualization. We recommend that the internet technology team explore several platforms to host a dashboard for visualizing their metrics. Since large clinics may face scalability or data integration issues, it could be beneficial to partner with technology vendors or research collaborators. In a collaborative effort between Stanford’s 4T team and Tidepool (a diabetes technology non-profit), a new clinic-agnostic, turnkey solution called Tidepool-TIDE was made available to any clinic in the United States [35]. This may help simplify the KPI framework implementation process for other clinics and hospitals.

Because of the success of our KPI framework, more metrics will be added in the future. We plan to add metrics related to CDCES time in TIDE, CDCES messaging activity, patient-clinician contact counts, PROs, and physical activity. These metrics have been researched as part of retrospective analyses, but it may be helpful to monitor them continuously as part of the KPI dashboard [2,7-9,18,22,24-26,36]. The above metrics have not yet been incorporated into the KPI framework because they are not integrated with the current workflow. We hope that the KPI framework supports the transition toward additional automation, such as using large language models to allow the platform to send some types of messages directly to the family [35,37]. The TIDE team continues to explore avenues for integration.

We acknowledge that the 4T Pilot and Study 1 populations are not representative of all other pediatric diabetes clinics, but were representative of the population diagnosed at our clinic [9,38]. This bias may limit the generalizability of our findings. Our study had high rates of early CGM adoption due to the design of the 4T intervention. Future implementation studies should include more representative samples outside of the context of broader research studies to validate the applicability of our framework across a wider range of clinical contexts. We also note that our study population is pediatric T1D, an autoimmune condition, in contrast to type 2 diabetes. T1D has the highest prevalence among non-Hispanic Whites, while type 2 diabetes has the highest prevalence among Native Americans and other non-White populations [39,40].

Although TIDE and the 4T Study have clearly improved patient outcomes and efficiency, we need to identify patients for whom repeated outreach is ineffective [2,8,9,17,21,22,25,26,32]. In future work, we will consider incorporating criteria to flag individuals who are repeatedly prioritized for review but are unreachable or do not seem to benefit from this prioritization. These patients may be switched from TIDE to a more traditional care model of quarterly reviews at clinic visits.

Data from CGMs, insulin pumps, and automated insulin delivery systems play a growing role in the management of T1D. To best use the data, clinics increasingly turn to trusted, transparent, and user-friendly algorithms to translate patient data into insights that inform patient care. As the role of algorithms grows, the proposed framework may offer clinics a quantitative approach to monitor and potentially adjust the care delivery model.