Published on in Vol 6, No 1 (2021): Jan-Mar

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/23687, first published .
Application of the National Institute for Health and Care Excellence Evidence Standards Framework for Digital Health Technologies in Assessing Mobile-Delivered Technologies for the Self-Management of Type 2 Diabetes Mellitus: Scoping Review

Application of the National Institute for Health and Care Excellence Evidence Standards Framework for Digital Health Technologies in Assessing Mobile-Delivered Technologies for the Self-Management of Type 2 Diabetes Mellitus: Scoping Review

Application of the National Institute for Health and Care Excellence Evidence Standards Framework for Digital Health Technologies in Assessing Mobile-Delivered Technologies for the Self-Management of Type 2 Diabetes Mellitus: Scoping Review

Review

1Medical Sciences Division, University of Oxford, Oxford, United Kingdom

2Bodleian Health Care Libraries, University of Oxford, Oxford, United Kingdom

3Nuffield Department of Primary Care Health Sciences, University of Oxford, Oxford, United Kingdom

Corresponding Author:

Laura C Armitage, MB BCh, MRCGP

Nuffield Department of Primary Care Health Sciences

University of Oxford

Radcliffe Primary Care Building

Radcliffe Observatory Quarter, Woodstock Road

Oxford, OX2 6GG

United Kingdom

Phone: 44 1865 617942

Email: laura.armitage@phc.ox.ac.uk


Background: There is a growing role of digital health technologies (DHTs) in the management of chronic health conditions, specifically type 2 diabetes. It is increasingly important that health technologies meet the evidence standards for health care settings. In 2019, the National Institute for Health and Care Excellence (NICE) published the NICE Evidence Standards Framework for DHTs. This provides guidance for evaluating the effectiveness and economic value of DHTs in health care settings in the United Kingdom.

Objective: The aim of this study is to assess whether scientific articles on DHTs for the self-management of type 2 diabetes mellitus report the evidence suggested for implementation in clinical practice, as described in the NICE Evidence Standards Framework for DHTs.

Methods: We performed a scoping review of published articles and searched 5 databases to identify systematic reviews and primary studies of mobile device–delivered DHTs that provide self-management support for adults with type 2 diabetes mellitus. The evidence reported within articles was assessed against standards described in the NICE framework.

Results: The database search yielded 715 systematic reviews, of which, 45 were relevant and together included 59 eligible primary studies. Within these, there were 39 unique technologies. Using the NICE framework, 13 technologies met best practice standards, 3 met minimum standards only, and 23 technologies did not meet minimum standards.

Conclusions: On the assessment of peer-reviewed publications, over half of the identified DHTs did not appear to meet the minimum evidence standards recommended by the NICE framework. The most common reasons for studies of DHTs not meeting these evidence standards included the absence of a comparator group, no previous justification of sample size, no measurable improvement in condition-related outcomes, and a lack of statistical data analysis. This report provides information that will enable researchers and digital health developers to address these limitations when designing, delivering, and reporting digital health technology research in the future.

JMIR Diabetes 2021;6(1):e23687

doi:10.2196/23687

Keywords



Background

Digital technologies are now integral to the delivery of health care and feature in policies for the future of national [1] and global [2] health care systems. The World Health Organization (WHO) defines a health technology as “the application of organized knowledge and skills in the form of devices, medicines, vaccines, procedures and systems, developed to solve a health problem and improve quality of lives” [3]. Typically, digital health technologies (DHTs) include apps, software, and web-based platforms intended to benefit people or the wider health care system [4]. DHTs are increasingly supporting or being used as an adjunct to face-to-face clinical care by facilitating remote health care.

Many DHTs are intended to support chronic disease management, where self-management and preventative medicine are key components of effective care. Approximately 500 million people use mobile device apps to manage their health [5], and diabetes is the condition most commonly targeted by commercial apps [6]. With an increasing global prevalence of type 2 diabetes, mobile device apps offer a potential means of supporting diabetes care, particularly in the context of increasing demands against limited resources. It is imperative that the quality, safety, and effectiveness of such mobile device apps are assessed before deployment in clinical practice. In 2019, the WHO cautioned that amid increasing interest, digital health has been characterized by interventions being implemented without careful examination of the evidence base on their benefit and harms [7]. In the same year, the National Institute for Health and Care Excellence (NICE) published the Evidence Standards Framework for DHTs to guide clinicians, researchers, and policy makers in assessing whether the published literature evaluating these technologies provides the required level of evidence for their intervention to be considered for use in the UK health care setting [4].

There are several existing guidelines on evaluating the use of DHTs, including guidelines by policy makers such as the WHO, the United States’ Federal Drug Association, and National Health Service England [8-11] as well as frameworks developed by independent research groups [12,13]. However, the NICE framework is unique in explicitly suggesting a quality standard in relation to a technology’s functionality. Although the NICE framework was developed for DHTs used in a UK health care setting, the framework has the advantage of being research oriented rather than reliant on nation-specific commercial standards. This provides an opportunity for applying the framework to broader settings. First, the research-based focus may allow the framework to be used to evaluate the effectiveness of both consumer-driven and clinician-prescribed DHTs. Second, the framework may also be adapted to other health care systems by adjusting the requirement for development and testing in the United Kingdom to that of the DHT’s host country. Therefore, the NICE Evidence Framework may be used to guide assessment of and make comparisons between scientific literature regarding a variety of DHTs developed and applied internationally.

The NICE framework classifies apps by function and stratifies them into tiers (tiers 1, 2, 3a, or 3b). The tier framework corresponds with the evidence level required to support use of the technology; requirements are cumulative, becoming increasingly rigorous from tier 1 to 3 and divided into best practice and minimum standards. Stakeholders are encouraged to assess the evidence against these standards, which include, for example, whether the study measures important outcomes for users, whether the intervention works independently of health care professionals’ input, and the extent to which the intervention guides diagnosis, management, and treatment of a disease.

To date, there has been no review exploring whether peer-reviewed scientific literature regarding DHTs meets these evidence requirements. We investigated this in the context of DHTs designed to support the self-management of type 2 diabetes, as it is the most common chronic condition targeted by self-management DHTs [6].

Objectives

The objectives of this review are (1) to systematically identify peer-reviewed publications on mobile device DHTs intended to support or encourage the self-management of type 2 diabetes mellitus (T2DM), (2) to use the NICE Evidence Standards Framework to allocate each DHT to the appropriate intervention tier based on their described technology and function, and (3) to examine the extent to which the evidence reported for the identified DHTs meets the NICE framework level of evidence required according to its tier.


Review Design

We performed a scoping review [14] to understand the literature to date and explore the application of research methodology in relation to the NICE evidence standards. The review is reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement [15]. 

Data Sources

A total of 5 databases (MEDLINE, Embase, PsycINFO, CINAHL, and Cochrane Database of Systematic Reviews) were searched for systematic reviews published between January 2000 and August 2019 that evaluated mobile device DHT interventions for T2DM. Our database choice and search strategy were developed through consultation with a medical information specialist to identify the most relevant sources for peer-reviewed medical and clinical research studies. An example search strategy is provided in Multimedia Appendix 1.

Screening for Systematic Reviews

Two reviewers (JF and LA) independently screened all citations for systematic reviews by title and abstract and excluded those that clearly did not meet the eligibility criteria. Decisions were then unblinded, and any conflicting decisions were arbitrated by a third reviewer (AF). Full-text articles for all included citations were then screened against the inclusion criteria by 2 reviewers (JF and LA). 

Reviews were eligible if they included primary studies evaluating mobile apps designed to support adults with the self-management of diabetes mellitus. Reviews were excluded if they included studies in which the study population included people with type 1 diabetes, an undifferentiated mix of people with type 1 diabetes or type 2 diabetes, gestational diabetes, childhood diabetes or prediabetes, or focused on diagnosing diabetes (due to our focus on assessing DHTs designed to support self-management). Reviews that focused exclusively on telemedicine or telehealth interventions were also excluded, owing to our focus on technologies that support self-management and therefore require some degree of functionality independent of a clinician.

Screening for Primary Studies and Technologies

Relevant primary studies were then identified from eligible systematic reviews. The eligible reviews were equally divided between the 4 reviewers (JF, LA, HC, and AF) who then screened the title and abstract of each primary study included in each review. When a primary study was excluded, the study was double screened by a second reviewer, and in the instance of any conflict, a third reviewer arbitrated (LA or AF). Primary studies included at this stage were then divided between the 4 reviewers who reviewed the full text of each study for eligibility. Furthermore, when a study was excluded, the study was double screened by a second reviewer, and any conflict was arbitrated by a third reviewer (LA or AF).

Primary studies were eligible for inclusion if they met the following inclusion criteria:

  1. Population: adults with a diagnosis of T2DM.
  2. Intervention: a mobile device–delivered DHT designed to support the self-management of T2DM, which provides support independent of a clinician.

Data Extraction

Data were extracted from the included primary studies by 4 reviewers (JF, LA, HC, and AF). We designed a custom data extraction form using the evidence for effectiveness tables from the NICE framework [4] and additional guidance in the framework; an explanation of this approach can be found in Multimedia Appendix 2.

We extracted the following items from primary studies: (1) DHT investigated, (2) year of study, (3) study nation, (4) study design, (5) study setting, (6) outcomes of interest, (7) study duration and follow-up period, (8) sample size, (9) recruitment setting, (10) comparator group, (11) improvement in outcome with intervention, (12) justification of sample size, (13) statistical methods, and (14) follow-up rate. For tier 3a studies, we also extracted the following item: (15) description of and reference to a behavior change technique. Where more than one article that investigated the same DHT intervention was identified, data were extracted separately for each article.

Assigning Technologies and Intervention Tier

Descriptions of each technology were extracted from the primary studies, and we assigned each app a tier according to the NICE framework, as described in Multimedia Appendix 2. Where an app had more than one function, the function with the highest applicable tier was considered when assigning an overall tier. Tier 3b was considered as a higher tier to 3a owing to its more rigorous evidence requirements, as detailed in Multimedia Appendix 2.

Assessment of Evidence According to Tier

We used the NICE framework to evaluate each DHT against evidence levels, referring to evidence in the primary studies for each DHT, as described in Multimedia Appendix 2. We assessed each technology against its highest relevant tier to determine whether the DHT met the framework’s minimum and best practice evidence requirements. Where a technology was reported in more than one primary study, we analyzed each primary study separately against the framework and selected the strongest supporting evidence for the technology reported across the primary studies.

We also compared the NICE evidence standards outcome for a DHT against the income status of the study nation (as defined by the World Bank [16]). This was done to explore whether the NICE framework could be applied to DHTs designed for a different health care structure and system outside of the United Kingdom; a need for more empirical approaches to assess DHTs in low- and middle-income countries has been highlighted in recent literature [17,18].

Tier 3a guidance requires evidence of a referenced behavioral change technique (BCT) in the development or use of a technology that encourages behavioral change. For the purposes of this review and evidence assessment, we took a pragmatic decision to exclude this requirement in our overall decision on whether a tier 3a technology met the evidence requirements, accounting for the fact that our search methods may not have identified all relevant development studies reporting on a technology’s design.

In addition, the framework defines data quality as the presence of “statistical considerations such as sample size and statistical testing.” A pragmatic decision was made that statistical testing of some degree was needed as the minimum evidence requirement for all studies. However, the framework accommodates observational and quasi-experimental study designs, where it is impractical to statistically justify the sample size. Therefore, when making an assessment of evidence for studies of these designs, a statistical justification of sample size was not needed to meet minimum standards (but was required for experimental studies or randomized controlled trials [RCTs]).


Screening for Systematic Reviews

The initial database search returned 715 citations. After removal of duplicates, 709 citations were screened by title and abstract. We identified 68 relevant systematic reviews for which we screened the full-text articles. Of these, 45 reviews were included (Figure 1).

Figure 1. Flow diagram showing the inclusion and exclusion of systematic reviews and primary studies to yield eligible technologies.
View this figure

Screening for Primary Studies and Technologies

From these 45 reviews, we identified 145 relevant primary studies and screened their full-text articles. Of these, 61 primary studies met the inclusion criteria described above. We subsequently excluded 2 studies because there was insufficient information describing their technology to allocate a tier. The remaining 59 studies described 39 unique technologies and were included for data extraction (Figure 1).

The characteristics of the 59 included studies are presented in Multimedia Appendix 3 [19-77]. The publication year of the included studies ranged from 2007 to 2017. Of the included 59 studies, 36 (61%) were RCTs (of which 7 were identified as feasibility or pilot studies) and 23 (39%) were observational cohort studies (of which 19 were identified as feasibility or pilot studies). Qualitative data were reported alongside 6 RCTs and 13 observational cohort studies. The study nation varied, with 23 studies conducted in the United States, 6 in Norway, 4 in Korea, 3 studies each in Canada, the United Kingdom, and Saudi Arabia, 2 studies each in the Netherlands, Japan, Iran, and India, and 1 study each in Singapore, Mexico, Finland, Iraq, Bangladesh, the Democratic Republic of Congo, and China. Of the 39 technologies included for data analysis, 17 (44%) were mobile apps, 2 (5%) were personal digital assistant apps, and 20 (51%) were automated SMS.

Assigning Technologies to an Intervention Tier

All DHTs identified and included in this review were classified as tier 3 technologies. Descriptions of the technologies and their assigned subtiers are presented in Table 1 for tier 3a and Table 2 for tier 3b.

Of the 39 technologies, 23 (59%) were assigned to tier 3a. Tier 3a describes DHTs used for preventing and managing diseases and is divided into preventative behavior change and self-manage. Of these 23 technologies, 6 were apps and 17 were SMS based. Of the tier 3a technologies, 12 were classified as preventative behavior change only, 3 were classified as self-manage only, and 8 had both 3a preventative behavior change and self-manage characteristics.

We assigned 16 (41%) of the 39 technologies to tier 3b. Tier 3b describes technologies used as tools for treatment, diagnosis, and management decisions and is divided into treat, active monitoring, calculate, and diagnose. Of these 16 technologies, 13 were apps and 3 were SMS based. Of the tier 3b technologies, 7 were active monitoring only, 3 were treat and active monitoring, 1 was treat and calculate, 1 was active monitoring and calculate, and 4 had all 3 of the 3b treat, active monitoring, and calculate characteristics.

Table 1. Tier 3a digital health technologies: descriptions and subtier allocation (N=23).
Digital health technology and descriptionSelf-managePBCa
Tier 3a app technologies

Diabetes Pilot [19-22]PDAb app: patient inputs health data, displayed graphically, optionally sent to HCPcdN/Ae

Few Touch App (FTA) [23-28]Mobile app: patient inputs health data, displayed graphically. Features: personal goal setting, general diabetes informationN/A

Unnamed (Sevick) [29]PDA app: patient inputs diet data, feedback on nutritional composition. Features: calorie target goal set by HCP, no data accessN/A

Monica [30]Mobile app: patient inputs data, displayed graphically, automatic informational and/or behavioral skills feedback

iDecide [31] Mobile app: patient inputs HbA1cf at start. Features: education, personalized complication risk, medication review, personalized goals

Diabetes 101 [32]Mobile app: no data input by patient. Features: 5 educational T2DMg self-management videos with quiz. Automatic self-care remindersN/A
Tier 3a SMS technologies

NICHE system [33]SMS: patients upload BGh and pedometer data onto web server: SMS summary to patient

Unnamed (Shetty) [34]SMS: unidirectional nonpersonalized SMS (every third day), informing and reinforcing health behaviorsN/A

Diabetech [35]SMS: BG automatically uploaded to server: automated SMS summary, suggestions to contact HCP where relevant

Unnamed (Goodarzi) [36]SMS: unidirectional nonpersonalized SMS (weekly) informing and reinforcing health behaviorsN/A

Real-Time Medication Monitoring [37,38]SMS: unidirectional SMS reminder if oral antidiabetic medication not taken (linked to electronic medication dispenser)N/A

Care4Life [39,40]SMS: unidirectional nonpersonalized daily SMS, informing and reinforcing health behaviors. Two-way messaging to HCP for feedback

SMS-DMCare [41]SMS: SMS medication reminders, unidirectional informational texts weekly about health behaviors and appointment reminders

MEssaging for Diabetes (MED) [42]SMS: unidirectional informational SMS on medications and bidaily SMS requesting adherence response (yes or no). HCP call every 2 weeksN/A

TExT-MED [43,44]SMS: unidirectional nonpersonalized bidaily SMS informing and reinforcing health behaviorsN/A

Unnamed (Haddad) [45]SMS: unidirectional nonpersonalized weekly SMS informing and reinforcing health behaviorsN/A

Unnamed (Argay) [46]SMS: unidirectional medication reminder SMS (up to 3 times daily)N/A

Unnamed (Bin Abbas) [47]SMS: unidirectional nonpersonalized daily SMS informing and reinforcing health behaviorsN/A

Unnamed (Islam) [48]SMS: unidirectional nonpersonalized SMS every other day informing and reinforcing medication compliancesN/A

Text to Move [77]SMS: patient self-uploads pedometer data: 2 unidirectional text messages daily based on step count and preset goals

Unnamed (Peimani) [49]SMS: unidirectional SMS informing and reinforcing health behaviors. Personalized to individual at start of studyN/A

Unnamed (Fang) [50]SMS: unidirectional nonpersonalized SMS informing health behaviorsN/A

Dulcedigital [51]SMS: unidirectional nonpersonalized SMS 2-3 daily reinforcing health behavior. Patient inputs BG in SMS which alerts HCP if abnormal

aPBC: preventative behavior change.

bPDA: personal digital assistant.

cHCP: health care professional.

dDigital health technology falls within the subtier.

eN/A: not applicable.

fHbA1c: glycated hemoglobin.

gT2DM: type 2 diabetes mellitus.

hBG: blood glucose.

Table 2. Tier 3b digital health technologies: descriptions and subtier allocation (N=16).
Digital health technology and descriptionTreatActive monitoringCalculate
Tier 3b app technologies

BPa telemanagement [52]Mobile app: patient BP automatically uploaded. HCPb accesses all data. Alert to patient and HCP if critical. Automatic BP reminders to patientN/AcdN/A

WellDoc [53-58]Mobile app: patient BG automatically uploaded, medication dose and diet self-inputted: automated personalized feedback on medication dose and behavior. HCP accesses all data

t+ Diabetes [59-61]Mobile app: patient BG automatically uploaded and insulin dose self-inputted: displayed graphically, decision aids for self-titration. HCP accesses all data and messages through the appN/AN/A

Mobil Diab [62]Mobile app: patient BG automatically uploaded, displayed graphically. HCP accesses all data and sends feedback through the appN/AN/A

Health Coach App [63,64]Mobile app: patient self-inputs health data: displayed graphically. Goal setting function. HCP accesses all data, individualized feedback, and two-way communication through the appN/AN/A

Dialbetics app [65,66]Mobile app: patient self-inputs BG data: behavioral feedback and alerts if abnormal. HCP accesses all data; abnormal readings flagged. Features: later version includes dietary feedbackN/A

SANAD [67]Mobile app: BGe automatically uploaded. Features: social networking module and CBTf module. HCP accesses all data; sends feedback through appN/A

SAED system [68]Mobile app: BG automatically uploaded. Features: weekly educational message. HCP accesses all data; two-way communication through the appN/AN/A

Diabetes Pal [69]Mobile app: patient self-inputs BG: app suggests insulin dose (within the preset range). Features: educational information. Research staff access all data; flag to HCPN/A

CollaboRhythm [70]Mobile app: patient self-inputs medication and BG displayed graphically. HCP accesses all data and suggests insulin correction; two-way communication through the app

PSDCS [71]Mobile app: BG automatically uploaded, diet and exercise self-inputted—feedback and suggested insulin changes based on algorithm. Features: automated daily recommendations for calorie intake and exerciseN/A

Brew app [72]Mobile app: patient self-inputs health data. Features: daily SMS reminders, educational information. HCP accesses summary of data and sends alerts for BG or missed appointmentsN/AN/A

Gather Health [73]Mobile app: patient self-inputs BG: displayed graphically. Features: daily reminders and self-care advice. HPC accesses all data; two-way communication through the appN/A
Tier 3b SMS technologies

UCDC system [74]SMS: patient BG automatically sent to server, automated summary SMS with behavioral suggestions. Patient sends BP and exercise via SMS. Informational SMS trice daily. HCP accesses all dataN/AN/A

Unnamed SMS (Kim) [75]SMS: patient BG automatically sent to server, automated SMS suggestions to adjust insulin based on an algorithm. If hypoglycemic, emergency SMS sent to patient and caregiver

CDSS u-health care [76]SMS: Patients BG automatically uploaded to server, automated daily SMS summaries, suggestions to adjust insulin based on algorithm, weekly and monthly summaries

aBP: blood pressure.

bHCP: health care professional.

cN/A: not applicable.

dDigital health technology falls within the subtier.

eBG: blood glucose.

fCBT: cognitive behavioral therapy.

Assessment of Evidence According to Tier

The assessment of evidence level according to the assigned tier is presented in Table S1 [22,28-36,38,39,41-43,45-51,77] in Multimedia Appendix 4 for tier 3a technologies and in Table S2 [52,54,61,62,64,65,67-76,78] in Multimedia Appendix 4 for tier 3b technologies. Across all 39 technologies, 11 demonstrated best practice standards for the evidence level assigned, 3 technologies demonstrated minimum standards, and 25 did not report methods or findings that met minimum standards.

Tier 3a Technologies

Of the 23 tier 3a technologies, 7 met the best practice standards, 3 met the minimum evidence standards, and 13 did not report methods or findings reaching minimum standards. Of the 13 technologies that did not provide evidence for minimum standards, there were several common reasons for falling short of the minimum standard. First, 7 technologies did not provide statistical justification of sample size where the study design was appropriate, with this being the only reason for not meeting minimum standards in all 7 technologies. Second, 6 technologies did not provide comparative data, with this being the only reason for not meeting the minimum standards in the 2 technologies. Finally, 3 technologies did not conduct any statistical testing on the data set.

For the 3 tier 3a technologies that met the minimum evidence standards, there were 2 common reasons why these technologies did not meet the best practice standards. First, 2 technologies showed no improvement in condition-relevant outcomes, with this being the only reason for both technologies not meeting the best practice. Second, 1 technology’s comparator group did not represent usual care, with this being the only reason for not meeting the best practice.

Tier 3b Technologies

Of the 16 tier 3b technologies, 4 met best practice standards, none met only minimum evidence standards, and 12 did not report methods or findings reaching minimum standards. Of the 12 technologies that did not provide evidence for minimum standards, there were several common reasons for falling short of the minimum standard. First, 3 technologies used a single-arm cohort study design that lacked a comparator group and failed to meet the requirement of design being quasi-experimental or higher, with inappropriate study design being the only reason for not meeting minimum standards in all 3 technologies. Second, 7 technologies had no statistical justification of sample size where the study design was appropriate, with this being the only reason for 5 of these technologies. Third, there were 2 technologies that did not conduct any statistical testing on the data set. Finally, 2 technologies had a follow-up period of less than 3 months, which is the accepted minimum clinically relevant follow-up period for type 2 diabetes.

Evidence Standard by Host Country

Table 3 shows the DHTs arranged according to the income status (as defined by the World Bank [16]) of the study nation and the outcome of the DHT’s NICE evidence assessment. There were considerably more DHTs from high-income economies (n=30) than upper middle-income (n=5), lower middle-income (n=3), or low-income (n=1) economies. In addition, there was no evidence of studies from high-income nations being more or less successful in meeting NICE evidence standards than lower-income nations: only 9 out of 30 DHTs investigated in high-income economies met either minimum or best practice standards, compared with 3 out of 5 DHTs investigated in upper middle-income economies, 2 out of 3 DHTs investigated in low- and middle-income economies, and 0 out of 1 DHTs investigated in low-income economies.

Table 3. Digital health technologies arranged by World Bank income status of host country and the digital health technology evidence outcome (N=39).
CountryDHTaNICEb evidence level met
Low-income economies

Democratic Republic of CongoMobil DiabNo
Lower middle-income economies

BangladeshUnnamed (Islam)Best practice

IndiaUnnamed (Shetty)No

IndiaGather HealthBest practice
Upper middle-income economies

ChinaUnnamed (Fang)Minimum

IranUnnamed (Haddad)No

IranUnnamed (Goodarzi)Best practice

IraqUnnamed (Peimani)Best practice

MexicoBrew appNo
High-income economies

CanadaBP telemanagementNo

CanadaHealth Coach AppNo

FinlandMonicaNo

HungaryUnnamed (Argay)No

JapanDialbetics appBest practice

KoreaCDSS-based u-health careNo

KoreaPSDCSNo

KoreaUCDC systemNo

KoreaUnnamed (Kim)Best practice

NetherlandsReal-Time Medication MonitoringNo

NorwayFew Touch ApplicationMinimum

Saudi ArabiaSANADNo

Saudi ArabiaSAEDNo

Saudi ArabiaUnnamed (Bin Abbas)No

SingaporeDiabetes PalNo

United Kingdomt+DiabetesNo

United StatesCare4lifeNo

United StatesCollaboRhythmNo

United StatesDiabetechNo

United StatesDulcedigitalNo

United StatesDiabetes 101No

United StatesMEDNo

United StatesNICHE systemNo

United StatesSMS-DMCareNo

United StatesUnnamed (Sevick)Minimum

United StatesDiabetes PilotBest practice

United StatesiDecideBest practice

United StatesTExT-MEDBest practice

United StatesText to MoveBest practice

United StatesWellDocBest practice

aDHT: digital health technology.

bNICE: National Institute of Care Excellence.


Principal Findings

We aimed to evaluate whether peer-reviewed literature investigating the use of mobile device DHTs for the self-management of T2DM met the required evidence level set out in the NICE Evidence Standards Framework for DHTs. The framework aims to ensure that new technologies introduced to clinical health care settings are effective and offer economic value. We identified 39 mobile device DHTs designed to support self-management of T2DM in the scientific literature; these were a mix of app-based and SMS-based technologies. We found that all technologies fell into tier 3a or tier 3b (the highest tiers) of the NICE framework, with tier 3 interventions targeting disease management and requiring the most rigorous evidence. When assessing a technology using the NICE Evidence Standards Framework, we assessed all primary studies supporting a DHT individually against the framework and selected the strongest supporting evidence for the technology reported across the primary studies.

For more than half of the technologies identified, the underpinning literature did not meet the evidence standards to demonstrate effectiveness, as recommended by the NICE framework for the technology’s tier. Of the 39 technologies identified, only 16 met minimum or best evidence standards, with 23 not meeting the minimum requirements. The most common reasons for not meeting the NICE standards included a lack of an appropriate comparator group that reflected usual care, no statistical justification of sample size, a lack of measurable improvement in condition-related outcomes, and no statistical data analysis. Given the high proportion of RCTs among the identified studies (36/59, 61%), it was surprising that such a large number did not meet the minimum evidence standards due to these reasons. We found that the evidence framework could easily be applied to a variety of study nations and that studies from a range of economic settings were able to meet evidence standards for the DHT. From the results of this study, we suggest that the application of DHT evidence standards are globally relevant.

Using the NICE Evidence Standards Framework to Evaluate Evidence

We encountered several challenges in interpreting and using the NICE framework. First, we found that for diabetes, there was ambiguity in distinguishing technology for healthy living and technology for disease management. The same technology that targeted diet and exercise could be considered tier 2 for people without diabetes as a healthy living app but tier 3 for those with T2DM as a disease management app. There are several terms used in the NICE framework that can be ambiguous in their application and may require greater clarity, including the phrases high quality data and clinically relevant follow-up period. The framework does not include guidance as to how either of these points should be assessed.

As the NICE Evidence Framework was designed in the United Kingdom, the standards reference the UK health care setting when assessing the development and effectiveness of a technology. We found that adaptation of the NICE framework to assess a DHT in its host country, rather than specifically in the United Kingdom, allowed the analysis and comparison of DHTs in an international context. We also noted that the UK-specific requirement may restrict UK policy makers, commissioners, and clinicians from adopting and implementing DHTs that have been rigorously evaluated in another health care setting and do not require substantial adaptation. This could be considered overly restrictive for DHTs that target self-management and may not need integration with a health care system.

Finally, we observed a potential mismatch between the level of risk associated with an intervention and the level of evidence required according to the intervention’s associated tier. For example, Real-Time Medication Monitoring [37,38], which would be categorized under tier 3a (preventative behavior change due to explicit suggestions by the DHT to the patient for actions or behavior change) might be considered a low-risk technology, involving automatic SMS reminders to take medication when a patient’s pill box remains unopened. However, Health Coach App [63,64], also classified under tier 3a (self-management for symptoms, health or disease related data, or medication tracking over time) might be considered as having higher risk, tracking multiple health behaviors, holding sensitive data, and facilitating two-way messaging. Despite this difference in the level of risk, both technologies fall under the same tier and require the same standard of supporting evidence. The evidence framework also stipulates that any technology where there is automatic transfer of data (regardless of type) to a health care professional should be categorized as tier 3b rather than tier 3a under active monitoring, requiring more rigorous evidence for clinical input without any apparent additional risk. Therefore, tier levels may need to be adjusted to reflect clinical risk rather than function alone.

Strengths and Limitations

Although this is a scoping review, we took a systematic approach to identify peer-reviewed articles, adding rigor to our methods. We included reviews of all study design types, including experimental, observational, and qualitative study designs. However, while we identified several experimental and observational studies, this approach may not have captured all developmental studies and recently published studies that are less likely to be included in systematic reviews. However, we would have expected developmental studies to be cited in subsequent experimental and observational clinical studies, and we hand-searched full-text articles for such studies. We adapted our evidence assessments where appropriate (eg, excluding requirements for BCT evidence in tier 3a).

We identified technologies that have been investigated and published in the scientific literature and did not review app catalogs or commercial publications for relevant technologies. We feel this approach was appropriate, as we did not have the resources to obtain and evaluate these sources and assess the extent to which they meet evidence standards, as described in the NICE framework. In addition, although the NICE framework was developed for DHTs used in a clinical setting, we did not differentiate between commercial and commissioned DHTs in this study. However, we encountered no challenges in applying the tier 3 evidence requirements to technologies scientifically evaluated either by clinical or commercial teams; indeed, the evidence framework could be used to design studies to evaluate the use of commercial apps within a clinical setting. Although we assessed the income status of the study nation to explore the applicability of the framework in a variety of health care settings, this did not take into account the scenario where a technology was developed in a high-income country but delivered in a low-income population [31,42-44,51,63,64]. Although beyond the scope of this review, future work could explore the effect of sociodemographic factors of the target population (such as economic status, access to health care, and technology literacy) in using the framework to evaluate the effectiveness of DHTs.

Due to potential ambiguity and subjectivity applying the NICE framework, we acknowledge that our interpretation will have affected decisions around classification and evidence evaluation and consequently the number of DHTs meeting evidence standards. We have highlighted that greater clarity of key terms in the framework would be valuable. We also acknowledge that the scope of our analysis was limited to the evidence requirements in the NICE framework, but other considerations for study quality (ie, prospective registration, retention rate) and intervention effect (ie, technology literacy, impact on behavior) are interesting and relevant in evaluating the effectiveness of DHTs.

We identified several evidence-level criteria as described by NICE that studies of DHTs commonly failed to meet. This offers a useful resource for digital health researchers and developers who may use this information in designing and reporting DHT research in the future. This might aid in the translation of research into clinical care by ensuring that the required information is measured and reported. This in turn will enable commissioners, policy makers, and clinicians to readily assess whether a technology is suitable for implementation in the UK health care setting.

Comparison With Previous Work

Previous studies have identified a lack of evidence of an effect in apps for diabetes. Recently, Veazie et al [79] identified 15 studies evaluating 11 apps for the self-management of diabetes and found that only 5 technologies were supported by evidence showing significant clinical improvement with use. Our study supported this finding as well as identifying many more apps and several other aspects of evidence that could be improved. In addition, a previous study highlighted challenges in applying the NICE Evidence Framework tiers in classifying DHTs. Nwe et al [80] used the NICE framework to classify 76 apps from the National Health Service (NHS) app library into their relevant technology tier and assessed the classification agreement between 2 mobile health (mHealth) researchers. They found a disagreement on the classified tier in 45% (34/76) of technologies [80]. Our study complements the author’s recommendation that greater clarity in the framework may be needed to improve the consistency of its application. To our knowledge, this is the first study to assess the evidence supporting DHTs against the NICE Evidence Framework. Previous reviews evaluating DHTs in other clinical settings, such as technologies for stroke rehabilitation and virtual reality tools in pediatric care, have highlighted the need for a set of recognized standards in the field with specific mention to the NICE framework [81,82]. Therefore, it would be of interest to assess and compare the application of the NICE framework with DHTs in other health care settings in addition to chronic disease management. Given that the NICE framework is relatively new, it would be valuable to conduct similar reviews in the future to assess the potential impact of the framework on rigor and quality of studies over time.

Conclusions

This review evaluated a defined group of mobile-delivered DHTs designed for use by people with T2DM, using the NICE Evidence Standards Framework for DHTs. Over half of the identified DHTs did not meet the minimum evidence standards required for their intervention tier, as defined by the NICE Evidence Standards Framework. This may pose a major barrier to the translation of mHealth interventions into the UK health care setting. However, we have highlighted the most common areas in which DHT evaluations do not meet the standards set out by NICE, and this provides an opportunity for researchers and DHT developers to address these points when designing and reporting DHTs in the future. In addition, we identified the potential scope for development of the NICE framework so that the evidence tiers correlate more closely with the associated risk of an intervention. Above all, commissioners, clinicians, and patients need to have confidence in the safety of DHTs for these to be implemented into everyday chronic disease management, and increased risk should be underpinned by the most rigorous scientific research.

Acknowledgments

This review did not receive any funding. This research was supported by the National Institute for Health Research (NIHR) Oxford Biomedical Research Centre (BRC). The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health. AF is an NIHR Senior Investigator, and both AF and LA receive support from the NIHR Oxford Biomedical Research Centre.

Conflicts of Interest

AF is Program Director of the NIHR Health Technology Assessment Programme.

Multimedia Appendix 1

Example of full search strategy for the Medline database.

PDF File (Adobe PDF File), 734 KB

Multimedia Appendix 2

An explanation of the classification strategy for digital health technologies using the technology tier and evidence level in the National Institute of Health and Care Excellence Framework.

PDF File (Adobe PDF File), 724 KB

Multimedia Appendix 3

Characteristics of primary studies included for data extraction.

PDF File (Adobe PDF File), 589 KB

Multimedia Appendix 4

Overall technology assessments against the National Institute for Health and Care Excellence Evidence Framework.

PDF File (Adobe PDF File), 608 KB

References

  1. Digitally-enabled primary and outpatient care will go mainstream across the NHS Internet. NHS Long Term Plan. -.   URL: https:/​/www.​longtermplan.nhs.uk/​online-version/​chapter-1-a-new-service-model-for-the-21st-century/​4-digitally-enabled-primary-and-outpatient-care-will-go-mainstream-across-the-nhs/​ [accessed 2021-01-16]
  2. Health intervention and technology assessment in support of universal health coverage internet. World Health Assembly Resolution. 2014.   URL: https://www.who.int/medical_devices/assessment/resolutionsearo_searc66r4.pdf [accessed 2021-01-16]
  3. What is a health technology? World Health Organization. 2015.   URL: https://www.who.int/health-technology-assessment/about/healthtechnology/en/ [accessed 2021-01-16]
  4. Nice evidence standards framework for digital health technologies. NICE. 2019.   URL: https:/​/www.​nice.org.uk/​Media/​Default/​About/​what-we-do/​our-programmes/​evidence-standards-framework/​digital-evidence-standards-framework.​pdf [accessed 2021-01-16]
  5. Rho MJ, Kim HS, Chung K, Choi IY. Factors influencing the acceptance of telemedicine for diabetes management. Cluster Comput 2014 Mar 12;18(1):321-331. [CrossRef]
  6. Martínez-Pérez B, de la Torre-Díez I, López-Coronado M. Mobile health applications for the most prevalent conditions by the World Health Organization: review and analysis. J Med Internet Res 2013 Jun 14;15(6):120 [FREE Full text] [CrossRef] [Medline]
  7. WHO. WHO guideline Recommendations on Digital Interventions for Health System Strengthening. WHO Guidelines Approved by the Guidelines Review Committee 2019 [FREE Full text] [Medline]
  8. WHO monitoring and evaluating digital health interventions: a practical guide to conducting research and assessment. WHO. 2016.   URL: https://www.who.int/publications/i/item/9789241511766 [accessed 2021-01-16]
  9. BETA - NHS digital, data and technology standards framework. NHS digital. 2020.   URL: https:/​/digital.​nhs.uk/​about-nhs-digital/​our-work/​nhs-digital-data-and-technology-standards/​framework [accessed 2021-01-16]
  10. Multiple function device products: policyconsiderations guidance for industryfooddrug administration staff internet. FDA. 2018.   URL: https://www.fda.gov/media/112671/download [accessed 2021-01-16]
  11. Policy for device software functionsmobile medical applications guidance for industryfooddrug administration staff preface public comment internet. FDA. 2019.   URL: https://www.fda.gov/media/80958/download [accessed 2021-01-16]
  12. Wilhide Iii CC, Peeples MM, Anthony Kouyaté RC. Evidence-based mhealth chronic disease mobile app intervention design: development of a framework. JMIR Res Protoc 2016 Feb 16;5(1):25 [FREE Full text] [CrossRef] [Medline]
  13. Stoyanov SR, Hides L, Kavanagh DJ, Zelenko O, Tjondronegoro D, Mani M. Mobile app rating scale: a new tool for assessing the quality of health mobile apps. JMIR Mhealth Uhealth 2015 Mar 11;3(1):27 [FREE Full text] [CrossRef] [Medline]
  14. Munn Z, Peters MDJ, Stern C, Tufanaru C, McArthur A, Aromataris E. Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Med Res Methodol 2018 Nov 19;18(1):143 [FREE Full text] [CrossRef] [Medline]
  15. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Br Med J 2009 Jul 21;339:2535 [FREE Full text] [CrossRef] [Medline]
  16. World bank country and lending groups. World Bank Data Help Desk.   URL: https:/​/datahelpdesk.​worldbank.org/​knowledgebase/​articles/​906519-world-bank-country-and-lending-groups [accessed 2021-01-16]
  17. Long L, Pariyo G, Kallander K. Digital technologies for health workforce development in low- and middle-income countries: a scoping review. Glob Health Sci Pract 2018 Oct 10;6(Suppl 1):S41-S48 [FREE Full text] [CrossRef] [Medline]
  18. Holeman I, Cookson TP, Pagliari C. Digital technology for health sector governance in low and middle income countries: a scoping review. J Glob Health 2016 Dec;6(2):020408 [FREE Full text] [CrossRef] [Medline]
  19. Forjuoh SN, Reis MD, Couchman GR, Ory MG, Mason S, Molonket-Lanning S. Incorporating PDA use in diabetes self-care: a central Texas Primary Care Research Network (CenTexNet) study. J Am Board Fam Med 2007;20(4):375-384 [FREE Full text] [CrossRef] [Medline]
  20. Forjuoh SN, Reis MD, Couchman GR, Ory MG. Improving diabetes self-care with a PDA in ambulatory care. Telemed J E Health 2008 Apr;14(3):273-279. [CrossRef] [Medline]
  21. Vuong AM, Huber JC, Bolin JN, Ory MG, Moudouni DM, Helduser J, et al. Factors affecting acceptability and usability of technological approaches to diabetes self-management: a case study. Diabetes Technol Ther 2012 Dec;14(12):1178-1182 [FREE Full text] [CrossRef] [Medline]
  22. Forjuoh SN, Bolin JN, Huber JC, Vuong AM, Adepoju OE, Helduser JW, et al. Behavioral and technological interventions targeting glycemic control in a racially/ethnically diverse population: a randomized controlled trial. BMC Public Health 2014 Jan 23;14:71 [FREE Full text] [CrossRef] [Medline]
  23. Arsand E, Tatara N, Ostengen G, Hartvigsen G. Mobile phone-based self-management tools for type 2 diabetes: the few touch application. J Diabetes Sci Technol 2010 Mar 01;4(2):328-336 [FREE Full text] [CrossRef] [Medline]
  24. Nes AAG, van Dulmen S, Eide E, Finset A, Kristjánsdóttir OB, Steen IS, et al. The development and feasibility of a web-based intervention with diaries and situational feedback via smartphone to support self-management in patients with diabetes type 2. Diabetes Res Clin Pract 2012 Sep;97(3):385-393. [CrossRef] [Medline]
  25. Tatara N, Arsand E, Skrøvseth SO, Hartvigsen G. Long-term engagement with a mobile self-management system for people with type 2 diabetes. JMIR Mhealth Uhealth 2013 Mar 27;1(1):1 [FREE Full text] [CrossRef] [Medline]
  26. Chomutare T, Tatara N, Årsand E, Hartvigsen G. Designing a diabetes mobile application with social network support. Stud Health Technol Inform 2013;188:58-64. [Medline]
  27. Torbjørnsen A, Jenum AK, Smastuen MC, Arsand E, Holmen H, Wahl AK, et al. A low-intensity mobile health intervention with and without health counseling for persons with type 2 diabetes, part 1: baseline and short-term results from a randomized controlled trial in the norwegian part of renewing health. JMIR Mhealth Uhealth 2014 Dec 11;2(4):52 [FREE Full text] [CrossRef] [Medline]
  28. Holmen H, Torbjornsen A, Wahl AK, Jenum AK, Smastuen MC, Arsand E, et al. A mobile health intervention for self-management and lifestyle change for persons with type 2 diabetes, part 2: one-year results from the norwegian randomized controlled trial renewing health. JMIR Mhealth Uhealth 2014 Dec 11;2(4):57 [FREE Full text] [CrossRef] [Medline]
  29. Sevick MA, Korytkowski M, Stone RA, Piraino B, Ren D, Sereika S, et al. Biophysiologic outcomes of the enhancing adherence in type 2 diabetes (ENHANCE) trial. J Acad Nutr Diet 2012 Aug;112(8):1147-1157 [FREE Full text] [CrossRef] [Medline]
  30. Orsama A, Lähteenmäki J, Harno K, Kulju M, Wintergerst E, Schachner H, et al. Active assistance technology reduces glycosylated hemoglobin and weight in individuals with type 2 diabetes: results of a theory-based randomized trial. Diabetes Technol Ther 2013 Aug;15(8):662-669. [CrossRef] [Medline]
  31. Heisler M, Choi H, Palmisano G, Mase R, Richardson C, Fagerlin A, et al. Comparison of community health worker-led diabetes medication decision-making support for low-income Latino and African American adults with diabetes using e-health tools versus print materials: a randomized, controlled trial. Ann Intern Med 2014 Nov 18;161(10 Suppl):13-22 [FREE Full text] [CrossRef] [Medline]
  32. Wood FG, Alley E, Baer S, Johnson R. Interactive multimedia tailored to improve diabetes self-management. Nurs Clin North Am 2015 Sep;50(3):565-576. [CrossRef] [Medline]
  33. Faridi Z, Liberti L, Shuval K, Northrup V, Ali A, Katz DL. Evaluating the impact of mobile telephone technology on type 2 diabetic patients' self-management: the NICHE pilot study. J Eval Clin Pract 2008 Jun;14(3):465-469. [CrossRef] [Medline]
  34. Shetty AS, Chamukuttan S, Nanditha A, Raj RKC, Ramachandran A. Reinforcement of adherence to prescription recommendations in Asian Indian diabetes patients using short message service (SMS)-a pilot study. J Assoc Physicians India 2011 Nov;59:711-714. [Medline]
  35. Roblin DW. The potential of cellular technology to mediate social networks for support of chronic disease self-management. J Health Commun 2011;16 Suppl 1:59-76. [CrossRef] [Medline]
  36. Goodarzi M, Ebrahimzadeh I, Rabi A, Saedipoor B, Jafarabadi MA. Impact of distance education via mobile phone text messaging on knowledge, attitude, practice and self efficacy of patients with type 2 diabetes mellitus in Iran. J Diabetes Metab Disord 2012 Aug 31;11(1):10 [FREE Full text] [CrossRef] [Medline]
  37. Vervloet M, van Dijk L, Santen-Reestman J, van Vlijmen B, van Wingerden P, Bouvy ML, et al. SMS reminders improve adherence to oral medication in type 2 diabetes patients who are real time electronically monitored. Int J Med Inform 2012 Sep;81(9):594-604. [CrossRef] [Medline]
  38. Vervloet M, van Dijk L, de Bakker DH, Souverein PC, Santen-Reestman J, van Vlijmen B, et al. Short- and long-term effects of real-time medication monitoring with short message service (SMS) reminders for missed doses on the refill adherence of people with Type 2 diabetes: evidence from a randomized controlled trial. Diabet Med 2014 Jul;31(7):821-828. [CrossRef] [Medline]
  39. Capozza K, Woolsey S, Georgsson M, Black J, Bello N, Lence C, et al. Going mobile with diabetes support: a randomized study of a text message-based personalized behavioral intervention for type 2 diabetes self-care. Diabetes Spectr 2015 May;28(2):83-91 [FREE Full text] [CrossRef] [Medline]
  40. Georgsson M, Staggers N. An evaluation of patients' experienced usability of a diabetes mHealth system using a multi-method approach. J Biomed Inform 2016 Feb;59:115-129 [FREE Full text] [CrossRef] [Medline]
  41. Nundy S, Dick JJ, Solomon MC, Peek ME. Developing a behavioral model for mobile phone-based diabetes interventions. Patient Educ Couns 2013 Jan;90(1):125-132 [FREE Full text] [CrossRef] [Medline]
  42. Osborn CY, Mulvaney SA. Development and feasibility of a text messaging and interactive voice response intervention for low-income, diverse adults with type 2 diabetes mellitus. J Diabetes Sci Technol 2013 May 01;7(3):612-622 [FREE Full text] [CrossRef] [Medline]
  43. Arora S, Peters AL, Burner E, Lam CN, Menchine M. Trial to examine text message-based mHealth in emergency department patients with diabetes (TExT-MED): a randomized controlled trial. Ann Emerg Med 2014 Jun;63(6):745-754. [CrossRef] [Medline]
  44. Burner ER, Menchine MD, Kubicek K, Robles M, Arora S. Perceptions of successful cues to action and opportunities to augment behavioral triggers in diabetes self-management: qualitative analysis of a mobile intervention for low-income Latinos with diabetes. J Med Internet Res 2014 Jan 29;16(1):25 [FREE Full text] [CrossRef] [Medline]
  45. Haddad NS, Istepanian R, Philip N, Khazaal FAK, Hamdan TA, Pickles T, et al. A feasibility study of mobile phone text messaging to support education and management of type 2 diabetes in Iraq. Diabetes Technol Ther 2014 Jul;16(7):454-459. [CrossRef] [Medline]
  46. Argay M, Mesko A, Zelko R, Hanko B. Therapy reminder message for Hungarian patients with type 2 diabetes. Acta Pol Pharm and Drug Res 2015;72(6):1293.
  47. Bin Abbas B, Al Fares A, Jabbari M, El Dali AC, Al Orifi F. Effect of mobile phone short text messages on glycemic control in type 2 diabetes. Int J Endocrinol Metab 2015 Jan;13(1):18791 [FREE Full text] [CrossRef] [Medline]
  48. Shariful Islam SM, Niessen LW, Ferrari U, Ali L, Seissler J, Lechner A. Effects of mobile phone sms to improve glycemic control among patients with type 2 diabetes in bangladesh: a prospective, parallel-group, randomized controlled trial. Diabetes Care 2015 Aug;38(8):112-113. [CrossRef] [Medline]
  49. Peimani M, Rambod C, Omidvar M, Larijani B, Ghodssi-Ghassemabadi R, Tootee A, et al. Effectiveness of short message service-based intervention (SMS) on self-care in type 2 diabetes: a feasibility study. Prim Care Diabetes 2016 Aug;10(4):251-258. [CrossRef] [Medline]
  50. Fang R, Deng X. Electronic messaging intervention for management of cardiovascular risk factors in type 2 diabetes mellitus: a randomised controlled trial. J Clin Nurs 2018 Feb;27(3-4):612-620. [CrossRef] [Medline]
  51. Fortmann AL, Gallo LC, Garcia MI, Taleb M, Euyoque JA, Clark T, et al. Dulce digital: an mHealth SMS-based intervention improves glycemic control in Hispanics with type 2 diabetes. Diabetes Care 2017 Oct;40(10):1349-1355 [FREE Full text] [CrossRef] [Medline]
  52. Logan AG, McIsaac WJ, Tisler A, Irvine MJ, Saunders A, Dunai A, et al. Mobile phone-based remote patient monitoring system for management of hypertension in diabetic patients. Am J Hypertens 2007 Sep;20(9):942-948. [CrossRef] [Medline]
  53. Quinn CC, Clough SS, Minor JM, Lender D, Okafor MC, Gruber-Baldini A. WellDoc mobile diabetes management randomized controlled trial: change in clinical and behavioral outcomes and patient and physician satisfaction. Diabetes Technol Ther 2008 Jun;10(3):160-168. [CrossRef] [Medline]
  54. Quinn CC, Shardell MD, Terrin ML, Barr EA, Ballew SH, Gruber-Baldini AL. Cluster-randomized trial of a mobile phone personalized behavioral intervention for blood glucose control. Diabetes Care 2011 Sep;34(9):1934-1942 [FREE Full text] [CrossRef] [Medline]
  55. Katz R, Mesfin T, Barr K. Lessons from a community-based mHealth diabetes self-management program:. J Health Commun 2012;17 Suppl 1:67-72. [CrossRef] [Medline]
  56. Quinn CC, Sareh PL, Shardell ML, Terrin ML, Barr EA, Gruber-Baldini AL. Mobile diabetes intervention for glycemic control: impact on physician prescribing. J Diabetes Sci Technol 2014 Mar;8(2):362-370 [FREE Full text] [CrossRef] [Medline]
  57. Quinn CC, Khokhar B, Weed K, Barr E, Gruber-Baldini AL. Older adult self-efficacy study of mobile phone diabetes management. Diabetes Technol Ther 2015 Jul;17(7):455-461 [FREE Full text] [CrossRef] [Medline]
  58. Quinn CC, Shardell MD, Terrin ML, Barr EA, Park D, Shaikh F, et al. Mobile diabetes intervention for glycemic control in 45- to 64-year-old persons with type 2 diabetes. J Appl Gerontol 2016 Feb;35(2):227-243. [CrossRef] [Medline]
  59. Turner J, Larsen M, Tarassenko L, Neil A, Farmer A. Implementation of telehealth support for patients with type 2 diabetes using insulin treatment: an exploratory study. Inform Prim Care 2009;17(1):47-53 [FREE Full text] [CrossRef] [Medline]
  60. Larsen ME, Turner J, Farmer A, Neil A, Tarassenko L. Telemedicine-supported insulin optimisation in primary care. J Telemed Telecare 2010;16(8):433-440. [CrossRef] [Medline]
  61. Nagrebetsky A, Larsen M, Craven A, Turner J, McRobert N, Murray E, et al. Stepwise self-titration of oral glucose-lowering medication using a mobile telephone-based telehealth platform in type 2 diabetes: a feasibility trial in primary care. J Diabetes Sci Technol 2013 Jan 01;7(1):123-134 [FREE Full text] [CrossRef] [Medline]
  62. Takenga C, Berndt R, Musongya O, Kitero J, Katoke R, Molo K, et al. An ICT-based diabetes management system tested for health care delivery in the African context. Int J Telemed Appl 2014;2014:437307 [FREE Full text] [CrossRef] [Medline]
  63. Wayne N, Ritvo P. Smartphone-enabled health coach intervention for people with diabetes from a modest socioeconomic strata community: single-arm longitudinal feasibility study. J Med Internet Res 2014 Jun 06;16(6):149 [FREE Full text] [CrossRef] [Medline]
  64. Wayne N, Perez DF, Kaplan DM, Ritvo P. Health coaching reduces HbAc in type 2 diabetic patients from a lower-socioeconomic status community: a randomized controlled trial. J Med Internet Res 2015 Oct 05;17(10):224 [FREE Full text] [CrossRef] [Medline]
  65. Waki K, Fujita H, Uchimura Y, Omae K, Aramaki E, Kato S, et al. Dialbetics: a novel smartphone-based self-management support system for type 2 diabetes patients. J Diabetes Sci Technol 2014 Mar;8(2):209-215 [FREE Full text] [CrossRef] [Medline]
  66. Waki K, Aizawa K, Kato S, Fujita H, Lee H, Kobayashi H, et al. Dialbetics with a multimedia food recording tool, foodlog: smartphone-based self-management for type 2 diabetes. J Diabetes Sci Technol 2015 May;9(3):534-540 [FREE Full text] [CrossRef] [Medline]
  67. Alanzi T, Istepanian R, Philip N. Design and usability evaluation of social mobile diabetes management system in the gulf region. JMIR Res Protoc 2016 Sep 26;5(3):93 [FREE Full text] [CrossRef] [Medline]
  68. Alotaibi MM, Istepanian R, Philip N. A mobile diabetes management and educational system for type-2 diabetics in Saudi Arabia (SAED). Mhealth 2016;2:33 [FREE Full text] [CrossRef] [Medline]
  69. Bee YM, Batcagan-Abueg APM, Chei C, Do YK, Haaland B, Goh S, et al. A smartphone application to deliver a treat-to-target insulin titration algorithm in insulin-naive patients with type 2 diabetes: a pilot randomized controlled trial. Diabetes Care 2016 Oct;39(10):174-176. [CrossRef] [Medline]
  70. Hsu WC, Lau KHK, Huang R, Ghiloni S, Le H, Gilroy S, et al. Utilization of a cloud-based diabetes management program for insulin initiation and titration enables collaborative decision making between healthcare providers and patients. Diabetes Technol Ther 2016 Feb;18(2):59-67 [FREE Full text] [CrossRef] [Medline]
  71. Kim EK, Kwak SH, Baek S, Lee SL, Jang HC, Park KS, et al. Feasibility of a patient-centered, smartphone-based, diabetes care system: a pilot study. Diabetes Metab J 2016 Jun;40(3):192-201 [FREE Full text] [CrossRef] [Medline]
  72. Anzaldo-Campos MC, Contreras S, Vargas-Ojeda A, Menchaca-Díaz R, Fortmann A, Philis-Tsimikas A. Dulce wireless tijuana: a randomized control trial evaluating the impact of project dulce and short-term mobile technology on glycemic control in a family medicine clinic in northern Mexico. Diabetes Technol Ther 2016 Apr;18(4):240-251 [FREE Full text] [CrossRef] [Medline]
  73. Kleinman NJ, Shah A, Shah S, Phatak S, Viswanathan V. Improved medication adherence and frequency of blood glucose self-testing using an m-health platform versus usual care in a multisite randomized clinical trial among people with type 2 diabetes in india. Telemed J E Health 2017 Sep;23(9):733-740. [CrossRef] [Medline]
  74. Yoo HJ, Park MS, Kim TN, Yang SJ, Cho GJ, Hwang TG, et al. A Ubiquitous Chronic Disease Care system using cellular phones and the internet. Diabet Med 2009 Jun;26(6):628-635. [CrossRef] [Medline]
  75. Kim CS, Park SY, Kang JG, Lee SJ, Ihm SH, Choi MG, et al. Insulin dose titration system in diabetes patients using a short messaging service automatically produced by a knowledge matrix. Diabetes Technol Ther 2010 Aug;12(8):663-669. [CrossRef] [Medline]
  76. Lim S, Kang SM, Shin H, Lee HJ, Won Yoon J, Yu SH, et al. Improved glycemic control without hypoglycemia in elderly diabetic patients using the ubiquitous healthcare service, a new medical information system. Diabetes Care 2011 Feb;34(2):308-313 [FREE Full text] [CrossRef] [Medline]
  77. Agboola S, Jethwani K, Lopez L, Searl M, O'Keefe S, Kvedar J. Text to move: a randomized controlled trial of a text-messaging program to improve physical activity behaviors in patients with type 2 diabetes mellitus. J Med Internet Res 2016 Nov 18;18(11):307 [FREE Full text] [CrossRef] [Medline]
  78. Harman NL, Wilding JPH, Curry D, Harris J, Logue J, Pemberton RJ, SCORE-IT Study Team. Selecting core outcomes for randomised effectiveness trials in type 2 diabetes (Score-It): a patient and healthcare professional consensus on a core outcome set for type 2 diabetes. BMJ Open Diabetes Res Care 2019;7(1):000700 [FREE Full text] [CrossRef] [Medline]
  79. Veazie S, Winchell K, Gilbert J, Paynter R, Ivlev I, Eden KB, et al. Rapid evidence review of mobile applications for self-management of diabetes. J Gen Intern Med 2018 Jul;33(7):1167-1176 [FREE Full text] [CrossRef] [Medline]
  80. Nwe K, Larsen ME, Nelissen N, Wong DC. Medical mobile app classification using the National Institute for Health and care excellence evidence standards framework for digital health technologies: interrater reliability study. J Med Internet Res 2020 Jun 05;22(6):17457 [FREE Full text] [CrossRef] [Medline]
  81. Parker J, Powell L, Mawson S. Effectiveness of upper limb wearable technology for improving activity and participation in adult stroke survivors: systematic review. J Med Internet Res 2020 Jan 08;22(1):15981 [FREE Full text] [CrossRef] [Medline]
  82. Ashmore J, Di Pietro J, Williams K, Stokes E, Symons A, Smith M, et al. A free virtual reality experience to prepare pediatric patients for magnetic resonance imaging: cross-sectional questionnaire study. JMIR Pediatr Parent 2019 Apr 18;2(1):11684 [FREE Full text] [CrossRef] [Medline]


BCT: behavior change technique
DHT: digital health technology
mHealth: mobile health
NICE: National Institute of Care Excellence
NIHR: National Institute for Health Research
NHS: National Health Service
RCT: randomized controlled trial
T2DM: type 2 diabetes mellitus
WHO: World Health Organization


Edited by D Griauzde; submitted 20.08.20; peer-reviewed by D Wong, K Waki, N Wayne, L Artavia-Mora; comments to author 15.11.20; revised version received 16.12.20; accepted 31.12.20; published 16.02.21

Copyright

©Jessica R Forsyth, Hannah Chase, Nia W Roberts, Laura C Armitage, Andrew J Farmer. Originally published in JMIR Diabetes (http://diabetes.jmir.org), 16.02.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Diabetes, is properly cited. The complete bibliographic information, a link to the original publication on http://diabetes.jmir.org/, as well as this copyright and license information must be included.