The first part of the study consisted of an exploratory qualitative study using face-to-face semistructured interviews with DM specialist nurses from hospitals and community health centers across West Midlands, United Kingdom. Diabetes Specialist Nurses who worked at local National Health Service (NHS) facilities and had a minimum of 2 years’ experience working with people with DM were eligible for the study.

The recruitment was performed mainly via emails or word-of-mouth. Once eligibility and consent were confirmed, an interview was scheduled. Interviews lasted 15-30 minutes and took place in a meeting room or office at the hospital or practice where the nurses worked. The interview topic was the use of mobile CDSS apps that assist nurses in managing aspects of diabetes. All interviews were recorded using a digital recorder. The recruitment, choice of interview type (face-to-face, semistructured), and further details are provided in Multimedia Appendix 2.

A minimum sample size estimate of 10 DM specialist nurses was chosen, with continued sampling until the saturation point [7,25,26]. Saturation was reached after 15 interviews at which time the recruitment was stopped.

Thematic analysis was used, and interview recordings were transcribed verbatim by the researcher [9,27]. The computer-assisted qualitative data analysis software NVivo (version 10) was used to assist in the analysis process. In an attempt to minimize bias in the interpretation of the data, an experienced qualitative researcher was consulted on the conduct and analysis of this research.

The nurses’ feedback at the “requirements gathering” stage, other requirements gathered from the literature, and requirements from feedback and suggestions from 2 diabetes and endocrinology consultants were considered during step 2. The total requirements were divided into 3 categories: functional, technical, and medical. These are analyzed further in Multimedia Appendix 3.

The app was built by a software developer (Medic Genie), and the development process included two parts: design and coding. Both parts were done by visual programming, while a junior doctor provided well-defined guidance and verification of the correctness of the management pathway. Development of the app involved generating management pathways using the most recent National Institute for Health and Care Excellence (NG28) guidelines on DM, CKD, and hypertension. The guidelines used in the development of the management pathways, developed decision algorithm, and table of dose adjustments for CKD are provided in Multimedia Appendix 4, Multimedia Appendix 5, and Multimedia Appendix 6, respectively.

This component used multiple methods and quantitative and qualitative designs. The 3 main methods used for this part of the study included a randomized controlled experiment, usability testing, and a satisfaction questionnaire (Multimedia Appendix 7), with an aim of demonstrating the impact and acceptability of the app.

Two types of end users were considered as participants: junior doctors and DM specialist nurses.

The junior doctors were recruited for the randomized controlled experiment from the University Hospitals Birmingham NHS Foundation Trust as part of a teaching session on DM and renal complications. At the teaching session, 39 doctors were recruited through convenience sampling. During the piloting phase, junior doctors were randomly divided into 2 groups using software-generated random numbers. The intervention group had access to the developed app “Diabetes & CKD,” while the control group had access to paper-based guideline algorithms that informed the app’s development. At this stage, 2 case scenarios were prepared by a diabetes and endocrinology consultant from the University Hospitals Birmingham NHS Foundation Trust. The evaluation was conducted to assess how the app could support health care providers in terms of (1) workflow efficiency (measured by time to complete the tasks) and (2) adherence to clinical guidelines (measured by accuracy of the decision made, compared to the use of paper-based guideline algorithms). Those in the control group had access to paper-based guideline algorithms that informed the app development, while participants in the intervention group were given a link to the app. Both groups were asked to deal with 2 simulation-based case scenarios (Multimedia Appendix 8). Decisions made in each group were written on the provided answer sheets. At the end, intervention participants were asked to complete the satisfaction questionnaire (Multimedia Appendix 9).

The DM specialist nurses from the interview study who expressed interest in taking part were included. Although 15 DM specialist nurses were invited via email to take part in the usability testing session, only 3 were recruited at the Sandwell & West Birmingham Hospitals NHS Trust, United Kingdom. During the testing phase, nurses were asked to perform tasks using the case scenarios and to verbalize what they are doing while they were doing it. Broad questions were used to explore participants’ views and opinions during the session.

Interviews were conducted with 15 DM specialist nurses from 4 hospitals and 2 community health centers across the West Midlands regarding the use of the mobile CDSS app. All were female, with an average age of 45 years and an average duration as a nurse specializing in DM of 10 years. The participants’ characteristics are summarized in Multimedia Appendix 10. They all were owners of a tablet or smartphone. Moreover, 14 of 15 (93%) used a device provided by the Trust during clinical practice.

The interviews with nurses identified varying themes related to 5 main areas: prior experience with using apps, perceptions and views of using apps in clinical practice, challenges in DM management, willingness to use apps, and one app does not fit all. The “one app does not fit all” area emerged during the interviews but did not fit under any of the 4 main investigated areas. People have different needs and preferences, range of skills, and degree of motivation, so one app will not fit all, as believed by most nurses. The themes identified by the interview and their relationships are shown in Figure 1. Examples of the DM specialist nurses’ answers that helped formulate the conclusions are provided in Multimedia Appendix 11.

Generally, nurses urged for apps that are simple, short, and to the point. Clinical support apps need to work across multiple mobile platforms, not require a WiFi connection, be visual, be interactive, and not require the inputting of too many details. Finally, they agreed that clinical apps need to be customized locally. Overall, nurses expressed a strong willingness to use apps in clinical practice.

In the controlled experiment, 39 junior foundation year 1 doctors were included (17/39, 44% male). Of them, 44% (17/39) were allocated to the intervention group, and 56% (22/39) were allocated to the control group. The usability testing session involved 3 DM specialist nurses. All were female, 30-46 years old, and working in the DM field for 6-16 years.

As the intervention and control groups were normally distributed, an independent samples t test was performed. There was no significant difference in the scores between the intervention group (mean 7.24, SD 2.46) and control group (mean 7.39, SD 2.56; t₃₇=–0.19, P=.85; maximum score was 13). The frequency distribution of the scores for both groups is presented in Multimedia Appendix 12.

These results suggest that, for the given 2 case scenarios, no difference was observed in the accuracy of the decision made using the app compared to the use of paper-based guideline algorithms.

A score of 8 was determined as the minimum standard of safe care for the given 2 case scenarios. In the intervention group, 9 doctors (9/17, 53%) scored ≥8, whereas only 7 doctors (7/22, 32%) in the control group scored ≥8. However, when comparing their scores for each case individually, 10 doctors (10/17, 59%) and 17 doctors (17/22, 77%) from the intervention and control groups, respectively, scored a minimum of 3 in the first case scenario, while 10 doctors (10/17, 59%) and 11 doctors (11/22, 50%) from the intervention and control groups, respectively, got a minimum score of 5 in the second case scenario.

All 17 questionnaires were completed, providing a 100% response rate (intervention arm). Nearly 50% (9/17) of the junior doctors indicated prior experience using a CDSS app. With regard to their overall impression, 83% (14/17) were satisfied or somewhat satisfied with the app. Ease and simplicity of the app were the most emphasized features: easy to use, user-friendly, straightforward, quick, simple flow, good presentation, intuitive user interface, clear design, easy to input information, gives a good recommendation based on results, not too wordy, good font size, and much easier to use than the algorithm. By contrast, the most common negative points were the ambiguity of the navigation between pages or recommendations and difficulty scrolling up and down. Regarding usability, some respondents reported that they encountered some technical bugs during the session.

In terms of suggestions for improvement, participants expressed the need for easy and clear navigation; information on doses supported with links to evidence; additional information on drug side effects, for example weight loss or gain; the possibility to save previous searches to go back to them easily; the ability to enter patient’s current medications to help streamline the options at the end; and more specific advice regarding which combinations of dual or triple therapy would be more appropriate.

All participants thought that the app seemed useful and were willing to use such apps in their clinical practice, specifically for more complex patients when they are uncertain and to avoid searching through guidance.

The DM specialist nurses faced several usability problems. The ambiguity of the navigation between pages or recommendations as well as app crashes were described as the most frustrating part of the app. Additionally, some DM specialist nurses felt that some data items were irrelevant, for example, when the app asks for blood pressure while looking at glycemic control. By contrast, they asked to add some data items such as BMI.

Nurses indicated several positive aspects of the app. They found it very useful that they could click on the drug names for further information. The DM specialist nurses stated also that the app was a good idea, particularly from a practice point of view. Although usability issues were experienced with the app, they did not hinder completion of the tasks. Moreover, they liked having the button “how did I get here,” which allowed them to check if they have inputted something incorrectly at any point. They also suggested having this screen compulsory in order to enable users to make sure their information was correct.

Several further suggestions were given by the DM specialist nurses to improve the app; for instance, when the app recommended monitoring the patient, nurses wondered about how to monitor; they indicated a preference to use other gestures when communicating with the app such as swiping; and there were thoughts that providing background information (such as basic guidelines or treatment pathways) on the home page would be helpful.