All patients at Galway University Hospitals must attend an education session with a Clinical Nurse Specialist before starting semaglutide injectable therapy. This session covers basic semaglutide pharmacotherapy and the safe self-administration of injections using a semaglutide pen. During the study period, allocated, eligible participants received their mandatory education from a “digital clinician” with human oversight. The control group received current standard-of-care, human, nurse-led education.

Once a clinical decision was made that treatment of overweight or obesity with semaglutide injections would be commenced, eligibility was assessed. Overweight and obesity were classified as a BMI greater than 25 kg/m². Eligibility criteria included being aged 18 years or older, without an intellectual or physical disability, which would interfere with a participant’s ability to self-administer semaglutide injections. All eligible patients were invited to participate in the study. Given the nature of the intervention (“digital clinician”), blinding was not feasible.

Eligibility criteria included patients aged older than 18 years, without an intellectual or physical disability, which would interfere with a participant’s ability to self-administer semaglutide injections. All eligible patients were invited to participate in the study. Given the nature of the intervention (“digital clinician”), blinding was not feasible.

To inform design, clinicians were observed educating patients. Their mannerisms, behavior, and conversations were recorded. Transcripts of educational sessions given by the local obesity nurse specialist were generated.

A 10- to 15-minute educational script was generated with multiple conversation streams. Information on medication name, dosages, mechanism of action, pen preparation, administration, side effects, and storage was included. User questioning was used to maximize engagement and knowledge retention.

Akin to natural clinical education, the “digital clinician” led the primary section of the tutorial, offering information and asking the patient questions throughout to test retention and assess understanding. The question types used by the “digital clinician” varied. Some were open, while others were multiple choice, true or false, yes or no, or numerical. After the clinician-led portion of the tutorial had ended, patients were invited to type or voice any question they wished. The option to see and hear the answers to some frequently asked questions was also offered by way of onscreen prompts.

Unclear or ambiguous responses were identified by the digital clinician as such, and this was communicated to the user who was then kindly asked to repeat themselves. If not understanding the user twice in a row, the “digital clinician” would simply offer the correct answer and continue. This prevented the possibility of an infinite loop. When offered the chance to ask general queries, the user may query infinitely if they wish, although a comprehensive list of frequently asked question prompts on screen, and an onscreen button to end the tutorial, were in place to reduce this need.

A careful selection of trigger words and combinations of “IF,” “AND,” “OR,” and “NOT” statements was used to design the conversational flow. Rigorous internal testing, including over 100 iterations of the conversational flow, was tested by the research team to ensure a fluent, cohesive, usable digital clinician with little risk of misinterpreting the user’s intent.

This “digital clinician” could be accessed with native web browsers on the participants’ phone, tablet, or laptop. In this case, the participants accessed the session under supervision on an iPad (Apple) device.

Two multiscene videos were recorded by the team and integrated into the tutorial using screen-in-screen projection. The videos offered human views and instructions of what preparing and using the injection pen entailed. The videos, while being demonstrated by members of the research team, were narrated by the “digital clinician.”

The software IBM Watson Assistant was used to generate conversation streams. Microsoft ClipChamp Video Editor was used to edit videos, then uploaded to Imgur, accessed via source code written on IBM Watson. The IBM Watson conversation code file was integrated with an animated avatar on the SoulMachines Creator Website. The “digital clinician” uses natural language processing and an evolving bank of AI-driven animation responses that monitor vocal tone and facial expression to regulate behavior. The “digital clinician” has a programmed personality, name, voice, and identity and is being used in a role traditionally considered to require human intelligence.

Logos of affiliated educational and health care institutions were included on the user interface screen to increase trust. Cinematic screening offered different angles of the avatar's face to make for a more interesting experience. Multiple avatars of different genders, races, and names were used to promote inclusivity. All possible conversation streams had been verified against official pharmaceutical literature. The individual scripts were not recorded, but each participant-AI interaction was supervised to monitor for technical issues or hallucinations. Multiple avatars of different genders, races, and names were used to promote inclusivity. All possible conversation streams had been verified against official pharmaceutical literature. The individual scripts were not recorded, but each participant-AI interaction was supervised to monitor for technical issues or hallucinations. A short video of the “digital clinician” in use is included in Multimedia Appendix 1

Minimization is a method used in clinical trials and experimental research to allocate participants [4,28]. Participants were allocated in order of enrollment to the study arm that minimized the difference across 4 selected numerical baseline variables, namely, age, BMI, pretutorial knowledge of semaglutide score, and pretutorial injection self-efficacy score. Order of enrollment in the study was determined by check-in time at the clinic. Allocation by minimization is deterministic and hence precludes traditional concealment.

The first participant was assigned at random using a coin toss. The second participant was subsequently assigned to the alternative group. Allocation was verified using Microsoft Excel by an independent assessor with access to study data in real-time at a separate location via a secure web-based suppository. The assessor then communicated the allocation to the research team on site.

Allocation was verified using Microsoft Excel by an independent assessor with access to study data in real-time at a separate location via a secure online suppository. The assessor then communicated the allocation to the research team on site.

Self-efficacy was measured using the 2-part validated Self Injection Assessment Questionnaire (SIAQ) [29]. The preinjection SIAQ was used as baseline data. The postinjection SIAQ was scored across 5 domains considering 2-week postintervention feelings about injections, self-image, self-confidence, pain and skin reactions, ease of use, and satisfaction. The SIAQ has been shown to be a valid, robust tool with sufficient validity, reliability, consistency, and sensitivity. Cronbach α and the test-retest coefficient were >0.70 for all domains [30].

Unvalidated knowledge assessments were designed specifically for the purpose of this research. The pre-educational assessment tool consisted of 4 questions. The posteducational assessment tool contained the same initial 4 questions and a further 8 questions. Questions were multiple choice and covered topics, such as drug name, drug class, mechanism of action, side effects, injection technique, and storage requirements.

The validated Patients’ Overall Satisfaction with Primary Care Physicians Scale (CPSS) [31] has criterion-related validity coefficients mostly in the 0.80s and 0.90s when considering empathy, physician recommendation, and general satisfaction. Cronbach coefficient α for the patient satisfaction scale is 0.98. The 7-point Likert scale was used in the study.

The “Trust between People and Automation Scale” (TPA) has undergone validation [32] and been in use for more than 20 years. It was adapted and integrated with the consultation satisfaction measure for this study.

The validated Technology Usability Questionnaire (TUQ) [33] was adapted. The questionnaire has high reliability with Cronbach Coefficient α more than 0.8 across all domains, and high validity expressed through its multiple native questionnaires [34]. A shortened version including questions from 4 of 5 domains was used. The domains were “Ease of use and Learnability,” “Interface Quality,” “Interaction Quality,” and “Satisfaction and future use.” Open-ended questions were used to record general themes and attitudes toward the “digital clinician.” The outcome measure tools are included as Multimedia Appendix 2.

When calculating sample size, a similar study using the validated SIAQ to measure self-efficacy was used [29]. It showed a mean 7.09 (SD of 1.5), on a 10-point Likert scale. A significant difference of two-thirds the SD was proposed to boundary noninferiority, with 80% power and α=.05; this would require 28 participants to be recruited into each arm of the study.

A randomized controlled noninferiority trial of “digital clinician”-led patient education was designed with ethical approval from Galway University Hospitals, Clinical Research Ethics Committee (Ref: CA 2920). The trial and trial protocol were guided by SPIRIT-AI (Standard Protocol Items: Recommendations for Interventional Trials involving Artificial Intelligence) [35] and CONSORT-AI (Consolidated Standards of Reporting Trials- Artificial Intelligence) [34] extensions. Further information on the study protocol can be found in Multimedia Appendix 3. It was ensured that all patients were offered at least the current standard of care. After data collection, all patients were seen by the clinical nurse specialist. The nurse assessed the patients who had received safety-net instructions before being discharged with a medication information leaflet. Contact with the nurse minimized disruption of the patient-clinician relationship. This established human oversight and accountability. Informed written consent was obtained from all participants before enrollment and randomization, after provision of a patient information leaflet and discussion with the research team. All patient data was anonymized. No patient identifiable information was included in data analysis and confidentiality was maintained. Data was stored on password-protected encrypted devices. No compensation was offered to participants.

Data were analyzed using Jamovi version 2.4.14 developed by Love, Droppman, and Selker [36,37] and R Studio, developed by Posit PBC [37]. All Likert scale data were appropriately aggregated in treatment and control arms and recalibrated to scales from 1 to 10 for self-efficacy, and 1-7 for trust, satisfaction, and usability.

The Mann-Whitney U test was used to assess differences in primary and secondary outcome variables. This nonparametric test is robust to the distribution of the outcome data. The Mann-Whitney U test compares the ranks of all the data points in 2 groups. P values below .05 were deemed statistically significant. Conventional content analysis was used to interpret qualitative data. Missing outcome data were omitted from analysis. Observed outcome data were analyzed as if representative of the entire cohort.

Knowledge, satisfaction, trust, and usability were measured immediately posttutorial. Subsequent loss to follow-up at 2 weeks did not affect the handling of this data.

Those lost to follow-up were unavailable for self-efficacy analysis. The profile of those with complete data and those with missing data were explored in Multimedia Appendix 4.

During the 7-week study period in the bariatric clinic, 53 patients were prescribed semaglutide self-administered injections for the treatment of overweight or obesity (see Figure 1). Of this cohort, 43 agreed to participate in the study. Of those enrolled, 100% (43/43) completed the pretutorial questionnaire, their assigned education session, and their posttutorial questionnaire. Only 41.9% (18) participants completed the 2-week follow-up self-efficacy questionnaire (Table 1).

Both groups had a median self-efficacy level of 10/10, with the “digital clinician” group having a marginally lower first quartile of 8, compared with the control group (8.2; Figure 2). This difference was not statistically significant (Table 2; P=.52).

Responses recording “Pain and Skin reactions” from the control group had a favorable, nonsignificantly higher mean rank.

Overall, the median test score of participants postconsultation in the “digital clinician” group was 10/12 (84%; IQR 10-11), while the median score in the control group was 8/12 (69%; IQR 7-9; P<.001; Figure 3, Table 3).

In total, 44% (12/27) respondents in the “digital clinician” group were more than 90% satisfied. The lowest score was 63%. In the control group, 94% (15/16) respondents were more than 90% satisfied with their consultation. The lowest score was 86%. The IQR of the “digital clinician group” was 6-7 (median 7), whereas it was 7-7 (median 7) in the control group (Mann-Whitney P<.001). This is shown in Figure 4 and Table 3 on a Likert scale between 1 and 7.

Figure 5 shows that participants in the “digital clinician” group scored their education provider lower for empathy (P=.003), clear intentions (P=.002), and overall trust (P<.001).

While Figures 4 and 5 show that the medians of the ordinal data are the same, the Mann Whitney U test is nonparametric and is not dependent on central tendency. As nonparametric data can be asymmetrical by nature, central tendency can be misleading. While the medians are the same, the IQR and ranks of the groups are shown to be significantly different with a P value <.05.

On a scale of 100, the “digital clinician” was found to be 86.667 usable (Table 4), scoring lowest for interface quality and highest for interaction quality.

Of the 27 participants in the intervention group, 22 (81%) of participants affirmed they would use the “digital clinician” for educational purposes in their own time, while 2 (7%) of participants recorded that “maybe” they would. A total of 3 (11%) said they would not.

Below are the described questions about the analysis of open-ended feedback.

Four responses made reference to a great invention or great possibility for future use. Seven said the idea was good, or very good. Four described it as “okay.” One respondent felt the avatar was “not ideal,” and another stated they would “prefer a human.” Two responses were of mixed sentiment:

Twelve responded that they would have no issues using the resource. Two expressed concerns over technical issues, such as freezing and audio quality. One expressed concern over language barriers. Four expressed concerns regarding the lack of empathy or possible loss of personal touch. One expressed concern over the ability of the avatar to answer the questions required. In total, 20%‐25% expressed some form of apprehension. The prevalence of triggering the wrong conversational stream was not formally measured in the study; however, users did not remark on it when providing open-ended feedback on usability.

Two responses:

Those with complete data had higher levels of education, female sex, were less trusting of (P=.04) and less satisfied (P=.03) with their health care professional than those lost to follow-up (Multimedia Appendix 3). However, the response rates of those in “digital clinician” and the “control” groups at follow-up were similar, as were measures of knowledge, self-efficacy, and usability.

Those who received education from the digital clinician were significantly more knowledgeable about their medication and its administration. They tended to have less stress and anxiety associated with using their injections, though this was not significant. Paradoxically, they had less confidence administering injections. They also trusted that their education provider was significantly less than those in the control group and were significantly less satisfied with their consultation. This suggests that from a health care psychology standpoint, participants’ injection-related confidence was more related to who they received their information from, rather than how well informed they were.

While being less satisfied than those in the control group, the intervention group still reported very high levels of trust and satisfaction at levels, with median levels of 7/7 in both measures. Participants were extremely positive about the intervention, with more than 80% of participants expressing that they would use the resource in their own time. If not used clinically, distribution of a “digital clinician” for home use could also add value. The study did not recruit adequate numbers to test noninferiority at the predetermined level of self-efficacy due to global semaglutide shortages hindering patient access. However, there were significant differences in a range of secondary outcomes.

Human clinician-patient interactions are not scripted. They are shaped by human factors, including rapport and variability. Consultations with the “digital clinician” are more uniform and consistent. This may explain why a human was more successful at ensuring trust and satisfaction, while the “digital clinician” was more effective at ensuring information was provided, tested, and retained.

Certain studies have compared physicians versus chatbots, in educational settings, by assessing accuracy. No study has been uncovered showing that patients themselves were better educated by a chatbot or an LLM than by a specialist clinician. It is one thing to assess what information the chatbot is providing; it is another thing to assess if a patient is receiving, retaining, and trusting that information. Moving forward, only the true benefit of “digital clinicians” can be judged when research examines the risks and benefits of integration on a systemic level.

Standards for new technologies should be established to enable regulation, safety, and trust. Bespoke task-specific conversation streams such as this, which mimic large language models, may be easier to control and regulate while taking advantage of growing trust and recognition in the public domain.

Automation may incorporate tradeoffs in patient satisfaction and health care system trust, but may offer benefits in comprehensiveness, efficacy, and uniformity. Tradeoffs could potentially be minimized through research, human oversight, and clear physician accountability. The opportunity cost of using these technologies or not should be considered in terms of productivity, finance, access, and workforce and patient health.

Limitations of the study include the loss of participants to follow-up questioning due to a global shortage of semaglutide, possibly contributing to some of the nonsignificant results in the study. There were significant missing data at the 2-week follow-up, which were omitted from analysis. Table S1 in Multimedia Appendix 4 shows the different profile of responders at the 2-week follow-up. Responders tended to have stronger views and were not a reliable representation of the initial cohort, making the conclusions regarding self-efficacy less applicable to the general population.

Nonvalidated measures of knowledge and adapted measures of usability, trust, and consultation satisfaction were used. While the trial design focuses on intergroup analyses, nonvalidated measures preclude direct comparison of the study cohort with the study population, as existing data for the latter does not exist using the same parameters. As a result, nonvalidated and adapted measures may not correlate well with other health outcomes such as compliance or the incidence of side effects as established measures.

This feasibility trial, while not measuring traditional health outcomes, would have been strengthened by prospective trial registration, a prerequisite for all interventional AI trials from January 01, 2025 [38]. Trial registration was initially omitted from the design process. Contributing factors included the novelty of the intervention and the relative sparsity of an established industry proforma at the time. During the trial and data analysis period, further recognition of AI use as a significant health care intervention led to the emergence of industry directives, prompting reflection and retrospective trial registration. While not included in this study, further longitudinal research could focus on clinical outcomes such as adherence to therapy or weight loss over longer periods (eg, 1 year).

There is a worsening shortage of health care workers globally. Systems are not adapting to keep up with demand in areas such as bariatric medicine. “Digital clinicians,”chatbots integrated with digital avatar and emotional intelligence technology, may be part of scalable AI solutions.

“Digital clinicians” may ensure higher levels of information retention among patients compared with humans. However, users have significantly lower levels of trust and satisfaction in their information provider. Despite not being as satisfied, users are still overwhelmingly positive about their consultation, with over 80% saying they would use the resource in their own time.

Offering a degree of automation feasibly allows existing health care workers to focus on more demanding tasks, reduce stress, and improve health care system performance. It may offer more patients and systems access to medication and resources, provided they have the necessary internet connection and operability.

Safety should be established through regulation and research to improve trust and satisfaction. Ethical principles such as human oversight and accountability should build on this. Rigorous policy analysis should be performed to assess the potential tradeoffs in using these technologies as AI becomes more widespread across resource-rich and resource-poor contexts.