This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Diabetes, is properly cited. The complete bibliographic information, a link to the original publication on http://diabetes.jmir.org/, as well as this copyright and license information must be included.
The burden of obesity is high among US veterans, yet many face barriers to engaging in in-person, facility-based treatment programs. To improve access to weight-management services, the Veterans Health Administration (VHA) developed
The primary aim was to establish preliminary evidence for the program by comparing outcomes for
We employed a formative mixed-methods design to evaluate the phased implementation of
Although stakeholders at 3 sites declined to be interviewed, objective program uptake was high at 2 sites, delayed-high at 2 sites, and low at 5 sites. At 6 months post enrollment, the mean weight loss was comparable for
This implementation evaluation of a clinical telehealth program demonstrated the value of partnership-based research in which researchers not only provided operational leaders with feedback regarding the effectiveness of a new program but also relevant feedback into contextual factors related to program implementation to enable adaptations for national deployment efforts.
In 2016, 42% of the patients receiving care in the Veterans Health Administration (VHA) were obese and 37% were overweight, putting these individuals at risk for obesity-related comorbidity, functional impairment, and diminished quality of life [
To address these access barriers, the VHA National Center for Health Prevention and Disease Prevention (NCP), which oversees MOVE!, collaborated with the VHA Telehealth Services Home Telehealth (HT) Program to develop a telehealth program called
The impetus for rapid
Although there was evidence for the efficacy of individual voice recognition (IVR) and phone coaching for weight management, studies of in-home messaging devices had not been rigorously evaluated for health promotion application before
This implementation evaluation had 2 aims. First, we sought to establish preliminary evidence for the impact of
We used a formative mixed-methods design to evaluate implementation of
To facilitate organizational learning,
Phase 2 began in October 2009, with an Web-based training conference about
Summary of pilot phases of
Phase | Participants | Implementation activity and evaluation method | |||
|
|||||
|
March 2009 |
NCPa, TSb, and regional network leaders |
Invitation for 10 HTc programs to submit written intent to volunteer to pilot Define collaborative roles and responsibilities for NCP/TS Create timeline for phased implementation plan Update implementation plan draft |
||
|
July-August 2009 |
Staff from 1 VAMCd NCP, TS, and regional network leaders |
Weekly planning meetings Track challenges and facilitators to Review readiness of cross-training modules for |
||
|
|||||
|
September-November 2009 |
1 VAMC Local staff |
Enroll 30-45 patients to develop implementation methods and toolkit |
||
|
|
|
|||
|
October 2009 |
Staff from 9 VAMCs NCP, TS, and regional network leaders |
Share early learnings/challenges from phase 1 site Disseminate program materials Share workflows and administrative procedures |
||
|
|
|
|||
|
November-February 2010 |
Staff from 9 VAMCs |
Enroll 30-60 patients per medical center Monitor and troubleshoot pilot implementation Identify key learnings; develop solutions to barriers Refine implementation plan for national rollout |
||
|
|||||
|
April 2010-September 2011 |
Interested VA facilities |
National goal to enroll 10,000 patients per year Enroll panels of 80-120 patients per medical center Funding for care for up to 300 patients per network |
aNCP: National Center for Health Promotion and Disease Prevention.
bTS: Telehealth Services.
cHT: Home Telehealth.
dVAMC: Veterans Health Administration medical centers.
Standard MOVE! treatment services were implemented throughout VHA in 2006 as a comprehensive, evidence-based lifestyle approach to weight management for veterans [
Upon completing MOVE! enrollment activities, patients who elected to choose the
The Health Buddy prompted participants to provide daily weight readings from their digital scale to encourage tracking of weight management progress. If a participant went 30 days or longer without losing half to 2 lb per week or lost weight too quickly, a trigger alert for re-evaluation would occur. Participants received 10- to 20-min telephone calls from a
The
The quantitative aspect of this mixed-methods study evaluated 2 cohorts of VHA patients who enrolled in either
Quantitative data was extracted from VHA patient care databases to describe patient characteristics, program use, and weight changes associated with program participation at each of the 9 demonstration sites. Participant demographic characteristics and program utilization data were extracted from the VHA Service Support Center-hosted visits ProClarity cube. Data pertaining to medical comorbidities and change in weight were extracted from the VHA Corporate Data Warehouse (CDW). Program use was characterized by 2 indicators: program enrollment and program engagement. Distinctions in program
Indicators of site implementation effectiveness were rated based on attaining targets for program enrollment
The evaluation plan called for conducting 2 rounds of semistructured interviews at each of the 9 phase 2 demonstration sites. The first round of interviews was conducted by phone, 3 to 6 months after phase 2 (June-August 2010) and the second round of in-person interviews were conducted 6 months after the start of phase 3 (November 2010-April 2011) to capture the dynamic nature of the implementation process. Key facility- and regional-level managers and program staff involved in
Verbal consent and permission to digitally audio-record interviews were obtained from participants at the start of their first interview. Staff at 3 sites declined to participate in both rounds of interviews. A total of 42 VHA stakeholders were invited to participate in an interview, and 66% (28/42) agreed to participate in at least one interview; 22 participated in the phone interviews, and 21 participated in on-site interviews. Interview ranged from 18 to 86 min in duration and was digitally audio-recorded and transcribed verbatim into Microsoft Word documents. Additionally, call minutes from biweekly conference calls held among the demonstration sites during the evaluation were analyzed to understand contextual factors affecting implementation effectiveness.
The CFIR [
This research study was approved by the VA Ann Arbor Healthcare System Institutional Review Board (2010-010042) with a waiver of signed informed consent for staff interviews and for secondary data analysis of deidentified patient-level outcome data.
There were significant differences in demographic characteristics between those who chose to enroll in each program modality (see
Preliminary indicators of clinical effectiveness are summarized in
Comparison of enrollment in evaluation cohorts for year 1 of implementation.
Cumulative enrollment across sites during year 1 of
Demographic characteristics of engaged
Characteristic | MOVE!a (n=1648) | |||
Age in years, mean (SD) | 57 (9.5) | 55 (11.0) | <.001 | |
Male, n (%) | 422 (84.9) | 1434 (87.01) | .23 | |
|
|
|
<.001 | |
|
White | 301 (80.1) | 853 (65.31) |
|
|
Black | 60 (16.1) | 413 (31.62) |
|
|
Other | 11 (3.1) | 40 (2.96) |
|
Ethnicityc (Hispanic), n (%) | 7 (1.7) | 66 (4.88) | .006 | |
Rural address, n (%) | 287 (57.9) | 691 (41.93) | <.001 | |
Baseline (lb), mean (SD) | 256 (51) | 243 (49) | <.001 | |
Baseline body mass index, mean (SD) | 37.5 (6.9) | 35.5 (6.3) | <.001 | |
Charlson score, mean (SD) | 1.7 (1.9) | 1.6 (1.9) | .39 |
aExcludes patients enrolled in
bPaired
cAvailable data to calculate % race/ethnicity variables were
Comparison of weight change outcomes in year 1 for engaged participants.
Characteristics | National MOVE! cohort (N=31,854) fiscal year 10a | MOVE! participants |
||
Six-month weight (lb), mean (SD) | −3.6 (0.1)c | −5.13 (12.4) | −5.22 (12.4) | .90 |
Six-month change (BMId), mean (SD) | −0.5 (0.0)c | −0.75 (1.8) | −0.70 (2.4) | .72 |
Change in body weight, n (%) | −1.4 (0.1)c | −2.02 (5.0) | −2.01 (5.6) | .95 |
Number of patients with >5% weight loss, n (%) | 5925 (18.60) | 372 (24.11) | 92 (22.1) | .31 |
aNormative in-person averages from national FY 2010 MOVE! report [
bPaired
cNote, all fiscal year 10 national
dBMI: body mass index.
Among the 6 sites that participated in qualitative interviews, we identified 5 CFIR constructs that illustrated key contextual factors that distinguished high from low implementation sites: complexity, patient needs and resources, networks and communications, leadership engagement, and reflecting and evaluating.
Complexity refers to the perceived difficulty of implementing an intervention especially with respect to the duration, scope, disruptiveness, and intricacy of the steps involved.
Across all 6 sites, respondents felt that although
Qualitative data illustrating contextual factors distinguishing
CFIRa construct | High-uptake site | Low-uptake site |
Complexity |
|
|
Patient needs and resources |
|
|
Networks and communication |
|
|
Leadership engagement |
|
|
Reflecting and evaluating |
|
|
aCFIR: Consolidated Framework for Implementation Research.
Although many patients were pleased with the telehealth devices, some patients were frustrated by connectivity issues and others were disappointed with the device’s simple interface. Often, these dissatisfied patients chose not to use their device, thereby reducing program productivity due to the time needed for program coordinators to contact these patients, return the devices back to the VA hospital, and to repurpose the device for another patient.
Coordinators across sites spent considerable time in calling patients to provide technical assistance, particularly for older veterans who were less confident in using the telehealth devices despite the relatively simple device interface. Coordinators also noted the need to address the ongoing issue of false alerts for issues such as low self-reported mood or errors in the transmission of weight data due to issues with the interface of the digital scales with the Health Buddy. Many of these complexity issues stemmed from the fact that most sites initially attempted to conduct screening and enrollment over the phone rather than conducting face-to-face orientations. With thoughtful experimentation, high-uptake sites found better ways during the phased implementation to mitigate complexity barriers by proactively preparing patients for the device use, whereas low-uptake sites continued to struggle with these issues without re-examining their workflow to identify areas to mitigate problems.
Low-uptake sites also found the
This construct reflects the extent to which patients’ needs, as well as barriers and facilitators to these needs are accurately known and prioritized by program staff. Accurately assessing patients’ home connectivity was a critical component of the recruitment and enrollment process. Over time, higher performing coordinators developed in-person protocols that carefully assessed this issue during screening and enrollment, recording declined patients’ names in a file for contact when
High-uptake sites were also more likely to report to patient needs by resourcing
Strong formal and informal social networks of program staff and leaders were essential to effectively implement
The presence of strong commitment and support by leaders is an indicator of an organization’s commitment to implementing an intervention. In contrast to low-uptake sites, among high-uptake sites, there was strong support by hospital leaders as well as by frontline supervisors and midlevel managers overseeing HT and MOVE! programming. This support was manifested by quick approvals for changes in workflows, staffing credit, and resources, with a shared consensus that
Effective implementations require the ability to regularly reflect and evaluate both quantitative and qualitative feedback regarding the progress and quality of an intervention implementation. This CFIR construct reflects a quality improvement mindset that was present to some degree in all high-uptake sites but absent at the low-uptake sites. High-uptake sites provided specific examples of monitoring various aspects of program implementation and then using these data to identify opportunities to improve care delivery and patient outcomes. Conversely, examples of reflecting and evaluating were largely absent at lower performing sites where program staff were more reactive and less innovative in identifying solutions to issues in implementing the program.
Our mixed-methods findings provide preliminary evidence for the clinical effectiveness of the
Our quantitative findings helped build a case for further adoption of
This evaluation benefited from the systematic assessment of stakeholders’ perspectives regarding
Stakeholders told us that it is important to consider what the CFIR identifies as outer setting factors interacting with the
Prior telehealth studies point toward leverage points to improve
Interviews also revealed how
This study is not without limitations. Notably, the study was a nonrandomized program evaluation of volunteer facilities to the implementation of clinical care program for an older generation of telehealth intervention platform. Generalizability was limited to a small number of sites and clinical stakeholders over a short period, and stakeholders at 3 sites with implementation challenges were unwilling to be interviewed regarding the specific barriers to implementing
The formative nature of this phased implementation program evaluation enabled operational decision makers to obtain real-time feedback from VHA implementation researchers to make several significant modifications to the program implementation guide and toolkit. Below are the recommendations made to operational leaders to inform national implementation efforts that were derived from our mixed-methods evaluation:
Conduct initial face-to-face screenings and enrollment sessions to assess patient ability and motivation, verify home connectivity status, and proactively address technical questions related to device use and installation.
Ensure telehealth devices from multiple vendors could all work from the same basic
Allow patients to enroll in another HT DMP while in
Revise implementation guides to emphasize the need for interservice care agreements between facilities MOVE! and HT services to answer and address specific implementation decisions regarding staffing, referrals, panel sizes, workload credit, staffing and funding needs, and procuring and mailing the telehealth and peripheral devices (scales and pedometers).
Advocate local coordinators to assess staff competencies and encourage staff to undergo recommended standardized trainings in motivational interviewing.
These program modifications were incorporated into the implementation plan used in the subsequent national
Although the telehealth technology highlighted by this mixed-methods program evaluation may seem dated by today’s standards, the barriers to implementation of new generations of eHealth and mHealth technologies largely remain the same [
We showed that an adaptation of telehealth technology could be adapted to promote clinically meaningful weight loss for veterans served by VHA, and formative qualitative data from program stakeholders could help guide national program implementation efforts when summarized by an implementation science framework. Our program evaluation highlights the benefit of implementation researchers partnering with operational initiatives to provide rigorous and rapid evaluation of the systematic deployment of promising innovation. This approach has direct application to the rapid scale-up of promising modes of telemedicine—mHealth and eHealth interventions that have the potential to help provide solutions to gaps in patient care and quality in a dynamic health environment.
Telephone interview guide evaluation of TeleMOVE program implementation.
Site visit follow-up interview guide evaluation of TeleMOVE program implementation.
Site-level indicators of TeleMOVE implementation over 2 years.
Summary of formative outcomes.
body mass index
community-based outpatient clinic
Corporate Data Warehouse
Consolidated Framework for Implementation Research
disease management protocol
fiscal year
Home TeleHealth
Individual Voice Recognition
National Center for Health Promotion and Disease Prevention
Veteran Affairs
VHA Medical Center
Veterans Health Administration
This study was funded by and initiated at the request of the VHA NCP. Funding was provided by the VHA Health Service Research and Development Quality Enhancement Research Initiative as a Rapid Response Project (RRP 10-177). The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs. The authors would like to acknowledge the role of Heather Elliot and Emily Renda in conducting site interviews and coding qualitative results. Preliminary results from this project were presented at the Annual VA Health Services Research and Development Conference in Baltimore, MD, in June 2012 and the Annual Meeting of the Society of Behavioral Medicine in San Francisco in March 2013.
None declared.