Mixed-Methods Research in Diabetes Management via Mobile Health Technologies: A Scoping Review

Introduction

The Increasing Need for mHealth and Mixed-Methods Research in Diabetes Management

Increases in diabetes incidence and prevalence are a major concern in today’s health care system. There are nearly 385 million diabetic patients in the world, and almost 90% of them have type 2 diabetes, which can be treated with appropriate lifestyle and nutrition changes. Diabetes is a complicated disease and requires a high level of patient involvement; 95% of its management is patient initiated. Considering the additional 235 million patients with type 2 diabetes expected worldwide by the year 2035, the development of diabetic patients’ self-management skills is critical [1].

Mobile health (mHealth) apps that offer personalized, fast, cost-effective, and engaging services to patients have been used increasingly in diabetes self-management [2,3]. The “disease and treatment” apps category currently includes more than 2250 diabetes apps (iOS and Android systems in total). The number of mobile apps for diabetes is growing rapidly and market penetration is expected to grow to 7.8% by 2018, reaching 24 million diabetic patients [4].

Despite the increase in the diabetes cases and rapid growth in the mobile app market, research does not yet provide sufficient information about the factors fostering or hindering the adoption of these apps, patients’ attitudes toward using them, or their effectiveness in terms of health behavior change [2,5,6,7,8]. Therefore, the controversy over whether research methodologies or mHealth-based interventions are ineffective [9,10] still remains. Alternative study designs and more rigorous methodologies to advance mHealth research, especially in diabetes management, are strongly suggested [2,11].

There is a growing interest in using mixed-methods research in mHealth-based diabetes management studies because both diabetes self-management and mHealth studies require using different approaches, techniques, and measures cooperatively within an integrated perspective [7,12,13]. This scoping review examined the extent of mixed-methods research used in mHealth-based diabetes management studies. The methodological approaches used to conduct mixed-methods studies were also investigated, and implications for future mHealth and diabetes management research are provided.

A Brief Review of Mixed-Methods Research

Mixed-methods research grew out of the “paradigm” controversy between positivist (quantitative) and constructivist (qualitative) research traditions in the 1980s and 1990s [14]. Mixed-methods research relies on both theory and practice to integrate knowledge from multiple approaches, perspectives, tools, positions, and opinions. The term “method” is used to cover a broad range of methodological (data collection techniques, design types, research methods, and so on) and related philosophical issues (eg, ontology, epistemology, axiology) [15].

The main goal of mixed-methods research is to use both quantitative and qualitative approaches to provide a better understanding of the research phenomena [16]. Because of the complementary strengths of different research paradigms, methodologies, and methods, mixed-methods research may provide new and different perspectives about an issue, expanding study findings beyond those produced by only one approach. Mixed-methods research increases the credibility of results when different approaches suggest the same conclusion, so it’s a value-added methodology [17,18].

Mixed-methods research is generally used for (1) triangulation of different approaches and methodologies to explain a single phenomenon; (2) complementarity of the research methods where one method is used to elaborate, illustrate, enhance, or clarify the results from another method; (3) sequential development of a study in which the results of one method are used to inform the other; (4) initiation of a study by using one method to find contradictions and paradoxes in findings from another method; and (5) expansion of a research study by using different methods to gain new perspectives or insights about a research problem [19].

Selecting an appropriate research design is a crucial step in a mixed-methods research. The purpose of a study and the nature of the research question help shape the design used in a mixed-methods study. In addition, using either concurrent or sequential time orientation affects study design, sampling, and data collection.

In a concurrent time orientation, data collection is completed for quantitative and qualitative phases of the study at approximately the same time to answer the same question. Both datasets are processed during data analysis and interpretation stages.

In a sequential time orientation, data can be obtained in stages, so the data from the first stage are used to shape the selection of data in the second stage (exploratory or explanatory sequential design). On the basis of their study purposes, questions, designs, and resources, researchers might place equal emphasis on both methods or use one of them primarily [16,20].

Integration is the central issue of mixed-methods research because it enables researchers to examine both types of data intensively [20]. Integration can be achieved at the method level or at the interpretation and reporting level. Bryman [21] reported that 57% of social science studies using mixed methods collected qualitative and quantitative data separately, while in approximately 27% both quantitative and qualitative data were derived from a single data source (eg, survey questionnaires including closed and open-ended questions). A growing scholarly interest has advanced mixed-methods research in various areas. Especially in health sciences, several important applications of mixed-methods research have been reported [22,23]. However, the majority of these studies failed to provide a detailed description of their data collection techniques, methods of analysis, stages of integration, and justifications for the use of mixed methods [24].

Methods

The 5-stage scoping review framework developed by Arksey and O’Malley [25] was used to identify and examine the related literature in this review. This framework allows researchers to clearly describe the methods used at each stage to increase the transparency and replicability of the studies.

Framework Stage 1: Identifying the Research Question

The research questions addressed in this review are as follows: (1) what is the extent and nature of mixed-methods research in mHealth-based diabetes management studies and (2) what are the current methodological approaches for designing and conducting mixed methods in these studies?

As a scoping review, this study did not intend to evaluate the scientific rigor of the selected studies as seen in systematic reviews [25]. It aimed to present the different ways in which researchers have used mixed methods.

Framework Stage 2: Identifying Relevant Studies

Because the key purpose of the study was scoping the area of research, a comprehensive search strategy using multiple sources was used to identify all the relevant studies. Content-specific electronic databases, key journals, Web-based research and knowledge centers, websites, and handsearches of reference lists from studies and reviews were included in the search. The search was run between May 15, 2016, and June 30, 2016, and included all articles published by July 1, 2016. Table 1 presents the literature sources used for this review.

Three sets of search term combinations were used in the search, and they were entered and combined using Boolean operators where applicable. Combination 1 was the most detailed and was used in each electronic database and on the mHealth Evidence website. It included “mobile health,” “mHealth,” “m-Health,” “mobile,” “mobile phone,” “smartphone,” “cellular phone,” “texting,” “text messaging,” “SMS,” “telemedicine,” “telehealth,” “telecare,” “telemonitoring,” “diabetes,” “diabetes mellitus,” “diabetic,” “mixed method,” “multi-method,” and “mix methodology.”

Because some studies do not explicitly indicate their methodologies as mixed methods, mix methodology, or multimethod, combination 2 was applied without using the methodology terms (mixed methods, multimethod, mix methodology). Because of the excessive number of records that resulted in the search with combination 2 in Google Scholar, PubMed, and Web of Science, this search was only completed in CINAHL (Cumulative Index to Nursing and Allied Health Literature) and PsycINFO databases.

Combination 3 used the terms most relevant to our research question. “Mobile health,” “diabetes,” and “mixed methods” search terms were used in US NIDDK (National Institute of Diabetes and Digestive and Kidney Diseases) Diabetes Research Center and in Google Scholar. Wide search options were used in terms of text availability, publication dates, and document formats, and only the English-language and academic journal filters were applied in electronic sources. ResearchGate was only used to search for the authors’ personal files and to obtain full-text articles. Table 2 summarizes the search term combinations used in each electronic source.

Framework Stage 3: Study Selection

Several inclusion and exclusion criteria were applied to the identified studies. The review included the primary research studies published in English-language academic journals that focused on using mHealth technologies in the context of type 1, type 2, or gestational diabetes management and included a combination of qualitative and quantitative methods.

In this review, the term “mobile health technologies” was used to cover mobile phones, portable monitoring devices (ie, accelerometer), and wireless devices used in medical care (ie, cell phones). Studies whose focus was Internet (computer)-based, telephone (landline)-based, or home-based monitoring were excluded. Examples of such studies include a mixed-methods diabetes telemonitoring study [26], using a Web-enabled glucometer for self-monitoring blood glucose, and a home-based monitoring study [27] including a randomized controlled trial and a series of interviews that used a transmission device to be attached to an analog telephone line or via USB to a computer to upload blood glucose and blood pressure measurements to a server. In addition, there were telephone coaching and counseling studies (ie, [28,29]) that did not match the inclusion criteria. The studies related to diabetic retinography were also excluded. Figure 1 presents the article selection process in this review.

‎View this figure

Framework Stage 4: Charting the Data

The descriptive-analytical method [25] was followed in order to standardize and chart key items of information acquired from the reviewed articles. The information collected and charted included the following elements:

• Authors, year of publication
• Main purpose of the study
• Recognition of mixed methods (mixed-methods terminology used or not)
• Purpose of mixing
• Formal mixed-methods research question formulated (yes or no)
• Prioritized method
• Design type
• Stage of integration
• Legitimation methods used
• Sources of qualitative data
• Sources of quantitative data
• Limitations described

Tables 3-5 present the definitions used to analyze the design types (Table 3), integration strategies (Table 4), and legitimation methods (Table 5) in this review.

Results

Framework Stage 5: Collating, Summarizing, and Reporting the Results

Collating and Summarizing the Results

The database search revealed 8 articles; with handsearches of reference lists and key journals, a total of 14 articles were identified. The final sample included studies focusing on feasibility and usability of mHealth diabetes technologies (n=7), behavioral measures related to mHealth apps (n=6), and challenges in patient recruitment, fidelity, and intervention delivery in the context of mHealth (n=1).

For this scoping review, the first author derived the data from the articles and completed the initial coding, which was verified by the second author. In addition, an independent researcher (an experienced researcher and doctoral student in social sciences) separately coded the articles based on the study design framework used in this review. Comparing the results revealed an interrater reliability level of more than 90%.

The characteristics of mixed-methods research used in the reviewed studies and the summary of the findings are presented in Tables 6 and 7. Table 6 presents the data gathered on formal recognition of mixed methods and purpose of using mixed methodology, indicating a formal mixed-methods question, priority of the methods used, stages of integration, and the design type used in the studies. Table 7 lists the data gathered on legitimation methods, qualitative and quantitative sources of data, and limitations as described in the studies.

Reporting the Results

Recognizing Mixed Methods and Purpose of Using Mixed Methods

According to Creswell and Plano Clark [47], mixed-methods studies should have a properly defined methodology and a formal terminology. This review shows that almost all the studies (13/14, 93%) defined their methodologies explicitly as either mixed methods or multimethod.

Only a few studies (4/14, 29%; Verwey et al [45], Georgsson and Staggers [40], Froisland et al [39], and Franklin et al [38]) used mixed-methods terminology or explained its purpose explicitly. For example, Franklin et al [38] stated that their purpose was to triangulate the messaging patterns and contents of an automated, scheduled SMS text messaging with diabetic patients’ clinical and demographic data, and Froisland et al [39] reported that they triangulated the usability assessments of a picture-based diabetes diary app and an SMS (short message service) text messaging with semistructured, in-depth interviews and field notes.

In other studies, information about the purpose of mixing methods had to be extracted from the methodology and discussion sections of the studies. Particularly for the embedded design studies, a decision rule was used based on the phases of a study because they could signify the purpose of mixing the methods in a study [30]. If different measurements using different types of data were addressing different parts of the research phenomena, it was evaluated as complementarity [30]. For example, in an embedded design study, the attitudes of a group of American Indian women toward potential Internet or mHealth interventions were examined using qualitative interviews and focus groups that were complementary to a cross-sectional study assessing their risk perceptions of diabetes [42]. If one method was embedded into the other method and used to compare and validate its results, it was evaluated as triangulation. For example, Nundy et al [43] conducted a longitudinal cohort study and compared and validated the results with in-depth interviews. In addition, if one type of data was used to inform the other type during the sequential development of a study, its mixing purpose was evaluated as sequential development. For example, motivational interviews before the intervention and follow-up phone interviews during the testing period were used in a study focusing on the development and feasibility of a text messaging and interactive voice response intervention [44].

In total, the purpose of mixing methods was triangulation in 7 studies [35,38,39,40,41,43,45], complementarity in 3 studies [33,34,42], sequential development in 3 studies [37,44,46], and initiation in 1 study [36].

Design Types

Because the studies did not use a formal mixed-methods terminology when describing their design types, stages of integration, and priority of the methods they used, information on these categories was extracted from the methodology sections of the studies.

The most common design used by the studies was embedded design (n=7). These studies (ie, [34,39,42,43,45]) were designed to mix different datasets at the design level. For example, Vervey et al [45] conducted a 3-armed cluster randomized controlled trial with nurses and patients to evaluate the physical activity counseling process with and without the use of mobile technology. During the intervention, data were gathered from nurses who interviewed the patients periodically, collected evaluation forms, and logged data. They conducted in-depth interviews with nurses after the trial to compare and cross-validate their findings.

There were 3 concurrent design studies in the review [38,40,41]. In their text messaging intervention, Franklin et al [38] conducted a content analysis and derived qualitative themes from the messaging patterns, which were also analyzed by quantitative methods. In addition, 3 sequential design studies were identified: 1 exploratory [37] and 2 explanatory design studies [33,36]. After completing the qualitative phase and establishing the framework of the research, Carroll et al [37] implemented a usability test of a diabetes monitoring system. On the contrary, Burner et al [36] conducted a focus group study to clarify the results of their quantitative study, measuring behavioral effects of an mHealth intervention in Latino diabetic patients.

The only multiphase design study in this review was the study by van der Weegen et al [46], which measured the usability of a monitoring tool targeting diabetic patients. They conducted different qualitative and quantitative measurements focusing on different aspects of a study including 4 phases.

Stages of Integration

All the studies enabled integration by connecting samples. The studies used either identical or nested sampling methods. On the basis of their designs and time orientation, they used samples sequentially or concurrently. Some studies (ie, [35,38,45,46]) made the integration at multiple points throughout their studies. They largely connected their samples, analyzed the data, merged their datasets, and interpreted their findings with an integrated view.

Some studies also enabled integration through the merging of both datasets in the data analysis stage. In an explanatory sequential design study on diabetes self-management [36], the themes from the qualitative analysis were merged with the demographic variables. In another study [33], descriptive measures of physical activity patterns and glucose levels of patients with type 2 diabetes were merged with the focus group data on the feasibility and acceptability of a new glucose monitoring and accelerometer technology.

The studies also presented various forms of data integration at the stage of interpretation or discussion. They identified themes or concepts and integrated their qualitative and quantitative findings into these themes [35,39,40,42,44,45]. In addition, the findings of a multiphase design study were reported in a series. After each stage was completed, its findings were written separately [46]. In some studies, specific findings were written in the separate parts of their reports and then an overall interpretation of the findings was made [34,38,43].

Prioritized Methods

More than half the studies evaluated their quantitative and qualitative methods equally; neither method was prioritized (8/14, 57%). Some studies with randomized controlled trial data [38,40,45] highlighted their comprehensive qualitative methodologies and their contributions to the results. For example, in order to completely measure young diabetic patients’ engagement with a new text messaging support system, a comprehensive content analysis of messages and message patterns of patients was performed [38]. Interestingly, in an explanatory design study [36], focus group results led researchers to reanalyze their quantitative data.

In another study [44], face-to-face interviews conducted before and after a text messaging trial provided important insights about the intervention’s technical features, tailored text messaging content, and motivational messaging. Researchers were guided to develop a better design for their apps. Three studies [34,35,43] primarily highlighted quantitative methods (QUAN + qual) and used qualitative methods to support quantitative data.

Data Collection Techniques

The most frequently used qualitative method in the studies was in-depth interviews (n=9). The studies conducted semistructured interviews [35,39,42,43,45,46], open-ended interviews [40], and motivational interviews [44]. Several studies conducted focus group meetings to collect data [33,36,37,41,42]. The interview or discussion duration varied from approximately 30 minutes to 90 minutes in these studies. In addition, 2 think aloud protocols [40,46], a content analysis [38], and a video-recording technique [46] were used to generate qualitative data.

Surveys were the most common quantitative data collection method (n=10), such as pre- and posttest questionnaires used to measure the usability of an interactive SMS text messaging intervention developed for patients with diabetes [40]. Perceptions of a group of American Indian women about gestational diabetes risk and related Internet and mHealth interventions were surveyed in a cross-sectional study [42]. In addition, there was a longitudinal cohort study [43] that collected survey data at multiple periods.

Quantitative data were also generated from descriptive measures, such as glucose and exercise levels of patients after a 72-hour trial of a continuous glucose monitoring system [33], usage data of patients after a 12-month trial of a text messaging support system [38], and interactive voice response calls and previous descriptive data [36,44].

The Methods of Legitimation and the Limitations Described

Although the studies in this review explicitly mentioned their procedures and strategies to increase the objectivity and credibility of their research, they did not use formal mixed-methods terminology. Therefore, the information on legitimation methods was extrapolated from the methodology and discussion sections of the studies.

Almost all the studies in this review applied an “inside and outside” legitimation method (13/14, 93%). For example, Nundy et al [43] assigned a team of 5 experienced research investigators to make an initial transcript coding and then randomly assigned 2 reviewers to code each transcript independently. In addition to peer reviews, team members’ views were important means for legitimation. For instance, conducting separate analyses helped the researchers compare their coding results and increased the validity of their study [39].

In addition to inside-outside legitimation, different methods of legitimation were identified in the studies. For instance, the qualitative and quantitative methods were used to complement each other and to minimize weaknesses (weakness minimization method) in a small-scale usability test study [37]. A small number of questionnaires were supported by conducting 10 different focus groups to increase the legitimacy of the results. The use of conversion legitimation was observed in 4 studies [35,43,45,46]. In 1 study [41], sequential legitimation was applied to minimize possible sequential order effects on the results.

Because the studies used either identical or nested sampling methods, sample integration legitimation was another frequently observed method of legitimation. Using similar samples in each phase minimizes lack of representativeness in qualitative findings and increases overall generalizability of the study [32]. For example, Grindrod et al [41] used identical samples in their qualitative and quantitative phases and performed multiple measurements to increase the validity of their small-scale intervention design, which lacked control group measurements.

Although various legitimation methods were used, they were also the source of study limitations. Many of the limitations were related to insufficient sample sizes and possible generalizability problems associated with them (ie, [35,36,38,44,46]). Sample selection problems and possible sample biases were also pointed out [33,39,40,42,43,45]. The duration of intervention [37,39,41], fidelity to the study [42], and other interventional issues, such as incentives (ie, lending cell phones, paying call charges) [37], and technical problems [39] were further limitations.

Discussion

Principal Findings

This scoping review identified comprehensive mixed-methods applications in the growing field of mHealth-based diabetes management. Mixed methods research can be effectively used if there is a strong and appropriate design dictated by research questions [48,49]. Creswell and Tashakkori [50] suggest using at least one clearly defined mixed methods question. Although none of the studies defined a separate mixed-methods research question and a very few studies (4/14, 29%) used formal mixed-methods terminology, the studies showed promise in their uses of multiple approaches and techniques. In addition to the common uses of mixed methods (triangulation and complementarity), studies with advanced designs presented various forms of combining different methods and techniques in sequential development of their studies. These studies are good examples for the field of diabetes management via mHealth technologies in which complicated interventions and multiple measurements are often required.

This study confirms the prominent role of qualitative methods in mixed-methods research. Qualitative measures were very important for the evolution of the majority of the papers reviewed, and they strongly influenced study findings. Because of the growing need for interpreting complex longitudinal studies and interventions [51] and big and sophisticated data coming from Web-based and mobile technology usage statistics, the importance of qualitative methods seems to be increasing.

The studies in this review presented various forms of integration at the method level. Some studies enabled integration at multiple points, linking their data collection and analysis at each point in the research. This is also noteworthy for mixed-methods studies in which integration is generally made at the level of data interpretation or discussion [52].

In terms of validity or legitimation strategies, little variation existed among the studies. In addition to inside-outside legitimation, these advanced designs should have also used weakness minimization and conversion legitimation methods because they are very specific to mixed-methods research.

The major limitations of the studies included in this review were related to data collection processes. Surveys were the most common data collection method used in the studies, but (as opposed to experimental studies) descriptive studies do not show causality, as they have poor control over external factors and external validity problems due to standardized question types. In addition, almost all studies acknowledged sampling problems (13/14, 93%). Small sample sizes, sampling bias, and use of convenient sampling are important barriers to the generalizability of findings in any type of study.

When various data collection techniques are used together, researchers are able to provide rich analysis and interpretations. This study shows that in addition to the common qualitative data collection techniques (focus group discussions and semistructured interviews), multiple data collection techniques (including think aloud protocols, interviews, field notes, content analysis, and evaluation forms) are used.

It is also noteworthy that these studies failed to discuss the advantages gained by using mixed-methods approaches. The majority of the studies (13/14, 93%) failed to provide a methodological discussion concerning the inconsistency or consistency between quantitative and qualitative data. Thus, this study supports Brown et al [24], who indicated that mixed-methods studies still lack justifications for using this approach.

Study Strengths and Limitations

This study is an early exploration into the scope of mixed-methods studies in the field of mHealth-based diabetes management. This paper itself provides a demonstration of how to classify, analyze, and evaluate mixed-methods research. Regardless of field of interest, we believe many researchers could benefit from the guidelines, criteria, and examples provided in this review.

Although the database search was quite comprehensive, it was limited to the papers published in scientific journals and written in English. To increase the consistency of the coding, some established frameworks were used to identify design types [20], integration methods [31], and legitimation methods [32]. Because related information was not always clearly reported in the studies, some decision rules were applied and the most relevant choices presented in the frameworks were adopted. Another limitation is related to the coding process. Although the design types were classified by 2 coders, and a high level of agreement between them was established, the rest of the coding was done by the first author. However, to increase the confirmability of our analysis, she took the role of a “devil's advocate” with respect to the results, checking and rechecking each coding and analysis throughout the study.

Conclusions and Future Recommendations

This review provided important insights into the evaluation of mixed-methods studies focusing on diabetes management via mHealth technologies. The prominent role of qualitative methods in mixed-methods research and tailored measures in diabetes self-management studies was confirmed, and the importance of using multiple methods and techniques in this field was emphasized. Future studies could continue to use different qualitative approaches and try to integrate new tools and approaches that seem to be effective in health care settings, such as participatory video, photovoice, online communities, and chat and discussion groups [3].

Considering the opportunities provided by mixed methods, it is surprising to see such a small number of studies in this field. The lack of systematic approaches in establishing rigor in mixed-methods studies seems to be a major barrier to the adoption and recognition of this research by a large group of social researchers [20]. However, in the field of mHealth-based diabetes management both usability studies and behavioral change interventions require complex measurements, so researchers could utilize the advantages of mixed methods more frequently. Future research should focus on the ways to improve precision in measurements by incorporating experimental designs with properly selected, adequate number of participants and new and creative data collection techniques.

In the future, researchers should incorporate formal mixed-methods terminology, starting with defining a mixed-methods question separately at the beginning, aiming to answer this question, and discussing the value of using the methodology at the end. In this growing field of diabetes management, future reviews should systematically analyze rigor in mixed-methods studies and compare the results with those of single-method studies.