Published on 06.02.17 in Vol 2, No 1 (2017): Jan-Jun
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/6667, first published Sep 19, 2016.
Mixed-Methods Research in Diabetes Management via Mobile Health Technologies: A Scoping Review
Background: Considering the increasing incidence and prevalence of diabetes worldwide and the high level of patient involvement it requires, diabetes self-management is a serious issue. The use of mobile health (mHealth) in diabetes self-management has increased, but so far research has not provided sufficient information about the uses and effectiveness of mHealth-based interventions. Alternative study designs and more rigorous methodologies are needed. Mixed-methods designs may be particularly useful because both diabetes self-management and mHealth studies require integrating theoretical and methodological approaches.
Objective: This scoping review aimed to examine the extent of the use of mixed-methods research in mHealth-based diabetes management studies. The methodological approaches used to conduct mixed-methods studies were analyzed, and implications for future research are provided.
Methods: Guided by Arksey and O’Malley’s framework, this scoping review implemented a comprehensive search strategy including reviewing electronic databases, key journal searches, Web-based research and knowledge centers, websites, and handsearching reference lists of the studies. The studies focusing on mHealth technologies and diabetes management were included in the review if they were primary research papers published in academic journals and reported using a combination of qualitative and quantitative methods. The key data extracted from the reviewed studies include purpose of mixing, design type, stage of integration, methods of legitimation, and data collection techniques.
Results: The final sample (N=14) included studies focused on the feasibility and usability of mHealth diabetes apps (n=7), behavioral measures related to the mHealth apps (n=6), and challenges of intervention delivery in the mHealth context (n=1). Reviewed studies used advanced forms of mixed-methods designs where integration occurred at multiple points and data were collected using multiple techniques. However, the majority of studies did not identify a specific mixed-methods design or use accepted terminology; nor did they justify using this approach.
Conclusions: This review provided important insights into the use of mixed methods in studies focused on diabetes management via mHealth technologies. The prominent role of qualitative methods and tailored measures in diabetes self-management studies was confirmed, and the importance of using multiple techniques and approaches in this field was emphasized. This review suggests defining specific mixed-methods questions, using specific legitimation methods, and developing research designs that overcome sampling and other methodological problems in future studies.
JMIR Diabetes 2017;2(1):e3
The Increasing Need for mHealth and Mixed-Methods Research in Diabetes Management
Increases in diabetes incidence and prevalence are a major concern in today’s health care system. There are nearly 385 million diabetic patients in the world, and almost 90% of them have type 2 diabetes, which can be treated with appropriate lifestyle and nutrition changes. Diabetes is a complicated disease and requires a high level of patient involvement; 95% of its management is patient initiated. Considering the additional 235 million patients with type 2 diabetes expected worldwide by the year 2035, the development of diabetic patients’ self-management skills is critical .
Mobile health (mHealth) apps that offer personalized, fast, cost-effective, and engaging services to patients have been used increasingly in diabetes self-management [, ]. The “disease and treatment” apps category currently includes more than 2250 diabetes apps (iOS and Android systems in total). The number of mobile apps for diabetes is growing rapidly and market penetration is expected to grow to 7.8% by 2018, reaching 24 million diabetic patients [ ].
Despite the increase in the diabetes cases and rapid growth in the mobile app market, research does not yet provide sufficient information about the factors fostering or hindering the adoption of these apps, patients’ attitudes toward using them, or their effectiveness in terms of health behavior change [, , , , ]. Therefore, the controversy over whether research methodologies or mHealth-based interventions are ineffective [ , ] still remains. Alternative study designs and more rigorous methodologies to advance mHealth research, especially in diabetes management, are strongly suggested [ , ].
There is a growing interest in using mixed-methods research in mHealth-based diabetes management studies because both diabetes self-management and mHealth studies require using different approaches, techniques, and measures cooperatively within an integrated perspective [, , ]. This scoping review examined the extent of mixed-methods research used in mHealth-based diabetes management studies. The methodological approaches used to conduct mixed-methods studies were also investigated, and implications for future mHealth and diabetes management research are provided.
A Brief Review of Mixed-Methods Research
Mixed-methods research grew out of the “paradigm” controversy between positivist (quantitative) and constructivist (qualitative) research traditions in the 1980s and 1990s . Mixed-methods research relies on both theory and practice to integrate knowledge from multiple approaches, perspectives, tools, positions, and opinions. The term “method” is used to cover a broad range of methodological (data collection techniques, design types, research methods, and so on) and related philosophical issues (eg, ontology, epistemology, axiology) [ ].
The main goal of mixed-methods research is to use both quantitative and qualitative approaches to provide a better understanding of the research phenomena . Because of the complementary strengths of different research paradigms, methodologies, and methods, mixed-methods research may provide new and different perspectives about an issue, expanding study findings beyond those produced by only one approach. Mixed-methods research increases the credibility of results when different approaches suggest the same conclusion, so it’s a value-added methodology [ , ].
Mixed-methods research is generally used for (1) triangulation of different approaches and methodologies to explain a single phenomenon; (2) complementarity of the research methods where one method is used to elaborate, illustrate, enhance, or clarify the results from another method; (3) sequential development of a study in which the results of one method are used to inform the other; (4) initiation of a study by using one method to find contradictions and paradoxes in findings from another method; and (5) expansion of a research study by using different methods to gain new perspectives or insights about a research problem .
Selecting an appropriate research design is a crucial step in a mixed-methods research. The purpose of a study and the nature of the research question help shape the design used in a mixed-methods study. In addition, using either concurrent or sequential time orientation affects study design, sampling, and data collection.
In a concurrent time orientation, data collection is completed for quantitative and qualitative phases of the study at approximately the same time to answer the same question. Both datasets are processed during data analysis and interpretation stages.
In a sequential time orientation, data can be obtained in stages, so the data from the first stage are used to shape the selection of data in the second stage (exploratory or explanatory sequential design). On the basis of their study purposes, questions, designs, and resources, researchers might place equal emphasis on both methods or use one of them primarily [, ].
Integration is the central issue of mixed-methods research because it enables researchers to examine both types of data intensively . Integration can be achieved at the method level or at the interpretation and reporting level. Bryman [ ] reported that 57% of social science studies using mixed methods collected qualitative and quantitative data separately, while in approximately 27% both quantitative and qualitative data were derived from a single data source (eg, survey questionnaires including closed and open-ended questions). A growing scholarly interest has advanced mixed-methods research in various areas. Especially in health sciences, several important applications of mixed-methods research have been reported [ , ]. However, the majority of these studies failed to provide a detailed description of their data collection techniques, methods of analysis, stages of integration, and justifications for the use of mixed methods [ ].
The 5-stage scoping review framework developed by Arksey and O’Malley  was used to identify and examine the related literature in this review. This framework allows researchers to clearly describe the methods used at each stage to increase the transparency and replicability of the studies.
Framework Stage 1: Identifying the Research Question
The research questions addressed in this review are as follows: (1) what is the extent and nature of mixed-methods research in mHealth-based diabetes management studies and (2) what are the current methodological approaches for designing and conducting mixed methods in these studies?
As a scoping review, this study did not intend to evaluate the scientific rigor of the selected studies as seen in systematic reviews . It aimed to present the different ways in which researchers have used mixed methods.
Framework Stage 2: Identifying Relevant Studies
Because the key purpose of the study was scoping the area of research, a comprehensive search strategy using multiple sources was used to identify all the relevant studies. Content-specific electronic databases, key journals, Web-based research and knowledge centers, websites, and handsearches of reference lists from studies and reviews were included in the search. The search was run between May 15, 2016, and June 30, 2016, and included all articles published by July 1, 2016.presents the literature sources used for this review.
|Source type||Literature sources used|
|Electronic databases||PubMed, CINAHLa, Web of Science, PsycINFO, Google Scholar|
|Key journal search||Journal of Medical Internet Research, The Journal of mHealth, The Diabetes Educator, The Journal of Mixed Methods Research, Journal of Diabetes, Journal of Diabetes Research, Journal of Telemedicine and Telecare, Telemedicine and e-Health|
|Research and knowledge networks||mHealth Evidence |
US NIDDKb Diabetes Research Center, ResearchGate
|Reference lists search||Systematic reviews, meta-analyses, narrative reviews, and similar article searches of the Web-based publishers|
aCINAHL: Cumulative Index to Nursing and Allied Health Literature.
bNIDDK: National Institute of Diabetes and Digestive and Kidney Diseases.
Three sets of search term combinations were used in the search, and they were entered and combined using Boolean operators where applicable. Combination 1 was the most detailed and was used in each electronic database and on the mHealth Evidence website. It included “mobile health,” “mHealth,” “m-Health,” “mobile,” “mobile phone,” “smartphone,” “cellular phone,” “texting,” “text messaging,” “SMS,” “telemedicine,” “telehealth,” “telecare,” “telemonitoring,” “diabetes,” “diabetes mellitus,” “diabetic,” “mixed method,” “multi-method,” and “mix methodology.”
Because some studies do not explicitly indicate their methodologies as mixed methods, mix methodology, or multimethod, combination 2 was applied without using the methodology terms (mixed methods, multimethod, mix methodology). Because of the excessive number of records that resulted in the search with combination 2 in Google Scholar, PubMed, and Web of Science, this search was only completed in CINAHL (Cumulative Index to Nursing and Allied Health Literature) and PsycINFO databases.
Combination 3 used the terms most relevant to our research question. “Mobile health,” “diabetes,” and “mixed methods” search terms were used in US NIDDK (National Institute of Diabetes and Digestive and Kidney Diseases) Diabetes Research Center and in Google Scholar. Wide search options were used in terms of text availability, publication dates, and document formats, and only the English-language and academic journal filters were applied in electronic sources. ResearchGate was only used to search for the authors’ personal files and to obtain full-text articles.summarizes the search term combinations used in each electronic source.
|Search terms||Web of Science||CINAHLa||PubMed||PsycINFO||Google Scholar||mHealth Evidence||US NIDDKb Diabetes Research Center|
aCINAHL: Cumulative Index to Nursing and Allied Health Literature.
bNIDDK: National Institute of Diabetes and Digestive and Kidney Diseases.
Framework Stage 3: Study Selection
Several inclusion and exclusion criteria were applied to the identified studies. The review included the primary research studies published in English-language academic journals that focused on using mHealth technologies in the context of type 1, type 2, or gestational diabetes management and included a combination of qualitative and quantitative methods.
In this review, the term “mobile health technologies” was used to cover mobile phones, portable monitoring devices (ie, accelerometer), and wireless devices used in medical care (ie, cell phones). Studies whose focus was Internet (computer)-based, telephone (landline)-based, or home-based monitoring were excluded. Examples of such studies include a mixed-methods diabetes telemonitoring study , using a Web-enabled glucometer for self-monitoring blood glucose, and a home-based monitoring study [ ] including a randomized controlled trial and a series of interviews that used a transmission device to be attached to an analog telephone line or via USB to a computer to upload blood glucose and blood pressure measurements to a server. In addition, there were telephone coaching and counseling studies (ie, [ , ]) that did not match the inclusion criteria. The studies related to diabetic retinography were also excluded. presents the article selection process in this review.
Framework Stage 4: Charting the Data
The descriptive-analytical method  was followed in order to standardize and chart key items of information acquired from the reviewed articles. The information collected and charted included the following elements:
- Authors, year of publication
- Main purpose of the study
- Recognition of mixed methods (mixed-methods terminology used or not)
- Purpose of mixing
- Formal mixed-methods research question formulated (yes or no)
- Prioritized method
- Design type
- Stage of integration
- Legitimation methods used
- Sources of qualitative data
- Sources of quantitative data
- Limitations described
- present the definitions used to analyze the design types ( ), integration strategies ( ), and legitimation methods ( ) in this review.
|Concurrent||This design aims to compare and contrast the results of both quantitative and qualitative findings, or to validate or expand quantitative results with qualitative data. This design is also labeled as parallel or convergent design. The researchers use different methods complementarily to investigate the same topic.|
|Exploratory||If a study has one dataset built on the results from the other, it is classified as a sequential design. It is a 2-phase mixed-methods design that collects only one type of data at a time. This design aims to explore first by placing a qualitative phase before a quantitative phase, and the data from the first phase are used to develop the second phase.|
|Explanatory||This design aims to clarify or interpret unexpected or confusing results from the quantitative phase with a follow-up qualitative phase. It can also be used to form groups based on quantitative results and monitor the groups through follow-up qualitative research.|
|Embedded||This includes the combination of both quantitative and qualitative data, but one data type has a supportive, secondary role within the overall design. One type of data is embedded within a methodology adapted by the other data type. It can be either a 1- or a 2-phase study. Unlike the conventional mixed-methods researchers, who think both methods should answer the same question in the research, some researchers have different questions requiring different types of data. Some complex interventions and experimental studies need embedded designs because this design is more manageable in terms of time and resources.|
|Multiphase||This design includes multiple qualitative, quantitative, or mixed measurement phases conducted over time and linked together so that one phase builds on another with a common overall objective. They usually include convergent and sequential elements.|
aDeveloped based on the frameworks presented in Creswell , Creswell and Plano Clark [ ], and Creswell et al [ ].
|Connecting||One dataset links to the other through sampling.|
|Building||One dataset informs the data collection approach of the other.|
|Merging||The two datasets are brought together for analysis.|
|Multiple integration||Data collection and analysis are linked at multiple points.|
aAdapted from Fetters et al .
|Inside-outside legitimation||Insider (emic) viewpoint refers to viewpoint of a group member while outsider (etic) viewpoint refers to an objective viewpoint gathered from an external source. Peer reviews, expert reviews, participant views, and team members’ views are used for legitimation.|
|Paradigmatic or philosophical validity||Successful integration of philosophical and methodological beliefs, defining the paradigm assumptions explicitly, and conducting research accordingly are the main indicators of legitimation.|
|Commensurability legitimation||This type of validity is obtained when researchers develop a third, mixed view, which helps them make broader and richer explanations about their study conclusions.|
|Weakness minimization||This type of validity is related to the integration of the research; researchers must continuously work to have nonoverlapping weaknesses while planning and designing their study.|
|Sequential legitimation||This type of validity is used to understand whether the sequential order of qualitative and quantitative phases in a study influences the results.|
|Conversion legitimation||By quantifying the narrative descriptions and creating a narrative profile for quantitative results (qualitizing), researchers can interpret their data in a broader perspective.|
|Sample integration||This type of validity refers to making appropriate generalizations from mixed samples. The relationship between the sampling designs in quantitative and qualitative phases is an important indicator of validity.|
|Sociopolitical legitimation||In order to achieve this type of legitimation, mixed-methods researchers should advocate pluralism of perspectives and try to build a practical theory or result that research consumers will find valuable.|
|Multiple legitimation||This type of validity indicates the extent to which all the pertinent validities (quantitative, qualitative, and mixed) are addressed and resolved successfully.|
aAdapted from Onwuegbuzie and Johnson .
Framework Stage 5: Collating, Summarizing, and Reporting the Results
Collating and Summarizing the Results
The database search revealed 8 articles; with handsearches of reference lists and key journals, a total of 14 articles were identified. The final sample included studies focusing on feasibility and usability of mHealth diabetes technologies (n=7), behavioral measures related to mHealth apps (n=6), and challenges in patient recruitment, fidelity, and intervention delivery in the context of mHealth (n=1).
For this scoping review, the first author derived the data from the articles and completed the initial coding, which was verified by the second author. In addition, an independent researcher (an experienced researcher and doctoral student in social sciences) separately coded the articles based on the study design framework used in this review. Comparing the results revealed an interrater reliability level of more than 90%.
The characteristics of mixed-methods research used in the reviewed studies and the summary of the findings are presented inand . presents the data gathered on formal recognition of mixed methods and purpose of using mixed methodology, indicating a formal mixed-methods question, priority of the methods used, stages of integration, and the design type used in the studies. lists the data gathered on legitimation methods, qualitative and quantitative sources of data, and limitations as described in the studies.
|Author, year||Main purpose of the study||Recognition of MMa||Purpose of mixing||Formal MM research question||Prioritized method||Stage of integration||Design type|
|Allen et al, 2009 ||To assess feasibility and acceptability of continuous glucose monitoring and accelerometer technology in exercising type 2 diabetic patients||Yes, as multimethod||Complementarity||No||Equal||Sampling and interpretation||Explanatory sequential|
|Baron et al, 2015 ||To identify the challenges related to recruitment, fidelity, implementation, and context of mobile telehealth interventions targeting diabetic patients||Yes||Complementarity: different measures for different parts of the research phenomenon||No||QUANb||Sampling, data collection, data analysis, and interpretation||Embedded|
|Baron et al, 2016 ||To examine the behavioral effects of a mobile phone–based home telehealth intervention in diabetic patients||No||Triangulation||No||QUAN||Sampling and interpretation||Embedded|
|Burner et al, 2013 ||To explore the attitudes of inner-city Latino patients toward TExT-MEDd program and other health information sources||Yes||Initiation: the qualitative study was conducted to understand the contradictory findings of the quantitative method||No||Equal||Sampling and interpretation||Explanatory sequential|
|Carroll et al, 2007 ||To evaluate user satisfaction with an mHealth diabetes monitoring system||Yes||Sequential development||No||QUAN||Discussion or interpretation||Exploratory sequential|
|Franklin et al, 2008 ||To explore the interactions of type 1 patients with SMS text messaging support||Yes||Triangulation||No||Equal||Sampling, data analysis, and interpretation||Concurrent|
|Froisland et al, 2012 ||To explore ways mobile apps can be used to monitor adolescents with type 1 diabetes||Yes||Triangulation||No||QUALc||Sampling, data collection, and interpretation||Embedded|
|Georgsson and Staggers, 2016 ||To test the feasibility of a multimethod approach for patients’ experienced usability of a diabetes mHealth system||Yes, as multimethod||Triangulation||No||Equal||Sampling, data analyses, and interpretation||Concurrent|
|Grindrod et al, 2014 ||To examine the usability and usefulness of mobile medication apps with older adults||Yes||Triangulation||No||Equal||Sampling, interpretation, and data analysis||Concurrent|
|Jones et al, 2015 ||To evaluate the attitudes of American Indian women toward postpartum intervention approaches (including mHealth) and risk factors for developing gestational diabetes||Yes||Complementarity: different measures for different parts of the research phenomenon||No||QUAL||Sampling and interpretation||Embedded|
|Nundy et al, 2014 ||To investigate the behavioral effects of a theory-driven mobile phone–based intervention using an automated, interactive SMS text messaging system||Yes||Triangulation||No||QUAN||Sampling and interpretation||Embedded design|
|Osborn and Mulvaney, 2013 ||To examine the capability of an SMS text messaging and interactive voice response intervention for low-income adults with type 2 diabetes mellitus||Yes||Sequential development||No||Equal||Sampling, data analysis, and interpretation||Embedded|
|Verwey et al, 2016 ||To examine the reach, implementation, and satisfaction with a counseling tool combining an accelerometer, a mobile phone, and a Web application||Yes||Triangulation||No||Equal||Sampling, interpretation, and data analysis||Embedded|
|van der Weegen et al, 2014 ||To test the usability of a monitoring and feedback tool targeting diabetic patients||Yes||Sequential development||No||Equal||Sampling, data collection, data analysis, and interpretation||Multiphase study|
aMM: mixed methods.
dTExT-MED: Trial to Examine Text Message–Based mHealth in Emergency Department Patients With Diabetes.
|Author, year||Legitimation methods described||Sources of qualitative data||Sources of quantitative data||Limitations described|
|Allen et al, 2009 ||Inside-outside legitimation||A 1-hour, structured, focus group interview (n=7) following the completion of the quantitative phase. Field notes were also taken on key discussion points and observations (eg, body language and group mood).||Descriptive measures of the sample (n=9). Wearable Continuous Glucose Monitoring System, activity monitor data, and activity counts reviewed for each participant.||Small sample size and lack of control group setting in quantitative phase were reported.|
|Baron et al, 2015 ||Inside-outside legitimation||Interviews, meetings, field notes, and communications between team members.||A 9-month randomized controlled trial (n=81) to assess intervention delivery and fidelity (patients and nurses).||A possible sample selection bias was indicated.|
|Baron et al, 2016 ||Inside-outside legitimation and conversion legitimation (quantizing the qualitative data)||Semistructured interviews (n=26) on perceived effects of mobile telehealth system on diabetes self-management.||A randomized controlled trial (n=81) with intervention and control groups was conducted. Self-report measures of self-efficacy, illness beliefs, and self-care were taken at baseline and 3- and 9-month points.||Sample size was indicated as insufficient to make generalizations.|
|Burner et al, 2013 ||Inside-outside legitimation||Two focus groups of 90-minute duration, one in English, one in Spanish (n=8), were conducted with a structured guide.||A 1-month bilingual diabetes SMS text messaging intervention (n=23). Quantitative data included demographic, clinical, and biometric data of patients, and measures of health behaviors, knowledge, and beliefs were taken.||Small sample size was indicated as a limitation to the generalizability of the results.|
|Carroll et al, 2007 ||Weakness minimization legitimation: large focus groups to support small-scale usability test||A series of focus groups (10; n=59) was conducted before testing a prototype cell phone with a glucose monitoring system.||A pilot usability test to evaluate satisfaction with the new system (n=10). A 15-item questionnaire is used.||Sample size and sample selection, intervention duration, and incentives to the participants were seen as barriers to generalizability of findings.|
|Franklin et al, 2008 ||Inside-outside legitimation and conversion legitimation||Content analysis of text messages and messaging patterns of a 12-month Sweet Talk intervention period.||A 12-month 3-armed randomized controlled trial of a text messaging support system, Sweet Talk (n=64), was conducted. Observational data on messaging patterns were triangulated with patient clinical and demographic data. Post hoc analyses combining qualitative data and demographic variables were made.||Small sample size was indicated as a limitation to generalizability of the results.|
|Froisland et al, 2012 ||Inside-outside legitimation||Semistructured in-depth interviews lasting between 45 and 90 minutes (n=12) were conducted at the end of the quantitative phase.||A pilot test of 2 mobile apps (n=12), after a 3-month trial.||Possible sampling bias, small sample size, and short intervention period were indicated.|
|Georgsson and Staggers, 2016 ||Inside-outside legitimation, conversion legitimation||Think aloud protocol and open-ended interviews (15-20 minutes) were conducted (n=20).||First, a brief demographic questionnaire, and, at the end of the intervention, a posttest questionnaire measuring the usability of an interactive SMSa- text messaging system for a randomly selected sample of patients with diabetes (n=10) were conducted.||Using a convenient sample frame and the novelty of the system to patients were indicated as a limitation to the generalizability of the findings.|
|Grindrod et al, 2014 ||Outside legitimation, sequential legitimation, sample integration legitimation||A 10-minute group discussion of what medication management meant before each usability evaluation and 30-minute focus group discussion after each session (n=35).||A 2-hour usability testing (n=35) of different mobile apps using a 10-item system usability scale and a visual analog scale was used.||Short intervention period was indicated as a limitation of the study.|
|Jones et al, 2015 ||Inside legitimation||Four focus groups consisting of 2-5 participants (n=11 in total) were conducted, maximum duration of 60 minutes. Individual interviews (n=15) ranged from 25-45 minutes.||A cross-sectional study (n=26) was conducted with eligible group of patients. The questionnaire included measures for personal and family health history and technology feasibility and acceptability.||The purposive sampling and small sample size were seen as barriers to the generalizability of the results.|
|Nundy et al, 2014 ||Inside-outside legitimation||Approximately 1-hour, semistructured, in-depth interviews (n=14) based on topic guides and open-ended questions after the intervention.||A longitudinal observational cohort study (n=74) was conducted and data were collected at baseline, 3 months (mid-intervention), and 6 months (end of intervention). Diabetes self-care, medication adherence, self-efficacy, health beliefs, and social support measures were used.||The limitations were described as the lack of control group measure and sole use of SMS text messaging intervention, which may create a causality problem. Sample size was small to make proper generalizations.|
|Osborn and Mulvaney, 2013 ||Inside-outside legitimation||Motivational interviewing, face-to-face interviews (n=20) before and after trial, at baseline, and after week 3.||Secondary research: previous descriptive data on target population obtained, and self-administered daily text messages and interactive voice response calls are collected for analysis (n=20).||Sampling size was indicated as small to generalize results.|
|Verwey et al, 2016 ||Inside and outside legitimation, conversion legitimation, sample integration||30-Minute semistructured telephone interviews with the nurses about the receipt of intervention and the evaluation forms regarding consultations were used.||A longitudinal 3-armed cluster randomized controlled trial in a total of 24 family practice locations; evaluation questionnaire after intervention with practice nurses (n=20) and patients (n=131; 71 with type 2 diabetes and 42 with chronic obstructive pulmonary disease).||A possible sample bias was indicated as a limitation of the study.|
|van der Weegen et al, 2014 ||Multiple legitimation: weakness minimization, conversion legitimation, inside-outside legitimation||Heuristic evaluation with 6 experts, thinking aloud procedure and video recordings of 5 patients at two different stages, a series of interviews with patients in the pilot test.||A usability test with 5 patients, a pilot test in real-life settings with 20 patients, and a poststudy system usability test were conducted.||The small sample size was indicated as a limitation to the generalizability of the findings.|
aSMS: short message service.
Reporting the Results
Recognizing Mixed Methods and Purpose of Using Mixed Methods
According to Creswell and Plano Clark , mixed-methods studies should have a properly defined methodology and a formal terminology. This review shows that almost all the studies (13/14, 93%) defined their methodologies explicitly as either mixed methods or multimethod.
Only a few studies (4/14, 29%; Verwey et al , Georgsson and Staggers [ ], Froisland et al [ ], and Franklin et al [ ]) used mixed-methods terminology or explained its purpose explicitly. For example, Franklin et al [ ] stated that their purpose was to triangulate the messaging patterns and contents of an automated, scheduled SMS text messaging with diabetic patients’ clinical and demographic data, and Froisland et al [ ] reported that they triangulated the usability assessments of a picture-based diabetes diary app and an SMS (short message service) text messaging with semistructured, in-depth interviews and field notes.
In other studies, information about the purpose of mixing methods had to be extracted from the methodology and discussion sections of the studies. Particularly for the embedded design studies, a decision rule was used based on the phases of a study because they could signify the purpose of mixing the methods in a study . If different measurements using different types of data were addressing different parts of the research phenomena, it was evaluated as complementarity [ ]. For example, in an embedded design study, the attitudes of a group of American Indian women toward potential Internet or mHealth interventions were examined using qualitative interviews and focus groups that were complementary to a cross-sectional study assessing their risk perceptions of diabetes [ ]. If one method was embedded into the other method and used to compare and validate its results, it was evaluated as triangulation. For example, Nundy et al [ ] conducted a longitudinal cohort study and compared and validated the results with in-depth interviews. In addition, if one type of data was used to inform the other type during the sequential development of a study, its mixing purpose was evaluated as sequential development. For example, motivational interviews before the intervention and follow-up phone interviews during the testing period were used in a study focusing on the development and feasibility of a text messaging and interactive voice response intervention [ ].
In total, the purpose of mixing methods was triangulation in 7 studies [, , , , , , ], complementarity in 3 studies [ , , ], sequential development in 3 studies [ , , ], and initiation in 1 study [ ].
Because the studies did not use a formal mixed-methods terminology when describing their design types, stages of integration, and priority of the methods they used, information on these categories was extracted from the methodology sections of the studies.
The most common design used by the studies was embedded design (n=7). These studies (ie, [, , , , ]) were designed to mix different datasets at the design level. For example, Vervey et al [ ] conducted a 3-armed cluster randomized controlled trial with nurses and patients to evaluate the physical activity counseling process with and without the use of mobile technology. During the intervention, data were gathered from nurses who interviewed the patients periodically, collected evaluation forms, and logged data. They conducted in-depth interviews with nurses after the trial to compare and cross-validate their findings.
There were 3 concurrent design studies in the review [, , ]. In their text messaging intervention, Franklin et al [ ] conducted a content analysis and derived qualitative themes from the messaging patterns, which were also analyzed by quantitative methods. In addition, 3 sequential design studies were identified: 1 exploratory [ ] and 2 explanatory design studies [ , ]. After completing the qualitative phase and establishing the framework of the research, Carroll et al [ ] implemented a usability test of a diabetes monitoring system. On the contrary, Burner et al [ ] conducted a focus group study to clarify the results of their quantitative study, measuring behavioral effects of an mHealth intervention in Latino diabetic patients.
The only multiphase design study in this review was the study by van der Weegen et al , which measured the usability of a monitoring tool targeting diabetic patients. They conducted different qualitative and quantitative measurements focusing on different aspects of a study including 4 phases.
Stages of Integration
All the studies enabled integration by connecting samples. The studies used either identical or nested sampling methods. On the basis of their designs and time orientation, they used samples sequentially or concurrently. Some studies (ie, [, , , ]) made the integration at multiple points throughout their studies. They largely connected their samples, analyzed the data, merged their datasets, and interpreted their findings with an integrated view.
Some studies also enabled integration through the merging of both datasets in the data analysis stage. In an explanatory sequential design study on diabetes self-management , the themes from the qualitative analysis were merged with the demographic variables. In another study [ ], descriptive measures of physical activity patterns and glucose levels of patients with type 2 diabetes were merged with the focus group data on the feasibility and acceptability of a new glucose monitoring and accelerometer technology.
The studies also presented various forms of data integration at the stage of interpretation or discussion. They identified themes or concepts and integrated their qualitative and quantitative findings into these themes [, , , , , ]. In addition, the findings of a multiphase design study were reported in a series. After each stage was completed, its findings were written separately [ ]. In some studies, specific findings were written in the separate parts of their reports and then an overall interpretation of the findings was made [ , , ].
More than half the studies evaluated their quantitative and qualitative methods equally; neither method was prioritized (8/14, 57%). Some studies with randomized controlled trial data [, , ] highlighted their comprehensive qualitative methodologies and their contributions to the results. For example, in order to completely measure young diabetic patients’ engagement with a new text messaging support system, a comprehensive content analysis of messages and message patterns of patients was performed [ ]. Interestingly, in an explanatory design study [ ], focus group results led researchers to reanalyze their quantitative data.
In another study , face-to-face interviews conducted before and after a text messaging trial provided important insights about the intervention’s technical features, tailored text messaging content, and motivational messaging. Researchers were guided to develop a better design for their apps. Three studies [ , , ] primarily highlighted quantitative methods (QUAN + qual) and used qualitative methods to support quantitative data.
Data Collection Techniques
The most frequently used qualitative method in the studies was in-depth interviews (n=9). The studies conducted semistructured interviews [, , , , , ], open-ended interviews [ ], and motivational interviews [ ]. Several studies conducted focus group meetings to collect data [ , , , , ]. The interview or discussion duration varied from approximately 30 minutes to 90 minutes in these studies. In addition, 2 think aloud protocols [ , ], a content analysis [ ], and a video-recording technique [ ] were used to generate qualitative data.
Surveys were the most common quantitative data collection method (n=10), such as pre- and posttest questionnaires used to measure the usability of an interactive SMS text messaging intervention developed for patients with diabetes . Perceptions of a group of American Indian women about gestational diabetes risk and related Internet and mHealth interventions were surveyed in a cross-sectional study [ ]. In addition, there was a longitudinal cohort study [ ] that collected survey data at multiple periods.
Quantitative data were also generated from descriptive measures, such as glucose and exercise levels of patients after a 72-hour trial of a continuous glucose monitoring system , usage data of patients after a 12-month trial of a text messaging support system [ ], and interactive voice response calls and previous descriptive data [ , ].
The Methods of Legitimation and the Limitations Described
Although the studies in this review explicitly mentioned their procedures and strategies to increase the objectivity and credibility of their research, they did not use formal mixed-methods terminology. Therefore, the information on legitimation methods was extrapolated from the methodology and discussion sections of the studies.
Almost all the studies in this review applied an “inside and outside” legitimation method (13/14, 93%). For example, Nundy et al  assigned a team of 5 experienced research investigators to make an initial transcript coding and then randomly assigned 2 reviewers to code each transcript independently. In addition to peer reviews, team members’ views were important means for legitimation. For instance, conducting separate analyses helped the researchers compare their coding results and increased the validity of their study [ ].
In addition to inside-outside legitimation, different methods of legitimation were identified in the studies. For instance, the qualitative and quantitative methods were used to complement each other and to minimize weaknesses (weakness minimization method) in a small-scale usability test study . A small number of questionnaires were supported by conducting 10 different focus groups to increase the legitimacy of the results. The use of conversion legitimation was observed in 4 studies [ , , , ]. In 1 study [ ], sequential legitimation was applied to minimize possible sequential order effects on the results.
Because the studies used either identical or nested sampling methods, sample integration legitimation was another frequently observed method of legitimation. Using similar samples in each phase minimizes lack of representativeness in qualitative findings and increases overall generalizability of the study . For example, Grindrod et al [ ] used identical samples in their qualitative and quantitative phases and performed multiple measurements to increase the validity of their small-scale intervention design, which lacked control group measurements.
Although various legitimation methods were used, they were also the source of study limitations. Many of the limitations were related to insufficient sample sizes and possible generalizability problems associated with them (ie, [, , , , ]). Sample selection problems and possible sample biases were also pointed out [ , , , , , ]. The duration of intervention [ , , ], fidelity to the study [ ], and other interventional issues, such as incentives (ie, lending cell phones, paying call charges) [ ], and technical problems [ ] were further limitations.
This scoping review identified comprehensive mixed-methods applications in the growing field of mHealth-based diabetes management. Mixed methods research can be effectively used if there is a strong and appropriate design dictated by research questions [, ]. Creswell and Tashakkori [ ] suggest using at least one clearly defined mixed methods question. Although none of the studies defined a separate mixed-methods research question and a very few studies (4/14, 29%) used formal mixed-methods terminology, the studies showed promise in their uses of multiple approaches and techniques. In addition to the common uses of mixed methods (triangulation and complementarity), studies with advanced designs presented various forms of combining different methods and techniques in sequential development of their studies. These studies are good examples for the field of diabetes management via mHealth technologies in which complicated interventions and multiple measurements are often required.
This study confirms the prominent role of qualitative methods in mixed-methods research. Qualitative measures were very important for the evolution of the majority of the papers reviewed, and they strongly influenced study findings. Because of the growing need for interpreting complex longitudinal studies and interventions  and big and sophisticated data coming from Web-based and mobile technology usage statistics, the importance of qualitative methods seems to be increasing.
The studies in this review presented various forms of integration at the method level. Some studies enabled integration at multiple points, linking their data collection and analysis at each point in the research. This is also noteworthy for mixed-methods studies in which integration is generally made at the level of data interpretation or discussion .
In terms of validity or legitimation strategies, little variation existed among the studies. In addition to inside-outside legitimation, these advanced designs should have also used weakness minimization and conversion legitimation methods because they are very specific to mixed-methods research.
The major limitations of the studies included in this review were related to data collection processes. Surveys were the most common data collection method used in the studies, but (as opposed to experimental studies) descriptive studies do not show causality, as they have poor control over external factors and external validity problems due to standardized question types. In addition, almost all studies acknowledged sampling problems (13/14, 93%). Small sample sizes, sampling bias, and use of convenient sampling are important barriers to the generalizability of findings in any type of study.
When various data collection techniques are used together, researchers are able to provide rich analysis and interpretations. This study shows that in addition to the common qualitative data collection techniques (focus group discussions and semistructured interviews), multiple data collection techniques (including think aloud protocols, interviews, field notes, content analysis, and evaluation forms) are used.
It is also noteworthy that these studies failed to discuss the advantages gained by using mixed-methods approaches. The majority of the studies (13/14, 93%) failed to provide a methodological discussion concerning the inconsistency or consistency between quantitative and qualitative data. Thus, this study supports Brown et al , who indicated that mixed-methods studies still lack justifications for using this approach.
Study Strengths and Limitations
This study is an early exploration into the scope of mixed-methods studies in the field of mHealth-based diabetes management. This paper itself provides a demonstration of how to classify, analyze, and evaluate mixed-methods research. Regardless of field of interest, we believe many researchers could benefit from the guidelines, criteria, and examples provided in this review.
Although the database search was quite comprehensive, it was limited to the papers published in scientific journals and written in English. To increase the consistency of the coding, some established frameworks were used to identify design types , integration methods [ ], and legitimation methods [ ]. Because related information was not always clearly reported in the studies, some decision rules were applied and the most relevant choices presented in the frameworks were adopted. Another limitation is related to the coding process. Although the design types were classified by 2 coders, and a high level of agreement between them was established, the rest of the coding was done by the first author. However, to increase the confirmability of our analysis, she took the role of a “devil's advocate” with respect to the results, checking and rechecking each coding and analysis throughout the study.
Conclusions and Future Recommendations
This review provided important insights into the evaluation of mixed-methods studies focusing on diabetes management via mHealth technologies. The prominent role of qualitative methods in mixed-methods research and tailored measures in diabetes self-management studies was confirmed, and the importance of using multiple methods and techniques in this field was emphasized. Future studies could continue to use different qualitative approaches and try to integrate new tools and approaches that seem to be effective in health care settings, such as participatory video, photovoice, online communities, and chat and discussion groups .
Considering the opportunities provided by mixed methods, it is surprising to see such a small number of studies in this field. The lack of systematic approaches in establishing rigor in mixed-methods studies seems to be a major barrier to the adoption and recognition of this research by a large group of social researchers . However, in the field of mHealth-based diabetes management both usability studies and behavioral change interventions require complex measurements, so researchers could utilize the advantages of mixed methods more frequently. Future research should focus on the ways to improve precision in measurements by incorporating experimental designs with properly selected, adequate number of participants and new and creative data collection techniques.
In the future, researchers should incorporate formal mixed-methods terminology, starting with defining a mixed-methods question separately at the beginning, aiming to answer this question, and discussing the value of using the methodology at the end. In this growing field of diabetes management, future reviews should systematically analyze rigor in mixed-methods studies and compare the results with those of single-method studies.
We would like to thank our JMIR DIabetes reviewers for their valuable comments and contributions.
Conflicts of Interest
- Conway N, Campbell I, Forbes P, Cunningham S, Wake D. mHealth applications for diabetes: user preference and implications for app development. Health Informatics J 2016 Dec;22(4):1111-1120. [CrossRef] [Medline]
- Hood M, Wilson R, Corsica J, Bradley L, Chirinos D, Vivo A. What do we know about mobile applications for diabetes self-management? A review of reviews. J Behav Med 2016 Dec;39(6):981-994. [CrossRef] [Medline]
- Lapointe L, Mignerat M, Vedel I. Using telecare for diabetic patients: a mixed systematic review. Health Policy Technol 2014;3(2):90-112.
- Research2guidance. 2015. mHealth App Developer Economics 2015 URL: http://research2guidance.com/product/mhealth-developer-economics-2015 [accessed 2017-01-16] [WebCite Cache]
- Free C, Phillips G, Galli L, Watson L, Felix L, Edwards P, et al. The effectiveness of mobile-health technology-based health behaviour change or disease management interventions for health care consumers: a systematic review. PLoS Med 2013;10(1):e1001362 [FREE Full text] [CrossRef] [Medline]
- Jones KR, Lekhak N, Kaewluang N. Using mobile phones and short message service to deliver self-management interventions for chronic conditions: a meta-review. Worldviews Evid Based Nurs 2014 Apr;11(2):81-88. [CrossRef] [Medline]
- Or CK, Tao D. Does the use of consumer health information technology improve outcomes in the patient self-management of diabetes? A meta-analysis and narrative review of randomized controlled trials. Int J Med Inform 2014 May;83(5):320-329. [CrossRef] [Medline]
- Arnhold M, Quade M, Kirch W. Mobile applications for diabetics: a systematic review and expert-based usability evaluation considering the special requirements of diabetes patients age 50 years or older. J Med Internet Res 2014;16(4):e104 [FREE Full text] [CrossRef] [Medline]
- Martin T. Assessing mHealth: opportunities and barriers to patient engagement. J Health Care Poor Underserved 2012 Aug;23(3):935-941. [CrossRef] [Medline]
- Cole-Lewis H. mHealthy behavior studies: Lessons from a systematic review. In: Donner J, Mecheal P, editors. mHealth in practice: Mobile technology for health promotion in the developing world. Bloomsbury Open Access: Bloomsbury Academic; 2013:32-47.
- Arambepola C, Ricci-Cabello I, Manikavasagam P, Roberts N, French DP, Farmer A. The impact of automated brief messages promoting lifestyle changes delivered via mobile devices to people with type 2 diabetes: a systematic literature review and meta-analysis of controlled trials. J Med Internet Res 2016;18(4):e86 [FREE Full text] [CrossRef] [Medline]
- Serrano-Gil M, Jacob S. Engaging and empowering patients to manage their type 2 diabetes, Part I: a knowledge, attitude, and practice gap? Adv Ther 2010 Jun;27(6):321-333. [CrossRef] [Medline]
- Matthew-Maich N, Harris L, Ploeg J, Markle-Reid M, Valaitis R, Ibrahim S, et al. Designing, implementing, and evaluating mobile health technologies for managing chronic conditions in older adults: a scoping review. JMIR Mhealth Uhealth 2016 Jun 09;4(2):e29 [FREE Full text] [CrossRef] [Medline]
- Muncey T. Does mixed methods constitute a change in paradigm? In: Andrew S, Halcomb EJ, editors. Mixed Methods Research for Nursing and the Health Sciences. Oxford, UK: Wiley-Blackwell; 2009:13-30.
- Johnson RB, Onwuegbuzie AJ, Turner LA. Toward a definition of mixed methods research. J Mix Methods Res 2007 Apr;1(2):112-133. [CrossRef]
- Johnson B, Christensen L. Educational Research: Quantitative, qualitative and mixed approaches. Thousand Oaks, CA: Sage Publications; 2014:978.
- Carroll LJ, Rothe JP. Levels of reconstruction as complementarity in mixed methods research: a social theory-based conceptual framework for integrating qualitative and quantitative research. Int J Environ Res Public Health 2010 Sep;7(9):3478-3488 [FREE Full text] [CrossRef] [Medline]
- Pluye P, Hong QN. Combining the power of stories and the power of numbers: mixed methods research and mixed studies reviews. Annu Rev Public Health 2014;35:29-45. [CrossRef] [Medline]
- Greene JC, Caracelli VJ, Graham WF. Toward a conceptual framework for mixed-method evaluation designs. Educ Eval Policy Anal 1989 Jan 01;11(3):255-274. [CrossRef]
- Creswell J. A Concise Introduction to Mixed Methods Research. Thousand Oaks, CA: Sage Publications; 2015:152.
- Bryman A. Integrating quantitative and qualitative research: how is it done? Qual Res 2006 Feb 01;6(1):97-113. [CrossRef]
- Creswell JW, Klassen AC, Plano Clark VL, Smith KC. Jabsom.hawaii. Best practices for mixed methods research in the health sciences URL: https://www2.jabsom.hawaii.edu/native/docs/tsudocs/Best_Practices_for_Mixed_Methods_Research_Aug2011.pdf [accessed 2017-01-19] [WebCite Cache]
- Zhang W, Creswell J. The use of “mixing” procedure of mixed methods in health services research. Med Care 2013 Aug;51(8):e51-e57. [CrossRef] [Medline]
- Brown KM, Elliott SJ, Leatherdale ST, Robertson-Wilson J. Searching for rigour in the reporting of mixed methods population health research: a methodological review. Health Educ Res 2015 Dec;30(6):811-839 [FREE Full text] [CrossRef] [Medline]
- Arksey H, O'Malley L. Scoping studies: towards a methodological framework. Int J Soc Res Methodol 2005 Feb;8(1):19-32. [CrossRef]
- Lee JY, Lee SW, Nasir NH, How S, Tan CS, Wong CP. Diabetes telemonitoring reduces the risk of hypoglycaemia during Ramadan: a pilot randomized controlled study. Diabet Med 2015 Dec;32(12):1658-1661. [CrossRef] [Medline]
- Koopman RJ, Wakefield BJ, Johanning JL, Keplinger LE, Kruse RL, Bomar M, et al. Implementing home blood glucose and blood pressure telemonitoring in primary care practices for patients with diabetes: lessons learned. Telemed J E Health 2014 Mar;20(3):253-260 [FREE Full text] [CrossRef] [Medline]
- Plotnikoff RC, Johnson ST, Luchak M, Pollock C, Holt NL, Leahy A, et al. Peer telephone counseling for adults with type 2 diabetes mellitus: a case-study approach to inform the design, development, and evaluation of programs targeting physical activity. Diabetes Educ 2010;36(5):717-729. [CrossRef] [Medline]
- McGloin H, Timmins F, Coates V, Boore J. A case study approach to the examination of a telephone-based health coaching intervention in facilitating behaviour change for adults with Type 2 diabetes. J Clin Nurs 2015 May;24(9-10):1246-1257. [CrossRef] [Medline]
- Creswell J, Plano Clark VL. Designing and Conducting Mixed Methods Research. Thousand Oaks, CA: Sage Publications; 2006:296.
- Fetters MD, Curry LA, Creswell JW. Achieving integration in mixed methods designs-principles and practices. Health Serv Res 2013 Dec;48(6 Pt 2):2134-2156 [FREE Full text] [CrossRef] [Medline]
- Onwuegbuzie AJ, Johnson RB. The validity issue in mixed research. Res Sch 2006;13(1):48-63 [FREE Full text]
- Allen NA, Jacelon CS, Chipkin SR. Feasibility and acceptability of continuous glucose monitoring and accelerometer technology in exercising individuals with type 2 diabetes. J Clin Nurs 2009 Feb;18(3):373-383 [FREE Full text] [CrossRef] [Medline]
- Baron J, Hirani S, Newman S. Challenges in patient recruitment, implementation, and fidelity in a mobile telehealth study. Telemed J E Health 2016 May;22(5):400-409. [CrossRef] [Medline]
- Baron JS, Hirani SP, Newman SP. Investigating the behavioural effects of a mobile-phone based home telehealth intervention in people with insulin-requiring diabetes: results of a randomized controlled trial with patient interviews. J Telemed Telecare 2016 Jul 03. [CrossRef] [Medline]
- Burner E, Menchine M, Taylor E, Arora S. Gender differences in diabetes self-management: a mixed-methods analysis of a mobile health intervention for inner-city Latino patients. J Diabetes Sci Technol 2013 Jan 01;7(1):111-118 [FREE Full text] [Medline]
- Carroll AE, Marrero DG, Downs SM. The HealthPia GlucoPack Diabetes phone: a usability study. Diabetes Technol Ther 2007 Apr;9(2):158-164. [CrossRef] [Medline]
- Franklin VL, Greene A, Waller A, Greene SA, Pagliari C. Patients' engagement with “Sweet Talk” - a text messaging support system for young people with diabetes. J Med Internet Res 2008;10(2):e20 [FREE Full text] [CrossRef] [Medline]
- Frøisland DH, Arsand E, Skårderud F. Improving diabetes care for young people with type 1 diabetes through visual learning on mobile phones: mixed-methods study. J Med Internet Res 2012;14(4):e111 [FREE Full text] [CrossRef] [Medline]
- Georgsson M, Staggers N. An evaluation of patients' experienced usability of a diabetes mHealth system using a multi-method approach. J Biomed Inform 2016 Feb;59:115-129 [FREE Full text] [CrossRef] [Medline]
- Grindrod KA, Li M, Gates A. Evaluating user perceptions of mobile medication management applications with older adults: a usability study. JMIR Mhealth Uhealth 2014;2(1):e11 [FREE Full text] [CrossRef] [Medline]
- Jones EJ, Peercy M, Woods JC, Parker SP, Jackson T, Mata SA, et al. Identifying postpartum intervention approaches to reduce cardiometabolic risk among American Indian women with prior gestational diabetes, Oklahoma, 2012-2013. Prev Chronic Dis 2015 Apr 02;12:E45 [FREE Full text] [CrossRef] [Medline]
- Nundy S, Mishra A, Hogan P, Lee SM, Solomon MC, Peek ME. How do mobile phone diabetes programs drive behavior change? Evidence from a mixed methods observational cohort study. Diabetes Educ 2014;40(6):806-819 [FREE Full text] [CrossRef] [Medline]
- Osborn CY, Mulvaney SA. Development and feasibility of a text messaging and interactive voice response intervention for low-income, diverse adults with type 2 diabetes mellitus. J Diabetes Sci Technol 2013 May;7(3):612-622 [FREE Full text] [Medline]
- Verwey R, van der Weegen S, Spreeuwenberg M, Tange H, van der Weijden T, de Witte L. Process evaluation of physical activity counselling with and without the use of mobile technology: a mixed methods study. Int J Nurs Stud 2016 Jan;53:3-16. [CrossRef] [Medline]
- van der Weegen S, Verwey R, Tange HJ, Spreeuwenberg MD, de Witte LP. Usability testing of a monitoring and feedback tool to stimulate physical activity. Patient Prefer Adherence 2014;8:311-322 [FREE Full text] [CrossRef] [Medline]
- Creswell J, Plano Clark VL. Designing and Conducting Mixed-Methods Research. Thousand Oaks, CA: Sage Publications; 2011:488.
- Teddlie C, Tashakkori A. Methodological thought since the 20th century. In: Teddlie C, Tashakkori A, editors. Foundations of Mixed Methods Research: Integrating Qualitative and Quantitative Approaches in The Social and Behavioral Sciences. Thousand Oaks, CA: Sage Publications; 2009:62-88.
- Bergman MM. Quality of inferences in mixed methods research: Calling for an integrative framework. In: Bergman MM, editor. Advances in Mixed Methods Research: Theories and Applications. Thousand Oaks, CA: Sage Publications; 2008:101-119.
- Tashakkori A, Creswell JW. Editorial: exploring the nature of research questions in mixed methods research. J Mixed Methods Res 2007 Jul;1(3):207-211. [CrossRef]
- Petticrew M, Rehfuess E, Noyes J, Higgins JP, Mayhew A, Pantoja T, et al. Synthesizing evidence on complex interventions: how meta-analytical, qualitative, and mixed-method approaches can contribute. J Clin Epidemiol 2013 Nov;66(11):1230-1243. [CrossRef] [Medline]
- Hanson WE, Creswell JW, Clark VLP, Petska KS, Creswell JD. Mixed methods research designs in counseling psychology. J Couns Psychol 2005;52(2):224-235. [CrossRef]
|CINAHL: Cumulative Index to Nursing and Allied Health Literature|
|mHealth: mobile health|
|NIDDK: National Institute of Diabetes and Digestive and Kidney Diseases|
|SMS: short message service|
|TExT-MED: Trial to Examine Text Message–Based mHealth in Emergency Department Patients with Diabetes|
Edited by G Eysenbach; submitted 19.09.16; peer-reviewed by J Baron, B Thompson; comments to author 10.10.16; revised version received 08.12.16; accepted 07.01.17; published 06.02.17
©Cigdem Sahin, Patti-Jean Naylor. Originally published in JMIR Diabetes (http://diabetes.jmir.org), 06.02.2017.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR mhealth and uhealth, is properly cited. The complete bibliographic information, a link to the original publication on http://diabetes.jmir.org/, as well as this copyright and license information must be included.