Thirty-six candidate items were pretested for ease of completion and understanding among people living with diabetes to ensure that items superficially made sense [20], and provided further support for content validity through ensuring that each theme identified in the qualitative interviews [19] was represented through the item content [21]. Participants were probed about their understanding of the proposed items and each item’s relevance to self-managing health and the use of technology [21,22]. In cases where items were ambiguous or repetitive, they were amended or removed.

Participants were aged 18 or over with a (self-reported) clinical diagnosis of type 2 diabetes and experience of using one or more diabetes-related web-based or mHealth technology. Participants who had previously taken part in an in-depth interview [19] to inform items and who consented to be contacted were emailed a participant information sheet. On agreeing to take part, participants could ask any questions about the research, asked to complete an online consent form, and given a link to the draft online survey containing the candidate items. Participants were given a GBP £20 (US $24.92) voucher for their participation.

Interviews were recorded and carried out over the telephone using a verbal probing method of cognitive interviewing to allow respondents an opportunity to give uninterrupted answers, which was then followed by a focused interview [23]. During the focused interview, participants were reminded of their answers to each item, and to gain a deeper understanding of their responses, the reasoning behind their answers was explored [24].

Participant comments were summarized and collated in an Excel document (Microsoft) according to each instruction and questionnaire item, allowing within-case (how the item fits within the questionnaire as a whole) and between-case (interpretation and consistency of items across the sample) analysis. Interpretation difficulties or inconsistencies were discussed among authors, amended where appropriate, and retested.

An expert panel consisting of three survey development and patient-reported outcome experts, one survey expert and user engagement manager for a national diabetes program, two diabetes experts specializing in digital health, one professor of diabetic medicine, and one consultant physician in diabetes were invited to review the candidate DSMT-Q items via email. Consulting experts sought to evaluate items from both health professionals and survey developers’ perspectives. Comments and feedback were received via email, and items amended where appropriate after discussion among the authors.

PPI representatives who were members of a volunteer list held by the Nuffield Department of Primary Care Health Sciences Coordinator of Patient & Public Involvement were emailed an invite to take part in questionnaire feedback. Representatives were required to have type 2 diabetes, but as questions were designed to be answerable to both those who use technology to manage their health and those who do not, they were not required to have experience using technology to manage their health. Feedback was given over the telephone after representatives had been allowed to review the online survey.

Participants were aged 18 or over with a (self-reported) clinical diagnosis of type 2 diabetes. Participants were recruited through a professional survey recruitment company. Eligible participants were provided with a participant information sheet and asked to confirm their consent to take part. Participants who confirmed they might be contacted again regarding the study while completing Survey 1 were sent an email 2 weeks later asking them to complete Survey 2.

We aimed to recruit 250 participants to complete Survey 1. Estimates suggest that meaningful psychometric tests require at least three times as many respondents as items [27], making this a conservative (large) sample size.

Descriptive statistics were used to present demographic data. DSMT-Q items were subjected to several initial data checks to confirm their suitability for inclusion in further analysis. Decision rules for item removal included items with high floor and ceiling effects (>40% of respondents selecting one of the extreme response options) [28,29] and items demonstrating a large number of weak correlations (<0.2) with other items. Exploratory factor analysis (EFA) was performed to group items into conceptually sound sub-scales. Suitability of using EFA on the dataset was assessed through performing the Bartlett Test of Sphericity (P<.05) [30] and calculating the Kaiser–Meyer–Olkin (KMO) statistic which has a recommended value of above 0.6 [31]. Factors with Eigenvalues >1 were rotated using an oblique, Direct Oblimin, rotation so that axes were not restricted to right angles, hence allowing correlation between the factors [32,33]. While both the Structure and Pattern matrices were used in interpreting output, and the Structure matrix offered primary guidance for interpretation [34].

Once domain structures were finalized, sub-scales floor and ceiling effects (>20% of responses scoring 0 or 100) were examined, and population characteristics were explored to identify potential covariate factors impacting the final scales. Convergent validity was examined using Pearson correlation coefficients (r) to compare relationships between the DSMT-Q sub-scales and the DSES [33,35]. The DSES was hypothesized to have moderate correlations with DMST-Q scores. Internal consistency, an indication of a scale’s reliability, was evaluated using the Cronbach alpha statistic (>0.7) [36]. External reliability was assessed using the test-retest procedure with the use of the ICC statistic to assess the stability of the scores [37].

Eight participants took part in two rounds of cognitive interviews. In the first round, participants (n=4) considered most questionnaire content to be relevant to the management of type 2 diabetes; however, considerable changes needed to be made to the arrangement of the items. Participants found it challenging to respond to the second set of mHealth specific items as they did not typically use one specific technology (for example, one mobile app) in isolation. Furthermore, many questionnaire items did not apply to every technology used, and it was, therefore, difficult for them to determine how they should respond. The second part of the questionnaire was removed, resulting in 12 items being deleted and 2 items, covering topics not already present in the first part, being restructured and included. Two further items, which asked about health care services, were deleted from part one due to participant feedback stating they were unrelated to the personal management of their type 2 diabetes. Twenty-two items were therefore retained.

Five items were amended following participant feedback. Two were amended to improve comprehension and clarity, one was amended to make it more suitable to the response options, one was changed to be more specific and capture the intended meaning better, and one item was changed to prevent duplication with a previous item.

In the second round of cognitive interviews, following participant (n=4) feedback, four items were deleted as they were considered to duplicate existing content. Five items were amended to improve clarity, and one item was revised to apply to a broader population. Two items were considered too broad and were made into four items to improve accuracy. This process resulted in 20 items for expert consultation.

Twenty items were circulated to the expert group. Following feedback, two items were amended to improve language for low literacy groups. One item was amended to be more inclusive to a broader range of people. Six items were amended to improve clarity, and one item was split into two items to try and find the best way to capture reassurance, resulting in 21 items.

PPI representatives (n=4) gave feedback on the 21-item questionnaire. All items were understood by representatives; however, some showed a preference for the further granularity of items. For example, one representative expressed a wish to have separate questions for how easily they can monitor their blood glucose levels, diet, and exercise. This change was omitted due to the likelihood of high frequencies of not applicable or missing data.

Survey 1 participants included 104 (41.6%) women and 146 (58.4%) men. The average age was 55.9 years old (SD 16.4, range 69 years). The modal time since diagnosis of type 2 diabetes was between one and five years ago (n=90, 36%). Most participants (n=232, 92.8%) described themselves as White British. Further sample characteristics can be viewed in Table 1.

Six items were removed due to ceiling effects of greater than 40%. The KMO value for the remaining 15 items exceeded the recommended value of 0.6 (KMO= 0.90), and the Bartlett Test of Sphericity reached statistical significance (P<.01), indicating there was a correlation between the items. Three factors were initially extracted, explaining 69.1% of the variance. One factor consisted of two items and had a poor Cronbach alpha level of 0.40. These two items were removed, and a second-factor rotation extracted two factors, explaining 63.6% of the variance. See Table 2 for the factor structure and loadings. Factor 1, entitled Understanding individual health and making informed decisions, consisted of seven items and had a Cronbach alpha level of 0.90. The second factor, entitled Confidence to reach and sustain goals, consisted of six items and had a Cronbach alpha level of 0.88.

Each scale was transformed to a 0-100 metric, where 0 indicated low levels of self-management, and 100 indicated high levels of self-management of type 2 diabetes. Scale scores were calculated by summing the final response values in each sub-scale and dividing the summed score by the maximum scale score. The raw scale score was then transformed into a 0-100 metric by multiplying the raw score by 100. Scale distribution statistics are reported in Table 3. Neither scale exhibited floor or ceiling effects, which was considered to be >20% of responses, achieving the minimum or maximum score. Minimal respondents achieved scores of 0, while 10.8% (n=27) of Factor 1 scores and 10.4% (n=26) of Factor 2 scores achieved the maximum score of 100.

Relationships between each DSMT-Q scale and a range of potential covariate factors were examined. No significant differences were found for either sex (Factor 1: t₂₄₈=–0.54, P=.59 and Factor 2: t₂₄₈=–0.35, P=.72) or age (n=248, Factor 1: r=–0.11, P=.10 and Factor 2: r=–0.06, P=.31) among either sub-scale scores. Nonparametric tests to examine sex (Mann-Whitney U test of significance; Factor 1, P=.59, Factor 2=0.71) and age (n=248; Factor 1: ρ=–1.37, P=.03, Factor 2: ρ=–0.09, P=.15) also demonstrated no significant differences. No significant differences were observed for time since diagnosis for using parametric (analysis of variance; Factor 1: F₂₄₆=2.14, P=.10, Factor 2: F₂₄₆=1.891, P=.13) or nonparametric tests (Kruskal-Wallis k independent samples; Factor 1: P=.13, Factor 2: P=.15).

Relationships between the DMST-Q scales and the DSES scale were examined to assess convergent validity. Correlations were high (Factor 1, r=0.67, P<.001 and Factor 2, r=0.75, P<.001), indicating the scales were tapping into similar but different concepts.

As predicted, those who did use technology to support self-management scored more highly on both new DMST-Q scales. For Factor 1, those using technology to support self-management (n=92) had a mean score of 78.81 (SD 16.64), while those who did not use technology (n=158) scored 72.60 (SD 18.42, t₂₄₈=3.09, P=.002). For Factor 2, those using technology to support self-management (n=92) had a mean score of 80.21 (SD 14.61), while those who did not use technology (n=158) scored 72.52 (SD 18.73, t₂₄₈=3.38, P<.001).

Factor 1 and 2 also demonstrated good test-retest reliability with ICC values, for those who had indicated their health had remained the same compared to two weeks ago (n=113), equal to 0.78 (P<.001) and 0.74 (P<.001), respectively.