Ecological momentary assessment (EMA) is a methodological approach used to study intraindividual variation over time, using repeated assessments of behavioral, physiological, and psychological processes during regular daily activities using electronic devices [1]. In recent years, the increasing reach of technology, have dramatically increased the feasibility and sophistication of approaches that use EMA [2,3]. The use of EMA overcomes many limitations of traditional study designs, such as retrospective response bias and undetected environmental influences on behavior, as well as allowing for real-time intervention [4,5]. However, as a relatively recent methodology, there are some challenges to the use of EMA, including the relative scarcity of specific evidence-based methodological guidelines for conducting studies. Moreover, few studies have described EMA methodology in sufficient detail for replication [6]. This lack of methodological guidance is particularly notable in the assessment of cognitive performance via EMA.

Individuals with type 1 diabetes (T1D) are a particularly important target population for EMA study designs given their vulnerabilities to and interactions among multiple dynamic physiological, behavioral, and psychological processes. T1D is a chronic autoimmune disease characterized by the destruction of insulin-producing β-cells in the pancreas, causing hyperglycemia [7] and requiring the use of exogenous insulin. Variations in blood glucose, which occur over the course of minutes to hours, are associated with short-term cognitive variability [8-12] in controlled studies and may indirectly impact psychological and other physiological states [13-15]. We are aware of 3 other ongoing studies that are using continuous glucose monitoring (CGM) coupled with EMA in adults with T1D: The Function and Emotion in Everyday Life with T1D study [16], Hypoglycaemia – Redefining Solutions for better lives project [17], and Towards a Better Understanding of Diabetes Distress, Depression and Poor Glycaemic Control study [18]. One of these studies does not include ambulatory cognitive assessment [18], and the other 2 have not yet reported results [16,17].

The primary goal of the Glycemic Variability and Fluctuations in Cognitive Status in Adults with Type 1 Diabetes (GluCog) study is to characterize the relationship between glycemic excursions and cognitive functioning in adults with T1D, with the secondary goal of determining how psychological state and diabetes-related factors mediate and moderate this relationship. The study, led by principal investigators Dr Laura Germine of McLean Hospital and Dr Naomi Chaytor of Washington State University, uses the EMA of cognitive performance and self-report data, coupled with blinded CGM. Four endocrinology centers (SUNY Upstate Medical University, University of Pennsylvania, Mayo Clinic, and AdventHealth Diabetes Institute), with central clinical site coordination by the Jaeb Center for Health Research, are currently recruiting participants for this study.

Here, we report the results of an initial EMA optimization pilot study of 20 participants with T1D. This optimization pilot was conducted before the finalization of the GluCog protocol to determine the appropriate EMA frequency for the detection of hypoglycemia and EMA completion rate and to refine our cognitive EMA battery. We describe how the initial optimization pilot results guided the design of the main GluCog study, which was launched in September 2020 and is ongoing. Although this study collected CGM data in adults with T1D, similar methodological considerations are applicable to any semicontinuous physiological or behavioral data collection coupled with discrete EMA data (eg, actigraphy, heart rate, and continuous electroencephalogram monitoring). To establish the optimal EMA frequency, we evaluated 2 EMA schedules (6 EMAs per day for 5 days vs 3 EMAs per day for 10 days) to determine which schedule (1) captured the highest number of hypoglycemic episodes within 60 minutes before each EMA and (2) resulted in higher EMA completion rates. We focused on EMA after hypoglycemia (rather than hyperglycemia) because of the established association with cognitive performance in controlled studies and given that hypoglycemia is less frequent than hyperglycemia [19,20].

In total, 20 adults with T1D were enrolled in the optimization study between February 2020 and May 2020 from SUNY Upstate Medical University based on the inclusion criteria that they must: be ≥25 years of age, be diagnosed with T1D, have T1D for >1 year, be fluent in English, have understood the EMA protocol and agreed to comply with it to the best of their ability, and have 24-hour access to a personal smartphone with reliable internet access. Exclusion criteria included the following: inability to complete cognitive assessments owing to significant visual, motor, hearing, or cognitive impairment; any medical or psychiatric condition or treatment that was determined by the principal investigators to interfere with the completion of the study; current use of real-time CGM; inability to complete EMA assessments during the study period (eg, night shift work, planned travel across time zones, or occupation that does not reliably allow time to complete assessments within a reasonable period).

The GluCog optimization pilot study was conducted in compliance with ethical principles that have their origin in the Declaration of Helsinki, including Regulations for the Protection of Human Participants of Research, and the standards of Good Clinical Practice. This study was approved by the Jaeb Center for Health Research Institutional Review Board. All the participants provided written informed consent.

Individual cognitive tests were evaluated for inclusion in the main study based on three sequential criteria: (1) completion rates and usability (eg, minimal participant burden or technical barriers), (2) minimal restriction of range, and (3) good between-person reliability in mobile format.

To calculate the between-participant reliability of each mobile cognitive test, we separately calculated the test scores for even and odd trials. For each test, unconditional multilevel mixed models were used to predict performance on each half of each mobile test, with random effects of EMA numbers nested within participants. Fitting these models allowed partitioning of variance to between- and within-person effects, which we entered into the following equation [36]: where Var (BP) is the total variance in scores between participants, Var (WP) is the variance in scores within participants (ie, variance between EMA sessions and residual variance), and n is the total number of measurements. For each test, we set n equal to the average number of measurements per participant, with a maximum possible n of 60 (a measurement of each half of the 30 mobile tests). CIs for the between-person reliability of each test were calculated using 10,000 bootstrap samples, with resampling at the participant level. Task psychometrics were compared across tasks and 3 EMA/day and 6 EMA/day schedules.

Here, we present the results that led to the choice of cognitive EMA tests used during the optimization pilot phase. Table 1 is based on the previous data collected from the TestMyBrain.org [21] website showing good sensitivity and internal reliability (see Multimedia Appendix 2 for score distribution across tests). All tests exceeded our reliability threshold of 0.4 for a single 30-to-60-second testing occasion. For clarity, the EMA versions of the TMB tests are designated as brief.

The optimization pilot sample (N=20) was recruited from a single clinic site (SUNY Upstate Medical University) from February 2020 to April 2020. Initially, 30% (6/20) of participants were enrolled before the widespread COVID-19 lockdowns. We modified the protocol, obtained International Review Board approval for fully remote study visits, and enrolled the remaining optimization study sample participants remotely. Subsequently, 50% (10/20) of participants were randomized into EMA group A (starting with 3 EMAs per day for 10 days and then switching to 6 EMAs per day for an additional 5 days), and 50% (10/20) of participants were randomized to the EMA group B (starting with 6 EMAs per day for 5 days and then switched to 3 EMAs per day for an additional 10 days). However, of the 10 participants, 2 (20%) participants in group A were excluded from the analyses as they completed <50% of the EMAs (prespecified minimum EMA completion). One participant reported sleeping through all morning EMAs, and the other participant declined to troubleshoot technical issues.

Completion of the remaining sample (N=18) was good across both EMA frequency schedules (3 and 6 EMAs per day) and randomization groups—groups A and B (Table 2). The reported mean for each test was calculated by first computing the mean of each participant across all completed EMA tests and then calculating the average and SD of those participant-level means.

Between-participant reliability was greater for the GluCog optimization pilot sample than for the TestMyBrain.org [21] sample (Table 1), consistent with the repeated testing approach of the EMA design, increasing the measurement precision of between-participant differences relative to a single testing session.

The psychometric characteristics of the mobile cognitive tests were similar across both EMA schedules (Table 3).

Paired samples 2-tailed t tests indicated that MOT accuracy was significantly better (P=.008) during the 3 EMA/day schedule than during the 6 EMA/day schedule (t₁₇=2.96; P=.009; Cohen d_z=0.70). For the other 5 cognitive tests, performance did not significantly differ between the 3 EMA/day and 6 EMA/day schedules (ChoiceRT medianRTc, P=.71; DSMmedianRTc, P=.57; FlickermedianRTc, P=.72; GradCPTD-Prime, P=.55; PSAT Accuracy, P=.75). The GradCPT reliability was much lower on the 6 EMA/day schedule than the 3 EMA/day schedule. This was because of the much higher between-person variance in scores (SD of scores) for the 3 EMA/day schedule than the 6 EMA/day schedule. However, the reason 3 EMA/day schedule captured more between-person variance in scores was unclear from our data.

EMAs were considered complete if the participants finished all 3 cognitive tests comprising the EMA. A paired samples 2-tailed t test found no significant difference in EMA completion rate between the 3 EMA/day schedule and the 6 EMA/day schedule (t₁₇=1.16; P=.26; Cohen d_z=0.27), with both schedules producing >80% EMA completion.

Another goal of the optimization pilot was to determine which EMA frequency schedule would be associated with more EMAs delivered within 60 minutes of a hypoglycemic event. Hypoglycemic episodes were considered only within 1 hour from the hours when EMAs were administered, that is, between 8 AM and 10 PM each day Hypoglycemia episodes were considered “captured” (overlapping with an EMA) if they met any of the following criteria: (1) any point of the hypoglycemic episode occurred within 60 minutes before the start of the EMA (2) the hypoglycemic episode began during the EMA, or (3) the hypoglycemic episodes began within 15 minutes following the end of the EMA. Of the 18 participants, 3 (17%) participants had no daytime hypoglycemic events during the entire pilot period (either 3 or 6 EMA/day periods, combined duration of wearing CGM for 15 d). During the 3 EMA/day period (10-day duration), 67% (12/18) of participants had 43 unique EMA-captured episodes of hypoglycemia (of 102 CGM-detected episodes). Two participants had no EMA-captured events despite having more than one event during the 10-day period. During the 6 EMA/day period (5-day duration), 50% (9/18) of participants experienced 16 unique EMA-captured episodes of hypoglycemia (out of 35 CGM-detected episodes). One participant had no EMA-captured events despite having >1 event during the 5-day period. When comparing periods within individual participants, 50% (9/18) of participants had more EMA-captured hypoglycemic events during the 3 EMA/day period compared to that of 6 EMA/day period, while only 11% (2/18) of participants had more EMA-captured events during the 6 EMA/day period. Overall, 39% (7/18) of participants had the same number of EMA-captured events in both periods.

A paired samples t test revealed that more hypoglycemic episodes occurred during the 3 EMA/day schedule than during the 6 EMA/day schedule (t₁₇=3.00; P=.008; Cohen d_z=0.71) as expected, given that the 3 EMA/day schedule spanned twice the amount of time as the 6 EMA/day schedule. Furthermore, more hypoglycemia episodes were captured by an EMA during the 3 EMA/day schedule (mean 2.4, SD 2.4) than during the 6 EMA/day schedule (mean 0.9, SD 1.0); t₁₇=2.57; P=.02; Cohen d_z=0.61 (Table 3).

The optimization pilot was incorporated into the planned study design for the GluCog study to determine the EMA frequency that would (1) result in higher EMA completion and (2) capture more hypoglycemic events, which are critical factors in the success of the main study. On the basis of comparable EMA completion and greater EMA capture of hypoglycemic episodes, we selected 3 EMA/day schedule over the 6 EMA/day schedule for the main GluCog study.

The optimization pilot also allowed us to evaluate the psychometric and usability properties of the cognitive EMA measures. Given the paucity of cognitive EMA studies on which to base test selection, we initially selected 6 cognitive EMA measures within 3 cognitive domains and alternated tasks within each domain at each EMA (to reduce the total EMA duration). The data from the optimization pilot allowed us to select one test within each cognitive domain that had the greatest likelihood of producing usable data for the main GluCog study. For the processing speed domain, the Brief TMB DSM and Brief TMB Choice Reaction Time had comparable reliability and usability. We selected the Brief TMB DSM owing to greater familiarity among clinicians and the use of Digit Symbol Matching tasks in prior studies of cognition in T1D. For the sustained attention domain, we selected the Brief TMB GradCPT owing to less restriction of range when compared with the Brief Paced Serial Addition Test. For the working memory domain, we selected the Brief TMB MOT because some participants had technical issues with touch sensitivity on their devices during the Brief TMB Flicker task. Specifically, some participants had issues with screen taps not immediately registering on the device and as a result, had longer reaction times.

The optimization pilot also revealed technical difficulties in the implementation of the EMA via the app-based notification system. Some participants had trouble installing and setting up the app, as well as keeping track of push notifications. To address this, in the main GluCog study, we switched to a system that did not involve installation of an app and relied on text messages rather than push notifications. This system required less technical support throughout the study.

Many aspects of EMA study design can affect adherence to the research protocol [49,50]. The optimal assessment strategy for capturing sufficient glycemic variability and ensuring adequate EMA completion rates was unknown before initiating the GluCog study. The EMA cognitive tasks that we selected were also refined via an optimization pilot. We found no difference in the completion rates between the 2 EMA frequencies. However, we found that a longer sampling duration (10 d) with less frequent EMAs (3 EMAs per day) resulted in more EMAs in close proximity to hypoglycemic episodes compared with short duration of testing (5 days) with high-frequency EMAs (6 EMAs per day). Our findings may guide future studies that include events of interest with relatively infrequent occurrence (mean=0.51, SD =0.44 events per day; range 0-1.47), such as hypoglycemia.

Given that the onset of the COVID-19 pandemic coincided with recruitment for the optimization pilot, we were able to develop procedures for completing clinic visits remotely via telehealth or telephone visits, or in-person as initially planned, and carry these procedures forward into the main GluCog study, along with a COVID-19 specific impact and stress questionnaire. Recruitment was slower than initially anticipated owing to the rapidly increasing clinical uptake of real-time CGM in the T1D population. The exclusion of participants using real-time CGM was removed in June 2021 after 50 participants were enrolled to ensure that concurrent real-time CGM use could be analyzed as a covariate. The lower age limit was also expanded to 18 years to maximize recruitment. All the other procedures have remained consistent with those reported above. The main GluCog study began recruitment in October 2020. Enrollment was completed on June 15, 2022. Our primary objective was to describe the research methodology used in this pilot study on glycemic and cognitive variability in adults with T1D, the results of which guided key decisions related to EMA use in the main study protocol. There are limited existing cognitive EMA data on which important study-design decisions can be made. Recent systematic reviews on EMA have pointed out a lack of important methodological information in the scientific literature [6,49-51]. Our report includes all recommendations from the guidelines adapted for EMA studies across disciplines (Checklist for Reporting EMA Studies, CREMA) [6] and provides a way forward for EMA studies until greater methodological consensus has been reached.

The current availability of smartphones with large and high-resolution screens has made rigorous mobile cognitive assessments possible. Critical dilemmas faced by EMA studies involving cognitive assessment include (1) selecting tests that have been linked to the phenomena of interest based on the traditional test literature (ie, impacted cognitive domains); (2) ensuring psychometric properties that are suited to high-frequency administration (ie, avoiding ceiling effects); (3) balancing the need for adequate reliability, while minimizing test length; and (4) formatting a test that is compatible with smartphone screen size and operating system variations. There are very limited empirical data to aid the selection of EMA cognitive tasks that account for all these factors. Thus, there is an advantage in using a large web-based test platform (such as the TestMyBrain [21] platform) to select high-performing EMA length tests with known device impacts.

Although not anticipated when designing the GluCog study, our exclusive use of remote assessment was useful in mitigating logistical challenges related to face-to-face assessments during the COVID-19 pandemic [52,53]. We were able to quickly pivot our initial in-person clinic enrollment visit to a teleconference visit (with CGM supplies mailed to the participants). In addition, remote assessment mitigates logistical challenges related to traditional face-to-face testing administration, such as costs (staff time, clinic and laboratory space, physical assessment materials), difficulties in getting to a study site for people with mobility limitations or who have transportation challenges such as those who live in rural or remote areas, necessary training for the examiner, and training of study personnel in complex test administration (which can be particularly challenging with multisite studies).

Among the most exciting aspects of EMA use is the opportunity to explore biopsychosocial mechanisms underlying human health and disease. Technological advances in physiological detection capability (eg, CGM) and data-driven machine learning techniques for predicting cognitive changes are advancing rapidly. Our understanding of real-world influences on cognitive performance will be exponentially increased by the ability to accurately measure fluctuations as they occur. Repeated measurements allow for the identification of mediators and moderators of cognitive change over time in real-world environments.

The EMA optimization pilot study described here responds to the urgent need for systematic and detailed information on EMA study designs. Recent advances in mobile technologies have resulted in new opportunities for EMA to examine cognition in everyday environments. When applied in a well-planned manner, EMA can be an important tool for research involving biopsychosocial mechanisms. In addition, cognitive EMA can assist in the early diagnosis of cognitive impairment as well as follow-up and intervention and can complement traditional neuropsychological assessments. EMA holds immense promise for understanding everyday conditions, both internal and external, that influence cognitive performance in individuals over time. This can be particularly useful in the assessment of populations with greater vulnerability to dynamic physiological, behavioral, and psychological interactions, such as individuals with T1D. Given the complexity of EMA studies, choosing the right instruments and assessment schedules is an important aspect of study design and subsequent data interpretation. Empirically determining these parameters in the target population will ensure adequate sampling of the phenomena of interest.