Abstract
Background:
Specialized education is critical for optimal insulin pump use but is not widely utilized or accessible. We aimed to (1) test the usability and acceptability of A1Control, a simulation platform supporting insulin pump education, and (2) determine predictors of performance.
Method:
Rural adult insulin pump users with type 1 diabetes (T1D) participated in a mixed methods usability study in 2 separate rounds. Participants navigated 3 simulations (ie, infusion site occlusion, hypoglycemia, exercise). Net Promoter Score (NPS) and Systems Usability Scale (SUS) were administered. Semi-structured interviews and direct observation were used to assess perceived usability, acceptability and performance. Synthetic Minority Oversampling Technique was used to fit predictive models for visualization of patterns leading to good or poor A1Control performance.
Results:
Participants (N = 13) were 28-70 years old, 10 used automated insulin delivery and 12 used continuous glucose monitoring (CGM). Mean NPS was 9.5 (range 9-10) and positive sentiment during interviews indicated very high acceptability. SUS (mean 88.5, range 70-100) indicted a high perceived usability. CGM percent wear ≥ 94%, time spent in hypoglycemia ≤ 54 mg/dl of <0.01%, and <70 mg/dl of 0.5% predicted successful site-occlusion scenario performance with 100% accuracy. BOLUS score ≥ 2, TDD ≥ 34, and technology brand predicted exercise scenario success with 100% accuracy. There were an insufficient number of failed hypoglycemia scenarios to assess predictors.
Conclusion:
A1Control shows potential to increase access and frequency of self-management and technology education. Additional study is needed to determine sustained engagement and benefit.
Keywords
Background
For the 1.5 million individuals with type 1 diabetes in the United States, the importance of glycemic control in the prevention of complications is well established. 1 However, most individuals do not meet glucose targets2,3 despite advances in therapeutics and technologies. Diabetes technologies (ie, insulin pump, continuous glucose monitor [CGM], automated insulin delivery [AID]) improve quality of life, glycemic control,4-7 and possibly mortality 8 ; however, safe and effective use requires technological sophistication, foundational diabetes self-management knowledge and access to care.7,9-14 Benefit from insulin pumps, even those providing AID, depends greatly on individual utilization and self-management behavior including frequency of self-monitored blood glucose (SMBG), 15 use of bolus calculators15-18 and utilization of advanced features.13-15,19
Diabetes self-management education (DSME) improves glycemic control,20-22 but over 50% of individuals with diabetes in the United States report that they have never received DSME. 23 Many rural counties face the highest
rates of diabetes yet have the most limited access to endocrinology clinics and educators. 23 For insulin pump users, expert groups recommend periodic reassessment and education.10,11,24,25 Insulin pump retraining is associated with significant improvements in patient knowledge and skills as well as reductions in A1C.26-28
Currently, insulin pump education is not standardized and largely relies on in-person visits.10-12,29 Initial training focuses on the technical aspects of insulin pump therapy (ie, entering pump settings, site insertion, bolus administration) at the expense of critical thinking, emergency management, and utilization of advanced features. 10 Traditional didactic education inadequately transfers knowledge to real world situations and prohibits application of knowledge to novel self-management dilemmas. 30 Multiple meta-analyses and systematic reviews of educational simulations demonstrate its effectiveness as an educational technique with advantages over traditional education formats in its ability to (1) facilitate lifelike training situations that would otherwise be unfeasible or too dangerous, (2) transfer tangible skills in addition to knowledge, (3) increase access to limited resources, and (4) enable distance learning.31-36 Numerous studies have demonstrated the positive findings associated with simulation education, including increased time-on-task, 31 enjoyment,37,38 motivation,39,40 deeper learning and long-term retention.41,42 Although simulation is being used increasingly in health care provider training, 43 particularly in surgical specialties, 44 it is under-utilized in patient education technologies.
Our objectives were to iteratively develop a diabetes education simulation platform while (1) testing A1Control’s usability and acceptability among a rural sample of insulin pump users, and (2) developing a monitoring function by which diabetes device data can be used to predict patient performance.
Methods
Study design
A prospective, mixed methods observational design was used to evaluate usability and acceptability during use of A1Control’s alpha prototype. We utilized an iterative design process in which testing was completed in 2 rounds with unique participants in each round allowing technology modifications between rounds. 45 After round 1, OSU’s College Diabetes Network (CDN) members participated in a workshop to interpret and translate round 1 data into technology modifications. 46
Participants
Eligibility criteria included diagnosis of T1D, >18 years of age, insulin pump user, and lived in a rural county at least 30 minutes from a diabetes education center. Participants meeting eligibility criteria were recruited from a single large academic medical center endocrinology practice. The study was approved by The Ohio State University’s Institutional Review Board. All participants provided written informed consent prior to data collection.
Intervention
Initial concept generation, co-design, and modifications of the platform were done in concert with the CDN through focus group interviews and concept co-design workshops. 46
Description of A1Control
A1Control is a diabetes education simulation platform designed to provide diabetes device specific education in concert with essential DSME in a virtual environment. Within A1Control, people with diabetes care for an avatar and are prompted to navigate challenges using a simulated insulin pump. The patient’s simulation experience is guided and mentored by a second avatar who can assist the user in making decisions and provide feedback on performance (Figure 1). Throughout the simulation, the patient is presented with common and emergency management scenarios and must navigate the simulation using by making self-management decisions (ie, exercise, eat, sleep, wait, bolus) (Figure 2). A variety of foods can be selected for consumption each featuring an accessible nutrition label. Differential equations, derived from previously published models,47,48 underpin the platform and facilitate physiologic glycemic responses to user actions (ie, carbohydrate consumption, insulin administration, physical activity). Diabetes devices, which currently include a full-scale virtual insulin pump and meter, are built directly into the platform. Discrete glucose values derived are displayed on the pump interface however trends and tracing data is not displayed. Patients can manipulate these devices (ie, enter carbohydrates, manipulate bolus doses, change settings) as they would physical versions of the technologies. The patient’s own insulin pump settings are used to tailor the simulation algorithms to mimic the users own physiologic patterns and insulin doses. The technology features both text and voice narration. Patient performance is tracked to assess competencies facilitating follow up education to target specific deficiencies in the future.
A1Control Simulation Platform.
A1Control Self-management Decisions.
Data Collection
Testing was initially conducted in clinic but converted to virtual testing using teleconference software in response to COVID-19 restrictions. During testing, participants navigated 3 simulations: infusion site occlusion, hypoglycemia, and exercise. Participant performance and response to simulation content was captured using video recording, platform database and observation. Participants were asked to use the think aloud method to distinguish between technology design vs self-management decisions. 49 Three questionnaires were administered including the Systems Usability Scale (SUS), a well-validated tool for the evaluation of technology usability. The SUS contains 10 items using a 5-point Likert scale. 50 The Net Promoter Score (NPS), a single item measure of acceptability, assessed how likely participants would be to recommend the platform to a friend. 45 The Revised Michigan Diabetes Knowledge Test (DKT2) was administered prior to usability testing. 51 The DKT is comprised of a General Knowledge and an Insulin subscale. Upon completion of usability testing, debriefing semi-structured interviews were conducted with each participant by a single interviewer (ERF). Interviews focused on participant’s views regarding usability, acceptability, and recommendations for the technology. The interviews contained 3 Likert scale questions asking participants how difficult the platform was to use; how they would feel about using this platform as part of their initial insulin pump training; and how realistic were the situations and outcomes in the scenarios. All interviews were audio recorded, professionally transcribed, and reviewed for accuracy.
The 28 days of data prior to testing were retrieved from each participant’s personal diabetes devices (ie, insulin pump, CGM, AID) using Tidepool software (Palo Alto, CA). Self-management and glucose variables were aggregated from raw device data. Aggregate daily measures included BOLUS score to assess frequency of mealtime bolus administration. 52 Participants received 1 point for each bolus administered at breakfast (0600 to 1000 h), lunch (1100 to 1500 h) and dinner (1600 to 2200 h) with a max score of 3. 52 Blood glucose (BG) input was defined as any glucose value used for bolus administration. These glucose values come from linked meter SMBG, linked CGM, or manual entry of CGM or SMBG value. BG input is a self-management variable measuring the way individuals use glucose values regardless of the source.53,54 CGM glucose values were aggregated into derived measures of glucose control consistent with ADA and international guidelines on glucose targets including time in range (TIR, 70-180 mg/dl), time above range (>180 mg/dl), time in hypoglycemia (<70 mg/dl & <54 mg/dl), and coefficient of variation (CV).55-57
Analysis
Our sample size was based on prior research suggesting that testing new technologies with 5-8 users will expose the majority of usability problems, using >50% of users reporting a problem as a threshold for modification.58,59
A1Control platform performance indicators included:
(1) scenario success rate defined as successful completion of a scenario (achievement of target glucose, 70-180 mg/dl) regardless of the number of “incorrect” choices as there may be several choices resulting in a positive
(2) number of platform confusion errors (count of incorrect or erroneous navigation moves attributable to A1Control design) vs. number of self-management errors (count of incorrect choices attributable to deficits in diabetes self-management knowledge) were computed for individual scenarios and across all scenarios per participant;
(3) Reason for scenario failure was evaluated using platform performance indicator 2 for those participants who did not successfully complete individual scenarios.
(4) Time to scenario completion was measured directly from the platform’s database for individual scenarios per participant.
Interview data, fieldnotes, and memos were imported into NVivoTM for data management and analysis.60-62 Interviews were initially coded by one of two coders, (CL) and (LJ) prior to a second coder, (ERF) reviewing 100% of interviews. Data were used in concert with observation, platform performance data and survey data to assess usability, acceptability and inform A1Control revisions between rounds.
Diabetes device data was used both to (1) describe participant self-management and glycemic control patterns and (2) in models predicting A1Control scenario success. Decision trees were fit for each scenario to (1) visualize the relationship between participant’s diabetes device data and A1Control performance and (2) to measure predictive performance for the relationship. Given the small sample size, Synthetic Minority Oversampling Technique was used and decision trees were run on a withheld test set of (N = 4) to determine accuracy. 63 To avoid picking up on random noise in the small simple size, a random forest was fit to determine features importance, and features with a mean increase in the gini index of greater than 20 were removed. The gini index is a measure which helps determine how important individual features are in determining performance. A decision tree was then fit on the subsetted dataset and predictive performance was calculated on the test set to gather baseline descriptive self-management data and to evaluate associations between personal device data and platform performance.
Results
Thirteen adults with T1D participated in the study, 5 in round 1 and 8 in round 2. Eight visits were conducted virtually, all in round 2. All participants lived >30 minutes from a diabetes education center and represented 10 rural Ohio counties. The mean age was 51.4 years (range 28-71), mean duration of diabetes was 30 years (range 11-61) and mean A1C was 7.5% (range 5.9%-9.9%). Most participants (n = 10), used AID devices and 12 used CGM. Twelve of the 13 participants had one or more diabetes complications (ie, retinopathy, neuropathy, nephropathy, cardiovascular disease). Table 1 depicts demographics, diabetes characteristics, and diabetes technology use for the participants stratified by round of testing. Demographics and A1C were similar for participants in round 1 and round 2 of testing.
Demographic, diabetes characteristics and baseline knowledge stratified by round of testing.
A1C in Group 2 N = 6 with no recent A1C for 2 participants.
Participants demonstrated a high degree of baseline diabetes knowledge with a mean DKT2 score of 20/22 (range 17-22), and a mean General Knowledge sub-section score of 12/13 (range 10-13) and a mean Insulin sub-section score of 8/9 (range 7-9) (Table 1).
Self-management and glycemic control variables were examined for all participants and by round of testing (Table 2). Participants performed a mean of 5.1 SMBG/day (range 3-10), mean 3.8 BG input/day (range 2-8), and mean 4.6 bolus/day (range 1-8). AID users (N = 10) perform a mean of 2.7 calibrations/day (range 0-5) and spent a mean of 62.3% time in auto-mode (range 0-98%). TIR, time above range, and time below range (<70 mg/dl and <54 mg/dl) were examined for participants who used CGM (N = 12). Participants spent a mean of 67.7% TIR (range 40.5-79.5%), mean 29.6% of time >180 mg/dl (range 20.5-59.5), mean 0.9% of time <70 mg/dl (range 0-5.32) and had a CV of 28%.
Diabetes device self-management and glucose characteristics stratified by round of testing.
(SMBG) Self-monitored blood glucose, (BOLUS-score) 1 point each for a mealtime bolus occurring between 0600 to 1000, 1100 to 1500, and 1600 to 2200, max score/day is 3.
Usability and acceptability
Mean NPS of 9.5 (range 9-10) and positive responses to interview questions indicated very high acceptability. SUS (mean 88.5, range 70-100) indicted a high degree of perceived usability (Table 3). The majority of participants reported the platform was extremely easy (n = 7) or very easy (n = 5) to use and found the simulation scenarios and glucose outcomes extremely realistic (n = 10) or very realistic (n = 3). When asked how they would have felt about using a simulation technology like the platform as part of their initial pump education, the majority reported they would have very much liked to use it (n = 11).
NPS, SUS and platform performance indicators stratified by round of testing.
Platform performance indicators are presented in Table 3. Time to scenario completion varied significantly depending on participant choices made throughout the simulation. Self-management errors observed during the simulation included choosing carbohydrate free foods to treat or prevent hypoglycemia (N = 2 participants with one participant making the error in both the exercise and hypoglycemia scenarios for a total of 3 errors), failure to check ketones (N = 3), failure to change pump infusion site (N = 2), and failure to prevent hypoglycemia with carbohydrates or temporary basal decrease prior to exercise (N = 7). Participants who failed the site occlusion scenario due to self-management errors (n = 2) stated they did not change the site because a visual inspection revealed no obvious problems with the site. Several other participants (n = 6) had a delay in changing the site but ultimately did successfully complete the scenario. In the exercise scenario, one participant consumed a carb free food to prevent hypoglycemia and stated this is how she typically treats and prevents hypoglycemia. Interestingly, this participant had the highest percentage of time in hypoglycemia (5.3% < 70 mg/dl and 3.4% < 54 mg/dl). Failure to locate a glucose value was the predominate platform confusion issue in the first round of testing with 3 of the 5 participants in round 1 reporting this issue. Other platform performance indicators including the number of prompts needed to complete scenarios and perceived difficulty completing tasks were evaluated to inform A1Control modifications.
Round 1 modifications included the addition of a glucose meter, playback button, algorithm adjustments and avatar upgrades to address platform confusion and functionality issues. Platform modifications made in round 1 appeared to successfully address functionality and confusion issues with none of the 8 participants in round 2 reporting the same problems. Modifications suggested from round 2 for future consideration included: addition of a virtual CGM, additional content library, increased personalized content, additional nutritional content, and brand specific devices.
Predictors of performance
CGM glucose and self-management variables (ie, time-in-range, CGM percent wear) were strongly associated with A1Control performance therefore only participants using CGM were included in the analysis (N = 12). Time-in-range 67%, mean SMBG <152 mg/dl, hypoglycemia, and time above target <30% were found to be highly predictive of site occlusion scenario success in the original model, but with a gini of >20% they were iteratively removed from the final model. In the final model constructed on subsetted data, CGM percent wear ≥94%, time spent in hypoglycemia ≤54 mg/dl of <0.01%, and time spent in hypoglycemia <70 mg/dl of 0.5% predicted site occlusion scenario success with 100% accuracy. Original exercise scenario models showed a high association between percent basal <67%, mean carb/bolus ≥34, and carbohydrates/day >104 were found to be highly predictive of exercise scenario success, however, each was found to have a gini of >20% and, therefore, each was iteratively removed to avoid overfitting. Available subsetted data showed BOLUS score ≥ 2, TDD ≥ 34, and brand (670G and Omnipod Dash compared to Medtronic 500 and 700 series) predicted exercise scenario success with 100% accuracy. Predictive models were not fit for hypoglycemia because only 1 participant failed the scenario leaving insufficient data for model development.
Discussion
The results of this study suggest a high degree of acceptance and perceived usability for diabetes education simulation technologies even among an older and rural cohort of adults with diabetes. All NPS scores were >8, indicative of an exceptionally high degree of technology acceptance,45,64 meaning that participants were extremely likely to recommend the technology to a friend with diabetes. Similarly, SUS scores were very high ranging from 70-100 indicating a high degree of perceived ease of use and satisfaction with the system.45,50 These scores were substantiated during debriefing interviews where participants expressed predominately positive sentiment regarding A1Control as a diabetes educational tool. Interestingly, NPS and SUS scores were high and feedback was generally positive for both rounds of testing despite modifications made prior to round 2.
The participants had a high degree of baseline diabetes knowledge as measured by the DKT2 survey. Additionally, they were an experienced cohort with a lengthy history of diabetes and history of pump use. Their diabetes device download data also showed a relatively high frequency of self-management behaviors (eg, SMBG/day, bolus/day, BG input/day) and a fair degree of glycemic control (ie, A1C, TIR). Their experience made them extremely well suited for design research in that they were able to compare platform scenarios to their real-world experience and contrast A1Control’s education to traditional didactic diabetes education. While the participants’ knowledge and patterns of self-management were a benefit at this stage of technology development, we acknowledge that the experiences of this cohort may not be generalizable to more novice diabetes device users. There is a need for accessible and personalized device education not just for ongoing education but perhaps most importantly for initial education. Therefore, future usability, and eventually efficacy testing should incorporate multiple cohorts with varied degrees of experience.
We identified one predominant platform confusion issue during round 1 with the majority of participants unable to locate a glucose value on the platform. During debriefing interviews, participants indicated they specifically wanted an option to test a meter SMBG. Round 1 participants were comprised of one non-CGM pump user and 4 AID users, though all still required routine SMBG testing. Participants’ recommendations in round 1 were, therefore, a reflection of their personal daily self-management experience. An indicator of success in our usability testing is that we were able to clearly identify a predominant platform confusion issue in round 1 and created a solution that addressed the problem in that the same issue was not identified in round 2. 45 Once a glucose meter was added to A1Control, all of the participants in round 2 were observed easily locating a glucose value when they desired.
Participants performed very well on the hypoglycemia scenario with successful completion for all but one participant, whereas more participants were unable to obtain a successful outcome in the exercise and site-occlusion scenarios. In each scenario, 2 participants were unable to achieve a successful outcome due to self-management errors. Participant’s comments indicating they did not change the site because no site issue could be externally visualized revealed deficits or gaps in knowledge. These participants stated that it had either been several years since they had a site issue or they had never had a failed site. They attributed this unfamiliarity with the situation as a reason for delay in site change. In the exercise scenario, the participant’s choice in treatment and prevention of hypoglycemia with a carbohydrate free food, illuminated real world self-management behaviors that are contrary to best practice and may have contributed to the participant’s high percentage of time in hypoglycemia. These scenario failures illustrate advantages in the use of simulation technologies. First, simulation technologies allow us to facilitate lifelike trainings for critical situations that would typically be unfeasible or too dangerous to create in real-world environments, thus providing a clear advantage over traditional didactic approaches. 65 Secondly, simulation technologies can serve as resources to assess patient knowledge and identify shortcoming.
Our analysis using personal device data to predict A1Control platform performance was limited by the small sample size. Our objective was to establish trends and examine what variables the models might use to make decisions in the future. Because of the limited sample size, it was imperative that decisions trees and models be evaluated not only for accuracy against a withheld set of participants, but also that models be evaluated from a clinical perspective to assess if the models “ring true” to provider experience.
Infusion site problems are common and not easily predictable, indicating the need for anticipatory patient education and training. 66 In the final site occlusion model, CGM percent wear ≥ 94% and time spent in hypoglycemia predicted site occlusion scenario success with 100% accuracy. A reasonable clinical case can certainly be made for the predictive relationship and direction of these variables. While there is little research evaluating self-management behaviors associated with DKA, CGM use is associated with lower risk of DKA. 67 To avoid overfitting of the models, variables found to be highly correlated with site occlusion success (eg, TIR, SMBG/frequency, time > 180 mg/dl) were removed; however given the small sample size we cannot definitively say these variables would not emerge as key indicators in future larger samples.
For the exercise scenario, BOLUS score≥ 2, TDD ≥ 34, and brand (670G and Omnipod Dash compared to Medtronic 500 and 700 series) were found to predict exercise scenario success with 100% accuracy. It is possible that brand may reflect more recent diabetes education as it is likely that 670G and Omnipod Dash participants had more recent pump training than the Medtronic 500 and 700 series users. Percent basal <67% was found to be highly correlated with successful completion of the exercise scenario but was removed to prevent overfitting. Again, due to the small sample size we cannot definitively say percent basal would be excluded from models incorporating larger sample sizes.
Conclusion
Our analysis shows a high degree of acceptance and usability of a simulation self-management technology, A1Control, that could improve access to and frequency of DSME and technology education. Our use of personal device data to establish predictive patterns provides a viable path for personalized assessment and education in the future. Future research is needed to determine sustained engagement and patient outcomes.
Footnotes
Acknowledgements
The authors thank Randy Sanborn for his extensive engineering contributions to A1Control’s design and development and all members of The Ohio State University College Diabetes Network for their continued research contributions.
Abbreviations
continuous glucose monitor (CGM), automated insulin delivery (AID); self-monitoring blood glucose (SMBG); diabetes self-management education (DSME); College Diabetes Network (CDN); Systems Usability Scale (SUS); Net Promoter Score (NPS); Revised Michigan Diabetes Knowledge Test (DKT2); time in range (TIR); coefficient of variation (CV)
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Eileen R. Faulds and Carl W. Noble are co-founders of A1Control a diabetes education startup
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The project described was supported by Award Number Grant UL1TR002733 from the National Center For Advancing Translational Sciences. Technology development was supported by the State of Ohio, Ohio Development Services Agency and Ohio Third Frontier through a Technology Validation Start-up Fund award (TECG20180459). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Center For Advancing Translational Sciences, the National Institutes of Health, Ohio Third Frontier or the State of Ohio.
Data Accessibility
The data that support the findings of this study are available from the corresponding author upon reasonable request.