Sage Journals: Discover world-class research

Abstract

Background:

Specialized education is critical for optimal insulin pump use but is not widely utilized or accessible. We aimed to (1) test the usability and acceptability of A1Control, a simulation platform supporting insulin pump education, and (2) determine predictors of performance.

Method:

Rural adult insulin pump users with type 1 diabetes (T1D) participated in a mixed methods usability study in 2 separate rounds. Participants navigated 3 simulations (ie, infusion site occlusion, hypoglycemia, exercise). Net Promoter Score (NPS) and Systems Usability Scale (SUS) were administered. Semi-structured interviews and direct observation were used to assess perceived usability, acceptability and performance. Synthetic Minority Oversampling Technique was used to fit predictive models for visualization of patterns leading to good or poor A1Control performance.

Results:

Participants (N = 13) were 28-70 years old, 10 used automated insulin delivery and 12 used continuous glucose monitoring (CGM). Mean NPS was 9.5 (range 9-10) and positive sentiment during interviews indicated very high acceptability. SUS (mean 88.5, range 70-100) indicted a high perceived usability. CGM percent wear ≥ 94%, time spent in hypoglycemia ≤ 54 mg/dl of <0.01%, and <70 mg/dl of 0.5% predicted successful site-occlusion scenario performance with 100% accuracy. BOLUS score ≥ 2, TDD ≥ 34, and technology brand predicted exercise scenario success with 100% accuracy. There were an insufficient number of failed hypoglycemia scenarios to assess predictors.

Conclusion:

A1Control shows potential to increase access and frequency of self-management and technology education. Additional study is needed to determine sustained engagement and benefit.

Keywords

Insulin pump continuous glucose monitoring automated insulin delivery devices diabetes self-management education simulation

Background

For the 1.5 million individuals with type 1 diabetes in the United States, the importance of glycemic control in the prevention of complications is well established.¹ However, most individuals do not meet glucose targets^2,3 despite advances in therapeutics and technologies. Diabetes technologies (ie, insulin pump, continuous glucose monitor [CGM], automated insulin delivery [AID]) improve quality of life, glycemic control,^4-7 and possibly mortality⁸; however, safe and effective use requires technological sophistication, foundational diabetes self-management knowledge and access to care.^7,9-14 Benefit from insulin pumps, even those providing AID, depends greatly on individual utilization and self-management behavior including frequency of self-monitored blood glucose (SMBG),¹⁵ use of bolus calculators^15-18 and utilization of advanced features.^13-15,19

Diabetes self-management education (DSME) improves glycemic control,^20-22 but over 50% of individuals with diabetes in the United States report that they have never received DSME.²³ Many rural counties face the highest

rates of diabetes yet have the most limited access to endocrinology clinics and educators.²³ For insulin pump users, expert groups recommend periodic reassessment and education.^10,11,24,25 Insulin pump retraining is associated with significant improvements in patient knowledge and skills as well as reductions in A1C.^26-28

Currently, insulin pump education is not standardized and largely relies on in-person visits.^10-12,29 Initial training focuses on the technical aspects of insulin pump therapy (ie, entering pump settings, site insertion, bolus administration) at the expense of critical thinking, emergency management, and utilization of advanced features.¹⁰ Traditional didactic education inadequately transfers knowledge to real world situations and prohibits application of knowledge to novel self-management dilemmas.³⁰ Multiple meta-analyses and systematic reviews of educational simulations demonstrate its effectiveness as an educational technique with advantages over traditional education formats in its ability to (1) facilitate lifelike training situations that would otherwise be unfeasible or too dangerous, (2) transfer tangible skills in addition to knowledge, (3) increase access to limited resources, and (4) enable distance learning.^31-36 Numerous studies have demonstrated the positive findings associated with simulation education, including increased time-on-task,³¹ enjoyment,^37,38 motivation,^39,40 deeper learning and long-term retention.^41,42 Although simulation is being used increasingly in health care provider training,⁴³ particularly in surgical specialties,⁴⁴ it is under-utilized in patient education technologies.

Our objectives were to iteratively develop a diabetes education simulation platform while (1) testing A1Control’s usability and acceptability among a rural sample of insulin pump users, and (2) developing a monitoring function by which diabetes device data can be used to predict patient performance.

Methods

Study design

A prospective, mixed methods observational design was used to evaluate usability and acceptability during use of A1Control’s alpha prototype. We utilized an iterative design process in which testing was completed in 2 rounds with unique participants in each round allowing technology modifications between rounds.⁴⁵ After round 1, OSU’s College Diabetes Network (CDN) members participated in a workshop to interpret and translate round 1 data into technology modifications.⁴⁶

Participants

Eligibility criteria included diagnosis of T1D, >18 years of age, insulin pump user, and lived in a rural county at least 30 minutes from a diabetes education center. Participants meeting eligibility criteria were recruited from a single large academic medical center endocrinology practice. The study was approved by The Ohio State University’s Institutional Review Board. All participants provided written informed consent prior to data collection.

Intervention

Initial concept generation, co-design, and modifications of the platform were done in concert with the CDN through focus group interviews and concept co-design workshops.⁴⁶

Description of A1Control

A1Control is a diabetes education simulation platform designed to provide diabetes device specific education in concert with essential DSME in a virtual environment. Within A1Control, people with diabetes care for an avatar and are prompted to navigate challenges using a simulated insulin pump. The patient’s simulation experience is guided and mentored by a second avatar who can assist the user in making decisions and provide feedback on performance (Figure 1). Throughout the simulation, the patient is presented with common and emergency management scenarios and must navigate the simulation using by making self-management decisions (ie, exercise, eat, sleep, wait, bolus) (Figure 2). A variety of foods can be selected for consumption each featuring an accessible nutrition label. Differential equations, derived from previously published models,^47,48 underpin the platform and facilitate physiologic glycemic responses to user actions (ie, carbohydrate consumption, insulin administration, physical activity). Diabetes devices, which currently include a full-scale virtual insulin pump and meter, are built directly into the platform. Discrete glucose values derived are displayed on the pump interface however trends and tracing data is not displayed. Patients can manipulate these devices (ie, enter carbohydrates, manipulate bolus doses, change settings) as they would physical versions of the technologies. The patient’s own insulin pump settings are used to tailor the simulation algorithms to mimic the users own physiologic patterns and insulin doses. The technology features both text and voice narration. Patient performance is tracked to assess competencies facilitating follow up education to target specific deficiencies in the future.

Figure 1.

A1Control Simulation Platform.

Figure 2.

A1Control Self-management Decisions.

Data Collection

Testing was initially conducted in clinic but converted to virtual testing using teleconference software in response to COVID-19 restrictions. During testing, participants navigated 3 simulations: infusion site occlusion, hypoglycemia, and exercise. Participant performance and response to simulation content was captured using video recording, platform database and observation. Participants were asked to use the think aloud method to distinguish between technology design vs self-management decisions.⁴⁹ Three questionnaires were administered including the Systems Usability Scale (SUS), a well-validated tool for the evaluation of technology usability. The SUS contains 10 items using a 5-point Likert scale.⁵⁰ The Net Promoter Score (NPS), a single item measure of acceptability, assessed how likely participants would be to recommend the platform to a friend.⁴⁵ The Revised Michigan Diabetes Knowledge Test (DKT2) was administered prior to usability testing.⁵¹ The DKT is comprised of a General Knowledge and an Insulin subscale. Upon completion of usability testing, debriefing semi-structured interviews were conducted with each participant by a single interviewer (ERF). Interviews focused on participant’s views regarding usability, acceptability, and recommendations for the technology. The interviews contained 3 Likert scale questions asking participants how difficult the platform was to use; how they would feel about using this platform as part of their initial insulin pump training; and how realistic were the situations and outcomes in the scenarios. All interviews were audio recorded, professionally transcribed, and reviewed for accuracy.

The 28 days of data prior to testing were retrieved from each participant’s personal diabetes devices (ie, insulin pump, CGM, AID) using Tidepool software (Palo Alto, CA). Self-management and glucose variables were aggregated from raw device data. Aggregate daily measures included BOLUS score to assess frequency of mealtime bolus administration.⁵² Participants received 1 point for each bolus administered at breakfast (0600 to 1000 h), lunch (1100 to 1500 h) and dinner (1600 to 2200 h) with a max score of 3.⁵² Blood glucose (BG) input was defined as any glucose value used for bolus administration. These glucose values come from linked meter SMBG, linked CGM, or manual entry of CGM or SMBG value. BG input is a self-management variable measuring the way individuals use glucose values regardless of the source.^53,54 CGM glucose values were aggregated into derived measures of glucose control consistent with ADA and international guidelines on glucose targets including time in range (TIR, 70-180 mg/dl), time above range (>180 mg/dl), time in hypoglycemia (<70 mg/dl & <54 mg/dl), and coefficient of variation (CV).^55-57

Analysis

Our sample size was based on prior research suggesting that testing new technologies with 5-8 users will expose the majority of usability problems, using >50% of users reporting a problem as a threshold for modification.^58,59

A1Control platform performance indicators included:

(1) scenario success rate defined as successful completion of a scenario (achievement of target glucose, 70-180 mg/dl) regardless of the number of “incorrect” choices as there may be several choices resulting in a positive

(2) number of platform confusion errors (count of incorrect or erroneous navigation moves attributable to A1Control design) vs. number of self-management errors (count of incorrect choices attributable to deficits in diabetes self-management knowledge) were computed for individual scenarios and across all scenarios per participant;

(3) Reason for scenario failure was evaluated using platform performance indicator 2 for those participants who did not successfully complete individual scenarios.

(4) Time to scenario completion was measured directly from the platform’s database for individual scenarios per participant.

Interview data, fieldnotes, and memos were imported into NVivo^TM for data management and analysis.^60-62 Interviews were initially coded by one of two coders, (CL) and (LJ) prior to a second coder, (ERF) reviewing 100% of interviews. Data were used in concert with observation, platform performance data and survey data to assess usability, acceptability and inform A1Control revisions between rounds.

Diabetes device data was used both to (1) describe participant self-management and glycemic control patterns and (2) in models predicting A1Control scenario success. Decision trees were fit for each scenario to (1) visualize the relationship between participant’s diabetes device data and A1Control performance and (2) to measure predictive performance for the relationship. Given the small sample size, Synthetic Minority Oversampling Technique was used and decision trees were run on a withheld test set of (N = 4) to determine accuracy.⁶³ To avoid picking up on random noise in the small simple size, a random forest was fit to determine features importance, and features with a mean increase in the gini index of greater than 20 were removed. The gini index is a measure which helps determine how important individual features are in determining performance. A decision tree was then fit on the subsetted dataset and predictive performance was calculated on the test set to gather baseline descriptive self-management data and to evaluate associations between personal device data and platform performance.

Results

Thirteen adults with T1D participated in the study, 5 in round 1 and 8 in round 2. Eight visits were conducted virtually, all in round 2. All participants lived >30 minutes from a diabetes education center and represented 10 rural Ohio counties. The mean age was 51.4 years (range 28-71), mean duration of diabetes was 30 years (range 11-61) and mean A1C was 7.5% (range 5.9%-9.9%). Most participants (n = 10), used AID devices and 12 used CGM. Twelve of the 13 participants had one or more diabetes complications (ie, retinopathy, neuropathy, nephropathy, cardiovascular disease). Table 1 depicts demographics, diabetes characteristics, and diabetes technology use for the participants stratified by round of testing. Demographics and A1C were similar for participants in round 1 and round 2 of testing.

Table 1.

Demographic, diabetes characteristics and baseline knowledge stratified by round of testing.

Characteristic	Total (N = 13)	Round 1 (N = 5)	Round 2 (N = 8)
Female, N (%)	7 (54)	3 (60)	4 (50)
Age (years), Mean (range)	51.4 (28-70)	51.2 (35-70)	51.4 (28-70)
White, N (%)	13 (100)	5 (100)	8 (100)
BMI, Mean (range)	34.1 (26.5-49.9)	32.2 (26.6-41.9)	35.2 (27.6-49.9)
Duration of diabetes (years), Mean (range)	29.8 (11-61)	32.8 (16-61)	28.1 (11-45)
Duration of pump use (years), Mean (range)	7.5 (2-21)	8.4 (2-21)	6.9 (2-17)
A1C, % (range)	7.5% (5.9-9.9%)	7.4% (5.9-8.8%)	7.5% (6.6-9.9%)*
AID, N (%)	10 (77)	4 (80)	6 (75)
CGM, N (%)	12 (92.3)	4 (80)	8 (100)
Diabetes complications
Nephropathy N (%)	4 (31)	1 (20)	3 (38)
Retinopathy N (%)	5 (39)	2 (40)	3 (38)
Neuropathy N (%)	7 (54)	4 (80)	3 (38)
Cardiovascular disease N (%)	10 (77)	4 (80)	6 (75)
Diabetes knowledge
DKT2, Mean (range)	20.2 (17-22)	20.2 (18-22)	20.3 (17-22)
DKT2 general sub-section, Mean (range)	11.9 (10-13)	11.8 (10-13)	12 (10-13)
DKT2 insulin sub-section, Mean (range)	8.3 (7-9)	8.4 (8-9)	8.3 (7-9)

A1C in Group 2 N = 6 with no recent A1C for 2 participants.

Participants demonstrated a high degree of baseline diabetes knowledge with a mean DKT2 score of 20/22 (range 17-22), and a mean General Knowledge sub-section score of 12/13 (range 10-13) and a mean Insulin sub-section score of 8/9 (range 7-9) (Table 1).

Self-management and glycemic control variables were examined for all participants and by round of testing (Table 2). Participants performed a mean of 5.1 SMBG/day (range 3-10), mean 3.8 BG input/day (range 2-8), and mean 4.6 bolus/day (range 1-8). AID users (N = 10) perform a mean of 2.7 calibrations/day (range 0-5) and spent a mean of 62.3% time in auto-mode (range 0-98%). TIR, time above range, and time below range (<70 mg/dl and <54 mg/dl) were examined for participants who used CGM (N = 12). Participants spent a mean of 67.7% TIR (range 40.5-79.5%), mean 29.6% of time >180 mg/dl (range 20.5-59.5), mean 0.9% of time <70 mg/dl (range 0-5.32) and had a CV of 28%.

Table 2.

Diabetes device self-management and glucose characteristics stratified by round of testing.

Characteristics, Mean (range)	Total (N = 12)	Round 1 (N = 5)	Round 2 (N = 7)
Self-management
SMBG/day, Mean (range)	5.08 (3-10)	6.4 (3-10)	4.14 (3-8)
BG pump entries/day	3.75 (2-8)	4.8 (2-8)	3 (2-6)
Bolus/day	4.58 (1-8)	6.2 (5-8)	3.4 (1-6)
BOLUS score/day	2.08 (1-3)	2 (2)	2.14 (1-3)
Carbohydrate (grams/day)	139.83 (33-273)	117.6 (33-177)	155.71 (86-273)
Carbohydrates/bolus	38.18 (18-75)	30.1 (18-34.5)	43.95 (32.25-75)
CGM (% wear/day)	85.19 (58-100)	95.2 (83.3-99.7)	91.62 (51.4-100)
Calibration/day (if AID user) (n = 10)	2.67 (0-5)	3.75 (3-5)	1.64 (0-3)
Time in auto-mode (if AID user) (n = 10)	62.3 (0-98)	73.69 (0-98.8)	53.2 (0-93.47)
Insulin delivery	Total (N = 12)	Round 1 (N = 5)	Round 2 (N = 7)
Total units insulin/day	43.8 (21.7-95.7)	49.4 (21.7-95.7)	39.9 (25.5-67.8)
Units/kg/day
Percent Basal/day	58.2 (38.4-96.1)	58.5 (44.0-78.5)	58.0 (38.4-96.1)
Glucose Characteristics; CGM users	Total (N = 12)	Round 1 (N = 5)	Round 2 (N = 7)
% Time > 180 mg/dl	29.57 (20.5-59.5)	24.6 (20.5-31.6)	32.4 (21.2-59.5)
% Time-in-target	67.7 (40.5-79.5)	71.8 (67.0-79.5)	65.4 (40.5-78.1)
% Time < 70 mg/dl	0.88 (0-5.32)	0.79 (0-2.1)	0.93 (0-5.3)
% Time < 54 mg/dl	0.31 (0-3.36)	0	0.48 (0-3.36)
CV (%)	28.0 (18.5-38.4)	24.2 (18.5-29.8)	30.2 (26.6-38.4)

(SMBG) Self-monitored blood glucose, (BOLUS-score) 1 point each for a mealtime bolus occurring between 0600 to 1000, 1100 to 1500, and 1600 to 2200, max score/day is 3.

Usability and acceptability

Mean NPS of 9.5 (range 9-10) and positive responses to interview questions indicated very high acceptability. SUS (mean 88.5, range 70-100) indicted a high degree of perceived usability (Table 3). The majority of participants reported the platform was extremely easy (n = 7) or very easy (n = 5) to use and found the simulation scenarios and glucose outcomes extremely realistic (n = 10) or very realistic (n = 3). When asked how they would have felt about using a simulation technology like the platform as part of their initial pump education, the majority reported they would have very much liked to use it (n = 11).

Table 3.

NPS, SUS and platform performance indicators stratified by round of testing.

Usability & Acceptability Survey data, Mean (range)	Total (N = 13)	Round 1 (N = 5)	Round 2 (N = 8)
NPS	9.5 (9-10)	9.6 (9-10)	9.5 (9-10)
SUS	88.5 (70-100)	92 (82.5-100)	86.3 (70-100)
Platform performance
Site-occlusion
Time to completion (min:sec), range	03:28-10:46	03:28-10:46	04:03-08:37
Scenario success rate, N (%)	9 (69.2)	4 (80)	5 (62.5)
Reason for scenario failure (N)	Self-management errors(2)	Self-management (1)	Self-management (1)
	Algorithm issue(1)		Algorithm issue (1)
	Platform confusion(1)		Platform confusion (1)
Exercise
Time to completion (min:sec), range	02:01-07:23	02:24-04:10	02:01-07:23
Scenario success rate, N (%)	7 (53.9)	2 (40)	6 (75)
Reason for scenario failure (N)	Self-management errors (3)Algorithm issue(1)Platform confusion (1)	Self-management (2)Platform confusion (1)	Algorithm issue (1)Self-management (1)
Hypoglycemia
Time to completion (min:sec), range	01:17-08:25	01:17-01:51	01:40-08:25
Scenario success rate, N (%)	12 (92.3)	4 (80)	8 (100)
Reason for scenario failure (N)	Self-management error (1)	Self-management error (1)

Platform performance indicators are presented in Table 3. Time to scenario completion varied significantly depending on participant choices made throughout the simulation. Self-management errors observed during the simulation included choosing carbohydrate free foods to treat or prevent hypoglycemia (N = 2 participants with one participant making the error in both the exercise and hypoglycemia scenarios for a total of 3 errors), failure to check ketones (N = 3), failure to change pump infusion site (N = 2), and failure to prevent hypoglycemia with carbohydrates or temporary basal decrease prior to exercise (N = 7). Participants who failed the site occlusion scenario due to self-management errors (n = 2) stated they did not change the site because a visual inspection revealed no obvious problems with the site. Several other participants (n = 6) had a delay in changing the site but ultimately did successfully complete the scenario. In the exercise scenario, one participant consumed a carb free food to prevent hypoglycemia and stated this is how she typically treats and prevents hypoglycemia. Interestingly, this participant had the highest percentage of time in hypoglycemia (5.3% < 70 mg/dl and 3.4% < 54 mg/dl). Failure to locate a glucose value was the predominate platform confusion issue in the first round of testing with 3 of the 5 participants in round 1 reporting this issue. Other platform performance indicators including the number of prompts needed to complete scenarios and perceived difficulty completing tasks were evaluated to inform A1Control modifications.

Round 1 modifications included the addition of a glucose meter, playback button, algorithm adjustments and avatar upgrades to address platform confusion and functionality issues. Platform modifications made in round 1 appeared to successfully address functionality and confusion issues with none of the 8 participants in round 2 reporting the same problems. Modifications suggested from round 2 for future consideration included: addition of a virtual CGM, additional content library, increased personalized content, additional nutritional content, and brand specific devices.

Predictors of performance

CGM glucose and self-management variables (ie, time-in-range, CGM percent wear) were strongly associated with A1Control performance therefore only participants using CGM were included in the analysis (N = 12). Time-in-range 67%, mean SMBG <152 mg/dl, hypoglycemia, and time above target <30% were found to be highly predictive of site occlusion scenario success in the original model, but with a gini of >20% they were iteratively removed from the final model. In the final model constructed on subsetted data, CGM percent wear ≥94%, time spent in hypoglycemia ≤54 mg/dl of <0.01%, and time spent in hypoglycemia <70 mg/dl of 0.5% predicted site occlusion scenario success with 100% accuracy. Original exercise scenario models showed a high association between percent basal <67%, mean carb/bolus ≥34, and carbohydrates/day >104 were found to be highly predictive of exercise scenario success, however, each was found to have a gini of >20% and, therefore, each was iteratively removed to avoid overfitting. Available subsetted data showed BOLUS score ≥ 2, TDD ≥ 34, and brand (670G and Omnipod Dash compared to Medtronic 500 and 700 series) predicted exercise scenario success with 100% accuracy. Predictive models were not fit for hypoglycemia because only 1 participant failed the scenario leaving insufficient data for model development.

Discussion

The results of this study suggest a high degree of acceptance and perceived usability for diabetes education simulation technologies even among an older and rural cohort of adults with diabetes. All NPS scores were >8, indicative of an exceptionally high degree of technology acceptance,^45,64 meaning that participants were extremely likely to recommend the technology to a friend with diabetes. Similarly, SUS scores were very high ranging from 70-100 indicating a high degree of perceived ease of use and satisfaction with the system.^45,50 These scores were substantiated during debriefing interviews where participants expressed predominately positive sentiment regarding A1Control as a diabetes educational tool. Interestingly, NPS and SUS scores were high and feedback was generally positive for both rounds of testing despite modifications made prior to round 2.

The participants had a high degree of baseline diabetes knowledge as measured by the DKT2 survey. Additionally, they were an experienced cohort with a lengthy history of diabetes and history of pump use. Their diabetes device download data also showed a relatively high frequency of self-management behaviors (eg, SMBG/day, bolus/day, BG input/day) and a fair degree of glycemic control (ie, A1C, TIR). Their experience made them extremely well suited for design research in that they were able to compare platform scenarios to their real-world experience and contrast A1Control’s education to traditional didactic diabetes education. While the participants’ knowledge and patterns of self-management were a benefit at this stage of technology development, we acknowledge that the experiences of this cohort may not be generalizable to more novice diabetes device users. There is a need for accessible and personalized device education not just for ongoing education but perhaps most importantly for initial education. Therefore, future usability, and eventually efficacy testing should incorporate multiple cohorts with varied degrees of experience.

We identified one predominant platform confusion issue during round 1 with the majority of participants unable to locate a glucose value on the platform. During debriefing interviews, participants indicated they specifically wanted an option to test a meter SMBG. Round 1 participants were comprised of one non-CGM pump user and 4 AID users, though all still required routine SMBG testing. Participants’ recommendations in round 1 were, therefore, a reflection of their personal daily self-management experience. An indicator of success in our usability testing is that we were able to clearly identify a predominant platform confusion issue in round 1 and created a solution that addressed the problem in that the same issue was not identified in round 2.⁴⁵ Once a glucose meter was added to A1Control, all of the participants in round 2 were observed easily locating a glucose value when they desired.

Participants performed very well on the hypoglycemia scenario with successful completion for all but one participant, whereas more participants were unable to obtain a successful outcome in the exercise and site-occlusion scenarios. In each scenario, 2 participants were unable to achieve a successful outcome due to self-management errors. Participant’s comments indicating they did not change the site because no site issue could be externally visualized revealed deficits or gaps in knowledge. These participants stated that it had either been several years since they had a site issue or they had never had a failed site. They attributed this unfamiliarity with the situation as a reason for delay in site change. In the exercise scenario, the participant’s choice in treatment and prevention of hypoglycemia with a carbohydrate free food, illuminated real world self-management behaviors that are contrary to best practice and may have contributed to the participant’s high percentage of time in hypoglycemia. These scenario failures illustrate advantages in the use of simulation technologies. First, simulation technologies allow us to facilitate lifelike trainings for critical situations that would typically be unfeasible or too dangerous to create in real-world environments, thus providing a clear advantage over traditional didactic approaches.⁶⁵ Secondly, simulation technologies can serve as resources to assess patient knowledge and identify shortcoming.

Our analysis using personal device data to predict A1Control platform performance was limited by the small sample size. Our objective was to establish trends and examine what variables the models might use to make decisions in the future. Because of the limited sample size, it was imperative that decisions trees and models be evaluated not only for accuracy against a withheld set of participants, but also that models be evaluated from a clinical perspective to assess if the models “ring true” to provider experience.

Infusion site problems are common and not easily predictable, indicating the need for anticipatory patient education and training.⁶⁶ In the final site occlusion model, CGM percent wear ≥ 94% and time spent in hypoglycemia predicted site occlusion scenario success with 100% accuracy. A reasonable clinical case can certainly be made for the predictive relationship and direction of these variables. While there is little research evaluating self-management behaviors associated with DKA, CGM use is associated with lower risk of DKA.⁶⁷ To avoid overfitting of the models, variables found to be highly correlated with site occlusion success (eg, TIR, SMBG/frequency, time > 180 mg/dl) were removed; however given the small sample size we cannot definitively say these variables would not emerge as key indicators in future larger samples.

For the exercise scenario, BOLUS score≥ 2, TDD ≥ 34, and brand (670G and Omnipod Dash compared to Medtronic 500 and 700 series) were found to predict exercise scenario success with 100% accuracy. It is possible that brand may reflect more recent diabetes education as it is likely that 670G and Omnipod Dash participants had more recent pump training than the Medtronic 500 and 700 series users. Percent basal <67% was found to be highly correlated with successful completion of the exercise scenario but was removed to prevent overfitting. Again, due to the small sample size we cannot definitively say percent basal would be excluded from models incorporating larger sample sizes.

Conclusion

Our analysis shows a high degree of acceptance and usability of a simulation self-management technology, A1Control, that could improve access to and frequency of DSME and technology education. Our use of personal device data to establish predictive patterns provides a viable path for personalized assessment and education in the future. Future research is needed to determine sustained engagement and patient outcomes.

Footnotes

Acknowledgements

The authors thank Randy Sanborn for his extensive engineering contributions to A1Control’s design and development and all members of The Ohio State University College Diabetes Network for their continued research contributions.

Abbreviations

continuous glucose monitor (CGM), automated insulin delivery (AID); self-monitoring blood glucose (SMBG); diabetes self-management education (DSME); College Diabetes Network (CDN); Systems Usability Scale (SUS); Net Promoter Score (NPS); Revised Michigan Diabetes Knowledge Test (DKT2); time in range (TIR); coefficient of variation (CV)

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: Eileen R. Faulds and Carl W. Noble are co-founders of A1Control a diabetes education startup

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The project described was supported by Award Number Grant UL1TR002733 from the National Center For Advancing Translational Sciences. Technology development was supported by the State of Ohio, Ohio Development Services Agency and Ohio Third Frontier through a Technology Validation Start-up Fund award (TECG20180459). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Center For Advancing Translational Sciences, the National Institutes of Health, Ohio Third Frontier or the State of Ohio.

ORCID iD

Eileen R. Faulds

Data Accessibility

The data that support the findings of this study are available from the corresponding author upon reasonable request.

References

Diabetes Control and Complications Trial Research Group; Nathan

Genuth

, et al. The effect of intensive treatment of diabetes on the development and progression of long-term complications in insulin-dependent diabetes mellitus. N Engl J Med. 1993;329:977-986. doi:10.1056/NEJM199309303291401

American Diabetes Association. Standards of medical care in diabetes-2015 abridged for primary care providers. Clin Diabetes. 2015;33:97-111. doi:10.2337/diaclin.33.2.97

Miller

Foster

Beck

, et al. Current state of type 1 diabetes treatment in the U.S.: updated data from the T1D Exchange clinic registry. Diabetes Care. 2015;38:971-978. doi:10.2337/dc15-0078

Bayrakdar

Noureddine

Farhood

Nasrallah

. Comparison of quality of life in a group of Lebanese type 1 diabetics on insulin pump and those on multiple daily injections. J Med Liban. 2014;62:22-26.

Enes

Martin-Frias

Alvarez

Yelmo

Alonso

Barrio

. Achievement of metabolic control goals set by the American Diabetes Association and the International Society for Pediatric and Adolescent Diabetes in pediatric patients with type 1 diabetes from Spain. Diabetes Res Clin Pract. 2015;107:300-305. doi:10.1016/j.diabres.2014.08.033

Golden

Sapir

. Methods for insulin delivery and glucose monitoring in diabetes: summary of a comparative effectiveness review. J Manag Care Pharm. 2012;18:S1-S17.

Juvenile Diabetes Research Foundation Continuous Glucose Monitoring Study Group; Tamborlane

Beck

, et al. Continuous glucose monitoring and intensive treatment of type 1 diabetes. N Engl J Med. 2008;359:1464-1476. doi:10.1056/NEJMoa0805017

Steineck

Cederholm

Eliasson

, et al. Insulin pump therapy, multiple daily injections, and cardiovascular mortality in 18,168 people with type 1 diabetes: observational study. BMJ. 2015;350:h3234. doi:10.1136/bmj.h3234

Clements

Foster

Maahs

, et al. Hemoglobin A1c (HbA1c) changes over time among adolescent and young adult participants in the T1D exchange clinic registry. Pediatr Diabetes. 2016;17:327-336. doi:10.1111/pedi.12295

10.

Grunberger

Abelseth

Bailey

, et al. Consensus statement by the American Association of Clinical Endocrinologists/American College of Endocrinology insulin pump management task force. Endocr Pract. 2014;20:463-489. doi:10.4158/EP14145.PS

11.

Heinemann

Fleming

Petrie

Holl

Bergenstal

Peters

. Insulin pump risks and benefits: a clinical appraisal of pump safety standards, adverse event reporting and research needs. A joint statement of the European Association for the Study of Diabetes and the American Diabetes Association Diabetes Technology Working Group. Diabetologia. 2015;58:862-870. doi:10.1007/s00125-015-3513-z

12.

Petrie

Peters

Bergenstal

Holl

Fleming

Heinemann

. Improving the clinical value and utility of CGM systems: issues and recommendations: a joint statement of the European Association for the Study of Diabetes and the American Diabetes Association Diabetes Technology Working Group. Diabetes Care. 2017;40:1614-1621. doi:10.2337/dci17-0043

13.

Joubert

Reznik

. Personal continuous glucose monitoring (CGM) in diabetes management: review of the literature and implementation for practical use. Diabetes Res Clin Pract. 2012;96:294-305. doi:10.1016/j.diabres.2011.12.010

14.

Wolpert

. Establishing a continuous glucose monitoring program. J Diabetes Sci Technol. 2008;2:307-310. doi:10.1177/193229680800200224

15.

Wilkinson

McFann

Chase

. Factors affecting improved glycaemic control in youth using insulin pumps. Diabet Med. 2010;27:1174-1177. doi:10.1111/j.1464-5491.2010.03068.x

16.

Shashaj

Busetto

Sulli

. Benefits of a bolus calculator in pre- and postprandial glycaemic control and meal flexibility of paediatric patients using continuous subcutaneous insulin infusion (CSII). Diabet Med. 2008;25:1036-1042. doi:10.1111/j.1464-5491.2008.02549.x

17.

Ziegler

Cavan

Cranston

, et al. Use of an insulin bolus advisor improves glycemic control in multiple daily insulin injection (MDI) therapy patients with suboptimal glycemic control: first results from the ABACUS trial. Diabetes Care. 2013;36:3613-3619. doi:10.2337/dc13-0251

18.

Glaser

Iden

Green-Burgeson

, et al. Benefits of an insulin dosage calculation device for adolescents with type 1 diabetes mellitus. J Pediatr Endocrinol Metab. 2004;17:1641-1651.

19.

Wen

Frampton

Wright

Fattore

Shadbolt

Perampalam

. A pilot study of factors associated with glycaemic control in adults with Type 1 diabetes mellitus on insulin pump therapy. Diabet Med. 2016;33:231-234. doi:10.1111/dme.13001

20.

Beck

Zhang

Shay

, et al. Diabetes knowledge in young adults: associations with hemoglobin A1C. Fam Syst Health. 2015;33:28-35. doi:10.1037/fsh0000101

21.

Elliott

Rankin

Jacques

, et al. A cluster randomized controlled non-inferiority trial of 5-day Dose Adjustment for Normal Eating (DAFNE) training delivered over 1 week versus 5-day DAFNE training delivered over 5 weeks: the DAFNE 5 x 1-day trial. Diabet Med. 2015;32:391-398. doi:10.1111/dme.12621

22.

DAFNE Study Group. Training in flexible, intensive insulin management to enable dietary freedom in people with type 1 diabetes: dose adjustment for normal eating (DAFNE) randomised controlled trial. BMJ. 2002;325:746.

23.

Centers for Disease Control and Prevention (CDC). National diabetes statistics report 2020: estimates of diabetes and its burden in the United States. 2020. Accessed on April 2, 2021. https://www.cdc.gov/diabetes/pdfs/data/statistics/national-diabetes-statistics-report.pdf2020

24.

Tanenbaum

Adams

Hanes

, et al. Optimal use of diabetes devices: clinician perspectives on barriers and adherence to device use. J Diabetes Sci Technol. 2017;11:484-492. doi:10.1177/1932296816688010

25.

Peters

Ahmann

Battelino

, et al. Diabetes technology-continuous subcutaneous insulin infusion therapy and continuous glucose monitoring in adults: an Endocrine Society clinical practice guideline. J Clin Endocrinol Metab. 2016;101:3922-3937. doi:10.1210/jc.2016-2534

26.

Scheiner

Sobel

Smith

, et al. Insulin pump therapy: guidelines for successful outcomes. Diabetes Educ. 2009;35 Suppl 2:29S-41S; quiz 28S, 42S-3S. doi:10.1177/0145721709333493

27.

Jenkins

Knott

Brooks

. Insulin pump users require recurrent education for the management of pump failure. J Diabetes Nurs. 2016;20:364-369.

28.

Ozgen Saydam

Yilmazmis

Aydin

, et al. The effect of retraining on treatment success, quality of life, and metabolic parameters in patients with type 1 diabetes using an insulin pump. Med Princ Pract. 2017;26:325-330. doi:10.1159/000475935

29.

Zhang

Jones

Klonoff

. Second insulin pump safety meeting: summary report. J Diabetes Sci Technol. 2010;4:488-493.

30.

Collins

Brown

Newman

. Cognitive Apprenticeship: Teaching the Craft of Reading, Writing, and Mathematics. Lawrence Erlbaum Associates; 1989.

31.

Kavanagh

Luxton-Reilly

Wuensche

, et al. A systematic review of Virtual Reality in education. Themes Sci Technol Educ. 2017;10:85-119.

32.

Vogel

Cannon-Bowers

Bowers

Muse

Wright

. Computer gaming and interactive simulations for learning: a meta-analysis. J Comput Assist Learn. 2006;34:229-243.

33.

Wouters

van Oostendorp

. Computer gaming and interactive simulations for learning: a meta-analysis. Comput Educ. 2013;60:412-425.

34.

Sitzmann

. A meta-analytic examination of the instructional effectiveness of the instructional effectivness of computer-based simulation games. Pers Psychol. 2011;64:489-528.

35.

Tobias

Fletcher

Chen

. Digital games as educational technology: promise and challenges in the use of games to teach. Educ Technol. 2015;55:3-12.

36.

Mikropoulos

Natsis

. Educational virtual environments: a ten-year review of empirical research (1999-2009). Comput Educ. 2011;56:769-780.

37.

Apostolellis

Bowman

. Evaluating the effects of orchestrated, game-based learning in virtual environments for informal education. In: Proceedings of the 11th Conference on Advances in Computer Entertainment Technology – ACE ’14, Funchal, Portugal, 11-14 November. ACM Press; 2014:1-10.

38.

Ferracani

Pezzatini

Del Bimbo

. A natural and immersive virtual interface for the surgical safety checklist training. In: Effelsberg

SGbW

, ed. Proceedings of the 2014 ACM International Workshop on Serious Games, Orlando, FL, 7 November. ACM Press; 2014:27-32.

39.

Cheung

SKS

Fong

Wang

Kwok

. Hybrid Learning and Continuing Education. Springer; 2013.

40.

Sharma

Agada

Ruffin

. Virtual reality classroom as a constructivist approach. In: Proceedings of the 2013 IEEE Southeastcon, Jacksonville, FL, 4-7 April. IEEE; 2013:1-5. doi:10.1109/SECON.2013.6567441

41.

Huang

Rauch

Liaw

. Investigating learners’ attitudes toward virtual reality learning environments: based on a constructivist approach. Comput Educ. 2010;55:1171-1182.

42.

Rizzo

Bowerly

Buckwalter

Klimchuk

Mitura

Parsons

. A virtual reality scenario for all seasons: the virtual classroom. CNS Spectr. 2006;11:35-44.

43.

Schwaab

Kman

Nagel

, et al. Using second life virtual simulation environment for mock oral emergency medicine examination. Acad Emerg Med. 2011;18:559-562. doi:10.1111/j.1553-2712.2011.01064.x

44.

Huang

H-M

Liaw

S-S

Lai

C-M

. Exploring learner acceptance of the use of virtual reality in medical education: a case study of desktop and projection-based display systems. Interact Learn Environ. 2016;24:3-19.

45.

Thomas Tullis

. Measuring the User Experience: Collecting, Analyzing, and Presenting Usability Metrics. Morgan Kaufmann Publishers Inc.; 2013.

46.

Faulds

Lewis

Dungan

Happ

. From consultant to co-investigator: one diabetes research team’s journey of patient engagement. West J Nurs Res. Published online February 5, 2021. doi:10.1177/0193945921992012

47.

Man

Breton

Cobelli

. Physical activity into the meal glucose-insulin model of type 1 diabetes: in silico studies. J Diabetes Sci Technol. 2009;3:56-67. doi:10.1177/193229680900300107

48.

Wong

Chase

Hann

, et al. Development of a clinical type 1 diabetes metabolic system model and in silico simulation tool. J Diabetes Sci Technol. 2008;2:424-35. doi:10.1177/193229680800200312

49.

Berry

Broadbent

. The role of instruction and verbalization in improving performance on complex search tasks. Behav Inf Technol. 1990;9:175-190.

50.

Brooke

. SUS: a “quick and dirty” usability scale. In: Jordan

Thomas

Weerdmeester

McClelland

, eds. Usability Evaluation in Industry. Taylor and Francis; 1996:189-194.

51.

Fitzgerald

Funnell

Anderson

Nwankwo

Stansfield

Piatt

. Validation of the Revised Brief Diabetes Knowledge Test (DKT2). Diabetes Educ. 2016;42:178-187. doi:10.1177/0145721715624968

52.

Patton

Clements

Fridlington

Cohoon

Turpin

Delurgio

. Frequency of mealtime insulin bolus as a proxy measure of adherence for children and youths with type 1 diabetes mellitus. Diabetes Technol Ther. 2013;15:124-128. doi:10.1089/dia.2012.0229

53.

Faulds

Hoffman

Grey

, et al. Self-management among pre-teen and adolescent diabetes device users. Pediatr Diabetes. 2020;21:1525-1536. doi:10.1111/pedi.13131

54.

Faulds

Militello

Tubbs-Cooley

Happ

. Evaluating feasibility of personal diabetes device data collection for research. Nurs Res. 2020;69:476-482. doi:10.1097/NNR.0000000000000464

55.

American Diabetes Association. 6. Glycemic targets: standards of medical care in diabetes-2021. Diabetes Care. 2021;44:S73-S84. doi:10.2337/dc21-S006

56.

Danne

Nimri

Battelino

, et al. International consensus on use of continuous glucose monitoring. Diabetes Care. 2017;40:1631-1640. doi:10.2337/dc17-1600

57.

Battelino

Danne

Bergenstal

, et al. Clinical targets for continuous glucose monitoring data interpretation: recommendations from the international consensus on time in range. Diabetes Care. 2019;42:1593-1603. doi:10.2337/dci19-0028

58.

Nielsen

. Usability Engineering. Academic Press; 1993.

59.

Rubin

. Handbook of Usability Testing: How to Plan, Design, and Conduct Effective Tests. John Wiley & Sons, Inc.; 1994.

60.

Hsieh

Shannon

. Three approaches to qualitative content analysis. Qual Health Res. 2005;15:1277-1288. doi:10.1177/1049732305276687

61.

Sandelowski

. Whatever happened to qualitative description? Res Nurs Health. 2000;23:334-340.

62.

Colorafi

Evans

. Qualitative descriptive methods in health science research. HERD. 2016;9:16-25. doi:10.1177/1937586715614171

63.

Chawla

Bowyer

Hall

Kegelmeyer

. SMOTE: synthetic minority over-sampling technique. J Artif Intell Res. 2002;16:321-357. doi:10.1613/jair.953

64.

MeasuringU. Measuring the reliability of the net promoter score. 2018. Accessed August 19, 2020. https://measuringu.com/nps-reliability/

65.

Chin

Dukes

Williams

. Assessment in simulation and gaming: a review of the last 40 years. Simul Gaming. 2009;40:553-568.

66.

Ross

Gray

Milburn

, et al. Insulin pump-associated adverse events are common, but not associated with glycemic control, socio-economic status, or pump/infusion set type. Acta Diabetol. 2016;53:991-998. doi:10.1007/s00592-016-0897-2

67.

Foster

Beck

Miller

, et al. State of type 1 diabetes management and outcomes from the T1D exchange in 2016-2018. Diabetes Technol Ther. 2019;21:66-72. doi:10.1089/dia.2018.0384

Simulation Platform Development for Diabetes and Technology Self-Management

Abstract

Background:

Method:

Results:

Conclusion:

Keywords

Background

Methods

Study design

Participants

Intervention

Description of A1Control

Data Collection

Analysis

Results

Usability and acceptability

Predictors of performance

Discussion

Conclusion

Footnotes

Acknowledgements

Abbreviations

Declaration of Conflicting Interests

Funding

ORCID iD

Data Accessibility

References