Benefits of Usability Evaluation in the Development Process of Diabetes Technologies Using the Example of a Continuous Glucose Monitoring System Prototype

Abstract

Background:

Usability engineering analyzes the interaction between the intended users and a device. Its implementation is mandatory for manufacturers to obtain regulatory approval for the European market. The aim of this evaluation was assessing the role of usability testing in the development process. For this purpose, a continuous glucose monitoring (CGM) device under development was investigated to determine whether it could be used safely and effectively by the intended users.

Methods:

Conduct of the usability testing was based on the international standard IEC 62366-1. Medical device use of CGM-experienced and non-experienced users (n = 15 each) was observed without initial training in use scenarios containing 18 tasks. The success rate of task completion was determined and the System Usability Scale (SUS) score was calculated from a questionnaire. A prototype of the FiberSense CGM System (EyeSense GmbH, Großostheim, Germany), comprising of a single-use sensor and a reusable detector, was investigated.

Results:

Most use errors made by both user groups were related to ease of handling of the reusable detectors. The SUS scores achieved in this study were below the pre-defined SUS score acceptance criterion of ≥68. The most frequently mentioned reason for use errors was an incomprehensible and non-chronological instructions for use (IFU).

Conclusions:

The evaluation provides valuable insights on how to improve usability of the prototype device and demonstrates the value of conducting structured usability testing prior to product finalization. The results reflected areas for improvement of the user interface, mainly by restructuring the IFU, provision of an additional leaflet, and device training prior to use.

Keywords

continuous glucose monitoring diabetes technology human factors SUS score usability use error

Introduction

Diabetes management has undergone a significant evolution,¹ and the use of continuous glucose monitoring (CGM) devices is expanding quickly. For persons with diabetes receiving extensive insulin therapy, maintaining near-normal glucose levels without crippling hypoglycemia is still a difficult task. For many of these people, improvements in CGMs have greatly lowered the burden of diabetes.^2-4 Usability testing is necessary to make sure that the product design and necessary training guarantee patient safety.

A prerequisite for the approval of any medical device is its safe and effective operation by the end user. Since the Medical Device Regulation (MDR) became effective in May 2021, usability engineering (or human factors engineering), which describes the analysis of the interaction between the intended users and a device, has gained importance in the development of medical devices, and manufacturers must demonstrate usability of their products in accordance with the international standard IEC 62366-1 for conformity assessment.^5,6 The usability process is primarily intended to eliminate use-related risks or to minimize them to an acceptable level.^7,8 Therefore, the usability engineering process is tightly connected to the risk management process according to ISO 14971.⁹ The manufacturer is required to identify hazard-related use scenarios and perform iterative, risk-based usability testing in the early phase of the development cycle (formative evaluations) in order to identify major design shortcomings and user interaction problems. At the end of the development phase, a final usability validation (summative evaluation) needs to be performed to confirm that the implemented user interface solutions and risk mitigation measures have been effective and prevent use errors that could result in harm to the intended users. As such, usability engineering is carried out throughout the entire design and development process of medical devices.⁶ In sum, identified use errors during a usability testing can support the development process as well as the design of safe and effective medical devices.⁷

In the field of diabetes technology, ease of use of devices is essential for optimal diabetes self-management, as these technologies need to be integrated into the daily lives of people with diabetes. For CGM systems, increased user-friendliness has been shown to promote acceptance and consistency of device use^10,11 emphasizing the importance of easy-to-use devices.

The focus of this study was a prototype FiberSense CGM system (FBS CGM system; EyeSense GmbH, Großostheim, Germany)^12,13 which was selected as a potential candidate by the Foundation for Innovative New Diagnostics (FIND), a non-profit organization supporting the development and establishment of novel, cost-effective diagnostic devices in low- and middle-income countries (LMICs),¹⁴ to increase the accessibility of CGM systems in these countries. In LMICs, the need for safe and cost-effective diabetes technologies is high: The International Diabetes Federation estimates that the number of people with diabetes is foreseen to increase from 537 million today to 700 million in 2045¹⁵ and this rise in prevalence will primarily affect LMICs.^16-18 Technology advances for diabetes management, such as CGMs, are not affordable to the majority of people with diabetes in LMICs, and more products with potential for cost-effectiveness through longer sensor wear time and transmitter/detector lifetime are needed.

Whereas performance assessment of the FBS CGM system has been conducted,¹³ the usability of the device has not finally been investigated at the time of this study. Therefore, this study focuses on whether the FBS CGM system can be safely and effectively used by CGM-experienced and CGM non-experienced users with special emphasis on how usability engineering can contribute to the development of safe and effective medical devices.

Methods

Design and Setting

This usability evaluation was performed at the Institut für Diabetes-Technologie Forschungs- und Entwicklungsgesellschaft mbH an der Universität Ulm, Ulm, Germany.

The study was designed according to the international standard, “IEC 62366-1:2015 Medical devices—Part 1: Application of usability engineering to medical devices”⁶ and “IEC 62366-2:2016 Medical devices—Part 2: Guidance on the application of usability engineering to medical devices”.¹⁹ However, the conduct of the evaluation deviated from the international standard in the following respects: Use scenarios were selected on the basis of the instructions for use (IFU) and not, as requested in the international standard, on the basis of the risk documentation of the manufacturer of the device under evaluation as the IFU available at the time of study conduct did not cover all hazard-related use scenarios of the risk analysis. The use scenarios depicted in Table 1 have been selected for usability evaluation on the basis of the IFU. The medical device under evaluation was an FBS CGM system prototype, characterized by a sensor lifetime of 28 days and a photometric glucose detection method.¹² The single-use sensor of the FBS CGM system consists of a glucose sensitive sensor fiber and a base plate which is attached to the skin with an adhesive tape. The reusable detector, a miniaturized fluorescence photometer, has to be coupled to the sensor during use and detached from the sensor for recharging and reuse with new sensors. A smartphone application (FiberSense App) is used to display the measurement result. The prototype of the FBS CGM system was not CE-marked at the time of testing; the devices used in the evaluation were provided by the manufacturer through FIND.

Table 1.

List of Selected User Tasks.

Task number		Task
1	Check the product components for completeness and the expiration date of the sensor on the package label.
2	Identify a possible insertion site. Inquiry: Which other sites are suitable for sensor insertion?
3	Prepare the insertion site at the upper arm for sensor application taking into account the necessary hygiene procedures (using an artificial skin).
4	Place the applicator on the prepared artificial skin and remove the injector safety seal.
5	Press the applicator button to detach the sensor from the applicator for attachment.
6	Remove the applicator and check the correct position of the sensor fiber.
7	Mount the detector to the base plate.
8	Switch on the smart device and locate and launch the App.
9	Pair the detector with the App.
10	Start the sensor in the App.
11	Inquiry: What should be done, if the displayed glucose values do not correspond to the symptoms, or no glucose values are shown?
12	Disconnect/Uncouple the detector from the App.*
13	Remove the detector for charging and attach the protective cap.
14	Charge the detector using the charging cradle.
15	During detector charging attach the over tape to the sensor.
16	Mount the detector to the base plate and pair the detector to the App again.
17	Stop the sensor.
18	Remove the detector and the sensor and dispose the used sensor properly.

Task 12 was classified as an optional task, as the timepoints for unpairing the detector from the App were insufficiently described in the IFU.

Participants

Two user groups of 15 participants each were included (Table 2). Participants of user group 1 were under intensive insulin therapy and used a CGM system regularly, whereas participants in user group 2 did not have any experience with CGM technology. Further inclusion criteria were 18 years of age or older, ability to speak, read and write German, and experience using a mobile device with graphical interface. Staff members or individuals who participated in usability tests for the particular medical device type regularly were ineligible to participate in the study.

Table 2.

Participant Characteristics.

Demographic		User Group 1n = 15	User Group 2n = 15
Gender, n (%)	Male	6 (40)	7 (46.7)
Gender, n (%)	Female	9 (60)	8 (53.3)
Age, years, (± SD)	Mean	48.5 (± 12.8)	60.7 (± 8.0)
	Minimum	24	38
	Maximum	68	73
Type of diabetes, n (%)	Type 1	15 (100)	0 (0)
Type of diabetes, n (%)	Type 2	0 (0)	15 (100)
Educational level, n (%)	Secondary School	9 (60)	12 (80)
	High School	3 (20)	1 (6.7)
	Bachelor	0	1 (6.7)
	Master / Diploma	2 (13.3)	1 (6.7)
	Doctorate	1 (6.7)	0
Native language, n (%)	German	15 (100)	15 (100)

Usability Testing

The method used for evaluation was a usability test, where the participants used the medical device within pre-defined test scenarios without the help of an instructor and completed a questionnaire on the usability of the FBS CGM system. The usability test simulated the given test scenarios in a general home setting. First, participants had unlimited time to familiarize with the IFU. Afterwards, they received the testing materials and the coordinator instructed the tasks to the participants, while an observer documented the task completion. After completing all test scenarios, a post-session interview was conducted which addressed questions on user experience of the device, when using the system and recommendations for improvement. Furthermore, the subjective root cause for each use error and use difficulty was identified. Comments and subjective statements from participants related to each use error and use difficulty were recorded. Finally, participants were asked to fill in a 12-item questionnaire on the usability of the medical device from which the first 10 items were used to calculate the System Usability Scale (SUS) score for estimation of the overall usability of the device. The ten-item SUS questionnaire is a validated and widely used instrument to assess the perceived usability of a variety of systems.²⁰ The two other items were analyzed separately.

Data Collection and Analysis

Task completion was documented by the study observer using an observation form, evaluating the success of task completion according to three parameters: correct use, use difficulty, and use error. All data were collected in form of observation records and questionnaires by the study staff. Data, answers and comments were transferred to an electronic data capture form as a double data entry. Personal data were processed in a pseudonymized form.

The rate of successful completion of each task by study participants was calculated. The acceptance criterion was defined to be at least 80% successful completion of each task (includes correct use and use difficulty). The SUS score was used to evaluate the overall device usability and should achieve a value ≥68. The SUS covers values from 0 (lowest usability) to 100 (highest usability), with 68 found to be the standard average SUS score.^20,21

Results

Usability Testing

Usability testing of the FBS CGM system was conducted with 30 participants, of whom 15 participants were experienced using a CGM system (user group 1), and 15 participants had not used a CGM system before (user group 2). Participants of user group 1 were people with type 1 diabetes (100%) with a mean age of 48.5 ± 12.8 years, whereas participants of user group 2 were people with type 2 diabetes (100%) with a mean age of 60.7 ± 8.0 years. Males and females were almost equally distributed in both user groups (see Table 2 for more details on participant characteristics).

Study participants’ medical device use was observed in use scenarios with 18 detailed tasks simulating the preparation to use the sensor, the attachment of the sensor, connection of the detector to the app, the use of the displayed glucose data, charging of the detector and the removal of the sensor (Table 1). The percentage of successful completion of each task is shown in Figure 1. In user group 1, 2 of the participants (13%) successfully performed all user tasks, 9 of the participants (60%) performed one to three use errors and four of the participants (27%) performed four to six use errors. In user group 2, three of the participants (20%) committed two to three use errors and 12 of the participants (80%) made four to seven use errors. The acceptance criterion of 80% successful completion was met for 14 and 9 tasks for user groups 1 and 2, respectively, with eight tasks in user group 1 and four tasks in user group 2 being 100% successfully completed (Figure 1).

Figure 1.

Side-by-side comparison of test results between user groups 1 and 2 (n = 15 each). Values on the bars indicate percentage of successful completion for each task written on the left. Tasks are displayed according to the order of completion. *Task did not meet the acceptance criterion of 80% successful task completion. Task 12 was only performed by three participants of each user group.

In user group 1, most of the use errors were observed in user tasks 13 (n = 9) and 6 (n = 7). Nine participants (60%) forgot to attach the protective cap after removing the detector from the sensor for charging (task 13) and seven participants (47%) had difficulties to remove the applicator from the sensor and to check the right position of the sensor fiber (task 6) (Figure 2). If the height of the sensor fiber is not checked after sensor application, the fiber could be damaged, bent or broken off when the detector is mounted. This in turn could lead to wrong or no glucose values being displayed.

Figure 2.

Features leading to use errors in user group 1 (a) and user group 2 (b). Bars indicate number of observed use errors for each task, values in parenthesis correspond to the percentage of observed use errors. The total amount of observed use errors (n) is 37 for user group 1 and 68 for user group 2. *Optional task, since the timepoints for unpairing the detector from the App were insufficiently described in the IFU. This task was performed by a total of three participants. Tasks are displayed according to decreasing percentage of observed use errors.

User group 2 participants made most of the use errors in user task 17 (n = 12) and 13 (n = 11). In both user groups, root cause analysis revealed that the most often mentioned reason for use errors related to these tasks was an incomprehensible and non-chronological IFU. The third most common use error in both groups was the pairing of the detector and the app. In this case, root cause analysis revealed that the most obvious reason for this use error was that the participants did not find the pairing code.

Questionnaire, SUS Score, and Post-Session Interview

Calculation of the SUS scores in Figure 3 showed a normal distribution in user group 1 and a positively skewed distribution in user group 2 with some outliers in the low range of the SUS score. The mean calculated SUS score for the questionnaire was 47 for user group 1 and 52 for user group 2 and therefore falls in the “not acceptable” and in the “marginally acceptable” range, respectively. The acceptance criterion that the system should achieve a SUS score ≥68 was not met for both user groups.

Figure 3.

SUS scores of user groups 1 and 2. Presented are median values, lower quartile values and upper quartile values (G1: 48 [33.8-55]; G2: 55 [50-62]). Mean SUS values are 47 and 52 for user groups 1 and 2, respectively.

Detailed results of the complete questionnaire are displayed in Figure 4. After having tested the device, 47% and 60% participants of groups 1 and 2, respectively, thought that most people learn to use the system quickly (question 7). Evaluation of questions concerning the comprehensibility of the IFU (question 11) and the size of the prototype device (question 12) revealed that 20% and 36% of the participants in user groups 1 and 2, respectively, considered the manual to be comprehensible, whereas 53% and 36% of the participants disagreed and 27% and 29% were neutral. The size of the device (question 12) was rated positively by 13% and 35% of the participants of user groups 1 and 2.

Figure 4.

SUS questionnaires for user groups 1 and 2. Questions 1 to 10 correspond to the SUS questionnaire. Questions 11 and 12 correspond to the general usability questionnaire. Values in the bar-graph indicate percentage of responses for each question.

The post-session interview revealed that experienced participants focused their criticism on the device handling, especially the detector handling, which was mainly perceived as difficult and complex, and this is largely due to the fact that it is a reusable system compared with many other systems that are single-use systems. Even though non-experienced participants perceived the device handling as complex, many rated the FBS CGM system as good.

Discussion

People with diabetes depend on the use of diabetes technologies in their daily self-management. Due to constant use, diabetes-related devices are an integral part of the daily life; therefore, even minor disruptions in the usability of the user interface due to multiple repetitions for tasks performed on a regular basis and tasks performed once or twice in a month can be inconvenient for the user.²² Hence, a well-designed user interface may contribute to increase user acceptance. The device presented in this evaluation is a prototype CGM system used to constantly monitor glucose levels in the interstitial fluid.

The clinical performance of the FBS CGM system has already been investigated and showed promising accuracy results.¹³ The current evaluation investigated the usability of an FBS CGM system prototype and used this prototype to demonstrate the importance of usability engineering in the development process of diabetes technologies. In this evaluation, two user groups were defined: (1) CGM-experienced participants using a CGM system for their regular therapy and (2) CGM non-experienced participants who have never used a CGM system before. The users of group 1 were recruited to represent typical users of CGM systems in Europe. In addition, user group 2 was added to assess usability of the system in CGM inexperienced users as an approximation of the intended users in LMICs, where CGM systems are scarce or not available at all.²³ Product characteristics of minimally invasive glucose monitoring systems have mostly been defined on requirements for people living in high-income countries, and usability and acceptability data are not widely available for this technology, especially from developing countries.²³ Overall, user group 1 managed better with the handling of the FBS CGM system, which was reflected by a higher rate of successful task completion and a reduced number of observed use errors compared with user group 2 (Figures 1 and 2).

According to the root cause analysis, a modification of the IFU may have prevented some of the use errors that occurred and could improve the interaction of the participant with the device. Furthermore, a leaflet highlighting most important handling steps would have been helpful for the users and is currently in preparation by the manufacturer for the final product. The testing and root cause analysis also revealed that use errors related to the pairing of the detector and the app could have been avoided by printing the PIN code for pairing on an additional site on the package or the charging tray or with a clear hint in the IFU or leaflet. Based on the results of this study, the manufacturer has now implemented this measure. In addition, the product design has been adapted to ensure that it is no longer necessary to check the height of the sensor fiber (task 6). In summary, these examples show that inadequacies in product design and IFU content as integral part of the user interface can have a considerable impact on the user-friendliness of the device. Therefore, the results of the usability testing are valuable for optimizing the device including the IFU (IFU used in the study was not intended to be the final version), provide important input for the improvement of the user interface through simple measures, and can thus help to reduce use-related risks.

The SUS scores achieved in this study were below the pre-defined SUS score acceptance criterion of ≥68. However, more than half of participants in group 2 found the system easy to use and around 60% of group 2 participants can imagine that most people learn to use the system quickly. Experienced participants mostly expressed neutrality and disagreement to feeling confident using the device as well as on the comprehensibility of the IFU. They showed a tendency to consider the FBS CGM system rather a complex device with some inconsistencies in device handling including the IFU, whereas the answers of non-experienced participants were more varied and included more positive than critical remarks. This resulted in a lower SUS score for experienced users despite a lower number of use errors compared with non-experienced users. These discrepancies could be attributed to the fact that experienced participants compare the device with their own approved, usually smaller device, which they are already familiar with its handling. Although the SUS score is a quick and cost-effective method to assess usability, the conduction of a usability study in which participants simulate the use of a medical device is considered one of the most effective methods in usability engineering.²⁴

A limitation of the study is a marked difference in age of the participants in the two user groups, with mean values of 48.5 years (user group 1) and 60.7 years (user group 2), respectively. First, this makes a direct comparison of the results between the two groups more difficult, as older adults may have reduced cognitive and visual abilities, which have been shown to often interfere with the use of self-monitoring and e-health devices.^25,26 Second, user group 2 does not ideally represent intended users in LMICs from an age perspective, as a younger cohort of people with type 1 diabetes without experience in CGM technology would have been better reflected the intended users in LMICs. However, in Germany, the vast majority of people with type 1 diabetes use a CGM system to monitor glucose levels which are also covered by the health insurance, whereas it is much less common for people with type 2 diabetes to wear a CGM system,²⁷ making it difficult to identify younger people with type 1 diabetes without CGM experience for this study.

A prior technical training, which is necessary for the use of a CGM system (and also foreseen for the final version of the FBS CGM system), is basically part of the user interface and should be considered in a final summative usability evaluation. In addition, a structured and evaluated CGM training by qualified members of a diabetes team is essential for successful therapy. Still, a lack of training might be the most effective way to assess usability and identify shortcomings of medical device prototypes that might not have been detected otherwise.

Conclusions

This evaluation highlights the value of usability testing in the development of medical devices. The results indicate that the CGM system prototype tested in this study was challenging to be operated by most of the participants without previous training. Usability testing identified several areas for improvement in the user interface, the optimization of which may improve the interaction between the intended user and the device. Specifically, a revision of the IFU might improve the usability and acceptability of the device. The results further emphasize the importance of a structured and well-designed IFU and leaflet as part of the user interface of a medical device and as risk mitigation measure.

Once the manufacturer has addressed areas for improvement for usability identified in this study, the FBS CGM system is a promising new market addition due to its long sensor wear time and rechargeable transmitter/detectors. A recent target product profile for new glucose self-monitoring technologies, developed by FIND in collaboration with technical and global health experts as well as LMIC users, includes a four-week duration of use for devices with adhesive, as well as the use of rechargeable transmitters/detectors.²⁸

Footnotes

Acknowledgements

The authors thank the participants as well as the study staff at the Institut für Diabetes-Technologie in Ulm including Tatjana Hummel, Christiane Kröner, and Pauline Schwarz.

Abbreviations

CE, Conformité Européenne; CGM, continuous glucose monitoring; FBS, FiberSense CGM system; FIND, Foundation for Innovative New Diagnostics; IEC, International Electrotechnical Commission; IFU, Instructions for use; ISO, International Organization for Standardization; LMIC, low- and middle-income country; MDR, Medical Device Regulation; SD, standard deviation; SUS, System Usability Scale

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: G.F. is general manager and medical director of the Institute for Diabetes Technology (Institut für Diabetes-Technologie Forschungs- und Entwicklungsgesellschaft mbH an der Universität Ulm, Ulm, Germany), which carries out clinical studies, eg, with medical devices for diabetes therapy on its own initiative and on behalf of various companies. G.F./IfDT have received research support, speakers’ honoraria, or consulting fees in the last three years from Abbott, Ascensia, Berlin Chemie, Boydsense, Dexcom, Lilly, Metronom, Medtronic, Menarini, MySugr, Novo Nordisk, PharmaSens, Roche, Sanofi, and Terumo. A.B., J.K., M.G.M., S.P., C.K., and G.F. are the employees of IfDT. B.V. and E.S. are the employees of FIND.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by FIND through a grant from the German Federal Ministry of Education and Research (Bundesministerium für Bildung und Forschung).

ORCID iDs

Anne Beltzer

Marta Gil Miró

Stefan Pleus

Beatrice Vetter

Guido Freckmann

References

Hirsch

Battelino

Peters

Chamberlain

Aleppo

Bergenstal

. Role of Continuous Glucose Monitoring in Diabetes Treatment. Arlington, VA: American Diabetes Association; 2018.

Beck

Riddlesworth

Ruedy

, et al. Effect of continuous glucose monitoring on glycemic control in adults with type 1 diabetes using insulin injections: the DIAMOND randomized clinical trial. JAMA. 2017;317(4):371-378.

Beck

Riddlesworth

Ruedy

, et al. Continuous glucose monitoring versus usual care in patients with type 2 diabetes receiving multiple daily insulin injections: a randomized trial. Ann Intern Med. 2017;167(6):365-374.

Lind

Polonsky

Hirsch

, et al. Continuous glucose monitoring vs conventional therapy for glycemic control in adults with type 1 diabetes treated with multiple daily insulin injections: the GOLD randomized clinical trial. JAMA. 2017;317(4):379-387.

Pelayo

Marcilly

Bellandi

. Human factors engineering for medical devices: European regulation and current issues. Int J Qual Health Care. 2021;33(suppl. 1):31-36.

IEC 62366-1. Medical devices—part 1: application of usability engineering to medical devices; 2015.

Cachemaille

Warren

Moss

. A pen device for injection of recombinant human growth hormone: a European usability engineering study. Expert Opin Drug Deliv. 2020;17(7):1041-1048.

Miclăuş

Valla

Koukoura

, et al. Impact of design on medical device safety. Ther Innov Regul Sci. 2020;54(4):839-849.

ISO 14971:2019. Medical devices—Application of risk management to medical devices.

10.

Liberman

Barnard

. Diabetes technology and the human factor. Diabetes Technol Ther. 2018;20(S1):S128-S38.

11.

Borges

Jr Kubiak

. Continuous glucose monitoring in type 1 diabetes. J Diabetes Sci Technol. 2016;10(3):633-639.

12.

Müller

Knuth

Nikolaus

Krivánek

Küster

Hasslacher

. First clinical evaluation of a new percutaneous optical fiber glucose sensor for continuous glucose monitoring in diabetes. J Diabetes Sci Technol. 2013;7(1):13-23.

13.

Kölle

Eichenlaub

Mende

, et al. Performance assessment of three continuous glucose monitoring systems in adults with type 1 diabetes. J Diabetes Sci Technol. 2023;12:19322968231159657.

14.

American Academy of Microbiology Colloquia Reports. Bringing the Lab to the Patient: Developing Point-of-Care Diagnostics for Resource Limited Settings: This Report is Based on a Colloquium Convened by the American Academy of Microbiology on September 15-16, 2011 in Annecy France. Washington, DC: American Society for Microbiology; 2012.

15.

International Diabetes Federation. IDF Diabetes Atlas (10th ed.). https://diabetesatlas.org/data/en/. Published 2021. Accessed 9 December, 2021.

16.

Niwaha

Rodgers

Carr

ALJ

, et al. Continuous glucose monitoring demonstrates low risk of clinically significant hypoglycemia associated with sulphonylurea treatment in an African type 2 diabetes population: results from the OPTIMAL observational multicenter study. BMJ Open Diabetes Res Care. 2022;10(2):e002714.

17.

Nadeem

Siddiqi

Martins

Badini

. Perceptions and understanding of diabetes mellitus technology in adults with type 1 or type 2 DM: a pilot survey from Pakistan. J Diabetes Sci Technol. 2021;15(5):1052-1058.

18.

Calliari

LEP

Krakauer

Vianna

AGD

, et al. Real-world flash glucose monitoring in Brazil: can sensors make a difference in diabetes management in developing countries. Diabetol Metab Syndr. 2020;12:3.

19.

IEC TR 62366-2. Medical devices—Part 2: guidance on the application of usability engineering to medical devices; 2016.

20.

Brooke

. SUS: a retrospective. J Usabil Stud. 2013;8(2):29-40.

21.

Lewis

Sauro

. Item benchmarks for the system usability scale. J Usabil Stud. 2018;13(3):158-167.

22.

Wilcox

Drucker

. Implications of the New Food and Drug Administration Draft Guidance on Human Factors Engineering for Diabetes Device Manufacturers. Thousand Oaks, CA: Sage; 2012.

23.

Bernabe-Ortiz

Carrillo-Larco

Safary

Vetter

Lazo-Porras

. Use of continuous glucose monitors in low-and middle-income countries: a scoping review. Diabet Med. 2023;40(6):e15089.

24.

Bergman

. Introduction to human factors. J Diabetes Sci Technol. 2012;6(2):229-230.

25.

Toschi

Munshi

. Benefits and challenges of diabetes technology use in older adults. Endocrinol Metab Clin North Am. 2020;49(1):57-67.

26.

Wilson

Heinsch

Betts

Booth

Kay-Lambkin

. Barriers and facilitators to the use of e-health by older adults: a scoping review. BMC Public Health. 2021;21:1-12.

27.

Kulzer

Heinemann

. Digitalisierungs- und Technologiereport Diabetes. Munich: Kirchheim-Verlag; 2022.

28.

Safary

Lepeska

Beran

, et al. Development of a target product profile for glucose self-monitoring technologies for use in low- and middle-income countries 2023. https://www.finddx.org/wp-content/uploads/2023/10/20231031_tpp_glucose_self-monitoring_FV_EN.pdf.