Abstract
Background:
Obesity is associated with multiple comorbidities and increased mortality, making it an important target for treatment. However, achieving and maintaining weight loss by diet and physical activity remains challenging, and may often require pharmacotherapy. Liraglutide 3.0 mg has recently been approved for weight management in the United States, Canada, and EU. The current analysis used a summative usability test to assess safety and effectiveness, ease of use, and training requirements for the novel liraglutide 3.0 mg pen injector.
Methods:
Of the 234 participants, half received instructions for use and video-based training and/or opportunity to handle the device. All participants (excluding pharmacists) performed 6 tasks followed by post- task interviews on task difficulty, device ease of use, and any use errors, close calls, and operational difficulties. Tasks included differentiation of correct box and pen injector, medication clarity assessment, normal, dose reversal, and end-of-content injection. Number/type of use errors, close calls, and operational difficulties were evaluated.
Results:
All assessed participants interpreted the instructions for use correctly. No potentially serious use errors, and low numbers of nonserious errors, were reported. Overall, participants committed 105 use errors related to handling, with no potential for harm. A total of 25 close calls and 44 operational difficulties were reported without any pattern indicative of a design flaw. Marked differences in the incidence of events were observed for trained versus untrained participants regardless of prior injection experience. Participants rated ease of use as 6.4/7.
Conclusions:
The liraglutide 3.0 mg pen injector is safe and easy to use for liraglutide administration. New device features allow for safe use after brief training.
Obesity is a chronic disease often encompassed by a multitude of serious comorbidities, making its prevention and treatment public health priorities. Globally, by 2030, the numbers of adults who are overweight (body mass index [BMI] ≥ 25 kg/m2) or obese (BMI ≥ 30 kg/m2) are projected to be 1.35 billion and 573 million, respectively. 1 Obesity imposes a significant and growing economic burden with considerable direct health care spending. An estimated $190.2 billion, or 20.6% of total national health care spending, is a result of obesity in the United States alone. 2
It has been shown that an increase in BMI is directly correlated with a reduction in life expectancy and increase in obesity-related comorbidities such as type 2 diabetes mellitus (T2DM).3-5 A 5–10% weight loss can provide clinically relevant improvements in obesity-related comorbidities and quality of life (QoL).6,7 Depending on BMI classification and the number/severity of comorbidities, behavioral therapy including dietary and physical activity changes, pharmacotherapy and (in the appropriate patient) bariatric surgery are recommended.8,9 However, achieving and maintaining weight loss by diet and physical activity alone can be challenging.10-12 Pharmacotherapy supports weight loss and maintenance, 13 but a limited number of treatments is currently available.
One of the newest weight loss medications is liraglutide—an analog of the incretin hormone glucagon-like peptide-1 (GLP-1) with 97% homology to human GLP-1—approved by the Food and Drug Administration (FDA) for the treatment of T2DM at doses of up to 1.8 mg. 14 Dose-dependent weight loss with once-daily subcutaneous injections of liraglutide at doses up to 3.0 mg has been reported in previous trials15,16 and is primarily mediated by reduced appetite and energy intake, rather than increased energy expenditure. Recently, liraglutide 3.0 mg has been approved for weight management by the FDA, in Canada, and in the EU.17,18
Pen injectors have been routinely used for treatment of diabetes since the 1980s and, since reports showed superior pen injector accuracy compared with vials and syringes, have undergone a continual process of refinement to improve ease of use.19-21 To ensure safe and effective use of pen injectors by patients, a number of key considerations must be incorporated in their design and development, including physical aspects such as the range of patient hand size, mobility, dexterity, coordination, flexibility, strength, and stamina. Furthermore, various sensory capacities (including vision and tactile sensitivities) as well as cognitive aspects (including the ability and willingness of patients to adapt to new devices) have to be considered. 22 Improvements in the delivery of injectable therapies have been shown to improve patient preference, QoL, and adherence.23-28 A key aspect of any pen device is that it should require minimal training on device initiation and maintained use, as the procedure should be easy to teach and to learn.29,30
The PDS290 pen injector (FlexTouch®) is a novel device designed to address specific usability issues and to be used safely and effectively for all intended users, uses and use environments. FlexTouch® is currently approved in combination with insulin detemir/insulin aspart in the United States and several other combinations in Europe.
Usability engineering is essential in developing injection devices to ensure safe and effective use by the target population, 22 and usability validation tests are required for the approval of medical devices. As mandated by the FDA for approval of medication in the United States, the design of the 2 usability tests was based on multiple guidelines supporting the regulatory filing for liraglutide 3.0 mg in order to validate the claim that the novel liraglutide 3.0 mg pen-injector, a PDS290 variant, could be used safely and effectively by patients who are overweight or obese, and with or without prior injection experience and training.
Methods
Objectives
Two similar summative usability tests were conducted evaluating device use throughout the test rather than solely evaluating test outcome. One test was performed in Europe and one in the United States, each focusing on the handling and differentiation of the liraglutide 3.0 mg pen injector (Saxenda® pen). For this analysis, results of both usability tests were combined and pooled data are reported.
Trial Design
The summative test methodology was based on applicable portions of the human factors process, usability testing and feedback from FDA on the FlexTouch variants previously submitted for review, including guidelines detailing best practice on the conduct of usability tests. 31 Risk management guidance was obtained for device safety, effectiveness and usability, 32 and medical device user interface design 31 to address the requirements of a human factors engineering program in conjunction with medical device development efforts. FDA guidance was consulted for linking risk analysis and human factors engineering activities, particularly with regard to how usability tests are planned and how test results are analyzed to determine a device’s safety. 33 The FDA reviewed and approved the US test protocol.
Subjects
Patients and health care professionals (HCPs; physicians, nurse practitioners, and pharmacists) were assessed.
Patients
Patients were either overweight (BMI of ≥27 kg/m2 and <30 kg/m2) or obese (BMI ≥30 kg/m2) with or without comorbidities such as hypertension and/or T2DM and/or dyslipidemia. Recruitment was aimed at achieving a balance of age and patients with and without prior injection experience. Patients were excluded if presenting with mental or extreme physical incapacity (eg, self-reported dementia and paralysis), inability to read or a language barrier that precluded adequate communication with the recruiter and/or test administrator. Furthermore, patients were excluded if they had any personal or family connection to a pharmaceutical company; had an occupation in the field of human factors engineering, market research, engineering, or pen injector development in general; had participated in any pen injector usability testing in the 2 years prior to this test; or were employed as an HCP.
HCPs
HCPs had to self-report fluency in English and be resident where the test was conducted. HCPs were involved in dispensing medication and were responsible for training or treating patients who are overweight or obese. They were excluded from the test for reasons of personal or family connection with a pharmaceutical company or partaking in any pen injector usability testing in the past 2 years.
Training and Usability Evaluation
Training and usability assessments are outlined in Figure 1. Briefly, following slightly different training regimens for the US and EU cohorts, trained participants eligible to take part in the usability test as assessed by a certified diabetes educator (CDE) were evaluated on knowledge of instructions for use (IFU), handling, and practicing with the liraglutide 3.0 mg pen injector, pen injector use, and subjective rating with a delay of 1-36 hours. The CDE assessed participants’ newly acquired skills and documented their competency using predefined training records. The 6 tasks performed during the usability test in a simulated home setting are summarized in Table 1.

Training and usability evaluation. Training was provided by NNAS-trained nurses or nurse practitioners experienced in training new users. Selection of patients to receive training was performed by the CRO. The duration and nature of the training provided participants with the representative training on launch of liraglutide 3.0 mg pen injector in the United States. CDE, certified diabetes educator; CRO, clinical research organization; EU, European Union; FDA, Food and Drug Administration; HCP, health care professional; IFU, instructions for use; NNAS, National Nursing Assessment Service.
Task Description.
Participants in the EU cohort were able to select from liraglutide 3.0 mg, Tresiba® U100 FlexTouch®, Tresiba® U200 FlexTouch®, Ryzodeg® U100 FlexTouch®, NovoRapid® FlexTouch®, Amplioza®, Victoza®, or Apidra® SoloSTAR® in the EU cohort and liraglutide 3.0 mg, Levemir® FlexTouch®, NovoLog® FlexTouch®, NovoLog® Mix 70/30 FlexPen®, Victoza®, or Apidra SoloSTAR® in the US cohort.
The choice of comparators varied between usability tests.
Assessment
A data analyst specialized in usability testing, who was present during individual tests with the test administrator, recorded use errors, close calls and operational difficulties using a checklist. The data collection forms used for event recording were developed by UL-Wiklund R&D to ensure their clarity and appropriateness. During the tests, the participants were not told when a use error was committed to reflect home use. Therefore, single errors could have been committed several times by the same patient. The test administrator documented key findings—use errors, close calls, and operational difficulties according to definitions described in the protocols according to ISO62366 and HE75 and the FDA draft guidance Applying Human Factors and Usability Engineering to Optimize Medical Device Design.31,32
Use Errors
A use error was defined as a case in which a user performs a step in an incorrect manner that will not lead to the intended outcome and can be categorized into (1) a use error (related to handling only, with no clinical impact), (2) a nonserious use error (a use error that potentially could be associated with a nonserious adverse event), and (3) a potentially serious use error (a use error that potentially could be associated with a serious adverse event).
Close Calls
Close calls were defined as cases in which users almost commit a use error, but catch themselves in time to avoid the use error. The term also describes a case in which a user commits a use error but detects it rapidly and recovers before the error becomes consequential.
Operational Difficulties
Operational difficulty is defined as a case in which a user appears to struggle to perform a task. Such a struggle might be indicated by multiple attempts to perform the task, anecdotal comments about the task’s difficulty, and facial expressions and vocalizations suggesting frustration or confusion, possibly contributing to higher than usual task performance times.
Post-task Rating
Participants rated the ease of performing each hands-on task on a 1-7 scale (1 = difficult, 7 = easy) and provided this rating immediately after task completion. After performing all tasks, participants (except pharmacists) rated pen injector overall ease of use, pen injector task speed (time required to perform an injection from start to finish) and legibility of pen injector display on a scale of 1 to 7 (1 = poor, 7 = excellent).
Data Analysis
Protocols for US testing were reviewed by independent investigational review boards (EU tests did not require approval when being performed in the UK). No statistical analyses were performed. Data are presented descriptively and were pooled from the US and European cohorts. Proportions of use errors, close calls, and operational difficulties are expressed as percentages of total events committed per each event category, and per participant/experience/training group.
Results
Participants
Participant characteristics are shown in Table 2. The test included 234 participants: 163 from the United States (79.1% patients and 20.9% HCPs) and 71 from Europe (36 patients, 35 HCPs). Although 61.1% of participants had injection experience, only 42.7% of test participants had previous experience in handling pen injectors. The mean age (range) of patients and HCPs was 57.1 (20-84) and 45.7 (25-66) years, respectively. Approximately half of the HCP group were pharmacists (52.2%). Overall, 44.8% of patients had injection experience; however, only 18.8% had previously handled pen injectors. Overall 62.4% of patients received training to use the liraglutide 3.0 mg pen injector for the first time. All HCPs had experience in administering, prescribing and/or dispensing pen injectors.
Participant Characteristics.
HCP, health care professional.
This group was composed of 2 age groups: adults aged 18-64 years and elderly aged ≥65 years.
This group included 16 physicians and 18 pharmacists in the United States and 17 physicians and 18 pharmacists in the European trial.
The evaluation of impairment was based on clinical practice without set parameters. Patients with blurred, spotty, or tunnel vision due to, eg, retinopathy, macular degeneration, and cataract, but able to perform the tasks, were included if nurse assessment deemed them capable of performing the task in real life. The same criteria were applied to participants with hearing and dexterity impairments.
Assessment of User Understanding of IFU
All participants in both the US (67 trained participants) and European (53 trained participants) cohorts interpreted all the IFU excerpts correctly, demonstrating their understanding.
Use Errors
The number of actual use errors versus potential use errors that could be committed by patients, nonpharmacist HCPs and pharmacists throughout the test were 101/6765, 10/1353, and 1/36, respectively (Figure 2). A total of 101 use errors were committed by patients, 6.9% were nonserious and 93.1% were related to handling with no clinical impact (Figure 3). The most common user errors were related to failure to check flow before first injection (40.6%), failure to set the dose correctly (10.9%), or resetting dose after flow check (7.9%). Failure to remove the inner/outer needle cap/mount pen injector cap (16.8%) and needle-related errors such as incurring needle-stick injury (6.9%), failure to remove needle after use (5.0%), and failure to mount needle correctly (4.0%) were also reported (Figure 4).

Use errors committed as a rate of the number of use errors possible to be committed for each task of the usability test for pooled US and EU cohorts. N and n represent the total number of participants and the number of participants for each study group, respectively. The total errors committed were 112 of a possible 8154. Possible use errors that could be committed by patients, nonpharmacist HCPs, and pharmacists throughout the test were 6765, 1353 and 36, respectively. HCP, health care professional.

Type of use error committed during task evaluation in pooled US and EU cohorts for individual participant groups. The total number of use errors was 112 (patients 101, HCPs 11). HCP, health care professional.

Incidence of use errors reported during task evaluation in pooled US and EU cohorts for individual participant groups. The total number of use errors was 112 (patients 101, HCPs 11). HCP, health care professional.
In the HCP group, 11 use errors related to handling with no potential for harm or impact were committed (Figure 3). The most common errors included failure to check pen injector flow (36.4%), failure to remove the inner/outer needle cap (27.3%), and incorrect dose setting (18.2%) (Figure 4). Root causes for use errors included test artifacts, performance anxiety/nervousness, and performing simulated injections only in a simulated use environment. Furthermore, forgetfulness, inattention, trust, reliance on memory, perceived lesser importance of certain steps, perceived safety attributes of the device, rushing to complete a task, overconfidence, misinterpretation of flow check process, and reluctance to waste medication were root causes for use errors in the usability test.
Test administrator assistance was required for 5 different use errors committed by 5 different patients for retrieving the liraglutide 3.0 mg pen injector box, avoiding injury when removing the needle, recognizing the medication was not cloudy, recognizing that removal of the inner and outer needle cap was required, and understanding the flow-check procedure.
Close Calls
A total of 21 participants (9.0%, 19 patients and 2 HCPs) encountered 25 close calls. These included (based on number of events) initially retrieving an incorrect product box from the fridge (32.0%), almost not setting dose correctly (24.0%), initially not performing flow check before first injection (12.0%), almost not removing inner needle cap before injection (12.0%), almost not holding dose button down until dose counter is back at 0 (12.0%), and initially failing to remove needle and/or mount pen injector cap after use (8.0%).
Operational Difficulties
Overall 29 participants (12.4%, 28 patients and 1 HCP) experienced 44 operational difficulties while performing the tasks. Operational difficulties included (based on number of events) attaching needle to pen injector (36.4%), removing needle after an injection (29.6%), checking pen injector flow (20.5%), removing inner or outer needle cap (9.1%), dose setting (2.3%), and inadvertently pressing the dose button (2.3%).
Effect of Training on Key Outcomes
With all EU participants (except pharmacists) receiving training in pen injector use, the impact of training on outcome has been assessed in the US cohort only. None of the HCPs in the US cohort received training, therefore only data for patients are displayed in Figure 5. Approximately half (48.1%) of patients in the US cohort were untrained in using the liraglutide 3.0 mg pen injector, however 50% of those patients had prior injection experience. A total of 138 events (88 use errors, 19 close calls, and 31 operational difficulties) occurred. In untrained injection-experienced and -naive participants, use error was the biggest contributor to rates of events with 1.5 and 0.9 use errors per patient, respectively. For trained patients, the numbers of any event were lower compared with untrained patients.

Use errors, close calls and operational difficulties by training status and previous injection experience for patients in the US cohort. Ratios have been calculated for use errors, close calls, and operational difficulties according to their individual occurrences based on assessment of the US cohort only (88, 19, and 31, respectively).
Post-test Task Ease-of-Use Ratings
All tasks received high mean ratings, indicating that they were completed with ease (Figure 6A). Participants rated checking the clarity of liraglutide as the easiest task, with mean ratings of 6.9 on a scale of 1-7. This task also had the lowest range of participant ratings of all assessed tasks. Normal injection and end-of-content tasks were also rated as easy, although with lower mean scores (5.8 in each case).

Post-task ease-of-use ratings (A) and post-test interview subjective ratings (B). Bars represent range of ratings.
Post-test Interview Subjective Rating
High mean scores of 6.4-6.8 were achieved on a scale of 1-7 for overall ease of use, display legibility and speed of task completion with the liraglutide 3.0 mg pen injector (Figure 6B).
Discussion
To our knowledge this is the first time that a study of this type has been reported. It provides important information for the practicing clinician on how studies of this nature are conducted. The data presented here add to the understanding of patient and clinician requirements prior to and during use of novel injection devices and has supported approval of liraglutide 3.0 mg for weight management by the FDA, in Canada, and in the EU. In addition, during the usability test actual patients had the opportunity to provide feedback on the injection process and device to ensure its safe use.
Overall participants showed a high level of satisfaction when using the liraglutide 3.0 mg pen injector. Even though approximately 80% of patients had not handled or used pen injectors before, satisfaction with task speed implies that, with repeated handling, participants quickly grew more confident in using the device, which is reflected in the high ratings for ease of all tasks and the display features of the pen injector. Furthermore, task ease-of-use ratings following the test, and subjective ratings from the post-test interview, support the low rate of task errors observed throughout and indicate that participants found the tasks easy to complete and the device easy to use.
The summative usability testing validated all use safety-related aspects of the device. No potentially serious use errors were observed even for participants without prior experience of handling/using a pen injector. Only 1 type of potentially nonserious use error, needle-stick injury, occurred (in patients only) and the remaining observed errors related to handling without significant medical consequences or long-term effects. However, when handling needles, a residual risk for user error remains even for skilled users. Therefore, a low level of use error was to be expected. The fact that participants caught themselves before committing any use errors indicates understanding and knowledge of how to operate the pen injector correctly even with the minimal amount of training provided.
In addition to low overall user error rates, no patterns in the type of close calls, potentially indicative of design faults, were observed, and evaluation of operational difficulties confirmed no patterns or operational difficulties caused by the design or the instruction of the pen injector.
The responsibility for initiating and overseeing the use of injectable therapies, typically both time- and resource-consuming activities, is increasingly placed on HCPs in primary care. 34 Although there were no or only slight differences in the rate or severity of events reported for injection-experienced and -naive participants within their training groups, it is important to note that training participants in the use of the pen injector resulted in vastly reduced numbers of events in both groups. Correct use of the pen injector by the majority of patients was achieved after ≤17 minutes of video-based and ≤45 minutes of practical training.
Ease of use and safe use are important characteristics of pen devices for both experienced and inexperienced patients, as well as their HCPs. Moreover, the current standard of pen injectors and needles mean that injections are almost pain free. Two previous usability studies in mixed populations of experienced and inexperienced patients have also found that the ease of use of the pen injector was rated significantly higher than other commonly used devices.35-37 These results were consistent when pen injector ease of use was assessed in patients and HCPs separately. 38
Since 2010, 4 prefilled insulin pen injectors from 3 major manufacturers have been available: 21 the FlexTouch and FlexPen pen injectors (Novo Nordisk), the Humalog® pen injector/Humalog KwikPen™ (Eli Lilly), and SoloSTAR (Sanofi). Studies show that pens are becoming increasingly more user-friendly and intuitive to use.29,30,39 The liraglutide 3.0 mg pen injector is a further development of the FlexPen platform and the variant of the device assessed in this usability test is a new model derived from the FlexTouch platform also used for insulin formulations. 14 It has novel features that render it the newest and most optimized device to achieve ease of use.
Novel features include absent push button extension, an end-of-dose click feature, a large display, color coding for different injectable agents and a spring-loaded delivery mechanism. Increasing pen length during dosing is likely to lead to difficulties in patients with impaired manual dexterity, which may be common in the elderly or those patients with diabetic neuropathy. 40 In contrast to other currently available pen devices, the push button of the pen injector does not extend with increasing doses. This is part of the reason why, in previous tests with insulin, the pen injector was rated as easier to use than comparators for reaching and depressing the push button and injecting at low, medium and high doses and also rated significantly higher for the suitability of the length of the device at all 3 doses.35,37 In addition, the liraglutide 3.0 mg pen injector has been designed with an end-of-dose click feature, and a large display, which may aid the injection and dose-selection process, particularly in those patients with sensory impairment. Additional color coding of pens for different injectable agents and insulin types is likely also to be helpful in these patient groups. A novel feature of the liraglutide 3.0 mg pen injector is the dose delivery technology which, based on a spring loaded mechanism, results in consistent injection force regardless of dose. Furthermore, dose delivery can be stopped by releasing the button and continued where it was left off by depressing it once again. Both of these innovations may be of use for patients with the physiological consequences of a BMI of 27-55 kg/m2 (as was the case in this test) and with issues of manual dexterity and stamina, and are in line with the principles of the FDA medical devices guidelines.22,31
Conclusions
The liraglutide 3.0 mg pen injector is safe and easy to use for administration of liraglutide for weight management in adults who are overweight or obese. The device was rated highly for ease of use and, owing to new features, safe use without serious errors was achieved after brief training.
Footnotes
Acknowledgements
The authors thank UL LLC/UL-Wiklund Research & Development for performing the utility testing and all the participants who took part in this usability test. The authors also thank Dr Juliane Moloney, UK, for medical writing support, funded by Novo Nordisk A/S.
Abbreviations
BMI, body mass index; EU, European Union; FDA, Food and Drug Administration; GLP-1, glucagon-like peptide-1; HCP, health care professional; NNAS, National Nursing Assessment Service; QoL, quality of life; T2DM, type 2 diabetes mellitus.
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: KF has acted in an advisory capacity for Novo Nordisk. His contributions to this manuscript include the analysis of data and writing of the manuscript. TS, LYS, and SK are employees of Novo Nordisk A/S. TS is also a shareholder of Novo Nordisk A/S and designed the trials, analyzed data, and reviewed the manuscript. LYS and SK contributed to the research performed in both trials, data analysis, and manuscript development. RFK has supported Novo Nordisk through research and advisory panels. His contributions to this manuscript include analysis of data, outline development and review of the manuscript.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This summative usability test was supported financially by Novo Nordisk A/S, Denmark, which was also responsible for the design, analysis and reporting of the test, with input from the authors.
