Abstract
Background:
In recent randomized clinical trials, an unusual reporting pattern of glycemic data and hypoglycemic events potentially related to an internet enabled blood glucose meter (MyGlucoHealth, BGM) was observed. Therefore, this clinical study was conducted to evaluate the system accuracy of the BGM in accordance with the ISO15197:2015 guidelines with additional data collection.
Methods:
To investigate system accuracy, 10 of 3088 devices and 6 of 23 strip lots, used in the trials, were selected by a randomization procedure and a standard repeatability assessment. YSI 2300 STAT Plus was used as the standard reference method. The samples were distributed as per the ISO15197:2015 recommendations with 20 additional samples in the hypoglycemic range. Each sample was tested with 6 devices and 6 strip lots with double determinations.
Results:
Overall, 121 subjects with blood glucose values 26-423 mg/dL were analyzed, resulting in 1452 data points. In all, 186/1452 readings (12.8%) did not meet the ISO acceptance criteria. Data evaluated according to the FDA guidelines showed that 336/1452 (23.1%) readings did not meet the acceptance criteria. A clear bias toward elevated values was observed for BG <100 mg/dL (MARD: 11.0%).
Conclusions:
The results show that the BGM, although approved according to standard regulatory guidelines, did not meet the level of analytical accuracy required for clinical treatment decisions according to ISO 15197:2015 and FDA requirements. In general, caution should be exercised before selection of BGMs for patients and in clinical trials.
The main objective of modern diabetes therapy is to achieve near normal blood glucose (BG) levels in order to prevent primary and secondary complications. Self-monitoring devices allow for metabolic control by patients and doctors and for a flexible therapy adjustment by the patients themselves. As insulin therapy is guided by the measured values, high-quality blood glucose monitoring systems (BGM) are required. 1 Therefore, the accuracy of the BGM is of critical importance to optimize insulin treatment. However, recent studies have demonstrated that the performance of marketed BGMs may not meet the minimum accuracy criteria required by the International Organization for Standardization (ISO) 15797:2015 guidelines.2-6 Several environmental and technological factors related to the devices as well as the strip lots could contribute to these findings. These among others include medications, temperature and humidity, blood composition and viscosity, and measurement technology. Klonoff et al speculate that the performance of the BGM could deteriorate over time owing to manufacturing issues resulting in lower analytical accuracy compared to the devices used to the determine initial accuracy data. 3 Nevertheless, inaccurate BGM measurements, especially overestimation of blood glucose values, pose a significant risk of hypoglycemia in patients using these devices.
In three recent trials (NCT03078478, NCT03377699, NCT03268005) conducted by Novo Nordisk, routine safety surveillance led to the detection of unusual reporting pat
Therefore, this study was designed to test the system accuracy of the BGM used by the patients in the aforementioned trials. The accuracy testing was performed according to the ISO15197:2015-recommended 8 protocol with additional data collection and the YSI 2300 STAT Plus glucose analyzer as the reference method.
Methods
Study Design
This open-label, nonrandomized clinical study to verify the system accuracy of the device was conducted from September-October 2018 at the Pfützner Health & Science Institute. Two visits were scheduled for the study participants. During the screening visit (visit 1) conducted at least one day prior to the first study procedures, the participants were informed about the study and signed informed consent was obtained. Subsequently, system accuracy evaluation was performed at visit 2.
This study was performed in accordance with the suggested system accuracy protocol in the ISO15197:2015 guidelines 8 with additional data collection to increase the specificity of the study in the hypoglycemic range. The study was conducted in compliance with the guidelines for Good Clinical Practice, Declaration of Helsinki, and all applicable legal and ethical requirements. The responsible IRB approved the study protocol and participants signed written informed consent prior to study procedures.
Subjects
The study was planned to include 120 male or female subjects (≥18 years). The majority of subjects were diagnosed with type 1 or type 2 diabetes mellitus. In addition, healthy volunteers provided samples to be adjusted for extreme BG concentrations. At screening, the BG values for these subjects were within one of the ISO 15197:2015-specified distribution categories between 50 and 400 mg/dL and hematocrit values were between 30% and 55%. Subjects with a history of hypotonic reactions with unconsciousness, major cardiovascular events, hyperuricemia or gout, clear and severe signs of hypoglycemia or hypoglycemic unawareness, or infusion site reactions were excluded. In addition, subjects with known insulin resistance and daily insulin dose >80 units were not included. Female subjects who were pregnant or breast feeding were also excluded. Intake of acetaminophen, salicylic acid, ascorbic acid, mannitol, maltose, galactose, xylose, ∝ -lipoic acid or use of anticoagulants within 2 days prior to blood collection was not permitted.
Study Device Selection
The device used by patients in recent trials (NCT03078478, NCT03377699, NCT03268005) was evaluated for system accuracy. All maintenance, adjustment and control procedures were followed in accordance with the manufacturer’s instructions. The sponsor randomly selected 120 of 3088 blood glucose meters (BGMs) used in the trials and these devices were shipped to the Pfützner Health & Science Institute directly from the trial sites. The BGMs were then subjected to a standard repeatability procedure as set forth in the FDA guidance for over-the-counter meters. 9 Each meter was used to perform a test with one strip lot (10 readings with 5 blood samples each: 30-50 mg/dL; 51-110 mg/dL, 111-150 mg/dL, 151-250 mg/dL, 251-400 mg/dL = 50 readings/meter and a total of 6000 determinations). After analyzing the raw data to obtain within-sample measurement precision, 10 devices were selected—3 devices with the lowest coefficient of variation (CV, 3.7%, 3.7%, and 3.8%), 3 devices with the highest CV (6.8%, 6.8%, and 7.0%), and 4 devices and with median CV (4×, 5.2%).
Strip Lot Selection
Three out of the 10 selected devices (one of each with low, median, and high CV) were used to perform another set of standard repeatability tests with 18 of 23 strip lots used in the trial since 5 strip lots had reached the expiration date or the number of strips were insufficient. After analyzing the raw data to obtain within-sample measurement precision, 6 strip lots were selected with the lowest CV (3.7% and 4.0%), highest CV (5.0% and 5.6%), and median CV (4.4% and 4.5%).
Testing Procedure
This study protocol to investigate the system accuracy of the BGM was developed in accordance with the ISO15197:2015 guidelines with additional data collection described below. YSI 2300 STAT Plus (YSI Inc, Yellow Springs, OH, USA) was employed as the reference method.
After subjects signed informed consent, blood was drawn for determination of hematocrit and oxygen pressure using ABL80 Flex/COOX (Radiometer, Willich, Germany). Thereafter, a first capillary sample was acquired for the YSI reference method and a health care professional used 6 assigned devices to invasively measure BG with the 6 strip lots selected (double determinations). Finally, a second sample was obtained for another YSI reference measurement. Subsequently, samples were distributed within the ISO 15197:2015-specified BG distribution categories listed in Table 1. In order to explore device performance in the hypoglycemic range (50-80 mg/dL), 20 additional samples were included in this category. Samples with glucose concentration between 10-600 mg/dL (measured by the YSI reference method) and those without endogenous and exogenous interferents were included in the analysis. Only unaltered samples were used for BG concentrations >50 mg/dL and ≤400 mg/dL. In order to obtain sufficient samples for extreme BG values (≤50 mg/dL and >400 mg/dL), the glucose concentration was adjusted as specified in the ISO15197:2015 guidelines. 8 Oxygen saturation and hematocrit levels of these manipulated sample were confirmed to be within the required range prior to the experimental measurements. Each sample was measured by a health care professional with all (6) strip lots and 6 out of the 10 BGMs selected. Thus, the 6 BGMs selected differed from sample to sample. For each experiment, the devices were assigned to the strip lots based on a predefined assignment list to ensure that each device and strip combination was tested equally with the same frequency throughout the entire study.
Sample Distribution by Blood Glucose Concentration.
20 additional patients were included in this category.
Statistical Methods
Data analysis for system accuracy was performed according to ISO 15197:2015. 8 The minimum acceptable accuracy criteria according to the guidelines are that 95% of the measured BG values should fall within either ±15 mg/dL of the average measured values of the reference measurement procedure at glucose concentrations <100 mg/dL or within ±15% at glucose concentrations ≥100 mg/dL. System accuracy was also evaluated according to the FDA guidelines on submitting premarket notifications (510(k)) for self-monitoring BG test systems intended for over-the-counter home use. 9 The FDA guidelines specify 95% of all BG values should be within ±15% of the comparator results and that 99% of all BG values should be are within ±20% of the comparator results across the entire claimed measuring range of the device.
The Bland and Altman 10 analysis was conducted to determine the average bias for the difference between the BG reading and reference value. In addition, the ISO15197:2015-recommended consensus error grid analysis according to Parkes et al and Pfützner et al11,12 and the surveillance error grid (SEG) analysis 13 were also performed. For the analysis of extreme values with BG <50 mg/dL and ≥400 mg/dL, the BGM results were multiplied by 1.12 to account for the calibration differences. 14
Results
Subjects
Overall, 121 subjects were included in the analysis. The mean age of the subjects was 58.7 years and 44% were female. In all, 43 subjects had type 1 diabetes, 68 subjects had type 2 diabetes and 10 healthy volunteers provided samples for laboratory measurements to obtain the extreme BG values (5 samples each for BG <50 mg/dL and >400 mg/dL). In the course of the study, one subject presented with a BG value <50 mg/dL, leading to 6 measurements <50 mg/dL available for the analysis instead of the planned number of 5 measurements. The entire dataset comprised 1452 data points with 121 subjects tested using 6 BGMs and 6 strip lots.
The study procedures were well tolerated. No general adverse events or device-related adverse events were reported from the study visits or the previous calibration periods.
System Accuracy According to ISO15197:2015
The mean values of the YSI reference method taken before and after the BGM readings were calculated for all samples. No data set needed to be repeated due to a difference of more than 5% between the two YSI values. In all, 113/540 (20.9%) samples <100 mg/dL and 73/912 (8.0%) samples ≥100 mg/dL did not meet the minimum accuracy requirements specified by ISO 15197:2015 (Table 2). Thus, in total 186 of 1452 (12.8%) samples did not meet the ISO 15197:2015 specifications.
System Accuracy Results According to ISO15197:2015.
A graphical presentation of these results for the individual strip lots is provided in Figure 1. In general, all strip lots showed the same insufficient performance. According to these results, the BGM tested in this study did not meet the ISO15197:2015 acceptance criteria with any of the tested strip lots individually and also with the combined data set. Strip Lot 6 showed the lowest agreement with the ISO acceptance criteria.
System accuracy analysis for the individual lots and the entire data set. The dotted lines represent the upper and lower ISO threshold of acceptance.
System Accuracy Analysis by FDA Guidelines
The accuracy analysis for the entire data set according to FDA criteria is provided in Table 3. In all, 336/1452 (23.1%) of samples did not meet the FDA acceptance criteria. When stratified according to the BG value, 263/540 (48.7%) measurements <100 mg/dL and 73/912 (8.0%) measurements ≥100 mg/dL did not meet the FDA acceptance criteria. The strip lots achieved comparable accuracy values within 66.9% to 83.9% (at the ±15% level) and that there was no particular outlier, in any direction, among the strip lots tested. Similar to the ISO 15197:2015 accuracy analysis, none of the strip lots met the FDA acceptance criteria and Strip Lot 6 showed the lowest accuracy (Table 3).
Accuracy Analysis in Accordance With the FDA Guidelines. 6
Shaded row indicates FDA acceptance criteria.
Bias Analysis
The mean absolute percentage bias was calculated for the six strip lots and the total data set (total mean absolute relative deviation [MARD]). The MARD was also calculated for values ≥100 mg/dL and the mean absolute deviation (MAD) was computed for the values <100 mg/dL. The results are provided in Table 4.
Bias Analysis.
The mean total MARD for all strip lots was 11.0%. A separate MARD analysis for the glucose meters resulted in a range of 10.1% to 12.3%, indicating comparable performance of the individual devices.
The Bland Altman analysis was performed to identify specific glucose ranges for a potential bias. Figure 2 shows that the observed bias is predominant for BG values below 100 mg/dL. The MAD was 10.4 mg/dL for BG values below 100 mg/dL.
Bland-Altman plot for the entire data set.
Consensus and Surveillance Error Grid Analysis
The ISO15197:2015 acceptance criteria require that 99% of the data pairs have to be found in zones A and B of the consensus error grid, which represent laboratory performance with clinically acceptable differences. In the consensus error grid this was fulfilled with 100% of the measured data pairs in zones A (97.1%) and B (2.9%). The SEG model indicates that devices with ≥ 97% pairs inside the SEG no-risk “green” zone would meet the ISO 15197:2015 requirements, while higher percentages outside the SEG no-risk zone would indicate noncompliance with the standard. As displayed in Figure 3, 96% of data pairs were within the no-risk “green” zone and 4% of the data pairs were within the “slight” risk zone. In conclusion, the devices and strips met consensus error grid criteria, but did not pass SEG criteria.
Surveillance error grid analysis.
Discussion
This study was performed following a modified ISO15197:2015 protocol to analyze the system accuracy of the BGM used in recent clinical trials. Data from these trials indicated unusual reporting patterns for glycemic parameters and hypoglycemic events that were most likely related to the BGM. Standard laboratory tests conducted using the device indicated an overestimation of measurements for BG values <100 mg/dL 15 and this clinical evaluation of system accuracy further confirmed the findings.
To evaluate system accuracy, 10 devices and 6 strip lots were selected by a combination of randomization procedures and laboratory repeatability analysis. The range of BG values examined were in line with ISO 15197:2015 requirements with 20 additional samples with BG values between 50-80 mg/dL in order to strengthen the analysis power in the hypoglycemic range.
In order to meet acceptance criteria according to ISO15197:2015 guidelines, 95% of all BG readings should be within ±15 mg/dL for BG <100 mg/dL or ±15% for BG >100 mg/dL. The results from this study show that 12.8% of all readings did not meet the ISO acceptance criteria. None of the strip lots tested met the acceptance criteria for system accuracy of the ISO guideline. In addition, the bias was strongest for values below 100 mg/dL, where only 79.1% were within the acceptable range and none of the individual strip lots met the 95% requirement for this criterion. Accuracy analysis was also performed in accordance with the FDA guidelines that specify 95% of all BG values should be within ±15% of the comparator results. This analysis showed that 336/1452 (23.1%) readings did not meet the FDA acceptance criteria. None of the individual strip lots met the 95% requirement for these criteria (range: 66.9% to 83.9%). Thus, data evaluated according to both the ISO and FDA specifications, revealed that the BGM did not meet the minimum accuracy requirements.
Bias analysis indicated a MARD value which is above the results expected for such devices intended for patient self-measurement (~5-8%). An additional bias analysis by device did not reveal any particularly good or poor performing device amongst the investigated devices. Despite a higher MARD, the consensus-error-grid analysis showed that 100% of the measured data pairs for the BGM were in the acceptable zones A and B. The SEG analysis showed that 4% of the data pairs were outside the acceptable no-risk zone.
However, the Bland-Altman analysis indicated a pronounced bias toward too high measurements in the hypoglycemic range (<100 mg/dL). These observations are concerning with regards to patient safety since overestimation of BG levels in this range may result in a false perception of the real glycemic control. This could not only lead to misinformed treatment intensification but also result in delaying preventative action to avoid hypoglycemia. Importantly, the associated risk with using inaccurate devices for regular BG measurement by patients at home is elevated compared to use under controlled clinical trial conditions. In the latter case, patients are constantly monitored and safety surveillance protocols are implemented which assist in identification of unexpected data issues. Indeed, data from the Novo Nordisk clinical trials indicate the impact of these inaccurate measurements resulting in unusual reporting of glycemic data and hypoglycemic events. 7 In a previous laboratory investigation, we observed that the BGMs tested in this trial showed acceptable precision performance, but systematically displayed too high results with low glucose samples (<100 mg/dL) and had substantial hematocrit interference. 15
Similar to these observations, previous studies have shown that marketed BGMs do not always meet accuracy requirements following regulatory approval.2-6
Many factors discussed below could contribute to the inadequate performance of BGMs in general and to the findings with the tested BGM in this report. First, the device studied is based on a technology approved more than 15 years ago, which may not meet current device standards for glucose self-testing anymore. Unfortunately, there are no regulatory requirements to ensure that BGMs meet actual accuracy criteria after they are marketed. Therefore, it is likely that the performance of BGMs, including the investigated devices, deteriorates over time. This critical point has been elucidated in detail in a previous publication. 16 Second, it could be possible that devices and strips are not adequately release-tested, which could results in a general suboptimal performance of distributed product toward higher measurements. Third, their performance could have been affected by shipment and storage conditions at the different sites. However, all strip lots and devices received from different geographical locations displayed the same measurement bias, which votes against this hypothesis and rather suggests a systematic problem. Last, suboptimal storage during previous use by the patients in the clinical trials could not have influenced the results due to the same reasons.
Conclusions
In conclusion, the tested BGM did not meet the minimum accuracy requirements specified by ISO and FDA. In our opinion, the devices displayed falsely higher results in the hypoglycemic range, which can explain the unusual pattern of glycemic reporting as observed in the clinical trials. Hence device manufacturers should be mandated to provide actualized performance data on a regular basis in order to ensure patient safety and to facilitate selection of devices for routine treatment and for clinical trials.
Footnotes
Abbreviations
BG, blood glucose; BGMS, blood glucose monitoring systems; CE, Conformité Européenne; ISO, International Organization for Standardization; MAD, mean absolute deviation; MARD, mean absolute relative deviation; SEG, surveillance error grid.
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: AP received a research grant for study conduct from Novo Nordisk. AHP is the coworker and spouse of AP.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Novo Nordisk A/S, Copenhagen, Denmark.
