Abstract
Background:
In recent clinical trials, use of the MyGlucoHealth blood glucose meter (BGM) and electronic diary was associated with an unusual reporting pattern of glycemic data and hypoglycemic events. Therefore, the performance of representative BGMs used by the patients was investigated to assess repeatability, linearity, and hematocrit interference in accordance with regulatory guidelines.
Method:
Ten devices and 6 strip lots were selected using standard randomization and repeatability procedures. Venous heparinized blood was drawn from healthy subjects, immediately aliquoted and adjusted to 5 target blood glucose (BG) ranges for the repeatability and 11 BG concentrations for the linearity tests. For the hematocrit interference test, each sample within 5 target BG ranges was split into 5 aliquots and adjusted to hematocrit levels across the acceptance range. YSI 2300 STAT Plus was used as the laboratory reference method in all experiments.
Results:
Measurement repeatability or precision was acceptable across the target BG ranges for all devices and strip lots with coefficient of variation (CV) between 3.4-9.7% (mean: 5.7%). Linearity was shown by a correlation coefficient of .991; however, a positive bias was seen for BG <100 mg/dL (86% measurements did not meet ISO15197:2015 acceptance criteria). Significant hematocrit interference (up to 20%) was observed for BG >100 mg/dL (ISO15197:2015 acceptance criteria: ±10%), while the results were acceptable for BG <100 mg/dL.
Conclusions:
The BGM met repeatability requirements but demonstrated a significant measurement bias in the low BG range. In addition, it failed the ISO15197:2015 criteria for hematocrit interference.
Self-measured blood glucose (SMBG) monitoring is a mainstay of optimizing insulin therapy in diabetes. Therefore, blood glucose measurement systems used for SMBG testing must be accurate and reliable to allow correct clinical decisions, particularly related to insulin dose adjustments. In order to obtain regulatory approvals, the performance of blood glucose meters (BGMs) is required to be evaluated in comparison to standard laboratory tests. The standards for these tests are outlined in the International Organization for Standardization (ISO) 15197:2015 requirements for blood glucose monitoring systems for self-testing in managing diabetes mellitus 1 and the US Food and Drug Administration (FDA) guidance for self-monitoring blood glucose test systems for over-the-counter use. 2
It is expected that a BGM satisfies the minimum system accuracy requirements specified by these guidelines. However, previous studies have shown that over time several marketed BGMs tested for accuracy following approval fail to meet accuracy criteria.3-6 Furthermore, reflections of inaccurate testing in the hypoglycemic range with an approved Conformité Européenne (CE) and 510(k)-marked BGM were recently detected in three randomized clinical trials being conducted by Novo Nordisk (NCT03078478, NCT03377699, NCT03268005). In all three trials, the same BGM was employed for SMBG recording. In the course of the trials, unusual data reporting patterns for glycemic parameters and hypoglycemic events probably related to the device were observed following routine safety surveillance activities. 7 Therefore, it was necessary to evaluate the accuracy of the devices and strip lots used by patients in the trials using standard laboratory tests and in a clinical study (reported in the same issue of this journal). 8
Three laboratory tests were selected to determine imprecision, measurement bias and the influence of hematocrit when using the suspected BGMs for BG monitoring. These tests were performed in accordance with principles from the ISO 15197:2015 1 and FDA guidelines. 2 The aim of this manuscript is to describe the observations from this laboratory evaluation of these devices.
Materials and Methods
Investigational Device
The MyGlucoHealth BGM (Entra Health Systems, El Cajon, CA, USA) is a measurement device for self-monitoring of blood glucose (BG) by patients with type 1 and type 2 diabetes. The device is plasma calibrated. It measures glucose in the range of 10-600 mg/dL with capillary whole blood, for example, obtained from the fingertip by means of a lancing device (sample volume: 0.3 μL). The test strips employ a glucose-oxidase technology with wireless data transfer via the internet, and the system is approved for over-the-counter use in the United States and in the EU. The strip lots had expiration dates between November 2018 and October 2019.
Selection of Devices and Strip Lots
The devices used by patients in recent trials (NCT03078478, NCT03377699, NCT03268005) were selected using randomization and standard repeatability procedures. The sponsor randomly selected 120 of 3088 BGMs used in the trials. These devices were shipped to the Pfützner Science & Health Institute directly from the trial sites. The randomly selected devices were then subjected to a standard repeatability procedure as set forth in the FDA guidelines. 2 Each meter was used to perform a test with one strip lot (10 readings with 5 blood samples each: 30-50; 51-110, 111-150, 151-250, 251-400 mg/dL = 50 readings/meter and a total of 6000 determinations). After analyzing the raw data to obtain within-sample measurement precision, 10 devices were selected—3 devices with the lowest coefficient of variation (CV, 3.7%, 3.7%, and 3.8%), 3 devices with the highest CV (6.8%, 6.8% and 7.0%), and 4 devices and with median CV (4×, 5.2%). Three out of the 10 selected devices (one of each with low, median, and high CV) were used to perform another set of standard repeatability tests with 18 of 23 strip lots used in the trial since 5 strip lots had reached the expiry date or the number of strips were insufficient. After analyzing the raw data to obtain within sample measurement imprecision, 6 strip lots were selected with the lowest CV (3.7% and 4.0%), highest CV (5.0% and 5.6%), and median CV (4.4% and 4.5%). Thus, 10 representative devices and 6 strip lots were used for the subsequent laboratory tests.
Laboratory Performance Tests
The measurement of repeatability to evaluate imprecision was performed with each strip lot tested with the selected devices using samples with BG concentration within target ranges adjusted to combine recommendations from the ISO 15197:2015 1 and FDA 2 guidelines (30-50, 51-110, 111-144, 151-250, 280-400 mg/dL).
The assessment of linearity for precision evaluation as described in the FDA 2 guidance was performed with the selected devices and strip lots using blood samples within 11 evenly spaced BG concentrations between 50-500 mg/dL. To explore device performance in the hypoglycemic range, 3 additional samples were included with BG values 55, 65, and 70 mg/dL.
The abbreviated assessment of hematocrit interference was performed to investigate the influence of varying packed cell volume on the accuracy of BG measurements with the BGM. This test was performed using blood samples within target BG ranges 30-50, 51-110, 111-150, 151-250, 251-450 mg/dL and each sample was adjusted to 5 different hematocrit concentrations (20%, 30%, 40%, 50%, and 60%).
Testing Procedure
Blood sample collection was performed from healthy subjects, not on any medications, based on a blood sample collection study (PSHI-SAM-001) for diagnostic purposes in compliance with local ethical and legal requirements and approved by the Institutional Review Board. The procedure employed for all laboratory performance tests is presented in Table 1.
Testing Procedures for the Laboratory Performance Tests.
Samples were diluted with plasma to decrease HCT or plasma was extracted to increase HCT.
pO2 levels were adjusted by gently shaking the tubes.
The adjustments to target BG concentrations were carried out either by glycolysis or by spiking the sample with a 20% glucose solution in physiological sodium chloride. All tests were carried by trained personnel in a laboratory setting with controlled room temperature (22.8 ± 2°C) and humidity (44-68%).
Data Analysis
The measurement repeatability was assessed in accordance with the FDA 2 and ISO 15197:2015 1 requirements. The measurements collected across 5 BG concentrations were analyzed to determine the mean value of measurements per meter with the corresponding standard deviation (SD) and percentage CV as recommended by the FDA guidelines. The measurement results for every device with 6 strip lots was corrected for differences to the YSI 2300 STAT Plus device (Yellow Springs, OH, USA) calibration by a multiplication factor of 1.12. 10 Standard quality control procedures for the YSI reference method were applied. Calibration procedures were performed after every second measurement. Glucose standards from the National Institute of Standardization (NIST) were used on a daily basis to determine the analytical performance of the reference method (analytical imprecision: 1.86%, bias: 1.43%, total error: –1.38%). In addition, the mean value per glucose concentration with the corresponding SD and percentage CV was also analyzed as recommended by ISO 15197:2015 1 and FDA 2 guidelines. Stable measurement repeatability was assumed when the CV% was within ±10% for all glucose concentrations and for the combined data set.
The assessment of linearity was conducted in accordance with the FDA guidelines following comparison to the YSI 2300 STAT Plus reference method. For each strip lot and for the entire data set regression analysis was performed to calculate the coefficient of correlation. Subsequently, a Bland-Altman analysis of the entire data set was performed after adjustment for different calibration methods (plasma vs whole blood calibration adjustment factor: 1.12). 10 Since acceptance criteria are not mentioned in the FDA guidelines, standard clinical accuracy criteria specified by ISO 15197:2015 were applied. According to these criteria, clinical accuracy can be confirmed if 95% of the individual results are within ±15 mg/dL of the results from the reference method for BG values <100 mg/dL and within ±15% for BG values ≥100 mg/dL.
Hematocrit interference was assessed in accordance with the ISO 15197:2015 requirements. To determine the effect of hematocrit on the measured values from the study device, the difference between the average glucose bias and the average bias of the midlevel sample (40%) was calculated for each sample. Data were analyzed and are graphically presented as requested by the ISO15197:2015 guideline. The minimum acceptance criteria are defined as an observed mean difference within ±10 mg/dL for values ≤100 mg/dL and within ±10% for values >100 mg/dL.
Results
The variability in measurement repeatability per meter was generally acceptable with CVs in the range of 3.7-9.7% across all the target BG ranges (Table 2). Within-strip lot variability was also acceptable over the entire BG measurement range (data not shown). The mean CV across all BG concentrations, devices, and strip lots was 5.7%, and the overall results per BG concentration are shown in Table 3.
Measurement Repeatability Per Device.
Overall Measurement Repeatability.
Table 4 presents the linearity assessment results per strip lot compared to the YSI 2300 STAT Plus reference method. The regression analysis performed for the entire combined data set resulted in a correlation coefficient of 0.991 (slope: 0.87, intercept: 34 mg/dL). However, the results of the Bland-Altman analysis for the entire data set, after adjustment for the calibration differences, indicated that 86% of the measurements <100 mg/dL do not meet the ISO 15197:2015 acceptance criteria for system accuracy (Figure 1). The overall deviation for BG below 100 mg/dL was +38.5% (mean absolute deviation: 26 mg/dL). When analyzed according to the usual FDA criteria for system accuracy (±15%), 97.7% (293/300) of measurements for BG values <100 mg/dL did not meet the acceptance criteria.
Linearity Measurement Per Strip Lot.
The reported YSI reference values are mean values of 4 individual YSI measurements, and the mean study BGM results are derived from 60 individual readings with 10 devices and 6 strip lots.
Bland-Altman analysis for measurement of linearity across the entire data set. The range between the dotted lines represents ISO 15197:2015 acceptance criteria for clinical accuracy.
The mean glucose value determined at a hematocrit level of 40% was normalized to be 100% in order to determine the potential bias (% deviation) occurring at the other hematocrit levels. The average bias from the 40% hematocrit level for each strip lot following stratification by BG values below or above 100 mg/dL is displayed in Figure 2. For both strip lots, the results indicated significant hematocrit interference, not fulfilling the acceptance criteria of ±10%, for BG values ≥100 mg/dL. The mean bias for BG values <100 mg/dL was within the acceptance criteria limits (±10 mg/dL). Overall, the results failed to meet the acceptance criteria specified by ISO 15197:2015 with respect to hematocrit interference.
Hematocrit interference stratified by blood glucose concentration.
Discussion
It was necessary to evaluate the performance of the BGM in light of recent observations in clinical trials indicating inaccurate glycemic data reporting patterns. 7 Such findings could potentially increase the risk of hypoglycemia in patients relying on the SMBG monitoring system for insulin dose adjustments. Therefore, the imprecision and trueness of the device was evaluated using three laboratory tests specified by the ISO 15197:2015 1 and FDA 2 guidelines. The devices and strip lots tested were used by patients in the aforementioned clinical trials. The selection was based on a randomization and standard repeatability procedure to identify the most representative devices and strip lots based on their CV.
The evaluation showed that random error with the BGM calculated as measurement repeatability was acceptable. However, a significant measurement bias toward overestimation for BG values <100 mg/dL was demonstrated. Of note, the results from the Bland-Altman analysis showed that the majority of values <100 mg/dL did not meet the ISO 15197:2015 1 acceptance criteria for clinical accuracy. This observation is particularly concerning since relatively small measurement errors in the hypoglycemic BG ranges could severely compromise the users’ safety. It is important to emphasize that these results were independent of the individual strip lots and devices tested. The widely accepted YSI 2300 STAT Plus glucose analyzer was used as a reference method for this assessment of linearity.
Several additional parameters including chemical substances, blood composition and environmental conditions can impact the performance of BGMs depending on their core measurement technology.11,12 Although frequently underestimated, the influence of hematocrit is known factor that compromises the accuracy of BGMs.12,13 In a healthy population, hematocrit remains fairly constant at a level of 30-50%; however, in patients with diabetes mellitus, abnormal ranges are fairly common.12-15 In meters employing glucose oxidase-based technology, like the tested BGM, lower-than-normal hematocrit values (<35%) can result in overestimated readings, whereas hematocrit values higher-than-normal (>45%) may result in underestimated readings compared to laboratory values. The primary reason for this issue is the internal calibration of the analysis process, which is based on the assumption of a standardized 45% hematocrit level. 12 Hence the sensitivity of the BGM to hematocrit was evaluated using samples with varying hematocrit levels across a range of BG concentrations. The results revealed significant hematocrit interference for samples with BG ≥100 mg/dL with a mean bias greater than the ISO 15197:2015 1 acceptance criteria of ±10%.
The accuracy of measurement with the tested device is not influenced by the hematocrit percentage in the low BG range. Therefore, hematocrit interference can be ruled out as the reason for the measurement bias observed toward overestimation for BG values <100 mg/dL. In addition, the repeatability test demonstrated acceptable variability between measurements for all BGM tested. Taken together, these observations indicate a systematic error in measuring BG concentrations in the low BG range. The test of linearity demonstrating this error, specified in the FDA guidelines, is frequently used to define the measuring range of BGM for SMBG monitoring. Our findings suggest that the performance accuracy of the tested BGM may not be suitable for its approved entire measuring range of 10-600 mg/dL.
However, using only well performing devices for glucose measurement is of particular importance for clinical trials as confirmed in recently published studies.16,17 Obviously, good device accuracy is a prerequisite to ensure patient safety. However, a suboptimal device performance may lead to other study-related problems such as potential bias in endpoints, potential inconsistencies between SMBG values and HbA1c, and so forth. Potential measures to avoid these pitfalls are to select meters based on independent published meter accuracy evaluations and to introduce quality checks for meters and strips prior to using them for a trial.
A potential limitation of our results is that the experiments were performed under controlled conditions in a laboratory using artificially manipulated blood samples which is not reflective of real-life conditions. In addition, it can be contended that pO2 levels could influence the linearity results as they were not measured. However, the experimental methods followed were identical to a previous study 9 in which it was proven that the method ensures pO2 is within physiological limits even when samples are heavily manipulated. These data provide vital information on measurement accuracy using a wide BG range. However, the results were to be confirmed in a clinical study employing the manufacturer’s instructions for use prior to drawing clinical conclusions. The results of the clinical study are reported in the same issue of this journal. 8 Since, the devices and test strips were stored at different trial sites, quality assurance for storage cannot be controlled. However, the similarity of results observed across the devices and strips, originating from different geographical locations, vote against the notion that this factor has driven the results.
Conclusions
The criteria applied for the laboratory tests were based on ISO 15197:2015 and FDA requirements; however, the methods were modified to evaluate the primary aim of these experiments. The BGM tested in this laboratory evaluation showed acceptable imprecision according to the criteria specified in this study. However, a significant bias toward overestimation of measurements in the low BG range was observed. The bias was observed across all strip lots tested and did not meet the minimum acceptance criteria defined by ISO 15197:2015. This is an important observation as accuracy of SMBG monitoring is critical to patient safety in this hypoglycemic BG range. In addition, hematocrit interference for samples BG ≥100 mg/dL was higher than the acceptable proportion recommended by ISO15197:2015. This observation can be anticipated considering that the device is based on the glucose-oxidase technology and was approved more than a decade ago, when hematocrit testing for regulatory approval was not required. Our laboratory results are in line with the observations from the Novo Nordisk clinical trial.
Footnotes
Abbreviations
BG, blood glucose; BGM, blood glucose meter; CE, Conformité Européenne; CV, coefficient of variation; FDA, US Food and Drug Administration; HCT, hematocrit; hr, hour; ISO, International Organization for Standardization; NIST, National Institute of Standardization; rpm, rate per minute; SMBG, self-measured blood glucose.
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: This work was sponsored by a research grant from Novo Nordisk. AP has received research grants, travel support and consultancy fees from Novo Nordisk. AHP is the spouse of AP. All other authors have no conflict of interest.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Novo Nordisk A/S, Copenhagen, Denmark.
