Abstract
Background:
Blood glucose data are frequently used in clinical decision making, thus it is critical that self-monitoring of blood glucose (SMBG) systems consistently provide accurate results. Concerns about SMBG accuracy have prompted the development of newly proposed International Organization for Standardization (ISO) standards: ≥95% of individual glucose results shall fall within ±15 mg/dl of the results of the manufacturer's reference procedure at glucose concentrations <100 mg/dl and within ±15% for values >100 mg/dl. We evaluated seven marketed systems against the current and proposed ISO criteria (criterion A).
Method:
Capillary blood samples were collected from 100 subjects and tested on seven systems: Accu-Chek Aviva Plus, Advocate Redi-Code, Element, Embrace, Prodigy Voice, TRUEbalance, and WaveSense Presto. Results were compared with manufacturer's documented reference system, YSI or perchloric acid hexokinase; three different strip lots from each system were tested on each subject, in duplicate.
Results:
Compared against current ISO criteria (≥95% within ±15 mg/dl for values <75 mg/dl and ±20% for values ≥75 mg/dl) the Accu-Chek Aviva Plus, Element, and WaveSense Presto systems met accuracy criteria. However, only the Accu-Chek Aviva Plus met the proposed ISO criteria (criterion A) in all three lots. The other six systems failed to meet the criteria in at least two of the three lots, showing lot-to-lot variability, high/low bias, and variations due to hematocrit.
Conclusions:
Inaccurate SMBG readings can potentially adversely impact clinical decision making and outcomes. Clinicians can reduce controllable variables by prescribing accurate SMBG systems. Adherence to the proposed ISO criteria should enhance patient safety by improving the accuracy of SMBG systems.
Keywords
Introduction
An important component of diabetes management is use of self-monitoring of blood glucose (SMBG) to detect and treat glycemic excursions (hypoglycemia and hyperglycemia) and to guide pharmacologic therapy and health behaviors. Studies have shown that SMBG use is beneficial in achieving improved glycemic control in people with type 1 diabetes mellitus (T1DM) 1 and type 2 diabetes mellitus (T2DM).2–4
Self-monitoring of blood glucose data are frequently used in clinical decision making, thus it is critical that SMBG systems consistently provide accurate results. Although several variables (e.g., user technique) are known to affect the accuracy of SMBG results, the inherent accuracy and lot-to-lot variability of the SMBG systems themselves are often not considered in clinical practice.
Because all SMBG systems marketed in the United States must be approved by the U.S. Food and Drug Administration (FDA) as safe and effective, it is generally assumed that these systems are equal in providing accurate test results. However, although these systems have received FDA clearance, they do not all consistently meet the requirements outlined in International Organization for Standardization (ISO) 15197:2003(E) criteria for SMBG system accuracy from lot to lot. 5 A study found that 16 of the 27 SMBG systems tested did not meet ISO criteria. 6 Growing concerns about SMBG accuracy have prompted the development of newly proposed ISO criteria, whereby acceptance criteria for accuracy (criterion A) require that ≥95% of the individual BG results fall within ±15 mg/dl of the results of the manufacturer's reference procedure at BG concentrations <100 mg/dl and within ±15% for BG values ≥100 mg/dl.
Nevertheless, in response to the rising costs associated with diabetes and diabetes management, health care payers have begun scrutinizing the cost of SMBG. Many payers and policymakers, including the Centers for Medicare and Medicaid Services, are increasingly basing their SMBG reimbursement decisions solely on cost, often favoring lower-priced SMBG systems, many of which are produced by non-U.S. manufacturers and distributed through mail-order channels. Sales of these meters have been growing since entering the market in 2005. As of 2009, manufacturers of these lower-priced meters made up nearly one-third of the Medicare mail-order segment. 7
We evaluated seven SMBG systems currently marketed in the United States against both the existing ISO criteria (ISO 15197:2003[E]) and the proposed ISO criteria to assess their accuracy in measuring BG levels.
Methods
The study was performed September 2011 through February 2012 at Rainier Clinical Research Center, Renton, WA. The study protocol was approved by the Independent Investigational Review Board (Plantation, FL) and is in compliance with the Declaration of Helsinki. 8 Written informed consent was obtained from all subjects. The methodology applied in this study is described in detail in the ISO 15197:2003(E) standard. 5
Study Population
The study population comprised adult patients (>18 years old) with T1DM or T2DM. Exclusion criteria included pregnancy; caloric intake within 30 min prior to testing; ingestion of acetaminophen or medication containing acetaminophen (dose >1000 mg) within 4 h prior to testing; vitamin C (dose >250 mg) within 6 h prior to testing; tolbutamide within 6 h prior to testing; dopamine, L-dopa, or methyl-dopa within 7 days prior to testing; prior diagnosis of gout; current diagnosis of anemia, hemophilia, thrombocytopenia, leukemia or infection with a blood-borne pathogen; undergoing chemotherapy; or hematocrit <30% or >55%. Subjects were also excluded if they had a history of cardiac intervention, transient ischemic attack, cerebral vascular accident, myocardial infarction, unstable angina, cardiac arrhythmia, coronary heart disease, seizure disorder, or severe hypoglycemia (resulting in unconsciousness within the past 6 months).
Self-Monitoring of Blood Glucose Systems Evaluated
Seven SMBG systems, consisting of a BG meter and test strips, were compared. These included (1) Accu-Chek Aviva Plus (Roche Diagnostics, Indianapolis, IN); (2) Advocate Redi-Code (TaiDoc Technology Corporation, New Taipei City, Taiwan); (3) Element (Infopia Co. Ltd., Anyang, Republic of Korea); (4) Embrace (Apex Biotechnology Corp., Hsinchu, Taiwan); (5) Prodigy Voice (Diagnostic Devices Inc., Taipei, Taiwan); (6) TRUEbalance (Nipro Diagnostics, Ft. Lauderdale, FL); and (7) WaveSense Presto (AgaMatrix, Salem, NH). The systems displayed either whole blood BG values or plasma equivalent BG values in mg/dl.
Reference Methods
Two different methods were used to perform reference measurements on the SMBG systems: glucose oxidase (YSI; YSI 2300 STAT Plus glucose analyzer, YSI Life Sciences, Yellow Springs, OH) and hexokinase [perchloric acid hexokinase (PCA-HK); Hitachi 917, Roche Diagnostics GmbH, Mannheim, Germany]. The reference method used for each SMBG system was chosen according to the manufacturer's device labeling. Perchloric acid hexokinase was used for the Accu-Chek Aviva system, whereas YSI was used for the other six systems. Measurements with both reference methods were performed using whole blood and reported whole BG values in mg/dl.
Study Targets
The ISO 15197:2003(E) accuracy criteria specify that the BG concentrations of the blood samples shall be distributed into glucose concentration “bins” as follows: 5% of values at <50 mg/dl, 15% at 50–80 mg/dl, 20% at 81–120 mg/dl, 30% at 121–200 mg/dl, 15% at 201–300 mg/dl, 10% at 301–400 mg/dl, and 5% at >400 mg/dl. Blood samples were collected and distributed into different concentration bins based on the mean reference BG values of the first reference measurement taken immediately prior to SMBG testing and the second reference measurement (immediately following SMBG testing). For prespecified accuracy, the drift between the first and second reference measurements had to be ≤4 mg/dl at BG concentrations ≤100 mg/dl and ≤4% at BG concentrations >100 mg/dl. The protocol was amended to allow for further collection of samples focused on the very low and high glucose bins, without manipulating any of the samples, due to issues with sample manipulation. As a safety measure, the lowest targeted BG concentration was revised to ≤65 mg/dl, with the possibility of obtaining results <50 mg/dl if subjects could tolerate this degree of hypoglycemia safely and without significant symptoms.
Test Protocol
The BG tests were performed in a clinical setting with controlled room temperature maintained at 23 ± 5 °C and humidity levels specified by the SMBG systems. Quality control checks were run (per product literature) at the beginning of each day and at the beginning of each new vial.
The SMBG meter systems were tested in two groups. Group 1 included Accu-Chek Aviva Plus, Advocate Redi-Code, Embrace, and TRUEbalance. Group 2 included Accu-Chek Aviva Plus, WaveSense Presto, Element, and Prodigy Voice. Within each group, 6 meters per system were tested with three test strip lots (two meters per lot). A total of 20 Accu-Chek Aviva glucose meters were used in the study; 12 meters were used for testing (6 in group 1 and six in group 2), 6 meters were available as backup, and 2 were used for screening. For each of the other systems, 6 meters were used for testing and 3 meters were available as backup.
A trained technician performed finger punctures, obtained blood samples, and performed testing as follows: perform initial reference collection (YSI and PCA-HK), perform BG testing with the 24 SMBG meter systems from group 1, and perform final reference collection (YSI and PCA-HK). The time between collection of the initial and final reference samples was no more than 5 min. The procedure was repeated for each subject using the group 2 SMBG meter systems. Test strip vials were randomly selected and rotated every 10 subjects per the original study design and then every 6 subjects when collecting the extreme high and low samples without manipulation to eliminate the risk of all “>400 mg/dl” or “<65 mg/dl” samples being tested from a single vial. Meters were set up and tested in a predetermined, fixed order per the protocol, rotating the start position for dosing the meters by two for each subject (subject 1 started at meter 1, subject 2 started at meter 3, and so forth).
Statistical Analyses
A valid data set for each subject tested required a valid hematocrit reading, 24 valid BG results for each meter group, and two PCA-HK and two YSI reference samples per meter group. Data were excluded from statistical analysis if the PCA-HK and YSI reference duplicates were out of the specified acceptable range, the PCA-HK and YSI reference bins were different, the time between collection of the first YSI sample and second PCA-HK was more than 5 min, controls were out of tolerance, no results were obtained, samples were manipulated, or reference bins were full (as defined in the ISO 15197:2003 [E]). Results from each SMBG meter tested were compared with the mean result of the two duplicate reference measurements. Analyses were performed using both the current and proposed ISO accuracy criteria (criterion A).
Current International Organization for Standardization 15197:2003(E) Accuracy Criteria
At BG concentrations <75 mg/dl, the number of SMBG meter system results within ±15 mg/dl of the reference was tabulated. At BG concentrations ≥75 mg/dl, the number of SMBG meter system results within ±20% of the reference was tabulated. The percentage of SMBG meter system results within ±15 mg/dl at BG concentrations <75 mg/dl was added to the percentage of results within ±20% at BG concentrations ≥75 mg/dl to assess the overall accuracy of each SMBG system.
Proposed International Organization for Standardization Accuracy Criteria (Criterion A)
At BG concentrations <100 mg/dl, the number of SMBG meter system results within ±15 mg/dl of the reference was tabulated. At BG concentrations ≥100 mg/dl, the number of SMBG meter system results within ±15% of the reference was tabulated. The percentage of SMBG meter system results within ±15 mg/dl at BG concentrations <100 mg/dl was added to the percentage of results within ±15% at BG concentrations ≥100 mg/dl to assess the overall accuracy of each SMBG system.
Bias plots were used to demonstrate the accuracy of the various systems. The difference between the meter result and the mean of reference values was plotted as the dependent variable (y axis), and the mean of reference values was plotted as the independent variable (x axis). Based on the acceptance criteria as stated in ISO 15197:2003 (E), limitation lines were included on the graphs. Results were expressed as the percentage of values falling within the following intervals: ±5, ±10, and ±15 mg/dl of the reference result for glucose concentrations <75 mg/dl and those within ±5%, ±10%, ±15%, and ±20% of the reference result for glucose concentrations ≥75 mg/dl. Further analysis was performed to calculate the percentage of values that fell within ±5, ±10, and ±15 mg/dl of the reference result for glucose concentrations <100 mg/dl and those within ±5%, ±10%, and ±15% of the reference result for glucose concentrations ≥100 mg/dl. In this study, nonmanipulated, native samples were used to assess total system performance with regard to hematocrit interference (hematocrit was not manipulated ex vivo). Calculation of the influence of hematocrit on accuracy was also performed based on the bias of the meter result from the reference method (PCA-HK or YSI, depending on the system) versus hematocrit. 9
Results
A total of 162 subjects were enrolled at the research center. Of these subjects, only the first 100 subject samples that were not excluded from the data set due to the exclusion criteria were used for analysis. Samples were tested by clinical personnel who were trained on the various SMBG meter systems, the manufacturer's device labeling, the safety practices, and the test protocol. Nonmanipulated blood samples were obtained for all glucose concentration bins; however, in group 1, 2% (n = 2) and, in group 2, 4% (n = 4) of samples obtained were <50 mg/dl versus the 5% required by the ISO 15197:2003(E) criteria
Samples Used for Lowest Glucose Concentration Category
In comparing the systems against current ISO accuracy criteria for meter performance, three of seven SMBG systems met the requirements (Accu-Chek Aviva Plus, Element, and WaveSense Presto), passing the current acceptance criteria with all three test strip lots
Percentage and Number of Tests within Range: Current and Proposed (Criterion A) International Organization for Standardization Accuracy Criteria
Failed to meet current or newly proposed accuracy criteria.
With regard to the proposed ISO accuracy criteria, only the Accu-Chek Aviva Plus met the proposed ISO accuracy criteria requirement (criterion A)
Self-Monitoring of Blood Glucose Systems versus Reference: Lot and Pooled Bias
A hematocrit effect was seen in the Advocate Redi-Code, Embrace, TRUEbalance, Element, and Prodigy Voice systems, with moderate hematocrit effect in the WaveSense Presto system
Capillary blood bias from reference method versus hematocrit. CI, confidence interval; HCT, hematocrit.
Discussion
Our study revealed distinct differences in accuracy and lot-to-lot variability among the SMBG systems evaluated when compared against both the current ISO criteria (ISO 15197:2003[E]) and proposed ISO accuracy criteria (criterion A). While three of the seven SMBG systems met current ISO accuracy criteria, only one system, the Accu-Chek Aviva Plus, met the newly proposed ISO criteria. This has been confirmed in additional published studies that have also demonstrated lot-to-lot variability with a number of BG systems available in global markets.6,10 Further, our evaluation demonstrated that hematocrit levels influenced, to various degrees, glucose results obtained from six of the seven SMBG systems tested. Similar to earlier studies, we found meter-reported glucose values to be mostly higher at low hematocrit levels and vice versa, compared with reference values.11,12
One potential limitation of the study was our inability to strictly adhere to the ISO glucose concentration distribution table in the hypoglycemic range, which requires that 5% of the values must be <50 mg/dl; only 3% of samples tested in our analysis met this requirement.
As discussed earlier, in Methods (Study Targets), after a review of package labeling to determine the acceptable anticoagulant for each SMBG system, blood samples were initially manipulated (per the procedure outlined in the ISO 15197:2003[E] standard) to achieve the required percentage of <50 and >400 mg/dl samples as required by the ISO protocol. However, a preliminary data review revealed that the manipulated samples showed a negative bias at low glucose values with TRUEbalance, WaveSense Presto, and Element systems when compared with nonmanipulated samples within similar glucose concentrations. Further testing indicated that use of collection tubes containing either anticoagulant ethylenediaminetetraacetic acid or lithium heparin caused the bias. The protocol was amended to allow for further collection of samples focused on the very low and high glucose bins, without manipulating any of the samples. To obtain subject BG concentrations >400 mg/dl, study subjects were allowed to present with elevated BG, or site personnel could administer oral carbohydrates to increase the BG level. The lowest targeted BG concentration was revised to ≤65 mg/dl to minimize the risks associated with inducing hypoglycemia in subjects; we completed our analysis using the lowest 5% of test samples obtained naturally
A key strength of the study was use of three test strips lots, which is more representative of real-world patient use. If a patient uses the same SMBG system over a long period of time, or purchases test strips in various geographic locations, there is an expectation that the strips in each new vial will provide the same consistently accurate test results as those from previous vials.
Clinical Considerations for Clinicians and Patients
An important and clinically significant finding of our study was marked lot-to-lot variability and the influence of hematocrit effect found in most of the meters tested, many of which are directly distributed to Medicare beneficiaries. It has been shown that inaccurate glucose readings due to low hematocrit can have significant clinical implications for patients with renal disease 13 by potentially masking hypoglycemia. This is of great concern among elderly diabetes patients with chronic kidney disease in whom the risk of severe or fatal hypoglycemia associated with the use of oral agents or insulin increases exponentially with age.14,15
Our study highlights the limitations of the current ISO 15197:2003(E) standard in assessing the accuracy of today's BG monitoring systems, which requires testing of only one test strip lot, which is to be representative of the manufacturing process. This may explain the significant lot-to-lot variability shown in our study, which used post-approval, commercially available test strips in the BG meters we tested. This illustrates that not all companies have the appropriate quality system in place to ensure that the lots they produce meet the required acceptance criteria post-approval. The proposed ISO accuracy criteria, across multiple manufactured lots, would likely enhance patient safety, requiring improved performance of future SMBG systems.
However, there is also concern about patients' ability to use their SMBG data to safely and effectively manage their diabetes. Studies have demonstrated that patient competency in interpreting and appropriately acting upon SMBG data is essential to improving clinical outcomes.2–4 Review of manufacturers' Web sites found training and support materials for data interpretation and use for only four of the seven SMBG systems included in our evaluation (Accu-Chek Aviva Plus, TRUEbalance, WaveSense, and Presto). Therefore, even the most accurate SMBG systems cannot improve clinical outcomes without adequate patient training and guidance for accurate interpretation and appropriate use of glucose data.
Conclusion
Self-monitoring of blood glucose is an important component of diabetes management. Because SMBG data are often used in clinical decision making, it is critical that the data are accurate, with minimal lot-to-lot variability.
Our study demonstrated that only three of the seven SMBG systems tested met the current ISO accuracy criteria, whereas only one of seven met the proposed accuracy criteria. Because SMBG data are frequently used to make therapeutic decisions, inaccurate glucose readings can potentially adversely impact clinical outcomes in people with diabetes.
Footnotes
Abbreviations:
This research was sponsored by funding from Roche Diagnostics, Indianapolis, IN.
Ronald L. Brazg and Leslie J. Klaff have received research funding from Roche Diagnostics. Christopher G. Parkin has received consulting fees from Roche Diagnostics.