Abstract
Background:
Inaccurate blood glucsoe monitoring systems (BGMSs) can lead to adverse health effects. The Diabetes Technology Society (DTS) Surveillance Program for cleared BGMSs is intended to protect people with diabetes from inaccurate, unreliable BGMS products that are currently on the market in the United States. The Surveillance Program will provide an independent assessment of the analytical performance of cleared BGMSs.
Methods:
The DTS BGMS Surveillance Program Steering Committee included experts in glucose monitoring, surveillance testing, and regulatory science. Over one year, the committee engaged in meetings and teleconferences aiming to describe how to conduct BGMS surveillance studies in a scientifically sound manner that is in compliance with good clinical practice and all relevant regulations.
Results:
A clinical surveillance protocol was created that contains performance targets and analytical accuracy-testing studies with marketed BGMS products conducted by qualified clinical and laboratory sites. This protocol entitled “Protocol for the Diabetes Technology Society Blood Glucose Monitor System Surveillance Program” is attached as supplementary material.
Conclusion:
This program is needed because currently once a BGMS product has been cleared for use by the FDA, no systematic postmarket Surveillance Program exists that can monitor analytical performance and detect potential problems. This protocol will allow identification of inaccurate and unreliable BGMSs currently available on the US market. The DTS Surveillance Program will provide BGMS manufacturers a benchmark to understand the postmarket analytical performance of their products. Furthermore, patients, health care professionals, payers, and regulatory agencies will be able to use the results of the study to make informed decisions to, respectively, select, prescribe, finance, and regulate BGMSs on the market.
The Need for the DTS BGMS Surveillance Program
The use of blood glucose monitoring systems (BGMSs) plays a vital role in the management of type 1, type 2, and gestational diabetes. People with diabetes who perform self-monitoring of blood glucose (SMBG) depend on the accuracy of their BGMS to ensure proper diabetes therapy including insulin and other medication dosing, exercise, meal planning and other necessary activities. 1
In the USA, BGMS products are class II medical devices that require clearance from the Food and Drug Administration (FDA) to be marketed. Acceptable analytical accuracy data are critical for a meter to be cleared for the market.
Several recent standards and guidelines define accuracy requirements of BGMSs. ISO 15197:2013 2 requires that at least 95% of meter test results fall within ±15mg/dl (for BG < 100mg/dl) and within ±15% (for BG ≥ 100 mg/dl) of an accepted reference method procedure (ie, comparative method).
On January 7, 2014, the FDA released a draft guidance document for BGMSs intended for over-the-counter use that contains additional criteria. 3
One shortcoming of the current process for clearing a BGMS for market is that, although many companies perform their own postmarket surveillance throughout the lifespan of the meter systems and test strips, no additional testing and/or reporting is required to be submitted to regulatory authorities once the system is on the market. Exceptions would include FDA audits and follow up of any medical device reporting incident (MDR) that may occur and/or warning letters from the FDA. Concerns about poorly performing BGMSs on the market have been expressed by many stakeholders including: regulatory agencies, health care providers, patients and payers. Many manufacturers of BGMSs have worked diligently to improve the performance of these systems. However, poorly performing BGMSs are currently available on the market. 4 Recent efforts by the FDA to improve accuracy standards for clearance will have little positive impact if the performance of the cleared BGMS is not maintained postmarket.
Current Performance of Cleared BGMS Products
Klonoff and Prahalad reviewed the literature from 2010-2014 of BGMSs that were tested according to ISO 15197:2003 and ISO 15197:2013, which are the standards used by FDA to clear most BGMSs currently on the market. They found that 75% met ISO:15197:2003 and 48% met ISO 15197:2013. 4 Such inaccuracy noted in the literature can be due to: manufacturing errors; scale up; changes in components between strip lots; other production issues; or improper shipping or storage. Eventual performance of a system might no longer correspond to the sponsor’s initial accuracy data submitted to the FDA.
The performance standards stated in the ISO and FDA standards ensure thorough evaluation of blood glucose values. Consequently, noncompliance with these standards could cause incorrect evaluation of blood glucose levels. Unreliably measured blood glucose levels can lead to incorrect therapeutic decisions and adverse health outcomes. This Surveillance Program is intended to protect people with diabetes from such poorly performing, inaccurate BGMSs that are currently on the market in the United States. 4 The program will provide an independent assessment of the performance of BGMSs following clearance by the FDA. The Surveillance Program will (1) establish a benchmark performance level for manufacturers to understand the postmarket accuracy of their products; (2) generate information that can assist diabetes patients, health care providers, and payers in making educated selection of BGMS products; and (3) provide data that can be used by regulatory agencies to support postmarket surveillance activities.
Such a program is needed because, once a BGMS product has been cleared for use by the FDA, there is currently no systematic postmarket Surveillance Program that can ensure ongoing product quality. One limitation of the research studies in the literature is that they are limited in time and scope. Some of them have different designs making comparisons among studies as well as assessments over time very difficult. A Surveillance Program (rather than many independent research studies) is needed to assess analytical performance on an ongoing basis, using a generally accepted protocol that allows comparison of performance over time and among BGMSs.
The Creation of the Program
The mission of Diabetes Technology Society (DTS) is to promote development and use of technology to help people with diabetes. Because BGMS products are one of the most important technologies for diabetes management, DTS has addressed the issues outlined above by establishing the DTS BGMS Surveillance Program for postmarket surveillance of BGMSs. The protocol for testing the marketed BGMS is the subject of this article.
On May 21, 2013, DTS presented a public meeting in Arlington, Virginia, to determine whether BGMSs on the market still met regulatory standards. 5 The meeting consisted of 4 sessions in which FDA, diabetes experts and leading academic clinicians and clinical chemists participated. This meeting was devoted to determining whether BGMSs on the market maintain their performance after they have been cleared by FDA. This was a concern because several articles in the literature at that time 5 presented data showing that, in fact, in some cases BGMSs did not meet the standards under which they were cleared. Meeting participants stated that if products on the market do not meet standards for which they were cleared, then people with diabetes will receive incorrect information and possibly make incorrect decisions putting them at increased risk for hypoglycemia and hyperglycemia. The FDA participated in the meeting and expressed interest in reaching a better understanding of the problem.
DTS sponsored a subsequent public meeting on this issue in Bethesda, Maryland, on September 9, 2013, Verifying the Performance of Blood Glucose Monitors Following FDA Clearance, that was attended by academia, patients, industry, nonprofit diabetes organizations, the FDA, and other government agencies. The event generated much interest and agreement that verification of the accurate performance of cleared BGMSs is needed. Plans for a Surveillance Program were introduced by DTS at that meeting.
On May 20, 2014, following much preparatory work and many negotiations, DTS announced the launch of the DTS BGMS Surveillance Program for Cleared Blood Glucose Monitoring Systems. The data obtained with this program will help manufactures with identifying problems in their quality assurance (QA) systems and will provide other stakeholders with scientifically sound data to initiate actions to address problems and concerns. The lack of sound scientific data seems to be one major challenge preventing key stakeholders from acting on the accuracy problems. The data obtained with this new Surveillance Program can support current efforts and facilitate actions and activities aimed toward improving BGMS and patient safety. The FDA has stated that they will be interested in the information received from the DTS-BGM Surveillance Programs’ Protocol. No other program in the USA with an expert-developed consensus protocol currently exits to independently assess the performance of multiple BGMSs.
In summary, this program is intended to protect people with diabetes from inaccurate BGMSs that might be currently on the market. This program will provide independent assessments of the performance of BGMSs that have received FDA clearance and provide information that can assist diabetes patients, health care providers, and payers in making informed product selection and can provide important information for regulators on the postmarket performance of regulated products.
Methods
Steering Committee
A first step in the process was the formation of a Steering Committee consisting of experts in blood glucose monitoring from academia, medical practice, clinical chemistry, and industry, as well as US government agencies and medical organizations. The Steering Committee that developed the protocol described in this manuscript included members from
Government/Regulatory
Centers for Disease Control and Prevention (CDC)
Food and Drug Administration (FDA)
National Institutes of Health (NIH)
US Army
Voluntary Health Organizations
American Association of Clinical Chemists (AACC)
American Association of Diabetes Educators (AADE)
American Diabetes Association (ADA)
College of American Pathologists (CAP)
Diabetes Technology Society (DTS)
Juvenile Diabetes Research Foundation (JDRF)
The Endocrine Society (TES)
Academic Institutions
Albert Einstein College of Medicine
Isala, Zwolle, the Netherlands
Mills-Peninsula Health Services
New York-Presbyterian/Weill Cornell
University of Missouri School of Medicine
University of Virginia, Center for Diabetes Technology
Vanderbilt University School of Medicine
Consulting
Joan Lee Parkes Consulting, Inc
Sampson Consulting, Inc
Industry
Abbott Diabetes Care
Once this Steering Committee was formed, the committee assembled in-person for all-day meetings on 3 different occasions in the Washington, DC, area to create a protocol for testing the marketed BGMS with an optimal study design. In addition to the in-person meetings of the entire Steering Committee, three separate Steering Committee Subcommittees consisting of experts in various fields, were formed that met by way of teleconferences to achieve consensus on specific topics regarding the best practices for BGMS testing.
The committee is confident that the BGMS testing protocol created is unbiased, transparent and fair to persons with diabetes as well as health care professionals, regulatory agencies, payers and manufacturers of BGMS products.
Decisions
Some of the specific areas where the committees worked through various approaches to reach decisions about best practices for a surveillance screening program for postmarket surveillance were:
Number of subjects, number of samples per subject
Inclusion and exclusion criteria
Testing to be conducted by subject or study staff
Required glucose range to be tested
Number of test strips to be tested per BGMS
Numbers of test strip lots to be tested per BGMS
Number of blood glucose meters to be used in a study
Methods for procuring BGMS products to best represent real-world experience
Hematocrit requirements
Environmental conditions and requirements
Blood sampling method either directly from the fingertip or from a capillary tube
Qualification of glucose reference instruments (ie, comparative glucose analyzers)
Choice of analytic accuracy target levels for passing
Incorporation of confidence intervals into the passing and failing scores
Incorporation of clinical accuracy metrics (ie, Error Grid analysis) into BGMS performance evaluation
Requirements for the DTS seal of approval
Testing sites for BGMSs and comparative glucose analyzer to be at the same or different locations to reduce potential investigator bias
Qualifications for a clinical trial site and a reference laboratory site
An accurate measurement method of SMBG performance in the hypoglycemic range in glycolyzed blood samples without introducing experimental artifacts
Criteria to select and eliminate data points from analysis
Statistical methods to analyze data
Vehicles to communicate findings to professionals and lay public
Retesting criteria if a BGMS fails
All of the above issues and options, along with many other considerations, were carefully worked through to achieve a consensus protocol acceptable to all on the Steering Committee. The committee was confident that the final protocol can be carried out within reasonable cost and time constraints, especially compared to the greater cost associated with clearance of a BGMS and they hoped such a study could be carried out soon.
Specific Controversial Topics
Four specific topics that generated the most discussion to achieve consensus included (1) choice and qualification of glucose comparative instruments, (2) a method of blood sampling, (3) whether to test BGMS and comparative glucose analyzer results at the same or different locations, and (4) modification of blood samples. These topics are discussed in this manuscript.
A separate section describes the statistical analysis including pass-fail criteria for analytical accuracy and clinical accuracy. The protocol itself (Protocol for the Diabetes Technology Society Blood Glucose Monitor System Surveillance Program) is attached as supplementary material.
Results
Study Objectives
The primary objective is to determine whether BGMSs marketed in the US meet predetermined analytical accuracy performance criteria determined by consensus of the DTS BGMS Surveillance Program Steering Committee. The program is not designed to evaluate whether devices are FDA compliant.
Other objectives of the protocol are to
Determine whether BGMSs marketed in the US meet predetermined clinical performance criteria (determined by error grid analysis);
Obtain data on BGMS device failures or device related problems that may occur during the study;
Provide study results to all interested parties through creation of the DTS BGMS Surveillance Website.
Overview of the Study
The protocol covers a series of in vitro diagnostic, Phase 4, BGMS clinical studies to determine accuracy of the systems.
All activities will be conducted according to good clinical practice including institutional review board approval of the protocol and supporting materials, and subject informed consent.
At least 110 and not more than 125 subjects will be enrolled in the study to ensure that at least 100 evaluable natural BGMS test results are obtained.
Limited demographic and medical history information will be collected from subjects.
To facilitate obtaining a wide range of blood glucose values, a subject may be tested in a fasting or postprandial state.
Trained study staff will perform both shallow and deep finger punctures on subjects using lancing devices to collect capillary blood for, respectively, testing on the BGMSs and testing on a comparative glucose instrument.
Subjects will not do any self-testing. Understanding by the end user of instructions for use and human factors analysis are not within the scope of this protocol.
A subject’s capillary blood will be directly applied to the test strip of the BGMS as per labeling of the device.
All values obtained by the BGMS assayed at the clinical site will be compared with capillary glucose values of plasma from the same subjects.
Capillary blood plasma samples will be properly prepared at the clinical site, frozen and shipped on dry ice to a Clinical Laboratory Improvement Amendments (CLIA)–certified, accredited laboratory for analysis on a glucose comparative instrument.
Glucose comparative instruments will be tested with appropriate manufacturers controls as well as with National Institute of Standards and Technology (NIST) 965b standards to ensure accuracy and performance (analytical imprecision ≤ 2.9% a bias of ≤ 2.2%, and a total error ≤ 6.9%) according to ADA and National Academy of Clinical Biochemistry (NACB) guidelines. 6
In some cases, a subject’s capillary-fingerstick blood will also be collected into a micro-tube and glycolyzed to obtain blood samples in the hypoglycemic range not safely obtained directly from the subject. Glycolyzed blood samples will only be assayed on BGMSs that are insensitive to oxygen.
Testing of blood will include glucose concentration and hematocrit.
Trained personnel will oversee the studies, monitor the clinical and laboratory sites, conduct Part 11 compliant data management, perform statistical analysis, create reports and post results on clintrials.gov and on the DTS BGMS Surveillance Program website.
Study Design
This protocol describes a series of studies, to be conducted in a similar fashion, to test the accuracy of marketed BGMSs. Each study will test a different BGMS. The protocol allows for more than a single BGMS to be tested within any study. The first part of a study will be conducted at a clinical site where blood glucose will be measured by a BGMS. The second part of a study will be conducted at a reference clinical laboratory where blood glucose will be measured by a protocol-specified comparable method that is traceable to established reference methods and materials.
Subjects must be at least 18 years old. Most subjects will have a diagnosis of diabetes. Up to 33% of subjects with no diagnosis of diabetes may be recruited to obtain values in the low blood glucose range (ie, <81 mg/dl) where BGMS accuracy is critical.
For each study, subjects will make one visit to the clinical trial site. Study sessions will be conducted by study staff members who are familiar with all aspects of the protocol, use of the BGMSs, data collection, and good clinical practice (GCP). Subjects will complete the informed consent process before any study procedures as performed.
Limited demographic and medical information about the subjects will be collected, including: age, gender, ethnicity, race, and diabetes type (ie, type 1, type 2, prediabetes, or no diabetes). All tests will be performed by trained staff professionals.
The following information will be collected but will
Staff Prestudy Orientation and Training
Study staff training will be conducted prior to the initial BGMS assay for the DTS BGMS Surveillance Program and periodically thereafter, at no duration greater than 1-year intervals, but sooner if there is any question or cause for retraining.
The Study Manager will be responsible for overseeing the entire study, including but not limited to: study progress at both the clinical and laboratory sites, training of study staff on the protocol and acquisition/use of the products, adequate monitoring of the study, and the final close out of the study. The Study Manager will review all study documents, procedures, and specific instructions for use of the BGMSs being tested by the site staff.
GCP training will be provided for the study staff through the CITI Program (the Collaborative Institutional Training Initiative of the University of Miami) or another qualified organization or body. A GCP course completion report for each member of the study staff at the clinical and laboratory sites will be provided to the Sponsor.
Subject Enrollment
At least 110 and up to 125 subjects who meet the inclusion–exclusion criteria will be enrolled into each study to ensure approximately 100 evaluable natural samples results for each BGMS being tested.
Site study staff will perform the following steps when the subject arrives at the clinical trial site: complete the informed consent process; ensure the subject meets all inclusion–exclusion criteria and day of study criteria; assign a unique number to each subject; record subject demographic, medical, and diabetes history information as well as medications taken in the last 48 hours; and offer some subjects something to eat prior to testing to obtain blood glucose values in the high range.
Assaying Glucose in Natural Blood Samples
Staff will perform a shallow fingerstick on the subject; the capillary blood will be applied directly from the finger onto a test strip of the BGMS being tested. Blood from a separate, deep fingerstick will then be collected immediately (but no more than 10 minutes later) into a microtube for centrifugation (within 5 minutes of collection) to separate plasma from cells. The plasma will be carefully transferred to another labeled tube, frozen, and prepared for shipment to the research laboratory for analysis on the comparative glucose instrument.
To facilitate obtaining a wide range of blood glucose values, sometimes a subject may be asked to come to the clinic in a fasting state and have a test performed. Other times, a nonfasting subject may be given something to eat and then be tested within 60 to 120 minutes to obtain a high BG value.
In total, there will be approximately 100 evaluable natural blood sample results from at least 110 subjects and no more than 125 subjects for each type of BGMS tested. Blood will also be collected to determine the hematocrit.
There are no criteria in this program for the number of test strip lots to be tested. This study is designed to test whatever strip lots are obtained from the sourcing process in a manner similar to how a person with diabetes would obtain test strips. This method mimics the real life situation.
Assaying Glucose in Glycolyzed Blood Samples
A BGMS will be tested across a wide range of blood glucose values. To obtain an adequate numbers of test results in the hypoglycemic range (Table 1), some blood samples will also be allowed to glycolyze to lower the blood glucose concentration. This method for generating hypoglycemic specimens protects subjects from the risks of hypoglycemia associated with manipulating their blood glucose levels downward.
Required Distribution of Data Points per Protocol According to Capillary Blood Glucose Level for Oxygen-Sensitive and Oxygen-Insensitive BGMSs.
As described earlier in the Assaying Glucose in Glycolyzed Blood Samples section, the blood samples for some BGMSs cannot be modified to generate data points in the <81 mg/dl range.
This value is an absolute percentage and not a relative percentage of the total.
Glycolyzed blood samples will be assayed only on BGMS products that are oxygen insensitive. This is because glycolysis changes the oxygen concentration of blood samples and the accuracy of some BGMSs is affected by the ambient oxygen concentration. BGMSs using glucose oxidase chemistry are particularly sensitive to oxygen levels.
To achieve blood sample glycolyzation, after the subject participates in the initial testing period, staff will perform an additional deep fingerstick on a subject using a Tenderlett™ or similar device. The capillary blood will be collected into a microtube(s) containing lithium heparin and be allowed to glycolyze in a water bath at 32°C ±2 degrees to lower the glucose level.
A blood drop from the tube of glycolyzing blood will be placed onto parafilm and immediately transferred to the test strip of the BGMS. If the blood glucose value is not less than 81 mg/dl, the tube will be recapped and placed back into the water bath for further glycolysis. Once the desired glucose concentration of the modified blood in the tube is achieved, an additional meter test will be performed. The goal is to obtain a range of low glucose values between 20 and 80 mg/dl. The remaining glycolyzed blood in the tube will be centrifuged (within 5 minutes of collection) and the plasma placed into a new tube, properly labeled and frozen for shipment to the clinical laboratory for assay on the comparative glucose instrument.
Safety Procedures
Study staff will clean and disinfect the BGMS after each use according to the manufacturer’s instructions, and at any time a meter becomes contaminated with blood. A new or properly cleaned and disinfected meter must be used for each subject. A new lancing device and a new lancet must be used for each subject. Safety precautions will be used throughout the study to prevent exposure to biohazard material.
Capillary Blood Glucose Target Ranges
The goal is to obtain at least 100 capillary blood samples (natural and/or modified) distributed across 3 glucose ranges (<81 mg/dl, 81-250 mg/dl, and >250 mg/dl) as shown in Table 1.
It is desired to obtain at least 10
Device Failures
Any functional problems with a BGMS will be documented by study staff. Study staff will be specific about describing the problem and the sequence of events that led to it. All information will be documented, including BGMS (meter) serial number or identification number as well as the test strip lot number. It is important to distinguish user error from BGMS failures. If 5% or more of BGMS devices exhibit a true system failure, then that system will be defined as a failed system despite its accuracy.
See the attached supplementary material for specific details of all testing procedures.
Data Recording
Study staff at the clinical site will record meter results via Electronic Data Capture (EDC) and/or paper case record forms (CRFs). Meter memory (if available) will also be reviewed to ensure that values were correctly documented. Staff at the laboratory site will record results of the comparative glucose instrument on EDC and/or paper CRFs. It is critical that proper labeling is performed to ensure that BGMS results will be properly matched to corresponding comparative glucose assay results. The results recorded for both the BGMS and the corresponding comparative glucose assay results will be sent to the data manager at the Contract Research Organization (CRO) for processing and analysis by the statistician.
Temperature and Humidity Monitoring at the Clinical Site
Study staff will measure the temperature and humidity of the testing area twice per day during the BGMS testing using thermometer/hygrometer(s). Staff will record results and ensure that environmental conditions are within the labeling of the BGMS. If physical conditions at the testing site are out of range, then the environment will have to be changed.
Qualifications of the Research Laboratory
The laboratory will be CLIA-certified, accredited laboratory and must participate in CAP or other proficiency testing glucose surveys. The laboratory must be using a comparative glucose method with records of quality control for at least the previous 6 months.
Proficiency testing (PT) should be performed as required by law in the United States (CLIA 88). The PT provider will send samples to the laboratory, which needs to analyze the samples in a manner identical to that used for patient samples. Details are in CLSI document GP27A2. 7
Measurement of Glucose by Comparative Glucose Analyzer at Research Laboratory
It is imperative that the comparative methods to which the BGMSs are being compared are appropriately accurate, precise, and traceable to established reference methods and materials. All laboratory methods have bias. The bias of the comparative method will be determined by analyzing reference standards (also known as secondary reference materials) of known glucose concentration (established by mass spectrometry) with each batch of subject specimens. This will allow both direct comparison of BGMS results to a laboratory comparative method and estimation of BGMS bias to the mass spectrometry glucose reference method.
Before a research laboratory site can be qualified to participate in the study, the suitability of the instrument for use as the glucose comparative method at that site will be assessed by comparing the measurement bias with predefined performance criteria.
The glucose comparative method to which the BGMS will be compared should be calibrated following the manufacturer’s instructions. NIST 965 Standards (4 levels) will be used to assess the accuracy of the glucose comparative instrument with respect to the reference method. Each sample should be analyzed in duplicate in a single run and bias estimated by calculating the difference between the comparative method results and the expected results for the sample established by the reference mass spectrometry method. A NIST Certificate of Analysis will be included in the study file with all Standard Reference Material 965b Glucose in Frozen Human Serum that is used in this study. This Certificate of Analysis defines the performance of the Standard Reference Material.
Only an instrument/method that meets predefined performance criteria will be used as a glucose comparative analyzer for this study. Any comparative method that meets the predefined criteria for accuracy and precision is acceptable.
As per ADA and NACB guidelines predefined performance criteria for the comparative method are analytical imprecision ≤ 2.9%, a bias of ≤ 2.2%, and a total error ≤ 6.9%. 6 The comparative method analyzer results for that day’s batch analysis will not be used if either the manufacturer controls are outside the acceptable QC ranges or any of the 4 NIST standard reference samples do not meet predefined criteria for precision of sample replicates.
Statistical Analysis
Accuracy Testing
The DTS BGMS Surveillance Program will determine if performance of a marketed BGMS meets prespecified criteria defined below. If it does, then the BGMS will receive a DTS seal of approval. To grant or deny a seal of approval, the surveillance problem is formulated in terms of statistical hypothesis testing, which establishes to what extent the meter test results are consistent with ISO 15197: 2013. Two hypotheses are used to create 3 zones for the probability that a BGMS would perform in compliance with ISO 2013, as follows:
Intuitively
Clear Acceptance with at least 95% probability of adequate performance (
Indeterminate Performance with between 5% and 95% probability of performance (
Clear Rejection with less than 5% probability of adequate performance (
Following the testing of these hypotheses, each meter will receive (1) a
Number readings K within the ISO 15197:2013 Standard limits (out of N=100).
Retesting of BGMS Products That Fail Their Initial Accuracy Test
Upon initial testing, devices in the Clear Acceptance and Indeterminate Performance zones will receive the program seal of approval; devices in the Clear Rejection Zone will
Graphs and Additional Analyses
Several additional graphs and analyses will be computed and offered to further clarify the nature of device errors and to identify outliers in device readings:
Surveillance error grid (SEG) analysis
The SEG Analysis for clinical accuracy 8 will be computed using the SEG software 9 and the results will be presented graphically on the color-coded surveillance plot. Two separate SEG plots will be generated: one for natural and one for glycolyzed samples. The results will be also presented in table format as shown in Table 3 in the Statistical Analysis section of the protocol (Appendix A of the supplementary material).
Bland–Altman plots
Modified Bland–Altman plots (difference between evaluation device results and reference results plotted against reference results) will be constructed for visual inspection of device errors with respect to reference glucose levels. To reflect the clinical risk of large deviations, the Modified Bland–Altman plots will also be color-coded in the same way as the Surveillance Error Grid plot—each data point will be coded according to its Surveillance Grid coordinates. As detailed in the Statistical Analysis section of the protocol (Appendix A of the supplementary material), 2 separate modified Bland–Altman plots will be presented: one for natural and one for glycolyzed samples.
Additional accuracy zone chart
In addition to the ISO 15197: 2013 standard the number of readings within accuracy limits of ±15 mg/dl (< 100 mg/dl) and within ±15% (≥ 100 mg/dl) will be presented together with number of readings within narrower zones as suggested by the 2014 FDA Draft Guidance 3 (see Table 4 in the Statistical Analysis section of the protocol—Appendix A of the supplementary material).
Regression and correlation
The correlation and regression coefficients between reference values and device reading will be reported. This analysis will use all samples, not differentiating natural from glycolyzed samples (see Table 5 in the Statistical Analysis of the protocol—Appendix A of the supplementary material).
Future
The DTS BGMS Surveillance Program will assess the performance of cleared BGMSs, verify adherence to the protocol by clinical trial and reference laboratory test sites, and award a program seal of approval to accurate products. The published protocol will also be available to independent investigators who wish to test the performance of BGMSs outside of the DTS program.
Discussion: Achieving Consensus on Four Specific Controversial Topics
As mentioned above, 4 controversial topics in particular generated lively discussion to achieve consensus. These included: (1) choice and qualification of comparative glucose instruments; (2) a method of blood sampling; (3) whether to test BGMS and comparative glucose analyzer results at the same or different locations; and (4) modification of blood samples.
Choice of and Qualification of Comparative Glucose Instruments
There was much discussion concerning what comparative glucose instrument to employ. Many BGMS manufacturers prefer the glucose oxidase–based Yellow Springs Instrument (YSI) glucose analyzer not only to test the accuracy of their systems but also in the development of their systems. Other manufacturers use large laboratory glucose analyzers such as the hexokinase-based Hitachi 917 or Cobas 6000 c501 (Roche Diagnostics GmbH, Mannheim, Germany).
The Steering Committee decided to accept any comparative glucose analyzer that met prespecified accuracy criteria as follows: Glucose Comparative instruments will be tested with appropriate manufacturer’s controls as well as with NIST 965b standards to ensure accuracy and performance. The following must be met: analytical imprecision ≤2.9%; a bias of ≤2.2%; and a total error ≤6.9%. 6
The committee decided that each BGMS would be assayed at a clinical site, and the plasma samples would be shipped for testing at a different location (CLIA-certified, accredited laboratory) for analysis on a glucose comparative instrument.
Method of Blood Sampling
It was decided the subject’s capillary blood will be directly applied to the test strip of the BGMS as per labeling of the device. The method of collecting blood from a separate shallow fingerstick for assay on the BGMS and from a deep fingerstick for the comparative glucose instrument and then comparing the samples has its limitations. However, the method of applying blood directly from the subject’s finger onto the BGMS test strip conforms to the product labeling, thus it was chosen as the method for this program. The alternative would have been to collect capillary blood from a deep fingerstick into a tube, use a drop of blood from the tube to assay on the BGMS and centrifuge the remainder to obtain plasma to assay on the comparative glucose analyzer.
Whether to Test BGMS and Comparative Glucose Analyzer at the Same or Different Locations
Many BGMS accuracy studies are conducted at a single clinical site. After measurement of the subject’s capillary glucose on the BGMS, a sample of whole blood or plasma from the same subject is assayed on a YSI glucose analyzer at the same location.
The DTS BGMS Surveillance Steering Committee decided to separate the two parts of the study. That is, the BGMS testing will be conducted at a clinical trial site, while the plasma will be frozen and sent to a participating laboratory site for analysis on a comparative glucose instrument. One of the advantages of conducting the study in this manner is to reduce potential investigator bias. It allows for a three-way blind so that an independent third party assembles and collates the data from the BGMS and the corresponding comparative laboratory glucose generated value.
Modification of Blood Samples
A BGMS will be tested across a wide range of blood glucose values. BGMS accuracy in the low glucose range is critically important for patient self-management. It is expected that some naturally low glucose specimens (considered to be <81 mg/dl in this protocol) will occur in every test subject population. These natural hypoglycemic specimens will be tested by all BGMSs. To safely achieve adequate numbers of tests results across the hypoglycemic range, some blood samples (from at least 20 subjects) will also be modified, that is, allowed to glycolyze to lower the blood glucose concentration.
The accurate performance of some BGMS products is particularly sensitive to oxygen levels, especially those with glucose oxidase chemistry. Because glycolysis changes the oxygen concentration of a blood sample, glycolyzed blood samples will be assayed only on BGMSs that are oxygen insensitive. Most BGMSs with glucose dehydrogenase-based chemistry are oxygen insensitive.
Therefore, fewer hypoglycemic data points will be included in the distribution (by glucose level) for oxygen-sensitive BGMSs than for oxygen-insensitive BGMSs. Accordingly, caution is needed if results from surveillance of oxygen-sensitive systems and oxygen-insensitive systems are ever compared, because the distribution of data points in the hypoglycemic range will differ between oxygen-sensitive and oxygen-insensitive BGMSs.
Conclusions
The DTS BGMS Surveillance Program for Cleared Blood Glucose Monitoring Systems is intended to protect people with diabetes from inaccurate, poorly performing BGMS products that are currently on the market in the United States. 4 This Surveillance Program will provide an independent assessment of the analytical performance of BGMSs following clearance by the FDA as well as generate information that can assist people with diabetes, health care providers, and payers in making educated selection of BGMSs. Such a program is needed, because once a BGMS product has been cleared for use by the FDA, there is currently no systematic postmarket surveillance program that can ensure ongoing product quality. Poorly performing, inaccurate BGMS can negatively impact the ability of people with diabetes to reliably measure their blood glucose levels, and can lead to adverse health effects.
An explanation often given for the poor analytical accuracy of some marketed BGMSs and their failure to meet the analytical accuracy data that they originally submitted to the FDA, is intentional selection. According to this theory, the best performing meters and test strip lots were chosen by the manufacturers for their initial studies submitted for clearance. In an effort to avoid this type of “cherry-picking,” the DTS BGMS Surveillance protocol specifies that products to be tested will be obtained from the same types of channels that patients use to obtain their diabetes supplies, rather than directly from the manufacturers to represent a more “real-world experience.”
The results of this program are intended to alert people with diabetes to inaccurate, poorly performing BGMS products that are currently on the market in the United States. The Surveillance Program will provide an independent assessment of the performance of BGMS products following clearance by the FDA. The results of the program will assist manufacturers to understand how their products are performing in the “real world.” The program will also generate information to assist diabetes patients, health care providers, payers, and regulators to understand the performance of these products that they must routinely select, prescribe, finance, and regulate.
Footnotes
Acknowledgements
Annamarie Sucher provided expert editorial assistance.
Abbreviations
AACC, American Association of Clinical Chemists; AADE, American Association of Diabetes Educators; ADA, American Diabetes Association; BGMSs, blood glucose monitoring systems; CAP, College of American Pathologists; CDC, Centers for Disease Control and Prevention; CITI Program, Collaborative Institutional Training Initiative of the University of Miami; CLIA, Clinical Laboratory Improvement Amendments; CRF, case record form; CRO, Contract Research Organization; DTS, Diabetes Technology Society; EDC, electronic data capture; FDA, Food and Drug Administration; GCP, good clinical practice; JDRF, Juvenile Diabetes Research Foundation; MDR, medical device reporting; NACB, National Academy of Clinical Biochemistry; NIH, National Institutes of Health; NIST, National Institute of Standards and Technology; PT, proficiency testing; QA, quality assurance; SEG, surveillance error grid; SMBG, self-monitoring of blood glucose; TES, The Endocrine Society; YSI, Yellow Springs Instrument.
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: DCK has served on advisory panels for Insuline, Lifecare, Novartis, Roche, Sanofi, Tempramed and Voluntis, has received research support from Eli Lilly, Halozyme, Janssen, and Novo Nordisk, and owns stocks/shares in Tempramed. JLP is an independent consultant and president of Joan Lee Parkes Consulting Inc. She has consulted for Diabetes Technology Society and Lifescan in 2014 and 2015. BPK served as an advisor to Astra Zeneca, Becton, Dickinson, and Company and Sanofi-Aventis and has received research support from Animas Inc, BD, Dexcom, Insulet, Roche Diagnostics, Sanofi-Aventis, and Tandem Diabetes Care. Stock ownership: Inspark Technologies, Inc, and TypeZero Technologies. RAV was a consultant for Sanofi, Medtronic, and Bayer at the time of the development of this program. He received an Investigator Initiated Research Grant from Dexcom. Currently, he has no disclosures except that he is an employee of Medtronic. RDB is on the Board of Diabetes Technology Society. JN has received honoraria and travel expenses from IL, BioRad, Fujiribio, Radiometer and Becton Dickinson over the past year related to scientific presentations and professional consulting. SS is an employee of Abbott Diabetes Care. JJS attended an advisory board meeting for Bayer Diabetes Care on April 25, 2015. CL, SB, GAR, AK, RL, MP, KR, DS, ES, RS, and HWV have no relevant disclosures.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This project was supported by a grant from Abbott Diabetes Care. This study was funded in part by the Intramural Research Program of the National Institutes of Health (to DBS).
References
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
