Why the Diabetes Technology Society Surveillance Protocol for Glucose Meters Needs to Be Revised

Abstract

The Diabetes Technology Society surveillance protocol provides a seal of approval for a glucose meter if a sufficient number of a candidate glucose meter’s results meet ISO 15197:2013 limits. The protocol provides clear details about how to conduct this study and analyze the data but has two flaws. There is no specification about the size of glucose meter errors that are outside of ISO limits. A meter that has a result in the E zone of a glucose meter error grid could receive the DTS seal of approval. In addition, the protocol uses the ISO standard, which could be considered a “state of the art” standard instead of an error grid, which is a clinical standard. Remedies for these problems are to replace the ISO standard with an error grid and to include requirements for errors found in C or higher zones of an error grid.

The number of glucose meters available for sale greatly increased in the past few years, prompting scrutiny of the quality of glucose meter results, especially for meters after release for sale (eg, cleared meters). And in fact it was recently shown that only 48.3% of meters met the ISO 15197:2013 acceptance criteria in spite of FDA clearance. ¹ Krouwer provided a possible explanation why meters that receive FDA approval may not perform as well after release for sale. ² In any case, several events took place to try to deal with the problem of poor meter performance:

1. The FDA issued a glucose meter guidance in which use of ISO 15197:2013 was not recommended^3,4

2. The Diabetes Technology Society (DTS) published a surveillance glucose meter error grid based on a survey of 206 clinicians ⁵

3. DTS published a glucose meter surveillance protocol with acceptance criteria, which when met would allow the meter to have the DTS seal of approval ⁶

This article deals with the DTS surveillance protocol’s acceptance criteria and data analysis. Briefly, this protocol requires a minimum of 100 results across the glucose range with the differences between the candidate glucose meter and reference to be tallied as to whether the differences are inside of the ISO 15197:2013 limits or not.

The data are considered in a binomial fashion, meaning that each result is considered to be either acceptable (within ISO limits) or not (outside of ISO limits). Hypothesis testing is used with 95% probability to determine if a meter’s results are within the ISO limits. There are three possible outcomes: acceptable, indeterminate, and unacceptable. Results that are either acceptable or indeterminate are given the DTS seal of approval. Thus, for 100 results, if 91 are within ISO limits, the meter is given the DTS seal of approval.

Note that many other analyses are performed such as regression analysis, error grid analysis, Bland-Altman plots, and how many times the meter fails to obtain a result. But with the exception of the failure of the meter to obtain a result (with 5% failures too many to receive the DTS seal of approval), the other analyses are for information only.

There Are Two Problems With This Approach

Problem 1

Up to 9% of the results can fail the ISO limits for a meter that receives the DTS seal of approval. This is actually more lenient than the ISO standard. Whereas it would be a good thing if these failed results were close to the ISO limits, there is no guarantee that this will happen and there is nothing in the DTS protocol to specify the location of results outside of ISO limits. The ISO standard also requires 99% of results to be within the A and B zones of an error grid, but this requirement is not mentioned in the DTS protocol. Large glucose meter errors, such as those that fall in the D and E zones of an error grid can cause serious harm or death to patients. In essence, the DTS protocol allows for results whose errors are unspecified which have been previously critiqued as a problem with ISO standards.^7,8

To further illustrate this problem, consider a different hypothesis; namely that no results are in the E zone of a Parkes glucose meter error grid. If 100 samples are tested and no results are observed in the E zone, what is the confidence in this outcome? Using a 95% confidence interval for proportions, ⁹ these data prove that no more than 3.6% of results can be in the E zone. Clearly, this is not very comforting. This is not a critique of the DTS protocol, but merely the fact that it is difficult to have high confidence with a relatively small sample size that bad things don’t happen. As another way of looking at this issue, consider that 1 out of 100 results is in the E zone. Whereas one would like to call this an outlier (and disregard the result), this finding is nevertheless an unbiased estimate that 1% of the results for this glucose meter are in the E zone. Of course, the above also applies to results in any zones, such as the C zone which is also associated with harm although less harm than E zone results. Thus, meters that can cause harm to patients (and in some cases serious harm) can achieve the DTS seal of approval.

Problem 2

There are several purposes of specifications. ⁸ The ISO 15197:2013 standard can be considered as a “state of the art” specification. For example, as glucose meter performance has improved, the accuracy requirement has shrunk from 20% to 15% for values > 100 mg/dL from 2003 to 2013. On the other hand, glucose meter error grids are clinical standards. They provide limits which when met, will ensure that a clinician (including self-testing users) will not make a medical error based on the glucose meter result. It is unclear why the DTS protocol uses a state of the art specification rather than a clinical standard. A recent conference suggested that the best specification is based on clinical outcomes and only when neither clinical outcomes nor biological variation are possible, a state of the art specification could be used. ¹⁰ Moreover, it is hard to reconcile the use of ISO 15197:2013 in the DTS protocol since the FDA has declared “FDA believes that the criteria set forth in the ISO 15197 standard do not adequately protect patients using BGMSs in professional settings, and does not recommend using the criteria in ISO 15197 for BGMSs.” ³ A similar statement appeared in the self-testing FDA guidance. ⁴ Moreover, DTS has published the surveillance error grid, but does not use it as an acceptance criterion.

The solution to problem 2 is simply to use an error grid instead of ISO limits and to provide acceptance limits (see next paragraph).

These Problems Can Be Remedied

Regarding problem 1, there needs to be a method of dealing with results found in higher zones of a glucose meter error grid. One suggestion is that if at least 95% of results are in the A zone and the remainder in the B zone of an error grid, the meter would be acceptable. If a result were found in a zone higher than the B zone and a reason for the result was a defect in the product, then the meter would be considered unacceptable until it was proven that the defect was corrected. To help in this process, it would be helpful to repeat the test with the person whose test result was in a higher zone in the error grid. If repeatable, this would suggest a patient related systematic issue such as an interference. If no reason for the failed result could be found, then the protocol would be repeated.

Discussion

One could argue that the results in higher zones of an error grid are likely to be rare. Yet, there is no reason to have a protocol that would allow meters that can produce harmful results to receive the DTS seal of approval, just because such events are likely to be rare. The use of the DTS surveillance protocol fills an important role for manufacturers, diabetes patients, health care providers, payers, and regulators to understand the performance of these products especially in identifying underperforming meters after release for sale. The recommended improvements will strengthen the protocol.