Analysis of “Accuracy and User Performance Evaluation of a New,Wireless-Enabled Blood Glucose Monitoring System That Links to a Smart Mobile Device”

Abstract

In the study published in Journal of Diabetes, Science and Technology, Timothy S. Bailey and coauthors evaluated system accuracy and user performance of a new, wireless-enabled blood glucose monitoring system, the Contour Plus ONE blood glucose monitoring system. The authors declare that the study protocol was performed according to ISO 15197:2013 guidelines, in particular to clauses 6.3 and 8. Results show a high level of accuracy of the product in a laboratory setting as well as in the hands of lay users. However, differences between both study parts emerge that stimulate discussion about the used comparison method.

Blood glucose monitoring systems (BGMS) are widely used by people with diabetes to manage their disease and to improve control of blood glucose levels. Therefore many studies are performed to examine the accuracy of BGMS currently on the market or becoming available soon according to the applicable ISO standard 15197.^1,2

Studies that examine the accuracy of BGMS are quite difficult and often prone to significant errors. Error sources include the acquisition of meters and test strips, the performance of the comparison method and the correct interpretation and application of ISO 15197:2013.

Bailey and colleagues ¹ state that the presented evaluation was an accuracy and user performance evaluation of a new BGMS, performed according to ISO 15197:2013 prerequisites. One evaluation was performed according to clause 6.3 (ie, system accuracy evaluation by trained personnel) and in the other evaluation clause 8 (ie, user performance evaluation) of the technical standard were applied.

In these studies, the BGMS showed accurate results in the hands of laboratory personnel as well as in the hands of untrained users, fulfilling the accuracy criteria of ISO 15197:2013. The studies were well-conducted, taking into account the requirements of ISO 15197:2013. This standard addresses manufacturers of BGMS, but may also be used by researchers investigating available BGMS.

Although these studies were, in general, well designed, they face some limitations. The first limitation is that study funding was provided by the manufacturer, which is common in such studies and does not imply that the results are not to be trusted. In addition, the test strips and meters were provided by the device manufacturer. Ideally, such trials should be performed by an independent organization using testing strips purchased from the market place to exclude a preselection by the manufacturer. ³ In addition, funding of such studies should not depend on the manufacturer of the tested BMGS but rather be financed by funding pools.

Another limitation, that interestingly is becoming obvious due to the presented excellent performance of the BGMS, relates to the used comparison method. Every accuracy evaluation is based on the comparison of a BGMS to a comparison method; therefore one limiting factor in such analyses is the quality of this comparison method, which is frequently called “reference method.” The authors of the study use YSI analyzers, as recommended by the manufacturer, to “obtain reference values for comparison.” Ideally, a “gold standard” analytical method such as isotope dilution mass spectrometry (ID-GC/MS) should be used, but this method is not suitable for routine laboratory measurements. When an instrument like the YSI is used, is has to be traceable to a reference of higher order according to ISO 17511, ⁴ that is, an unbroken chain of calibrations should be demonstrated. Information about the traceability and quality controls of the YSIs used by Bailey and colleagues is missing in the article, but also in publications of other investigators, indicating a common issue.

Interestingly the ISO and radar plots show a negative bias for the system accuracy evaluation (Figure 1, parts A and C) data, whereas no bias is apparent in the user performance evaluation data (Figure 2, parts A and C). What is the cause for this bias, which is not shown in numbers in the publication? Is it at least partly caused by the use of different YSI instruments in different centers? The lead author coauthored an article titled “Fundamental Importance of Reference Glucose Analyzer Accuracy for Evaluating the Performance of Blood Glucose Monitoring Systems (BGMSs)” in this journal earlier this year, ⁵ indicating that this a known issue.

Because Figure 1 shows professional measurements, which are expected to provide accurate results, the negative bias might be attributed to differences between the study site’s and the device manufacturer’s YSIs used for calibration of the BGMS. Unfortunately it is not described if the same instruments and the same consumables (eg, calibration solutions and quality controls) were used in both studies.

A difference in bias can also be caused by lot-to-lot differences of the test strips, by differences in the procedures or by differences in the reference method. It is not clearly described whether the same test strip lots were used in both studies.

In a BGMS of such high analytical quality as the system evaluated by Bailey and colleagues, the impact of a potential bias of the comparison method may not be severe but in case of a less accurate BGMS, the bias might decide about fulfilling ISO criteria or not. In addition, deviations of up to 8% between the two common reference methods glucose oxidase and hexokinase were observed, even though traceability of both methods was established. ⁶ Accordingly, studies showed remarkable differences in accuracy of BGMS depending on which comparison method was applied.^7,8 The impact of the comparison method should thus not be underestimated and in any case be considered in the discussion of accuracy results of a BGMS.

One possible solution to this problem is to mathematically recalibrate the comparison method results. The Scandinavian evaluation of laboratory equipment for primary health care initiative (SKUP), for example, uses inverse calibration to correct their comparison method results. High-quality control material (eg, National Institute for Standards and Technology [NIST] Standard Reference Material® [SRM] 965b) is measured multiple times throughout the course of the study to obtain an appropriate number of results. These results are then used to establish a correction formula, which is applied to each comparison measurement result. Whereas SKUP measures NIST SRM 965b only sparsely, it should ideally be measured at least daily so that possible analyzer drifts over time can be accommodated.

While recalibration cannot eliminate imprecision, it could harmonize different laboratory devices and methods with respect to bias and subsequently, if implemented by device manufacturers, glucose monitoring systems as well.

In conclusion, the article by Bailey and colleagues presents well-performed studies that confirmed high analytical quality of a BGMS. However, a more detailed description and discussion of the used comparison method would have been desirable, especially to sensitize the readers and other investigators to the importance of well-managed comparison method.