Even Head-to-Head Comparison Studies of Continuous Glucose Monitoring System Performance Can be Biased

Hanson et al ¹ recently published results from the first head-to-head comparison study on the performance of the FreeStyle Libre 3 (FSL3; Abbott, Chicago, Illinois) and DexCom G7 (G7; DexCom, San Diego, California) continuous glucose monitoring (CGM) systems. Their results agreed well with the performance reported for the FSL3, but they showed substantial differences to the reported performance of G7. In this letter, we would like to raise awareness for the fact that even head-to-head CGM performance studies can be biased to favor one CGM system over the other.

The study protocol included in-clinic visits with comparator data obtained by collecting venous blood samples between days 1 and 10 after insertion of CGM sensors. Although this protocol adequately captured the beginning, middle, and end of the 10 day sensor lifetime of the G7, as recommended previously, ² it only covered about 70% of the 14 day sensor lifetime of the FSL3. This protocol clearly favors FSL3 as its accuracy has been reported to worsen toward the end of its sensor lifetime. ³ Nevertheless, we acknowledge the inherent difficulties when designing head-to-head studies for CGM systems with different sensor lifetimes.

Another aspect is the choice of comparator measurement approach that can substantially affect the observed accuracy of a given CGM system. ⁴ Hanson et al collected both venous and capillary blood for comparator glucose measurements using a laboratory analyzer (Yellow Springs Instruments [YSI], Yellow Springs, Ohio) and a home-use blood glucose measurement system (BGMS), respectively. For physiological reasons, capillary blood glucose (BG) concentrations are, on average, about 6% higher than venous BG concentrations. ⁴ However, the results reported by Hanson et al indicate the opposite, ie, that capillary BG concentrations were lower than venous BG levels (Figure 1, panel a). It is therefore highly likely that their chosen BGMS had a considerable negative bias, which agrees with previous findings. ⁵ Had they used an accurate BGMS, the bias would have been smaller for G7 in comparison to FSL3 (Figure 1, panel b). As the bias affects point accuracy, the choice of BGMS favored FSL3 over G7. Nevertheless, G7 showed a large, unexplained bias with respect to venous BG YSI measurements.

Figure 1.

Panel a: Using the bias results provided by Hanson et al for FSL3 and G7, it can be shown that the BGMS used in their study likely had a bias between −3.1% and −4.3% with respect to the YSI glucose analyzer, which is physiologically implausible. Panel b: Assuming an accurate BGMS had been used as a comparator, the bias of FSL3 and G7 would have been −5.4% and +3.4%, respectively. BGMS: blood glucose monitoring system, FSL3: FreeStyle Libre 3, G7: DexCom G7, YSI: Yellow Springs Instruments.

In this context, a general weakness of the study design by Hanson et al can be highlighted: the traceability of neither the YSI glucose analyzer nor the BGMS with respect to a glucose measurement method of higher metrological order was established, despite the fact that even laboratory glucose analyzers can show a bias toward methods of higher order. We therefore recommend this to be done in all CGM performance studies and further recommend a more careful choice of the comparator measurement approach in general. ⁶

In summary, the study by Hanson et al was for the most part well designed, but some aspects, in particular the choice of comparator measurement approach and the scheduling of in-clinic visits, favored FSL3 over G7, which is not surprising as study design, data analysis, and data interpretation were performed by Abbott Diabetes Care. We therefore argue that even when CGM systems are tested simultaneously with a head-to-head study design, biased results can be found. In the future, a standardized study design should be used to evaluate the performance of CGM systems to make sure that the results are unbiased and reliable, which is goal of the working group on CGM established by the International Federation of Clinical Chemistry and Laboratory Medicine. ²