Abstract
Systematic reviews and meta-analyses (SRMAs) provide unique insights into comparative effectiveness of diabetes treatments. However, use of these analyses may be inappropriate for assessing the value and utility of technologies that involve significant behavioral interventions and encompass rapidly evolving technologies such as real-time continuous glucose monitoring (RT-CGM). The rapid evolution of RT-CGM, compared with the time required for publication of clinical studies used in SRMAs, may preclude differentiation between past and current generations of devices. In addition, the effect of performance and usability differences between the various commercial devices on possible clinical outcomes associated with the devices are often not clearly discussed, and many of the RT-CGM studies assessed in SRMAs do not provide adequate information regarding whether and/or to what degree study subjects and clinicians were trained to use the RT-CGM and utilize the data to adjust therapy. Although numerous clinical studies have shown that the glycemic benefit of RT-CGM is related to the frequency and duration of use, a disproportionate number of RT-CGM studies included in recent SRMAs are based on the results of the intention-to-treat analyses and do not consider this fundamental behavioral component in their conclusions. Given these limitations, the generalizability of SRMA conclusions may be limited, and findings from these reports may significantly underestimate the potential glycemic benefit of current and future devices, posing challenges for coverage and reimbursement. We reviewed the potential limitations of the recent Cochrane Collaboration report on CGM, focusing on the 12 studies that assessed RT-CGM use in adults, children/adolescents or both.
New pharmacological agents, technologies, and innovative treatment approaches provide the potential to enable patients with diabetes to more safely and effectively manage their glycemic levels. One emerging decision-support technology is real-time continuous glucose monitoring (RT-CGM).
Numerous studies have shown that use of RT-CGM improves glycemic control in both children and adults with type 1 diabetes1-11 and in patients with type 2 diabetes who are managed with or without intensive insulin treatment.12-15 However, in many CGM studies, clinical benefit was primarily seen in those patients who regularly wore their CGM devices for approximately 6 days or more per week,2,3,5,6,16-18 and as such, these results are heavily dependent on subject behavior and motivation to comply with protocols. Problems associated with early generation RT-CGM devices may have resulted in poor protocol compliance and distorted the conclusions of such studies as to the magnitude of the potential clinical benefits or lack thereof of continuous glucose monitoring.
Despite the demonstrated benefits of RT-CGM, conclusions from recent systematic reviews and meta-analyses (SRMAs) regarding the efficacy and utility of RT-CGM use are somewhat mixed. Of the 9 SRMA reports we evaluated, 7 concluded that current evidence strongly supports the beneficial effects of RT-CGM on glycemic control in diabetes patients, particularly in adults;19-24 whereas, the 2 remaining reports were significantly less supportive of the strength of the current evidence for use in adults 25 or children/adolescents.25,26
In this article, we discuss potential limitations of SRMA assessments pertinent to RT-CGM efficacy and propose alternative approaches to research that may be more appropriate.
A recent article by Pickup discussed several limitations of SRMA in assessing the value and clinical utility of continuous subcutaneous insulin infusion (CSII). 27 In this article we present important challenges of utilizing SRMA relevant to RT-CGM. Because reports from the Cochrane Collaboration are often strongly considered by clinicians and payers, we have used the most recent Cochrane report on continuous glucose monitoring 26 for our discussion of the potential limitations of SRMA, focusing on the 12 studies that assessed RT-CGM use in adults, children, adolescents, or both (Table 1).
Studies of RT-CGM Included in the Cochrane Report. 26
Abbreviations: BG, blood glucose; CGM, continuous glucose monitoring; CSII, continuous subcutaneous insulin infusion; RT-CGM, real-time continuous glucose monitoring; SG, sensor group
Inability to Contend With Rapid Evolution of CGM Technology
While SRMA assessments provide unique insights into the efficacy of a given treatment versus clinical effectiveness, rapid advances in CGM technology may invalidate the power of SRMA techniques and make it difficult to draw clinically meaningful conclusions from the application of standard SRMA techniques. Conducting outcomes studies, which are required for SRMA, and then publishing findings in the peer-reviewed literature is a lengthy process. By the time reports from clinical trials utilizing earlier generation of devices are published, analyzed, and compiled, those devices have been replaced by a newer, improved generation of RT-CGM systems with different performance characteristics 28 that may affect patient use and outcomes.29,30
For example, among the 12 RT-CGM devices introduced between 2001 and 2014, only 4 are commercially available in the United States today: Dexcom G4 PLATINUM system (Dexcom, San Diego, CA, USA); Medtronic Guardian Real Time system (Medtronic, Northridge, CA, USA); Medtronic Paradigm Real Time insulin pump with Sof-Sensor; and Medtronic 530 G with Enlite Sensor. A number of other systems are in use or are awaiting regulatory approval in Europe. Although the accuracy and usability of these newer devices is significantly greater than earlier-generation systems, and the enhanced accuracy has recently been linked to reduced hypoglycemia, 31 none of the most recent devices were included in the studies used in the Cochrane meta-analysis.
Inability to Differentiate Between RT-CGM Systems
Not only do current RT-CGM systems differ considerably from earlier-generation devices, there are also significant differences between manufacturers in their devices in terms of sensor needle gauge, sensor/transmitter/receiver size, transmitter range, warm-up/calibration time, sensor duration and exogenous and endogenous chemical interferences. Table 2 includes performance data of RT-CGM devices that were available in the United States in 2013 and highlights the differences in several attributes that can directly impact user comfort and perceptions of convenience and usability, which can, in turn, influence users’ frequency of wear and impact clinical outcomes.1-3,5,6,16-18,29,30,32 A recent survey by Chamberlain and colleagues revealed significant differences in patient preferences and frequency of use between 2 earlier-generation RT-CGM devices. 33
JDRF RT-CGM Devices Compared With Current RT-CGM Devices.
Abbreviations: CGM, continuous glucose monitoring; JDRF, Juvenile Diabetes Research Foundation; MARD, mean absolute relative difference ; ROC, rate of change; RT-CGM, real-time continuous glucose monitoring; YSI, Yellow Springs Instrument.
Sensors used in the JDRF study.
Because SRMA reports do not differentiate between RT-CGM systems that may have distinct features and performance characteristics, their conclusions may be biased. Moreover, the majority of studies included in these SRMAs are limited to the assessment of only a few RT-CGM devices. When there is a preponderance of data from 1 device, the findings and conclusions from the SRMA may misrepresent the potential clinical benefits of the category as a whole. Among the 4 most recent clinical studies9,16,34,35 included in the Cochrane analysis, 26 3 evaluated the same RT-CGM device, the MiniMed Paradigm CGM system.9,34,35 The specific performance characteristics and subsequent user acceptance of a given RT-CGM device can impact the frequency of use and subsequent clinical outcomes.
Heterogeneity of Patient Training/Support Among Studies
Although CGM use may be intuitive to many individuals, some patients still may require additional training from health care professionals to use the CGM data more effectively. The level of training and support users received in previous studies may have influenced not only the frequency with which they wore their RT-CGM device but, just as important, whether they made appropriate use of their data to improve their self-management. As discussed previously, a key predictor of users’ satisfaction and perceived benefit from CGM is confidence and trust in their ability to use their data to safely manage their diabetes. 30 Thus, the scope and comprehensiveness of patient training should be considered when assessing studies that evaluate the efficacy of RT-CGM use. However, this important aspect of diabetes management relevant to RT-CGM utilization was not considered by the Cochrane investigators in their assessment of heterogeneity among the studies included in their report. 26
Among the 12 RT-CGM studies included in the Cochrane meta-analysis, only 7 provided any description of the training/support subjects received during in their study; this training ranged from complex written instructions for real-time CGM use to verbal guidance only (Table 1).3,4,9,16,34-36 In most studies it was unclear how subjects were educated to use their real-time CGM data to make acute management decisions, or how to retrospectively review their RT-CGM data. In essence, current SRMAs did not recognize the differences in patient training or the lack of information regarding this important behavioral component as a source of heterogeneity and a potential confounding factor in their cross-comparison of multiple studies.
Use of Intent-to-Treat Analyses to Assess Behavioral-Based Interventions
Another significant limitation is the reliance of standard SRMA methodology on intent-to-treat (ITT) analyses, which is often prerequisite for included studies. Although pharmaceutical studies have traditionally used ITT analyses to assess the efficacy of medications, the utility of RT-CGM is minimized when ITT analyses are applied to behavior-based interventions. For example, the intervention and evaluation mechanisms in most pharmaceutical clinical studies are fairly straightforward; subjects are administered a diabetes medication and investigators assess the medication’s metabolic mode of action in lowering glucose, utilizing the difference in HbA1c between intervention and control groups as the primary determinant of efficacy.
However, in studies of technologies such as RT-CGM, which require modification of patient behavior as part of the ongoing adjustment of therapy, the mechanisms of action are much more complex. Because the mode of action involves both device performance and behavioral response from study participants and/or their treating clinicians, the efficacy of the intervention is dependent on behavior change as well as technology, chemistry, and biology. When ITT analyses are applied to behavior-based studies, only the effects of “asking” subjects to use the device are measured—not the efficacy of the device when subjects actually wear it at the frequency prescribed and make appropriate therapy decisions based on the data generated.
Conclusions
Although SRMAs are generally considered the highest level of clinical evidence for evaluating therapeutic efficacy, the inherent design of SRMAs may render them inappropriate for assessing rapidly evolving technologies such as RT-CGM. Rapid evolution of CGM technology may invalidate the power of SRMA techniques and make it difficult to draw clinically meaningful conclusions and may significantly underestimate the efficacy and utility of current CGM systems in diabetes management. This is particularly relevant to the behavioral aspects of RT-CGM use because if 1 system is less accurate and less well tolerated by patients, the SRMA may be biased to a less favorable assessment than is deserved by the category of CGM as a whole. In addition, SRMAs that neglect both the duration of wear and the extent of patient training in the use of the data are, therefore, at risk of drawing inappropriate conclusions.
Given that diabetes management is primarily dependent on patient behavior, different research approaches and measures of efficacy are needed to more definitively assess the effects of behavior-based interventions. For example, comparison of the percentage of subjects in both study groups who showed significant clinical benefit would be a meaningful analysis. This approach not only provides opportunities for identifying and elucidating the characteristics of those patients who are most (or least) suited for the intervention but also readily lends itself to assessing the economic benefits of the intervention in a more meaningful way. For example, it would be helpful to know that 40% more patients can achieve clinically significant improvements in glycemic control using RT-CGM than patients who are basing decisions on standard self-monitoring of blood glucose (SMBG) devices. Payers could then calculate the potential cost savings within their specific patient population associated with the intervention in comparison with the incremental costs (if any) associated with the intervention. This analytical approach has been used in a recent assessment of the benefits of SMBG.37,38
Unless careful attention is directed to the specific devices used and the details of the behavioral interventions in the cited clinical studies, SRMAs of CGM systems may draw inappropriate or inconsistent conclusions from the results of those studies. Moreover, investigators, clinicians and financial decision makers should be open to a shift in the current evidence-based medicine paradigm to accommodate the unique characteristics of behavior-based interventions that utilize these rapidly evolving technologies.
Footnotes
Abbreviations
ITT, intent to treat; RT-CGM, real-time continuous glucose monitoring; SMBG, self-monitoring of blood glucose; SRMAs, systematic reviews and meta-analyses.
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: DP and CG are employees of Dexcom, Inc. CGP has received consulting fees from Animas, Boston Therapeutics, Inc, CeQur, Dexcom, Inc, Roche Diagnostics, and Sanofi. TAP has received consulting fees from Dexcom, Inc.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Funding for the development of this article was provided by Dexcom, Inc, San Diego, CA.
