Abstract
Background:
Results from accuracy assessments of systems for self-monitoring of blood glucose (SMBG) are often visualized in difference or regression plots. These approaches become more difficult to read as the number of data points displayed increases, thus limiting their use. In the recently presented rectangle target plot (RTP) approach, data from each reagent system lot or product are displayed graphically as a single rectangle, thus allowing the plot to remain comprehensible even when displaying system accuracy data from multiple reagent system lots or products.
Methods:
The RTP illustrates the accuracy of SMBG systems. Each rectangle shows the mean bias and the variability of a system. By use of statistical tolerance intervals, each rectangle most closely approximates the total error for lower (<100 mg/dL) and upper (≥100 mg/dL) glucose concentrations. RTPs were created for data from 8 different manufacturers of systems for SMBG. In total, the accuracy data of 87 different reagent system lots of 50 different SMBG systems were displayed in RTPs.
Results:
The RTP approach was suitable for 81 of the 87 reagent system lots analyzed. In the remaining cases, outliers caused excessive skewness of the distribution of measurements. The reagent system lots analyzed were grouped according to manufacturer in RTPs. Data from 3 to 15 different reagent system lots were displayed in each RTP.
Conclusion:
Applying the RTP approach to a large number of reagent system lots showed that it was suitable in more than 93% of cases analyzed. The display of system accuracy data in RTPs enables lot-to-lot variability within specific products and product reliability of specific manufacturers to be visualized in a comprehensible manner.
Nowadays, self-monitoring of blood glucose (SMBG) is an integral component of diabetes management. The therapeutic benefits of structured blood glucose (BG) control are unquestioned, particularly for insulin-dependent patients with diabetes using intensified insulin therapy in which they adjust their insulin dose based on their measured glucose values. However, even in non-insulin-dependent diabetic patients, regularly performed SMBG can have a positive impact on metabolic control, for example, an improvement in HbA1c values.1-3
For adequate therapeutic adjustment, high-quality SMBG systems that provide accurate and reliable BG values are indispensable. Various standards and guidelines are available which describe requirements for the assessment of the accuracy of SMBG systems. Examples include the International Organization for Standardization (ISO) standard ISO 15197:2003 and its successor ISO 15197:2013,4,5 the Scandinavian Evaluation of Laboratory Equipment for Primary Health Care (SKUP), 6 the United Kingdom Medicines and Healthcare products Regulatory Agency (MHRA) Point of Care Testing—Blood Glucose Meters guideline, 7 the Netherlands Organization for Applied Scientific Research (TNO) guideline PG/TG/2001.045, 8 the Clinical and Laboratory Standards Institute (CLSI) guideline EP09-A3, 9 and the recently proposed Food and Drug Administration (FDA) Draft Guidance for over-the-counter SMBG systems. 10 In most of these examples, testing of more than 1 reagent system lot is required or at least encouraged to account for lot-to-lot variability. However, the guidelines differ with respect to procedural recommendations or requirements and the minimum accuracy criteria specified.
Results obtained from such accuracy assessments are often visualized in difference plots, such as, for example, recommended by ISO 15197, or in regression plots. However, such plots become more difficult to read as the number of data points displayed increases. A new approach, the rectangle target plot (RTP), was presented by Stephan and colleagues. 11 In RTP, a single rectangle is created based on measurement data from any number of measurements. Thus, it enables data from different systems or from different reagent system lots to be displayed in the same graph, while the graph remains comprehensible.
In the analysis presented here, RTPs were created for a total of 50 different SMBG systems and 87 different reagent system lots from 8 different manufacturers.
Methods
Data from different system accuracy evaluations following ISO 15197,4,5 were retrospectively analyzed by creating RTPs. 11 The evaluations were performed between 2008 and 2014 at the Institut für Diabetes-Technologie Forschungs- und Entwicklungsgesellschaft mbH at the University of Ulm, Germany. Nearly 80% of the system accuracy results have been published.12-17
The objective of the analysis was to evaluate performance characteristics of SMBG systems per manufacturer. Therefore the analysis presented here is limited to manufacturers from which at least 3 different products were evaluated. A maximum number of products per manufacturer was not defined.
RTPs were created by calculating tolerance intervals for BG concentrations <100 mg/dL and ≥100 mg/dL that cover 90% of data points with a confidence of 95%, as described by Stephan and colleagues. 11 These calculations were performed for each reagent system lot separately. As described by Stephan and colleagues, 11 the basic assumption for the RTP is that the data are normally distributed. In the case of a strong deviation from normal distribution due to outliers, tolerance intervals were calculated based on robust estimators such as the median and the scale estimator Sn, and checked for consistency with the observed distribution of measurement data. 18 However, due to excessive skewness of data, even the robust estimators can be biased. In these cases, the rectangles were calculated using robust estimators but marked with dashed lines as the size of the rectangles may not fully correspond to real assessment data.
Results
A total of 87 different reagent system lots from 8 different manufacturers were included in the analysis. If data were available for multiple reagent system lots of a specific system, all lots were included. The analysis inclusion criteria were met by products from Abbott Diabetes Care (Figure 1), Bayer HealthCare/Bayer Consumer Care (Figure 2), Bionime (Figure 3), Infopia (Figure 4), LifeScan (Figure 5), Nipro Diagnostics (Figure 6), Roche Diabetes Care (Figures 7 and 8), and TaiDoc (Figure 9). Although an important benefit of the RTP is the ability to illustrate several lots or products in 1 graph, the large number of data sets available for the Roche Diabetes Care products would not suitably fit into the same RTP. These products were therefore separated into 2 figures. So, data from 3 to 15 different reagent system lots were displayed in each RTP. Information about the respective SMBG systems and reagent system lots can be found in Table 1.
Rectangle target plots (RTPs) for SMBG systems from Abbott Diabetes Care. Different colors stand for reagent system lots showing a certain percentage of differences between system measurements and comparison measurements within specified limits. Dark green: ≥95% of differences within ±10 mg/dL and ±10% at glucose concentrations <100 mg/dL and ≥100 mg/dL, respectively. Light green: ≥95% of differences within ±15 mg/dL and ±15% at glucose concentrations <100 mg/dL and ≥100 mg/dL, respectively. Yellow: ≥95% of differences within ±15 mg/dL and ±20% at glucose concentrations <75 mg/dL and ≥75 mg/dL, respectively. Red: <95% of differences within ±15 mg/dL and ±20% at glucose concentrations <75 mg/dL and ≥75 mg/dL, respectively. See also Table 1. Rectangles plotted in dashed lines mark reagent system lots where tolerance intervals are inconsistent with the distribution of results.
Rectangle target plots (RTPs) for SMBG systems from Bayer HealthCare/Bayer Consumer Care.
Rectangle target plots (RTPs) for SMBG systems from Bionime.
Rectangle target plots (RTPs) for SMBG systems from Infopia.
Rectangle target plots (RTPs) for SMBG systems from LifeScan.
Rectangle target plots (RTPs) for SMBG systems from Nipro Diagnostics.
Rectangle target plots (RTPs) for SMBG systems from Roche Diagnostics (Roche Diabetes Care), part 1.
Rectangle target plots (RTPs) for SMBG systems from Roche Diagnostics (Roche Diabetes Care), part 2.
Rectangle target plots (RTPs) for SMBG systems from TaiDoc.
SMBG Systems and Reagent System Lots Included in the Analysis.
These older Roche SMBG systems have now been upgraded to use a maltose-independent test strip chemistry.
The system name as displayed in this table is identical to the system name as displayed in the figures; if a system test was published under a different name, the name under which it was published is given as “(tested as . . .).”
Absolute differences for BG concentrations <75 mg/dL, relative differences for BG concentrations ≥75 mg/dL.
Absolute differences for BG concentrations <100 mg/dL, relative differences for BG concentrations ≥100 mg/dL.
The RTPs are displayed in Figures 1 to 9. The figures are composed of 2 parts. The larger, upper part shows the rectangles calculated from the measurement data. In the smaller, lower part, the mean absolute and relative differences (ie, the center of the rectangle) are indicated for each individual reagent system lot. Different colors represent reagent system lots showing a certain percentage of differences between system measurements and comparison measurements within the system accuracy limits as defined in ISO 15197:2013 or within and outside the system accuracy limits as defined in ISO 15197:2003. The dark green rectangles represent tolerance intervals for test strip lots that meet stringent criteria of ±10 mg/dL/±10% for system accuracy. The light green rectangles represent tolerance intervals for test strip lots that meet the ±15 mg/dL/±15% system accuracy limits as stipulated by ISO 15197:2013 (criterion A). The orange rectangles show tolerance intervals for test strip lots that meet the system accuracy criteria of ISO 15197:2003, but not ISO 15197:2013. The red rectangles show tolerance intervals for test strip lots that do not meet the system accuracy criteria of either ISO 15197:2013 or ISO 15197:2003.
Because of the statistical methods used in the RTP approach, it is not intended to be a substitute for a system accuracy assessment following ISO 15197. Although there is a high degree of consistency between compliance with ISO 15197:2013 criterion A and having a rectangle within the light grey area (±15 mg/dL/±15%), the RTP is a predictor for a system’s measurement performance, but does not provide a definitive statement regarding ISO 15197.
The RTP approach was suitable for 81 of the 87 reagent system lots analyzed. For the remaining 6 reagent system lots, the measurement data showed a pronounced deviation from normal distribution due to excessive skewness. In these cases, the rectangles were calculated using robust estimators but marked with dashed lines as the size of the rectangles may not fully correspond to real assessment data.
Thus, the RTP was fully applicable in more than 93% of cases analyzed. In the remaining 7% of cases, the calculated rectangles rather overestimated the real performance of the respective SMBG systems.
Discussion
Common approaches to graphical presentations such as difference plots and regression plots display each data point separately. The disadvantage of such plots is that they become difficult to comprehend as the number of data points displayed increases.
The RTP is a method of visualizing a large amount of measurement accuracy data from an SMBG system graphically as a single rectangle, thus facilitating a comprehensible display of system accuracy data from multiple reagent system lots or products in the same graph.
In the analysis presented here, the RTP approach was suitable for nearly all analytical accuracy data sets considered. The results of the analysis suggest that RTPs can be used to illustrate the analytical accuracy of a broad variety of SMBG systems and test strip lots. Therefore, the RTP approach allows for an easy comparison of reagent systems within and among manufacturers.
The size and location of the rectangles in the figures clearly show whether different products or different reagent system lots of the same product have consistent analytical accuracy. As described by Stephan and colleagues, 11 the size of the rectangle illustrates the measurement variability, and the center of the rectangle indicates the mean bias of the reagent system lot separately for the low and the high glucose concentration range relative to comparison method results. Small, centered rectangles represent better performance than large rectangles or rectangles positioned away from the center of the graph.
The graphs presented in Figures 1 to 9 show clearly different performance characteristics of SMBG systems per manufacturer. The RTPs suggest a wide variety in performance between manufacturers resulting from the observed level of heterogeneity of the rectangles. Comparisons of different products from the same or different manufacturers might help to estimate the effect of switching from 1 product to another. However, it should be borne in mind that this analysis was performed based on the data sets available to the authors and does not make a claim to completeness with respect to the respective manufacturers’ portfolios.
Some products for which multiple reagent system lots were tested exhibit only small lot-to-lot variation, whereas other products exhibit greater lot-to-lot variation. Lot-to-lot variation for a specific product is of direct relevance to the reliability of BG measurement results obtained by persons with diabetes. Since BG results are used by insulin-dependent persons with diabetes to adjust their diabetes therapy, the reliability of the measurement results is of great importance with regard to short-term and long-term complications. 15
Data sets used to illustrate the RTP approach in this article were obtained in evaluations assessing the system accuracy of SMBG systems following ISO 15197. Although the RTPs include shaded areas and color coding related to the system accuracy criteria of ISO 15197, the RTP approach is not intended to replace the analysis recommended in ISO 15197, and it is not limited to data sets obtained in this kind of accuracy evaluation.
Conclusions
In conclusion, the RTP approach allowed the analytical accuracy data of reagent system lots from different SMBG system manufacturers to be visualized in a comprehensible manner. Accuracy results were used to assess the consistency among reagent system lots for specific SMBG systems and the consistency among products of the same manufacturer. In both cases, qualitative differences could easily be identified using the RTP.
Footnotes
Acknowledgements
The studies in which data for this post hoc analysis were obtained were performed at the Institut für Diabetes-Technologie Forschungsund Entwicklungsgesellschaft mbH an der Universität Ulm, Ulm, Germany.
Abbreviations
BG, blood glucose; CLSI, Clinical and Laboratory Standards Institute; FDA, Food and Drug Administration; GOD, glucose oxidase; HK, hexokinase; ISO, International Organization for Standardization; MHRA, UK Medicines and Healthcare Products Regulatory Agency; RTP, rectangle target plot; SKUP, Scandinavian Evaluation of Laboratory Equipment for Primary Health Care; SMBG, self-monitoring of blood glucose; TNO, Netherlands Organization for Applied Scientific Research
Declaration of Conflicting Interests
The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: All authors are employees of Roche Diagnostics GmbH or Roche Diabetes Care GmbH, Mannheim, Germany.
Funding
The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Roche Diabetes Care GmbH, Mannheim, Germany, funded the studies in which data for this post hoc analysis were obtained.