System Accuracy Evaluation of Four Systems for Self-Monitoring of Blood Glucose Following ISO 15197 Using a Glucose Oxidase and a Hexokinase-Based Comparison Method

Abstract

Background:

The standard ISO (International Organization for Standardization) 15197 is widely accepted for the accuracy evaluation of systems for self-monitoring of blood glucose (SMBG). Accuracy evaluation was performed for 4 SMBG systems (Accu-Chek® Aviva, Contour®XT, GlucoCheck XL, GlucoMen® LX PLUS) with 3 test strip lots each. To investigate a possible impact of the comparison method on system accuracy data, 2 different established methods were used.

Methods:

The evaluation was performed in a standardized manner following test procedures described in ISO 15197:2003 (section 7.3). System accuracy was assessed by applying ISO 15197:2003 and in addition ISO 15197:2013 criteria (section 6.3.3). For each system, comparison measurements were performed with a glucose oxidase (YSI 2300 STAT Plus™ glucose analyzer) and a hexokinase (cobas® c111) method.

Results:

All 4 systems fulfilled the accuracy requirements of ISO 15197:2003 with the tested lots. More stringent accuracy criteria of ISO 15197:2013 were fulfilled by 3 systems (Accu-Chek Aviva, ContourXT, GlucoMen LX PLUS) when compared to the manufacturer’s comparison method and by 2 systems (Accu-Chek Aviva, ContourXT) when compared to the alternative comparison method. All systems showed lot-to-lot variability to a certain degree; 2 systems (Accu-Chek Aviva, ContourXT), however, showed only minimal differences in relative bias between the 3 evaluated lots.

Conclusions:

In this study, all 4 systems complied with the evaluated test strip lots with accuracy criteria of ISO 15197:2003. Applying ISO 15197:2013 accuracy limits, differences in the accuracy of the tested systems were observed, also demonstrating that the applied comparison method/system and the lot-to-lot variability can have a decisive influence on accuracy data obtained for a SMBG system.

Keywords

comparison measurement method glucose oxidase method hexokinase method system accuracy ISO 15197:2003 ISO 15197:2013 self-monitoring of blood glucose

The accuracy of blood glucose (BG) measurement results obtained with systems for self-monitoring of blood glucose (SMBG) by people with diabetes is crucial for adequate therapeutic decisions. Requirements for SMBG systems, for example, acceptable system accuracy, are established in the International Organization for Standardization (ISO) standard 15197. Manufacturers often provide evidence of conformity with this standard to obtain the Conformité Européenne (CE) mark for their system; in the European Union this mark, is a minimum requirement for being marketed. According to ISO 15197:2003¹ accuracy criteria, at least 95% of results measured with an SMBG system shall fall within ±15 mg/dl of comparison measurement values at BG concentrations <75 mg/dl and within ±20% at BG concentrations ≥75 mg/dl. Compliance with accuracy criteria need only be shown for 1 test strip lot. For comparison measurements, the manufacturer’s selected measurement procedure, usually a glucose oxidase (GOD) or hexokinase (HK) method will be used.¹

In the past decade, SMBG system technology has improved and experts from science and medicine have urged the establishment of more stringent accuracy requirements. Last year, the revision ISO 15197:2013² was published in which several requirements were modified including more stringent accuracy criteria with at least 95% of results within ±15 mg/dl of the comparison measurement result at BG concentrations <100 mg/dl and within ±15% at BG concentrations ≥100 mg/dl and at least 99% of results within zones A and B of the consensus error grid (CEG). The test procedure also has some important changes, for example, the number of test strip lots and the comparison method. Regarding the number of required test strip lots, compliance with system accuracy criteria must be shown for 3 different lots. For comparison measurements, any method that conforms to established traceability requirements³ may be used, not only the manufacturer’s selected measurement procedure.² Metrological traceability means that a measurement result can be related to a higher order standard through a documented unbroken chain of calibrations. However, insufficient information is available regarding the comparability of accuracy data obtained with different comparison methods/systems.

In this study, accuracy of 4 SMBG systems was assessed with 3 test strip lots each following test procedures and acceptance criteria of ISO 15197:2003 (section 7.3). In addition, accuracy criteria of ISO 15197:2013 (section 6.3.3) were applied for which compliance is required only after a 36-month transition period. To investigate a possible influence of the comparison method on system accuracy data, 2 established comparison methods were used.

Methods

The study (trial registry number [ClinicalTrials.gov] NCT01789021) was performed from March to April 2013 at the Institut für Diabetes-Technologie Forschungs- und Entwicklungsgesellschaft mbH an der Universität Ulm (IDT), Germany, in compliance with the German Medical Devices Act. The study protocol was approved by the Ethics Committee and the Federal Institute for Drugs and Medical Devices. All participants signed informed consent prior to the study commencing.

Study Population

A number of 114 subjects (≥18 years) with diabetes mellitus type 1, type 2, or without diabetes were included in the study to obtain 100 evaluable data sets for each system (see also the section on data exclusion). The study physician reviewed the subjects’ medical history, inclusion and exclusion criteria for study participation (eg, pregnancy or lactation period, severe acute disease, and/or chronic disease) and medication, and checked for interfering substances given in the manufacturers’ labeling.

SMBG Systems

Four SMBG systems with a CE mark were evaluated with 3 different test strip lots each: Accu-Chek® Aviva (system 1), Roche Diagnostics GmbH, Mannheim, Germany; Contour®XT (system 2), Bayer Consumer Care AG, Basel, Switzerland; GlucoCheck XL (system 3), aktivmed GmbH, Augsburg, Germany; and GlucoMen® LX PLUS (system 4), A. Menarini Diagnostics S.r.l., Firenze, Italy (Table 1). Meters and test strips were purchased by the IDT from a local pharmacy. All meters displayed plasma equivalent glucose concentrations in mg/dl. The systems were stored, used, and maintained following the manufacturers’ labeling. Control measurements according to the manufacturers’ instructions were performed daily prior to the test procedure to ensure the proper function of each system. One meter of System 4 had to be replaced after 78 valid measurements because of an equipment failure.

Table 1.

Evaluated SMBG systems.

System	Test strips			Comparison method	Test strip enzyme	Measurement range (mg/dl)	Measurement conditions
#	Lot		Expiry date	Comparison method	Test strip enzyme	Measurement range (mg/dl)	Temperature (°C)	Humidity (%)	Hematocrit (%)
1	1	491460	03/2014	HK	GDH	10-600	8-44	10-90	10-65
	2	491465	03/2014
	3	491472	03/2014
2	1	DP2FFEC04B	06/2014	GOD	GDH	10-600	5-45	10-93	10-70
	2	DP2FFEC02A	06/2014
	3	DP2EFED03A	05/2014
3	1	TD12I122-B0G	06/2014	GOD	GDH	20-600	10-40	<85	20-60
	2	TD12J102-B0H	07/2014
	3	TD12H117-B0F	05/2014
4	1	3212214249	08/2014	GOD	GOD	20-600	5-45	10-90	25-60
	2	3212145249	05/2014
	3	3212275249	09/2014

Abbreviations: GDH, glucose dehydrogenase; GOD, glucose oxidase; HK, hexokinase.

Comparison Measurements

Comparison measurements were performed with 2 different methods/systems, a GOD method/YSI 2300 STAT Plus™ glucose analyzer (YSI Inc, Yellow Springs, OH, USA) and a HK method/cobas® c111 analyzer (Roche Instrument Center, Rotkreuz, Switzerland). Both methods provided BG values in mg/dl. According to information provided by the manufacturers, the YSI 2300 STAT Plus™ glucose and the cobas c111 analyzers conform to traceability requirements of ISO 17511.³ To verify trueness and precision of the 2 analyzers, regular internal and external quality control measures were performed as required by the German national standard (RiliBÄK).⁴ In addition, daily quality control measurements were performed following IDT internal standard operating procedures. For the GOD method, mean biases of <±0.8% with coefficients of variation (CVs) <1.3% were found using aqueous control solutions that are traceable to NIST SRM 917c (YSI Bioanalytical Standards; YSI Life Sciences, Yellow Springs, OH). The HK method showed mean biases <2.2% with CVs <0.9% using serum-based controls traceable to the isotope dilution mass spectrometry method (Precinorm U/Precipath U; Roche Diagnostics GmbH, Mannheim, Germany). In addition, serum control samples were measured as part of a German external quality assessment. The GOD method showed a mean bias of -5.5%, whereas the HK method showed a mean bias of −0.7%. The manufacturer’s selected comparison measurement method and/or the method used by the manufacturer for accuracy evaluation as indicated in their system’s labeling is shown in Table 1.

Samples

For each system, at least 100 capillary samples from different subjects and with a defined distribution into different glucose concentration categories (slightly modified limits of these categories were used, because the limits are not clearly defined and differ between the English and the German version of ISO 15197:2003) were analyzed. Samples were assigned to the respective concentration category according to the mean glucose concentration measured with the comparison method. For glucose values ≥50 mg/dl to <400 mg/dl, only unaltered blood samples were used. For glucose concentrations <80 mg/dl and >300 mg/dl, BG values could be provoked by fasting or intake of a high-carbohydrate meal and/or by adjusting the injection–meal interval. The study physician monitored the provocation closely. To obtain higher (≥400 mg/dl) or lower (<50 mg/dl) glucose concentrations, samples could be adjusted. For this purpose, samples were collected in lithium heparin tubes and adjustment of the samples was performed by either incubation, to allow for glycolysis, or by supplementation of glucose (40% glucose in 0.9% NaCl). To ensure an oxygen partial pressure (pO₂) in adjusted samples comparable to the pO₂ in capillary blood,⁵ the pO₂ was determined (OPTI CCA-TS, OPTI Medical Systems, Inc, Roswell, GA, USA) immediately after the measurement procedure. The hematocrit value of each sample was checked to be within the range indicated in the manufacturer’s labeling (Table 1) by collecting capillary whole blood in heparinized capillaries, centrifuging, and determining the hematocrit values using an alignment chart.

Test Procedure

The measurements took place over a period of 21 days in a laboratory setting with controlled temperature (20 to 24°C) and humidity (31% to 47%) following procedures described in detail in ISO 15197:2003 (section 7.3), which differs in some points from the ISO 15197:2013 protocol. Measurements were performed by study personnel, trained to the limitations of the SMGB systems, the manufacturers’ labeling, safety practices, and the test protocol.

Before the measurement procedure, subjects were asked to wash and dry their hands. Then, samples were collected from fingertips by skin puncture. Each sample was measured in duplicate with each test strip lot of a system. Duplicate measurements were performed with 2 meters using test strips from the same vial. Vials were changed after approximately 10 subjects to ensure that test strips were taken from at least 10 different vials.

For unaltered samples, the measurement procedure was as follows: (1) Aliquots were collected for measurements with the comparison methods (first aliquot); (2) measurements were performed with the 4 systems (test strip lot 1); (3) aliquots were collected for measurements with the comparison methods (second aliquot). This procedure was repeated for test strip lots 2 and 3. Measurements with the SMBG systems were performed directly from the fingertip. A fresh drop of blood was generated before measurements with each system and before each aliquot collection for comparison measurements. Aliquots for measurements with the comparison methods were collected in lithium heparin tubes and centrifuged; the measurements were performed on separated plasma. The difference between the first (aliquot collected before the measurements with the SMBG systems) and second comparison measurement result (aliquot collected after the measurements with the SMBG systems) was checked to be ≤4 mg/dl at BG concentrations ≤100 mg/dl and ≤4% at BG concentrations >100 mg/dl to verify sample stability.

Adjusted samples were applied to the test strips from a sample drop placed on Parafilm® (American National Can Company, Chicago, IL, USA). The measurement procedure was as follows: (1) aliquots were removed for measurements with the comparison methods (first aliquot); (2) measurements were performed with the 4 systems (3 lots per system); (3) aliquots were removed for measurements with the comparison methods (second aliquot). The aliquots were processed as described above.

Data Exclusion

Samples were excluded from analysis for the following reasons: hemolysis or lipemia in plasma samples for comparison measurements; equipment failures; the pO₂ of adjusted samples was not determined; the change between first and second comparison measurement result exceeded the acceptance criteria for sample stability (as defined above); the quality control measurement result with the comparison method was outside the acceptable limits; the concentration category was different between lots and/or comparison methods; the data set for a subject was incomplete; the required number of samples in a BG concentration category was already reached.

Data Analysis

For each system 600 data (for each lot 200 data obtained from 100 capillary samples) were analyzed.

System Accuracy

System accuracy was evaluated for each individual lot of a system by comparison with both comparison methods, GOD and HK. For this purpose, the glucose concentration measured with the SMBG system was compared with the respective mean value of the comparison measurement (obtained from aliquots collected before and after the measurement with the system).

According to ISO 15197:2003, the manufacturer’s comparison method shall be used for determination of system accuracy acceptability. To assess system accuracy following ISO 15197:2003, the relative number of results within ±15 mg/dl of the mean comparison measurement result for BG concentrations <75 mg/dl, and within ±20% for BG concentrations ≥75 mg/dl was calculated. Acceptability was assessed for each individual test strip lot by adding the number of results within ±15 mg/dl (BG concentrations <75 mg/dl) to the number of results within ±20% (BG concentrations ≥75 mg/dl).

According to ISO 15197:2013, any comparison measurement procedure for which the manufacturer confirms traceability following requirements of ISO 17511³ can be used for determination of system accuracy.

For accuracy evaluation applying ISO 15197:2013 criteria (section 6.3.3), the relative number of results within ±15 mg/dl for BG concentrations <100 mg/dl and within ±15% for BG concentrations ≥100 mg/dl was calculated. The acceptability was determined for each individual test strip lot by adding the number of results within ±15 mg/dl (BG concentrations <100 mg/dl) to the number of results within ±15% (BG concentrations ≥100 mg/dl). For consensus error grid analysis as required by ISO 15197:2013, the number and percentage of results within zones A and B was calculated and the acceptability of a system was determined by combining the results of the 3 lots.

The system accuracy results of each system are presented following ISO 15197:2003/ISO 15197:2013 in a table and as difference plots in which the differences between each individual system measurement result and the respective mean comparison result are shown.

Bias Analysis According to Bland and Altman

The relative bias (%) of the measurement results according to Bland and Altman⁶ was calculated for each lot of a system by using the formula

\frac{1}{n} \sum^{n} 2 \times \frac{(B G - c o m p a r i s o n)}{(B G + c o m p a r i s o n)} \times 100

in which n is the number of all BG measurement results, BG is an individual SMBG system measurement result, and comparison is the mean result of the comparison measurements (obtained from aliquots collected before and after the measurements with the SMBG system). The relative bias is shown with 95% limits of agreement (≈ ±1.96 × standard deviation).

Regression Analysis According to Passing and Bablok

For each individual lot of a system, regression analyses according to Passing and Bablok⁷ were performed. The regression equation (y = ax + b) reveals proportional (regression line’s slope a) and constant (regression line’s intercept b) measurement differences between an SMBG system and the comparison method.

Results

The 4 systems fulfilled accuracy requirements of ISO 15197:2003 (section 7.3) with all 3 tested lots when compared to the respective manufacturer’s comparison method (GOD method: systems 2, 3 and 4; HK method: system 1). The systems showed 95.5% to 100% of results within the acceptable limits (±15 mg/dl or ±20%) (Table 2, Figure 1).

Table 2.

System Accuracy Results Applying Performance Criteria of ISO 15197:2003.

System		Manufacturer’s comparison method	System accuracy criteria of ISO 15197:2003
			Comparison method: GOD (%)								Comparison method: HK (%)
			Within ±15 mg/dl/±20%	Glucose concentrations <75 mg/dl			Glucose concentrations ≥75 mg/dl				Within ±15 mg/dl/±20%	Glucose concentrations <75 mg/dl			Glucose concentrations ≥75 mg/dl
	Lot		Within ±15 mg/dl/±20%	±15 mg/dl	±10 mg/dl	±5 mg/dl	±20%	±15%	±10%	±5%	Within ±15 mg/dl/±20%	±15 mg/dl	±10 mg/dl	±5 mg/dl	±20%	±15%	±10%	±5%
1	1	HK	100 (200/200)	100	97.5	60	100	98.1	91.9	63.1	100 (200/200)	100	100	82.5	100	99.4	94.4	66.3
	2	HK	99.5 (199/200)	100	97.5	85	99.4	98.8	88.1	58.1	100 (200/200)	100	97.5	92.5	100	98.1	88.8	56.9
	3	HK	100 (200/200)	100	95	82.5	100	98.8	88.8	63.1	100 (200/200)	100	97.5	85	100	100	91.9	63.8
2	1	GOD	100 (200/200)	100	100	95	100	100	96.3	56.9	100 (200/200)	100	100	100	100	100	100	91.3
	2	GOD	100 (200/200)	100	100	97.5	100	100	95.6	60	100 (200/200)	100	100	100	100	100	100	77.5
	3	GOD	100 (200/200)	100	100	92.5	100	100	93.8	53.8	100 (200/200)	100	100	97.5	100	100	98.8	91.9
3	1	GOD	98.5 (197/200)	100	72.5	20	98.1	95	71.9	43.1	97 (194/200)	92.5	60	17.5	98.1	90	73.8	38.8
	2	GOD	97.5 (195/200)	100	75	27.5	96.9	90	75	41.9	98.5 (197/200)	97.5	57.5	17.5	98.8	93.8	71.9	43.8
	3	GOD	95.5 (191/200)	85	52.5	7.5	98.1	94.4	81.3	45	94 (188/200)	85	42.5	5	96.3	85.6	63.1	26.9
4	1	GOD	100 (200/200)	100	92.5	70	100	97.5	84.4	46.3	99 (198/200)	100	97.5	65	98.8	90.6	60	16.9
	2	GOD	98 (196/200)	90	90	57.5	100	98.8	90	55.6	98 (196/200)	90	90	65	100	96.3	69.4	30.6
	3	GOD	100 (200/200)	100	92.5	67.5	100	94.4	77.5	41.9	97 (194/200)	100	95	67.5	96.3	81.3	44.4	13.8

Values in brackets are number of results within accuracy limits and total number of results. Abbreviations: GOD, glucose oxidase; HK, hexokinase.

Figure 1.

Difference plots of 4 SMBG systems with 3 test strip lots each. Left-hand side: the glucose oxidase method was used for comparison measurements. Right-hand side: the hexokinase method was used for comparison measurements. Black lines: Criteria for system accuracy following ISO 15197:2003. Dashed lines: Criteria for system accuracy following ISO 15197:2013. For lot 3 of System C, 1 data point is not shown because the difference exceeds the y-axis scale (comparison measurement result: 155.5 mg/dl (glucose oxidase) and 164.6 mg/dl (hexokinase), system measurement result: 314 mg/dl).

Accuracy criteria of ISO 15197:2013 (section 6.3.3) were fulfilled by 3 systems (1, 2, and 4) when compared to the manufacturer’s comparison method and by 2 systems (1 and 2) when compared to the alternative comparison method. Regarding all 4 systems, 92% to 100% of results were within the limits (±15 mg/dl or ±15%) when the respective manufacturer’s comparison method was applied and 85.5% to 100% of results when the alternative comparison method was applied (Table 3, Figure 1). All 4 systems showed at least 99.8% of results in consensus error grid zones A and B (Table 3). Accuracy of the systems was also analyzed by calculating mean absolute relative deviations, which is presented in a current publication.⁸

Table 3.

System Accuracy Results Applying Performance Criteria of ISO 15197:2013.

System		Manufacturer’s comparison method	System accuracy criteria of ISO 15197:2013
			Comparison method: GOD (%)								Comparison method: HK (%)
			Within ±15 mg/dl/±15%	Glucose concentrations <100 mg/dl			Glucose concentrations ≥100 mg/dl			CEG zones A + B	Within ±15 mg/dl/±15%	Glucose concentrations <100 mg/dl			Glucose concentrations ≥100 mg/dl			CEG zones A + B
	Lot		Within ±15 mg/dl/±15%	±15 mg/dl	±10 mg/dl	±5 mg/dl	±15%	±10%	±5%	CEG zones A + B	Within ±15 mg/dl/±15%	±15 mg/dl	±10 mg/dl	±5 mg/dl	±15%	±10%	±5%	CEG zones A + B
1	1	HK	99 (198/200)	98.4	95.3	62.5	99.3	93.4	63.2	100 (600/600)	99.5 (199/200)	98.3	96.6	81	100	95.1	66.2	100 (600/600)
	2	HK	99 (198/200)	100	93.8	71.9	98.5	89.7	59.6		98.5 (197/200)	100	98.3	87.9	97.9	87.3	54.9
	3	HK	99.5 (199/200)	100	93.5	74.2	99.3	88.4	63.8		100 (200/200)	100	96.6	79.3	100	92.3	64.1
2	1	GOD	100 (200/200)	100	100	81.3	100	96.3	58.1	100 (600/600)	100 (200/200)	100	100	98.3	100	100	90.8	100 (600/600)
	2	GOD	100 (200/200)	100	100	90.6	100	94.9	58.8		100 (200/200)	100	100	98.3	100	100	75.4
	3	GOD	100 (200/200)	100	100	82.3	100	92.8	53.6		100 (200/200)	100	100	96.6	100	98.6	91.5
3	1	GOD	96 (192/200)	100	68.8	25	94.1	74.3	44.9	99.8 (599/600)	91 (182/200)	91.4	55.2	17.2	90.8	79.6	41.5	100 (600/600)
	2	GOD	92 (184/200)	98.4	79.7	35.9	89	73.5	41.2		94.5 (189/200)	98.3	60.3	17.2	93	73.2	47.2
	3	GOD	92.5 (185/200)	88.7	62.9	19.4	94.2	81.2	46.4		85.5 (171/200)	87.9	48.3	8.6	84.5	64.8	28.9
4	1	GOD	98.5 (197/200)	100	92.2	64.1	97.8	83.8	47.1	100 (600/600)	92.5 (185/200)	98.3	94.8	56.9	90.1	56.3	14.1	100 (600/600)
	2	GOD	97.5 (195/200)	93.8	89.1	62.5	99.3	90.4	53.7		95 (190/200)	93.1	91.4	67.2	95.8	66.2	26.1
	3	GOD	95.5 (191/200)	100	90.3	66.1	93.5	76.8	38.4		85.5 (171/200)	100	91.4	60.3	79.6	40.1	10.6

Values in brackets are number of results within accuracy limits and total number of results. Abbreviations: CEG, consensus error grid; GOD, glucose oxidase; HK, hexokinase.

The results of the bias analysis and regression analysis are shown in Table 4. Regarding the results obtained by using the manufacturers’ comparison method, the relative bias of individual lots ranged from −6.2% (system 3) to +4.7% (system 2). The maximum difference in relative bias between any of the 3 lots of a system was 0.8% (system 2), 1% (system 1), 4% (system 4), and 4.1% (system 3) (Table 4, Figure 2). Regression analyses according to Passing and Bablok showed that the average constant measurement differences (ie, intercepts) between the systems and the manufacturer’s comparison method ranged from −14.45 mg/dl to +6.32 mg/dl and the average proportional measurement differences ranged from −9% to +11% (ie, slopes of 0.91 to 1.11) (Table 4).

Table 4.

Relative Bias According to Bland and Altman With a 95% Limit of Agreement Between the SMBG Systems’ Measurement Results and the Comparison Measurement Results and Linear Regression According to Passing and Bablok.

System		Comparison method: GOD			Comparison method: HK
	Lot	Bias (%)	95% limit of agreement (%)	Regression (mg/dl)	Bias (%)	95% limit of agreement (%)	Regression (mg/dl)
1	1	3.7	−6.3 to 13.7	y = 1.01x + 3.36	−0.4	−11.6 to 10.7	y = 0.95x + 5.43
	2	3.0	−8.3 to 14.2	y = 1.01x + 2.55	−1.4	−14.0 to 11.1	y = 0.94x + 5.49
	3	3.3	−7.0 to 13.6	y = 1.03x + 0.17	−1.3	−12.6 to 10.0	y = 0.95x + 3.46
2	1	4.4	−1.9 to 10.7	y = 1.05x − 0.87	0.3	−6.3 to 6.8	y = 0.98x + 2.27
	2	3.9	−2.7 to 10.5	y = 1.03x + 0.90	−0.5	−8.0 to 7.0	y = 0.96x + 3.79
	3	4.7	−2.0 to 11.4	y = 1.05x − 0.95	0.1	−6.6 to 6.8	y = 0.98x + 2.66
3	1	−2.9	−24.5 to 18.6	y = 1.10x − 13.69	−7.1	−26.9 to 12.7	y = 1.03x − 11.19
	2	−2.1	−23.4 to 19.1	y = 1.11x − 14.45	−6.5	−25.8 to 12.7	y = 1.03x − 10.86
	3	−6.2	−29.2 to 16.8	y = 1.06x − 12.92	−10.8	−31.6 to 10.1	y = 0.98x − 9.35
4	1	−2.3	−17.0 to 12.5	y = 0.93x + 4.66	−6.4	−22.3 to 9.5	y = 0.87x + 7.68
	2	0.2	−16.6 to 17.0	y = 0.94x + 6.32	−4.2	−22.5 to 14.1	y = 0.87x + 9.81
	3	−3.8	−19.7 to 12.1	y = 0.91x + 5.03	−8.3	−25.9 to 9.2	y = 0.85x + 8.15

Abbreviations: GOD, glucose oxidase; HK, hexokinase.

Figure 2.

Relative bias according to Bland and Altman for each test strip lot. The relative bias evaluated by using the glucose oxidase (GOD) method is shown in blue diamonds with the upper 95% limit of agreement, the bias evaluated by using the hexokinase (HK) method is shown in red triangles with the lower 95% limits of agreement.

Discussion

In this study, the system accuracy of 4 SMBG systems was evaluated with 3 test strip lots each in a standardized manner under controlled laboratory conditions, under which influencing factors with potential impact on BG measurements were reduced to a minimum. System accuracy and other performance data of the systems investigated have also been reported in other studies.^9-19 However, in our study, each system was evaluated by using 2 established comparison methods/systems, a GOD-based method/YSI 2300 STAT Plus glucose analyzer and a HK-based method/cobas c111 analyzer; the use of 2 methods allows for an investigation of possible impacts of the used comparison method/system on system accuracy.

ISO 15197:2003 stipulates the use of the manufacturer’s selected comparison method for system accuracy assessments. All 4 SMBG systems investigated in our study fulfilled ISO 15197:2003 accuracy criteria (at least 95% of results within ±15 mg/dl at BG concentrations <75 mg/dl and within ±20% at BG concentrations ≥75 mg/dl) with all 3 tested lots when evaluated against the respective manufacturer’s comparison method (GOD: 3 systems, HK: 1 system).

With the revision ISO 15197:2013, comparison method requirements have changed and any method that conforms to ISO 17511 traceability requirements³ can be used. For both systems used in our study for comparison measurements, the YSI 2300 STAT Plus glucose analyzer and the cobas c111 analyzer, compliance with traceability requirements was confirmed by the manufacturers. Regarding accuracy results, the more stringent accuracy limits of ISO 15197:2013 (at least 95% of results within ±15 mg/dl at BG concentrations <100 mg/dl and within ±15% at BG concentrations ≥100 mg/dl) were fulfilled by 3 systems (systems 1, 2, and 4; Table 3) when the respective manufacturer’s comparison method was applied. Only 2 systems (systems 1 and 2; Table 3) met the accuracy criteria when the alternative comparison method was used. Regarding these results, it should be taken into account that only the accuracy limits but not all of the testing procedures of ISO 15197:2013 were applied, because at the time of the study, the final version was not yet published. Differences assignable to comparison methods/systems also became evident when considering the results of the bias analysis. Thus, the results of our study indicate that the comparison method/system can be an influencing factor as to whether accuracy data obtained for a given system comply with established standards or not.

The comparison method/system is used for the calibration of SMBG systems by the manufacturer and for the evaluation of SMBG systems. Following ISO 15197:2013, the evaluation of SMBG systems is not limited to the comparison method/system used for the calibration. Therefore, when interpreting system accuracy data one should be aware that the use of different comparison methods/systems can have an influence on the results obtained. Until now, only a few studies investigated possible systematic differences between different comparison methods/systems. In these studies, a bias of up to 8% between different comparison methods/systems was found.^20,21 In a recent study performed at our institute following requirements of ISO 15197:2013, accuracy results also differed depending on the applied comparison method/system.²² Therefore, a harmonized calibration of comparison methods/systems used for the calibration/evaluation of SMBG systems to a higher order standard is important.

Another important but often neglected factor that can affect system accuracy considerably is the variation between different test strip lots in the same system. Following ISO 15197:2003, system accuracy evaluation need only be performed with 1 test strip lot. However, in several studies, remarkable variations between different test strips were found for some SMBG systems.^11,23,24 ISO 15197:2013 considers lot-to-lot variability more closely and stipulates that at least 3 different test strip lots of a system shall be included in the evaluation. For acceptable system accuracy, each individual lot must show 95% of results within the required limits (15 mg/dl/15%). In our study, minimal differences in relative bias between the 3 lots were found in 2 systems.

The results obtained in this study are comparable with recently published data.^{10,11,15,22,25,26}

To ensure constant adherence of available SMBG systems to established standards, postmarket surveillance evaluations are recommended as being discussed in the current revision of the European directive for in vitro diagnostic medical devices.²⁷

Conclusions

ISO 15197:2003 accuracy criteria were fulfilled by all 4 SMBG systems with the evaluated test strip lots. More stringent accuracy criteria of ISO 15197:2013, for which compliance is recommended in 2016, were met by 3 systems (Accu-Chek Aviva, ContourXT, GlucoMen LX PLUS) when the manufacturer’s comparison method was applied. Only 2 systems (Accu-Chek Aviva, ContourXT) fulfilled ISO 15197:2013 accuracy criteria when the alternative comparison method was used. The results of our study indicate that the comparison method/system and the lot-to-lot variability can have an influence on SMBG system accuracy data. According to ISO 15197:2013 any method that conforms to established traceability requirements can be used for system accuracy assessment. To provide comparable information about SMBG system accuracy, a harmonized calibration of comparison methods/systems used for the calibration/evaluation of SMBG systems to a higher order standard is important.

Footnotes

Acknowledgements

We would like to thank the study personnel and other employees of the IDT who conducted the study and helped preparing the manuscript. We would also like to thank Thorsten Petruschke for reviewing the manuscript and Anne Deveson for proofreading the manuscript. Parts of these data were presented at the Diabetes Technology Meeting 2013 in San Francisco, CA, USA, October 31-November 2, 2013, at the 7th International Conference on Advanced Technologies & Treatments for Diabetes in Vienna, Austria, February 6-8, 2014, and at the IFCC WorldLab Istanbul 2014 in Istanbul, Turkey, June 22-28, 2014.

Abbreviations

BG, blood glucose; CE, Conformité Européenne; CEG, consen-sus error grid; GDH, glucose dehydrogenase; GOD, glucose oxidase; HK, hexokinase; ISO, International Organization for Standardization; pO₂, partial pressure of oxygen; SMBG, self-monitoring of blood glucose.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: All authors are employees of the IDT, which carries out studies on the evaluation of BG meters and medical devices for diabetes therapy on behalf of various companies. GF is general manager of the IDT. GF/IDT have received speakers’ honoraria or consulting fees from Abbott, Bayer, Berlin-Chemie, Becton-Dickinson, Dexcom, Menarini Diagnostics, Novo Nordisk, Roche Diagnostics, Sanofi, and Ypsomed.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This study was funded by Bayer Vital GmbH, Leverkusen, Germany.

References

International Organization for Standardization. In vitro diagnostic test systems—requirements for blood-glucose monitoring systems for self-testing in managing diabetes Mellitus. EN ISO 15197:2003 E.

International Organization for Standardization. In vitro diagnostic medical devices—measurement of quantities in biological samples—metrological traceability of values assigned to calibrators and control materials. EN ISO 17511:2003 (D).

Bundesärztekammer. Richtlinie Der Bundesärztekammer Zur Qualitätssicherung Laboratoriumsmedizinischer Unter suchungen. Deutsches Ärzteblatt. 2008;105(7):341-355.

Freckmann

Schmid

Baumstark

Pleus

Link

Haug

. Partial pressure of oxygen in capillary blood samples from the fingertip. J Diabetes Sci Technol. 2013;7(6):1648-1649.

Bland

Altman

. Statistical methods for assessing agreement between two methods of clinical measurement. Lancet. 1986;1(8476):307-310.

Passing

Bablok

. A new biometrical procedure for testing the equality of measurements from two different analytical methods. Application of linear regression procedures for method comparison studies in clinical chemistry, part I. J Clin Chem Clin Biochem. 1983;21(11):709-720.

Freckmann

Pleus

Link

. Accuracy evaluation of four blood glucose monitoring systems in unaltered blood samples in the low glycemic range and blood samples in the concentration range defined by ISO 15197. Diabetes Technol Ther. 2015.

Freckmann

Baumstark

Jendrike

. System Accuracy Evaluation of 27 Blood Glucose Monitoring Systems According to DIN EN ISO 15197. Diabetes Technol Ther. 2010;12(3):221-231.

10.

Freckmann

Baumstark

Schmid

Pleus

Link

Haug

. Evaluation of 12 blood glucose monitoring systems for self-testing: system accuracy and measurement reproducibility. Diabetes Technol Ther. 2014;16(2):113-122.

11.

Baumstark

Pleus

Schmid

Link

Haug

Freckmann

. Lot-to-lot variability of test strips and accuracy assessment of systems for self-monitoring of blood glucose according to ISO 15197. J Diabetes Sci Technol. 2012;6(5):1076-1086.

12.

Bernstein

Parkes

Goldy

. A new test strip technology platform for self-monitoring of blood glucose. J Diabetes Sci Technol. 2013;7(5):1386-1399.

13.

Hasslacher

Kulozik

Platten

. Analytical performance of glucose monitoring systems at different blood glucose ranges and analysis of outliers in a clinical setting. J Diabetes Sci Technol. 2014;8(3):466-472.

14.

Kristensen

Monsen

Skeie

Sandberg

. Standardized evaluation of nine instruments for self-monitoring of blood glucose. Diabetes Technol Ther. 2008;10(6):467-477.

15.

Halldorsdottir

Warchal-Windham

Wallace

Pardo

Parkes

Simmons

. Accuracy evaluation of five blood glucose monitoring systems: the North American Comparator Trial. J Diabetes Sci Technol. 2013;7(5):1294-1304.

16.

Zueger

Schuler

Stettler

Diem

Christ

. Assessment of three frequently used blood glucose monitoring devices in clinical routine. Swiss Med Wkly. 2012;142:w13631.

17.

Pleus

Schmid

Link

. Accuracy assessment of two novel systems for self-monitoring of blood glucose following ISO 15197:2013. J Diabetes Sci Technol. 2014;8:906-908.

18.

Scandinavian evaluation of laboratory equipment for primary health care (SKUP). Accu-Chek Aviva—meter and test strips designed for glucose self-measurement. 2014. http://www.skup.nu/. Accessed January 13, 2015.

19.

Scandinavian evaluation of laboratory equipment for primary health care (SKUP). CONTOUR(R) NEXT and CONTOUR(R) XT Test strips and meter designated for self-monitoring of blood glucose. 2012. http://www.skup.nu/. Accessed September 1, 2014.

20.

Twomey

. Plasma glucose measurement with the Yellow Springs glucose 2300 STAT and the Olympus AU640. J Clin Pathol. 2004;57(7):752-754.

21.

Genter

Ipp

. Accuracy of plasma glucose measurements in the hypoglycemic range. Diabetes Care. 1994;17(6):595-598.

22.

Freckmann

Link

Baumstark

Schmid

Pleus

Haug

. System accuracy evaluation of 10 SMBG systems with 3 lots each following ISO 15197:2013 against 2 different comparison methods. Paper presented at: EASD Annual Meeting; September 15-19, 2014; Vienna, Austria.

23.

Kristensen

Christensen

Thue

Sandberg

. Between-lot variation in external quality assessment of glucose: clinical importance and effect on participant performance evaluation. Clin Chem. 2005;51(9):1632-1636.

24.

Kimberly

Vesper

Caudill

. Variability among five over-the-counter blood glucose monitors. Clin Chim Acta. 2006;364(1-2):292-297.

25.

Klaff

Brazg

Hughes

. Accuracy evaluation of contour next compared with five blood glucose monitoring systems across a wide range of blood glucose concentrations occurring in a clinical research setting. Diabetes Technol Ther. 2015;17:8-15.

26.

DeSalvo

Shanmugham

Wilson

Buckingham

. Accuracy evaluation of blood glucose monitoring systems in children on overnight closed-loop control. J Diabetes Sci Technol. 2014;8(5):969-973.

27.

European Commission. Proposal for a regulation of the European Parliament and of the council on in vitro diagnostic medical devices. European Commission; September 26, 2012. Available at: http://ec.europa.eu/health/medical-devices/files/revision_docs/proposal_2012_541_en.pdf. Accessed December 4, 2014.