Sage Journals: Discover world-class research

Abstract

In the European Union (EU), the ISO (International Organization for Standardization) 15197 standard is applicable for the evaluation of systems for self-monitoring of blood glucose (SMBG) before the market approval. In 2013, a revised version of this standard was published. Relevant revisions in the analytical performance requirements are the inclusion of the evaluation of influence quantities, for example, hematocrit, and some changes in the testing procedures for measurement precision and system accuracy evaluation, for example, number of test strip lots. Regarding system accuracy evaluation, the most important change is the inclusion of more stringent accuracy criteria. In 2014, the Food and Drug Administration (FDA) in the United States published their own guidance document for the premarket evaluation of SMBG systems with even more stringent system accuracy criteria than stipulated by ISO 15197:2013. The establishment of strict accuracy criteria applicable for the premarket evaluation is a possible approach to further improve the measurement quality of SMBG systems. However, the system accuracy testing procedure is quite complex, and some critical aspects, for example, systematic measurement difference between the reference measurement procedure and a higher-order procedure, may potentially limit the apparent accuracy of a given system. Therefore, the implementation of a harmonized reference measurement procedure for which traceability to standards of higher order is verified through an unbroken, documented chain of calibrations is desirable. In addition, the establishment of regular and standardized post-marketing evaluations of distributed test strip lots should be considered as an approach toward an improved measurement quality of available SMBG systems.

Keywords

blood glucose monitoring SMBG systems ISO 15197:2003 ISO 15197:2013 FDA draft guidance document system accuracy

Today, self-monitoring of blood glucose (SMBG) is an integral component of diabetes management. Particularly for patients with diabetes on intensified insulin therapy, therapeutic benefits of structured blood glucose (BG) measurements, such as detection of postprandial glucose excursions, identification of glucose patterns, and the improvement of HbA1c values and related complications, are well-established.^1-3 Also in patients not on insulin therapy, regular SMBG can have a positive impact on metabolic control, for example, an improvement in HbA1c values.^4-6 To make adequate therapeutic adjustments, the SMBG system (combination of a meter with the respective test strips) has to provide accurate and reliable BG readings.^7-9

In the European Union, medical devices such as SMBG meters must have a Conformité Européenne (CE) mark for their market launch. The manufacturer of an SMBG system declares with the application of the CE mark that its product is in compliance with the essential requirements of the in-vitro diagnostics directive (IVDD) (European directive 98/79/EG, Annex I)¹⁰ with regard to health protection, safety and environment protection. These requirements can be fulfilled by applying the harmonized standards which are developed by standardization organizations like the International Organization for Standardization (ISO). For SMBG systems that are intended to be used by lay persons (patients with diabetes) for therapy adjustments, the standard ISO 15197 is applicable. This standard, which was first published in 2003,¹¹ lists requirements for design and development, safety and reliability testing, analytical performance evaluation, information to be supplied by the manufacturer and user performance evaluation. Regarding analytical performance evaluations, particularly requirements on system accuracy are described in detail in this standard, including evaluation design and minimum accuracy criteria. In 2013, a revised version of ISO 15197¹² was published for which compliance is recommended only after a transition period of 36 months (2016). In this revision, relevant changes to the previous version ISO 15197:2003 were made, for example, more stringent accuracy criteria and some changes in the testing procedure (see below).

In the United States, the Food and Drug Administration (FDA) regulates the premarket notification for the market approval, the manufacturing and performance standards as well as the post-marketing surveillance of SMBG systems. The manufacturer has to submit a premarket notification to the FDA and the agency reviews if the system complies with applicable requirements, for example, performance requirements. Regarding SMBG system accuracy requirements, the FDA, hitherto, followed in their own internal guidance at least in part the recommendations of ISO 15197. Recently, the FDA published their own guidance document for the premarket evaluation of SMBG systems¹³ which differs in some parts, for example, system accuracy evaluation, considerably from ISO 15197.

This review aims at providing an overview of the analytical performance requirements described in ISO 15197 and of the relevant changes in the recently published revision. Particularly, requirements on SMBG system accuracy evaluation as an important and often discussed aspect of the analytical performance will be considered in detail, also taking into account the recommendations of the new FDA draft guidance document. In addition, certain critical aspects in the establishment of performance criteria for premarket evaluations will be discussed.

Evaluation of the Analytical Performance of SMBG Systems According to ISO 15197

The analytical performance is an important criterion for high-quality SMBG systems, which is usually evaluated by well-trained laboratory personnel under controlled and standardized conditions in which environmental influences are reduced to a minimum. The evaluation of analytical performance according to ISO 15197 includes the evaluation of measurement precision and the evaluation of system accuracy. Additional parts of the analytical performance evaluation in the new ISO 15197 standard are the evaluation of influence quantities, that is, hematocrit and interfering substances like medications, and the evaluation of the stability of reagents and materials that were not included in the analytical performance evaluation in the 2003 version (Figure 1). Another relevant revision is that analytical performance shall be evaluated for 3 test strip lots. The previous version of the ISO standard allowed the evaluation with 1 test strip lot if characterization data have demonstrated that lot-to-lot variability is a minor source of the total variability.

Figure 1.

Evaluation of the analytical performance of SMBG systems according to the ISO 15197 standard. The new version ISO 15197:2013 also considers the analysis of influence quantities and the evaluation of the stability of reagents and materials that were not considered in ISO 15197:2003 for analytical performance evaluation.

This review focuses on system accuracy evaluation. The evaluation of the stability of reagents and materials will not be considered and the evaluation of measurement precision and influence quantities will only be shortly introduced.

Measurement Precision

This parameter characterizes the closeness of agreement between replicate measurement results with an SMBG system on a given sample.¹² ISO 15197 describes testing procedures for the evaluation of measurement repeatability and intermediate measurement precision but does not define in detail acceptance criteria for such.

Measurement Repeatability

Measurement repeatability evaluation describes the closeness of agreement between a series of measurements with an SMBG system on a blood sample over a short period of time. The evaluation shall include a minimum of 5 venous blood samples with defined glucose concentrations in the hyperglycemic, euglycemic, and hypoglycemic range.

Intermediate Measurement Precision

Intermediate measurement precision evaluation considers the closeness of agreement between daily measurements over at least 10 days with an SMBG system on control solution. The evaluation shall include a minimum of 3 control solutions with defined glucose concentrations.

Influence Quantities

An important revision in ISO 15197:2013 is the inclusion of the evaluation of influence quantities, such as hematocrit and interfering substances in blood that can affect the analytical performance of an SMBG system. A list showing examples of interfering substances which could be present in the blood of the intended users is given in the annex of the standard. The preferred sample for the evaluation of influence quantities is venous blood. Hematocrit influences shall be investigated for a minimum of 5 different hematocrit levels at each of 3 defined glucose concentrations. Interfering substances shall be investigated for a minimum of 2 defined glucose concentrations. ISO 15197:2013 defines that influence quantities >10 mg/dl and >10% difference between the test sample and the respective control sample for glucose concentrations ≤100 mg/dl and >100 mg/dl, respectively, shall be reported in the instructions for use along with the respective hematocrit levels or interfering substance concentrations.

System Accuracy

In the last decade, the accuracy of SMBG systems has increasingly become the subject of public interest and is frequently discussed by experts from science, medicine, and industry. System accuracy is the closeness of agreement between a set of representative results from a measuring system and their respective reference values.

Accuracy Criteria

Regarding minimum accuracy criteria, ISO 15197:2013 stipulates that at least 95% of measurement results shall fall within ±15 mg/dl of the reference value at BG concentrations <100 mg/dl and within ±15% at BG concentrations ≥100 mg/dl and at least 99% of measurement results shall fall within the Consensus Error Grid (CEG) zones A and B. Thus, accuracy criteria are more stringent than in the first version of the standard (±15 mg/dl at BG concentrations <75 mg/dl and ±20% at BG concentrations ≥75 mg/dl) (Table 1, Figures 2 and 3). In addition, ISO 15197:2013 requires the accuracy evaluation of 3 different test strip lots (as stated above). The evaluation of different test strip lots allows for a more comprehensive assessment of a given SMBG system. Studies showed that the measurement accuracy can remarkably vary within 1 system, that is, lot-to-lot variations can occur.¹⁴ However, it needs to be emphasized that ISO 15197:2013 still requires that each individual test strip lot must comply with the 95% accuracy criteria (Table 1). Another relevant change in ISO 15197:2013 is the consideration of the clinical relevance of measurement deviations.¹² The CEG is divided into 5 zones with differing degrees of risk for the patient (Figure 3). Both versions of the ISO standard limit the number of analytically inaccurate results to 5% (= 95% within the accuracy limits). However, with the inclusion of the CEG into the minimum accuracy criteria (as described above), the new ISO standard limits the number of clinically inacceptable results to 1% (= 99% within CEG zones A and B).

Table 1.

Minimum SMBG System Accuracy Criteria According to the ISO 15197 Standard in Its 2003 and 2013 Versions and the FDA Draft Guidance Document From 2014.

	ISO 15197:2003		ISO 15197:2013		FDA^a
Relative number of results	95%		95%^b		95%	99%
Within	±15 mg/dl	±20%	±15 mg/dl	±15%	±15%	±20%
At BG concentrations	<75 mg/dl	≥75 mg/dl	<100 mg/dl	≥100 mg/dl	Entire range
			99% of results within CEG zones A + B

FDA draft guidance for Self-Monitoring Blood Glucose Test Systems for Over-the-Counter Use, published in 2014.

In ISO 15197:2013 this acceptance criterion is also applied for the user performance evaluation.

Figure 2.

Difference plot of an SMBG system (1 test strip lot, 200 data) following ISO 15197. According to ISO 15197:2003 (outer line), at least 95% of results shall be within ±15 mg/dl at BG concentrations <75 mg/dl and within ±20% at BG concentrations ≥75 mg/dl). According to the revision ISO 15197:2013 (inner line), at least 95% of results shall be within ±15 mg/dl at BG concentrations <100 mg/dl and within 20% at BG concentrations ≥100 mg/dl. The results obtained with the SMBG system displayed shows 98% of measurement results within ISO 15197:2003 accuracy limits but only 92.5% of measurement results within ISO 15197:2013 accuracy limits.

Figure 3.

Consensus error grid (CEG) analysis of an SMBG system (3 test strip lots, 600 data). According to ISO 15197:2013, 99% of measurement results shall be within CEG zones A and B. The SMBG system displayed shows 99.8% of results with in CEG zones A and B.

The rationale for the establishment of more stringent accuracy criteria in ISO 15197:2013 is the essential role of SMBG in diabetes management and the fact that currently SMBG is the most important practical means for detection of asymptomatic hypoglycemia.¹² Since the publication of the first version of the ISO 15197 standard more than a decade ago, SMBG systems’ technology has improved, and today, various systems are capable of meeting higher accuracy standards. From a clinical point of view, improved SMBG system accuracy is needed, for example, for an adequate use of other diabetes technologies such as bolus calculators and calibration of continuous glucose monitoring systems.

Evaluation Procedure

In the last decade, numerous evaluations of SMBG systems applying ISO 15197 system accuracy criteria were published.^15-26 However, most of these studies show remarkable deviations from the complex testing procedure recommended by the ISO standard, for example, number of samples included, sample type, and investigated glucose concentration categories.²⁷

According to ISO 15197, evaluation of SMBG system accuracy should be performed with at least 100 capillary blood samples from different subjects. The glucose levels in these samples shall be distributed into defined glucose concentration intervals between ≤50 mg/dl and >400 mg/dl (Table 2). Since human studies to obtain capillary blood samples in the hyperglycemic and hypoglycemic range represent a certain risk for patients, ISO 15197 allows the adjustment of the glucose levels in these samples into the target ranges by either glycolysis or glucose supplementation. According to the new ISO 15197 standard, samples with BG concentrations ≤80 mg/dl and >300 mg/dl can, at least partly, be adjusted. In the previous version of the standard, only samples with BG concentrations <50 mg/dl and >400 mg/dl were allowed to be adjusted (Table 2). However, the preparation procedure to adjust the BG concentration of capillary blood samples is complex. Everyone should be aware that sample matrix changes may result in additional influence factors, for example, changes in the oxygen content of the blood sample, which can affect the measurement results.

Table 2.

SMBG System Accuracy Evaluation: Test Procedure Requirements Described in the ISO 15197 Standard in Its 2003 and 2013 Versions.

	ISO 15197:2003		ISO 15197:2013
Number of test strip lots	1		3
Performance of measurements with the SMBG system	Laboratory/medical personnel^a		Laboratory/medical personnel^a
Reference measurement procedure	Manufacturer’s measurement procedure		Any method with verified metrological traceability
Sample type/number	Capillary whole blood/100		Capillary whole blood/100
Glucose concentrations	Number of samples	Glucose concentrations (mg/dl)	Number of samples	Glucose concentrations (mg/dl)
Glucose concentrations	5%	<50	5%	≤50
	15%	50-80	15%	>50-80
	20%	80-120	20%	>80-120
	30%	120-200	30%	>120-200
	15%	201-300	15%	>200-300
	10%	301-400	10%	>300-400
	5%	>400	5%	>400
	• Only unaltered samples between 50 and 400 mg/dl		• Only unaltered samples between >80 and 300 mg/dl
			• At least 8 unaltered samples between >50 and 80 mg/dl
			• At least 5 unaltered samples between >300 and 400 mg/dl

Not clearly defined but indicated (“trained operators” in ISO 15197:2013).

Reference Measurement Procedure

An important but often neglected aspect that can affect SMBG system accuracy results is the reference measurement procedure used. A potential systematic measurement difference of the reference measurement procedure with respect to higher-order procedures (see below) adds to SMBG system-inherent inaccuracies. Today, most manufacturers of SMBG systems calibrate and evaluate their systems by using either a glucose oxidase (GOD) or a hexokinase (HK) based reference measurement procedure. ISO 15197:2003 required the use of the manufacturer’s measurement procedure for system accuracy evaluations, thus ensuring a certain degree of comparability. This was changed in ISO 15197:2013 so that any method that conforms to established traceability requirements²⁸ can be used for reference measurements, not only the manufacturer’s measurement procedure (Table 2). However, the comparability of different reference measurement procedures is not yet scientifically proven. On the contrary, in 2 studies, systematic differences of up to 8% between a GOD method and a HK method were observed.^29,30 In 2 recent SMBG system accuracy evaluations following the requirements of the ISO 15197 standard, accuracy results differed depending on the applied reference measurement procedure.^31,32 Thus, the reference measurement procedure used in such evaluations can be the deciding factor whether accuracy data obtained for a given SMBG system comply with the ISO 15197 accuracy criteria or not; which becomes particular critical the more stringent the criteria are.

An example that highlights the importance of a high quality reference method is the recommendation by the American Diabetes Association,³³ that SMBG systems should not exceed an analytical error (systematic and random measurement difference) of 5% which can only be achieved if the reference method shows no or only a minimal systematic measurement difference (= bias).

Therefore, the establishment of a harmonized reference measurement procedure for all SMBG systems for which the traceability to standards of higher order is verified through an unbroken, documented chain of calibrations²⁸ should be considered. In this context the implementation of the isotope dilution gas chromatography mass spectrometry (IDGCMS) method in the calibration and evaluation process of SMBG systems by the manufacturer has to be mentioned. This is a highly accurate, but time-consuming and expensive primary reference measurement procedure^34,35 which is listed in the JCTLM (Joint Committee for Traceability in Laboratory Medicine) database.³⁶

User Performance Evaluation Following ISO 15197

Analytical performance evaluations under controlled and standardized conditions when measurements are performed by well-trained laboratory personnel are important to provide standardized and reproducible information of an SMBG system’s measurement quality. However, even with a high-quality SMBG system it can be difficult for patients to obtain accurate results if the system is difficult to handle or if the instructions for use are incomplete or unclear. Therefore, ISO 15197 also requires a user performance evaluation to show if patients with diabetes are able to obtain accurate measurement results with a given system. For this purpose, measurements shall be performed by the users following the instructions of use, without any training or assistance. The 2013 version of the ISO standard describes the testing procedure for the user performance evaluation much more precisely than the 2003 version. Also some changes in the testing procedure were made (Table 3). The most important revision is the definition of acceptance criteria which follow closely the minimum criteria for system accuracy: 95% of measurement results obtained by the patients shall be within ±15 mg/dl of the reference value at BG concentrations <100 mg/dl and within ±15% at BG concentrations ≥100 mg/dl.

Table 3.

User Performance Evaluation Following ISO 15197 Standard.

	ISO 15197:2003	ISO 15197:2013
Acceptance criteria		95% of measurements obtained within ±15 mg/dl of the reference measurement results at glucose concentrations <100 mg/dl and within ±15% at glucose concentrations ≥100 mg/dl
Number of test strip lots	3	1
Performance of measurements with the SMBG system	At least 50 subjects	At least 100 diabetic subjects
Sample type	Capillary whole blood	Capillary whole blood
Determination of accuracy	Results measured by the subjects shall be compared to	Results measured by the subjects shall be compared to
	• results obtained with the reference measurement procedure	results obtained with the reference measurement procedure
	• results obtained by medical/laboratory personnel from the same sample
Reference measurement procedure	Manufacturer’s measurement procedure	Any method with verified metrological traceability^a
Collection of samples for reference measurements	Within 5 minutes after the subject’s measurement with the SMBG system by medical/laboratory personnel	Within 5 minutes after the subject’s measurement with the SMBG system by medical/laboratory personnel
Training material/practice tests	Instructions for use, etc routinely provided with the system	Instructions for use, etc routinely provided with the system/subjects may be allowed a limited number of practice tests
Incorrectly performed measurements		The measurement can be repeated if the subject reports a mistake (a maximum of 3 data exclusions due to incorrectly performed measurements shall be allowed)
Human factors	User technique observation by the medical/laboratory personnel	User technique observation by the medical/laboratory personnel
Evaluation of the instruction for use	Ease of understanding^b	Ease of understanding^c

Not clearly defined but indicated.

The evaluation method is not described in detail.

The evaluation method is described in detail.

FDA Draft Guidance Document

In 2014, the FDA published draft guidance documents with requirements for SMBG systems intended for professional health care settings ("Blood Glucose Monitoring Test Systems for Prescription Point-of-Care Use”; POCT)³⁷ and for SMBG systems intended for self-monitoring by lay-persons (“Self-Monitoring Blood Glucose Test Systems for Over-the-Counter Use”; OTC).¹³ Subsequently, the draft guidance for POCT will not be discussed here.

According to the draft guidance document for OTC SMBG systems, the analytical performance evaluation includes the evaluation of measurement precision (measurement repeatability, intermediate measurement precision), the evaluation of linearity over the claimed measuring range, the evaluation of system accuracy by a method comparison/user evaluation and the evaluation of interferences.¹³ Currently, the FDA recommendations for system accuracy are extensively discussed and we also focus on these. The FDA recommendations for the other analytical evaluations will not be considered.

System Accuracy Evaluation

The accuracy criteria recommended by the FDA for SMBG systems for OTC use and the respective testing procedures differ considerably from that provided by the ISO 15197 standard (Tables 1 and 4).

Table 4.

SMBG System Accuracy Evaluation: Test Procedure Requirements Described in the FDA Draft Guidance for Self-Monitoring Blood Glucose Test Systems for Over-the-Counter Use, Published in 2014, for Method Comparison/User Evaluation and Accuracy Evaluation at Extreme Glucose Values.

	FDA
General study design
Number of test strip lots	3
Reference measurement procedure	Any method with verified metrological traceability
Evaluation of the instruction for use	Readability assessment using a computer program
Method comparison/user evaluation
Performance of measurements with the SMBG system	At least 350 intended users (at least 10% naïve to the use of SMBG systems)
Training material	Instructions for use, etc routinely provided with the system
Collection of samples for reference measurements	After the subject’s measurement with the SMBG system by medical/laboratory personnel
Sample type/number	Capillary whole blood/at least 350 each site (fingertip, forearm, palm, etc)
Glucose concentrations	• Samples shall span the measuring range of the system (minimum range of 50-400 mg/dl is required)
	• At least 10 unaltered samples <80 mg/dl
	• At least 10 unaltered samples ≥250 mg/dl
Accuracy evaluation at extreme glucose values
Performance of measurements with the SMBG system	laboratory/medical personnel^a
Glucose concentrations	• At least 50 capillary samples adjusted to <80 mg/dl
	• At least 50 capillary samples adjusted to >250 mg/dl
	• Samples shall cover the lower and upper limits of the claimed measuring range (minimum range of 50-400 mg/dl is required)

Not clearly defined but indicated (“laboratory setting”).

Accuracy Criteria

According to the FDA draft guidance document, at least 95% of measurement results shall fall within ±15% and at least 99% within ±20% of the reference measurement values across the entire claimed measurement range of the SMBG system (Table 1). Experts from industry, science and medicine are concerned about these quite strict minimum accuracy criteria.^38
-40 The difficulty to meet these accuracy criteria becomes particularly obvious for BG concentrations at the low end of the measurement range. Most current SMBG systems have an indicated measurement range of about 10 or 20 mg/dl to 600 mg/dl. Applying the FDA criteria for BG concentrations of, for example, 20 mg/dl means that only 5% of measurement results are allowed to exceed the acceptable deviation of ±3 mg/dl and 1% of measurement results are allowed to exceed the maximum acceptable deviation of ±4 mg/dl. In comparison, ISO 15197:2013 allows a deviation of ±15 mg/dl for BG concentrations <100 mg/dl. According to the FDA draft guidance document, manufacturers have to adapt the claimed measurement range if the system does not fulfill the FDA accuracy criteria. Glucose values that fall outside of the claimed measurement range should be indicated with an error code, for example, “low—less than 50.” The FDA requires a measurement range of at least 50 mg/dl to 400 mg/dl. Also at BG concentrations of 50 mg/dl, the requirement that 95% of measurements shall be within ±7.5 mg/dl (42.5 to 57.5 mg/dl) is comparably strict, and published data indicate that many available systems may not reliably achieve this accuracy.^15,16,19,25

The FDA also proposes that 3 test strip lots shall be included in the accuracy evaluation. However, it remains unclear if the criteria must be fulfilled with each of the 3 lots, as required by ISO 15197:2013, which is more stringent than taking all 3 lots together. When setting such stringent accuracy requirements, a high accuracy of the laboratory method used for the reference measurements is a prerequisite (see above). Regarding the reference measurement procedure, the FDA draft guidance allows, as the ISO 15197:2013, any measurement procedure with verified traceability to standards of higher order.

Test Procedure to Obtain System Accuracy

In the FDA draft guidance document, the test procedure to evaluate system accuracy is quite different from the ISO 15197 test procedure because the FDA recommends that the intended users perform the measurements for the evaluation. Only data at extreme glucose values (lower and upper measurement range) can be obtained using adjusted samples in a laboratory setting (Table 4). For both parts, the method comparison/user performance evaluation and the accuracy evaluation at extreme glucose values, the accuracy criteria as mentioned above shall be applied.

Method Comparison/User Evaluation

For this evaluation part, measurements with the SMBG system shall be performed by the intended users, that is, people with diabetes (Table 4). At least 350 subjects, comprising naïve and non-naïve users, shall perform the measurements with the SMBG system following the instructions of use, without any training or assistance. Capillary BG concentrations shall be distributed over the entire SMBG system’s measurement range, including at least 10 unaltered samples <80 mg/dl and ≥250 mg/dl, respectively (Table 4). However, no recommendations are made with respect to a defined distribution into different BG concentration intervals—as given in the ISO 15197 standard.

Test Procedure to Obtain System Accuracy

To evaluate sufficient accuracy data at the lower and upper measurement range, the FDA recommends additional studies conducted in a laboratory setting with adjusted samples (Table 4). The additional study comprises at least 50 samples adjusted to glucose concentrations <80 mg/dl and at least 50 samples adjusted to glucose concentrations >250 mg/dl. Such samples shall evenly cover the lower and upper limits of the measurement range.

SMBG System Accuracy—Status Quo

Premarket Requirements

It is clear that high measurement quality, for example, system accuracy, is an important aspect for the market approval of SMBG systems. The stipulation of more stringent accuracy criteria standards applicable for the premarket evaluation, such as in ISO 15197:2013 or in the FDA draft guidance document is a possible approach to further improve the measurement quality of SMBG systems. For this purpose, different aspects, for example, the effectiveness of current SMBG systems in diabetes management, existing standards, current technologies and recommendations of experts should be taken into account. In addition, criteria should be clinically appropriate for the intended use, that is, self-monitoring of BG by patients in daily life conditions. Furthermore, the stipulation of more stringent accuracy criteria must be balanced with cost increases and potential decreases in measurement convenience and system usability that can negatively affect the frequency with which people with diabetes perform measurements.

Currently, the premarket evaluation of SMBG systems is a 1-time procedure which is usually conducted by the manufacturer itself and often criticized to be a nontransparent process.^41,42 Since independent premarket evaluations are required neither in the United States nor in the European Union, a selection of test strip lots by the manufacturer for the premarket evaluations can not be excluded. The FDA stipulates that each released test strip should conform to the labeled performance and requires a description of the lot release criteria in the premarket notification. In contrast, in Europe, regular and standardized evaluations of test strip lots from routine production are not required once an SMBG system is available on the market.

Postmarketing Performance Data

Over the last years a number of evaluation studies were published showing that not all available SMBG systems comply with currently still applicable ISO 15197:2003 accuracy standards.^{14,16,18,23,43} Unsurprisingly, a remarkable number of systems failed to meet the more stringent accuracy criteria of the new version of this standard.^14,16,23,44 However, accuracy data of such postmarketing evaluations do not allow for a general conclusion regarding the measurement performance of a given SMBG system as this evaluation is usually limited to a small number of lots. Nevertheless, these study results show that considerable variations in the measurement quality of distributed test strip lots can occur, between different SMBG systems as well as in the same system. In practice, most of the patients and health care professionals are even not aware of lot-to-lot variations when they interpret BG measurement results and make therapeutic decisions. Therefore, the implementation of independent postmarketing surveillance programs is important to ensure sufficient accuracy of all test strip lots available for patients.

Conclusion

In the EU, the ISO standard 15197 is applicable for the premarket evaluation of SMBG systems. In 2013, a revision of this standard was published for which compliance is recommended in 2016. Regarding analytical performance evaluations, some relevant changes were introduced in the new version of this standard, for example, the evaluation of influence quantities, the number of test strip lots being evaluated, and the reference measurement procedure. In addition, system accuracy criteria are more stringent. In the United States, the FDA recently published its recommendations with stricter SMBG system accuracy criteria and evaluation procedures that differ considerably from ISO 15197. The establishment of more stringent accuracy criteria for the premarket evaluation is an approach to improve the measurement quality of SMBG systems; however, this requires very close attention to the respective evaluation procedure, for example, the reference measurement procedure used. Therefore, the establishment of a harmonized reference measurement procedure for all SMBG systems with verified traceability to standards of higher order should be considered. In addition, the establishment of independent postmarketing controls should also be implemented as an approach toward improved measurement quality of SMBG systems in daily life of patients with diabetes. To ensure similar requirements for SMBG systems’ measurement performance and the respective evaluation procedures all over the world, the establishment of an international consensus standard would be helpful.

Footnotes

Acknowledgements

The authors thank Stefan Pleus for reviewing the manuscript.

Abbreviations

BG, blood glucose; CE, Conformité Européenne; CEG, consensus error grid; FDA, Food and Drug Administration; GOD, glucose oxidase; HK, hexokinase; IDGCMS, isotope dilution gas chromatography mass spectrometry; ISO, International Organization for Standardization; IVDD, in-vitro diagnostics directive; JCTLM, Joint Committee for Traceability in Laboratory Medicine; OTC, over-the-counter use; POCT, point-of-care use; SMBG, self-monitoring of blood glucose.

Declaration of Conflicting Interests

The author(s) declared the following potential conflicts of interest with respect to the research, authorship, and/or publication of this article: GF is general manager of the IDT (Institut für Diabetes-Technologie Forschungs- und Entwicklungsgesellschaft mbH an der Universität Ulm, Ulm, Germany), which carries out studies on the evaluation of BG meters and medical devices for diabetes therapy on behalf of various companies. GF/IDT have received speakers’ honoraria or consulting fees from Abbott, Bayer, Berlin-Chemie, Becton-Dickinson, Dexcom, Menarini Diagnostics, Novo Nordisk, Roche Diagnostics, Sanofi, and Ypsomed. CS, AB, and CH are employees of the IDT. LH has received speakers’ honoraria or consulting fees from Abbott, Becton-Dickinson, Dexcom, Roche Diagnostics, Sanofi, and other companies. He is shareholder of Profil Institut für Stoffwechselforschung, Neuss Germany and Profil Institute for Clinical Research, San Diego, CA. MR was employee of Roche Diagnostics GmbH until 2013.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

References

Diabetes Control and Complications Trial Research Group. The effect of intensive treatment of diabetes on the development and progression of long-term complications in insulin-dependent diabetes mellitus. N Engl J Med. 1993;329(14):977-986.

American Diabetes Association. Standards of medical care in diabetes—2014. Diabetes Care. 2014;37(suppl 1):S14-S80.

Bergenstal

Gavin

III . The role of self-monitoring of blood glucose in the care of people with diabetes: report of a global consensus conference. Am J Med. 2005;118(suppl 9A):1S-6S.

Polonsky

Fisher

Schikman

. Structured self-monitoring of blood glucose significantly reduces A1C levels in poorly controlled, noninsulin-treated type 2 diabetes: results from the Structured Testing Program Study. Diabetes Care. 2011;34(2):262-267.

Virdi

Daskiran

Nigam

Kozma

Raja

. The association of self-monitoring of blood glucose use with medication adherence and glycemic control in patients with type 2 diabetes initiating non-insulin treatment. Diabetes Technol Ther. 2012;14(9):790-798.

American Diabetes Association. Standards of Medical Care in Diabetes—2011. Diabetes Care. 2011;34(S1):S11-S61.

American Diabetes Association. Consensus statement on self-monitoring of blood glucose. Diabetes Care. 1987;10(1):95-99.

American Diabetes Association. Self-monitoring of blood glucose. Diabetes Care. 1994;17(1):81-86.

Boyd

Bruns

. Quality specifications for glucose meters: assessment by simulation modeling of errors in insulin dose. Clin Chem. 2001;47(2):209-214.

10.

Richtlinie 98/79/EG Des Europäischen Parlaments Und Des Rates Vom 27. October 1998. Über In-Vitro-Diagnostika. 1998

11.

International Organization for Standardization. In vitro diagnostic test systems—requirements for blood-glucose monitoring systems for self-testing in managing diabetes mellitus. EN ISO 15197:2003 E.

12.

13.

Food and Drug Administration. Self-monitoring blood glucose test systems for over-the-counter use—draft guidance for industry and food and drug administration staff. Food and Drug Administration; 2014. Available at: http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM380327.pdf. Accessed January 20, 2015.

14.

Baumstark

Pleus

Schmid

Link

Haug

Freckmann

. Lot-to-lot variability of test strips and accuracy assessment of systems for self-monitoring of blood glucose according to ISO 15197. J Diabetes Sci Technol. 2012;6(5):1076-1086.

15.

Freckmann

Baumstark

Zschornack

. Evaluation of the system accuracy of 34 blood glucose monitoring systems according to EN ISO 15197. Poster presented at: ADA; June 8-12, 2012; Philadelphia, PA.

16.

Freckmann

Baumstark

Schmid

Pleus

Link

Haug

. Evaluation of 12 blood glucose monitoring systems for self-testing: system accuracy and measurement reproducibility. Diabetes Technol Ther. 2014;16(2):113-122.

17.

Hasslacher

Kulozik

Platten

. Analytical performance of glucose monitoring systems at different blood glucose ranges and analysis of outliers in a clinical setting. J Diabetes Sci Technol. 2014;8(3):466-472.

18.

Kuo

Hsu

Wang

. Accuracy and precision evaluation of seven self-monitoring blood glucose systems. Diabetes Technol Ther. 2011;13(5):596-600.

19.

Tack

Pohlmeier

Behnke

. Accuracy evaluation of five blood glucose monitoring systems obtained from the pharmacy: a European multicenter study with 453 subjects. Diabetes Technol Ther. 2012;14(4):1-8.

20.

Kristensen

Monsen

Skeie

Sandberg

. Standardized evaluation of nine instruments for self-monitoring of blood glucose. Diabetes Technol Ther. 2008;10(6):467-477.

21.

Essack

Hoffman

Rensburg

Van Wyk

Meyer

Erasmus

. A comparison of five glucometers in South Africa. JEMDSA. 2009;14(2):102-105.

22.

Zueger

Schuler

Stettler

Diem

Christ

. Assessment of three frequently used blood glucose monitoring devices in clinical routine. Swiss Med Wkly. 2012;142:w13631.

23.

Brazg

Klaff

Parkin

. Performance variability of seven commonly used self-monitoring of blood glucose systems: clinical considerations for patients and providers. J Diabetes Sci Technol. 2013;7(1):144-152.

24.

Halldorsdottir

Warchal-Windham

Wallace

Pardo

Parkes

Simmons

. Accuracy evaluation of five blood glucose monitoring systems: the North American Comparator Trial. J Diabetes Sci Technol. 2013;7(5):1294-1304.

25.

Klaff

Brazg

Hughes

. Accuracy evaluation of contour next compared with five blood glucose monitoring systems across a wide range of blood glucose concentrations occurring in a clinical research setting. Diabetes Technol Ther. 2015;17:8-15.

26.

DeSalvo

Shanmugham

Wilson

Buckingham

. Accuracy evaluation of blood glucose monitoring systems in children on overnight closed-loop control. J Diabetes Sci Technol. 2014;8(5):969-973.

27.

Thorpe

. Assessing the quality of publications evaluating the accuracy of blood glucose monitoring systems. Diabetes Technol Ther. 2013;15(3):253-259.

28.

International Organization for Standardization. In vitro diagnostic medical devices—measurement of quantities in biological samples—metrological traceability of values assigned to calibrators and control materials. EN ISO 17511:2003 (D).

29.

Twomey

. Plasma glucose measurement with the Yellow Springs glucose 2300 STAT and the Olympus AU640. J Clin Pathol. 2004;57(7):752-754.

30.

Genter

Ipp

. Accuracy of plasma glucose measurements in the hypoglycemic range. Diabetes Care. 1994;17(6):595-598.

31.

Link

Schmid

Pleus

Baumstark

Rittmeyer

Haug

Freckmann

. System accuracy evaluation of four systems for self-monitoring of blood glucose following ISO 15197 using a glucose oxidase and a hexokinase based comparison method. J Diabetes Sci Technol. 2015; [accepted for publication].

32.

Freckmann

Link

Baumstark

Schmid

Pleus

Haug

. System accuracy evaluation of 10 SMBG systems with 3 lots each following ISO 15197:2013 against 2 different comparison methods. Paper presented at: EASD; September 15-19, 2014; Vienna, Austria.

33.

American Diabetes Association. Self-monitoring of blood glucose. Diabetes Care. 1996;19(suppl 1):S62-S66.

34.

Andreis

Kullmer

Appel

. Application of the reference method isotope dilution gas chromatography mass spectrometry (ID/GC/MS) to establish metrological traceability for calibration and control of blood glucose test systems. J Diabetes Sci Technol. 2014;8(3):508-515.

35.

Hagvik

. Glucose measurement: time for a gold standard. J Diabetes Sci Technol. 2007;1(2):169-172.

36.

Joint Committee for Traceability in Laboratory Medicine. Available at: http://www.bipm.org/jctlm/viewResults.do?type=isRMP&searchString=Glucose&searchStringIUPAC=&searchStringMixed=&analyteCategory=&matrixCategory=&uniqueNominationNumber=&sortBy=Analyte_Name&status=P&id=C3RMMP19*1*10&id=NRMeth+4*2*10&x=22&y=9. Accessed January 20, 2015.

37.

Food and Drug Administration. Blood Glucose Monitoring Test Systems for Prescription Point-of-Care Use—Draft Guidance for Industry and Food and Drug Administration Staff. Food and Drug Administration; 2014. Available at: http://www.fda.gov/downloads/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/UCM380325.pdf. Accessed January 20, 2015.

38.

AdvaMedDx. Available at: http://advameddx.org/download/files/sections/Policy/Regulation/Comments%20to%20FDA%20on%20Blood%20Glucose%20Monitoring%20Test%20Systems%20for%20Prescription%20Point-of-Care%20Use.pdf. Accessed January 15, 2015.

39.

Endocrine Society. Available at: https://www.endocrine.org/~/media/endosociety/Files/Advocacy%20and%20Outreach/Society%20Letters/FDA%20BGM%20Guidance%20Final.pdf. Accessed January 15, 2015.

40.

American Association of Diabetes Educators. AADE analysis on draft guidance for blood glucose monitoring systems. http://www.diabeteseducator.org/export/sites/aade/_resources/Advocacy/FDA_BGM_GUIDANCE_COMMENTS_ANALYSIS_2014.pdf. Accessed January 15, 2015.

41.

Boulton

Del

. Regulation of medical devices used in diabetology in Europe: time for reform? Diabetologia. 2012;55(8):2295-2297.

42.

Heinemann

Freckmann

Koschinsky

. Considerations for an institution for evaluation of diabetes technology devices to improve their quality in the European Union. J Diabetes Sci Technol. 2013;7(2):542-547.

43.

Freckmann

Baumstark

Jendrike

. System accuracy evaluation of 27 blood glucose monitoring systems according to DIN EN ISO 15197. Diabetes Technol Ther. 2010;12(3):221-231.

44.

Hasslacher

Kulozik

Platten

. Accuracy of self monitoring blood glucose systems in a clinical setting: application of new planned ISO-standards. Clin Lab. 2013;59(7-8):727-733.

Analytical Performance Requirements for Systems for Self-Monitoring of Blood Glucose With Focus on System Accuracy

Abstract

Keywords

Evaluation of the Analytical Performance of SMBG Systems According to ISO 15197

Measurement Precision

Measurement Repeatability

Intermediate Measurement Precision

Influence Quantities

System Accuracy

Accuracy Criteria

Evaluation Procedure

Reference Measurement Procedure

User Performance Evaluation Following ISO 15197

FDA Draft Guidance Document

System Accuracy Evaluation

Accuracy Criteria

Test Procedure to Obtain System Accuracy

Method Comparison/User Evaluation

Test Procedure to Obtain System Accuracy

SMBG System Accuracy—Status Quo

Premarket Requirements

Postmarketing Performance Data

Conclusion

Footnotes

Acknowledgements

Abbreviations

Declaration of Conflicting Interests

Funding

References