Evaluation of Four HbA1c Point-of-Care Devices Using International Quality Targets: Are They Fit for the Purpose?

Abstract

Background:

Point-of-care (POC) testing is becoming increasingly valuable in health care delivery, and it is important that the devices used meet the same quality criteria as main laboratory analyzers. While external quality assessment (EQA) provides a great tool for assessing quality, many POC devices are not enrolled in these schemes and standard laboratory evaluations are needed to assess performance.

Methods:

The Clinical and Laboratory Standards Institute (CLSI) protocols EP-5 and EP-9 were applied to investigate imprecision, accuracy and bias. We assessed bias using the mean of 4 certified secondary reference measurement procedures (SRMPs).

Results:

The Afinion2™ and the Quo-Lab had CVs of ≤1.7 and ≤2.4% respectively in IFCC SI units (≤1.2 and ≤1.7% NGSP) and a bias ≤2 mmol/mol (≤0.2% NGSP) at 48 and 75 mmol/mol (6.5 and 9.0% NGSP). Sigma for the Afinion2 was 5.8 and for the Quo-Lab 4.0. Both methods passed the NGSP criteria with 2 instruments when compared with 4 individual SRMPs. The HbA1c 501 had a CV of 3.4% and 2.7% in IFCC SI units (2.1% and 1.7% NGSP) and a bias ≤2.4 mmol/mol (≤0.2% NGSP) and passed the NGSP criteria with 2 instruments compared with 4 individual SRMPs except for instrument 2 compared with the Tosoh G8. Sigma was 2.1. The A1Care had a sigma of 1.4 and failed all criteria mainly due to a high CV (6.2% and 4.1% in IFCC SI units [4.1% and 2.9% NGSP] at 48 and 75 mmol/mol [6.5 and 9.0% NGSP]).

Conclusions:

The analytical performance was excellent for the Afinion2 and the Quo-Lab, acceptable for the HbA1c 501 and unacceptable for the A1Care according to different used criteria, demonstrating that whilst performance is improving there are still areas for considerable improvement.

Keywords

diabetes HbA1c Hb-variants point of care sigma metrics

Point-of-care (POC) testing provides rapid test results facilitating treatment decisions to be made in a single visit to the doctor’s office and thus, potentially improving the patient’s experience and outcomes.^1-3 Currently there are in excess of 30 HbA1c POC instruments on the market. Many of these systems will have achieved certification that they are fit for purpose as a POC test from whichever regulatory body controls distribution of the instrument in a particular country or region (eg, CE [conforms with relevant EU directives regarding health and safety or environmental protection], US Food and Drug Administration [FDA], NGSP). However the evaluation of these instruments, to meet the relevant certification criteria, is generally performed by the manufacturers and under ideal conditions, which does not reflect real-life performance in the field.⁴ External quality assessments (EQA) with accuracy based value assignment is the ideal way to investigate the real analytical performance of a method in the hands of the end/intended users. Unfortunately many users of POC instruments do not participate in EQA schemes for various reasons such as cost, lack of legislative requirement and waiver programs that allow less stringent monitoring approaches, and therefore the real analytical performance of these instruments is not clearly defined.⁵ To further compound this issue some devices are designed in a such a way that internal quality control (IQC) is hard to undertake routinely and is a particular issue with single use test kits, giving even less information on the performance of these devices.

In 2015 the IFCC Task Force for the implementation of the standardization of HbA1c published performance criteria for HbA1c methods based on sigma metrics.⁶ The criteria focus on CV and bias at a concentration of 50 mmol/mol (6.7% NGSP) as this is “in between two important medical decision points” (48 mmol/mol [6.5% NGSP] and 53 mmol/mol [7.0% NGSP]). More detailed evaluations may also focus on a wider range of HbA1c values. The NGSP manufacturer certification criteria focuses on relative differences between the test instrument compared with a single NGSP SRMP and not directly on imprecision.⁷

The European Reference Laboratory for Glycohemglobin (ERL) provides manufacturers with reference materials and also assists in performing evaluation studies for medical devices to provide an independent review of the analytical performance of an instrument. The ERL works with manufacturers to assess quality and advise on improvements at all stages of the development process and preferably before a new device comes to the market. The ERL can, for example, provide the manufacturer with reference materials or patient samples with values assigned by a reference method, to aid in calibration and performance analysis.

The ERL has undertaken many evaluations of different HbA1c methods over the years and have developed additional performance criteria to enhance the evaluation of HbA1c instruments.⁸

The aim of this study was to evaluate 4 POC instruments according to the CLSI protocols and determine how the instruments perform when different criteria are applied using 4 certified IFCC and NGSP secondary reference measurement procedures (SRMPs).^9,10 In addition, we investigated the presence of potential interference from common Hb-variants on the instruments that were capable of analyzing frozen material and if there was a statistical difference between 2 instruments of the same manufacturer.

Methods

Prestudy

Before starting with a full evaluation a small prestudy was undertaken to gain an overview of the analytical performance of each POC instrument. The prestudy consisted of analyzing 12 fresh patient samples in duplicate on one day and calculation of a %CV from the duplicate values were assigned using 4 IFCC and/or NGSP certified SRMPs:

Roche Tina-quant Gen.3 HbA1c on Cobas c513, immunoassay, IFCC and NGSP SRMPs (Roche Diagnostics, Rotkreuz, Switzerland)

Premier Hb9210, affinity chromatography HPLC, IFCC and NGSP SRMPs Biotech, Bray, Ireland)

Tosoh G8, cation-exchange HPLC, IFCC SRMP (Tosoh Bioscience, Tessenderlo, Belgium)

Abbott Enzymatic method on Architect c4000, IFCC and NGSP SRMPs (Abbott Diagnostics, Lake Forest, IL, USA)

The results of the prestudy were provided to the individual manufacturers, who then decided whether or not to proceed to a full evaluation (EP-5, EP-9, Hb-variant interference, etc). The manufacturer can then either choose to use the data internally or to sign a contract with the ERL to publish the results. The decision to publish or not must be made prior to starting a full evaluation. The following 4 manufacturers gave permission to use the results for publication:

The Afinion2™ (Abbott, Oslo, Norway), which is based on boronate affinity separation, with results available in 3 min. This instrument is the successor of the Alere Afinion AS100 Analyzer. The difference between the AS100 Analyzer and the Afinion2 is that the Afinion2 instrument has a built-in connectivity unit and a new lid design. The user interface, the assays and the test procedure is the same as for the AS100 Analyzer.

The Quo-Lab (EKF Diagnostics PLC, Cardiff, UK), is based on boronate affinity separation and the use of fluorescence quenching with results available in under 4 min.

The HbA1c 501 (HemoCue Diagnostics, Ängelholm, Sweden), is based on boronate affinity separation with results available in 5 min.

The A1Care (i-SENSE, Seoul, Korea), which is based on enzymatic determination of HbA1c with results available in 5 min.

Full Evaluation

Imprecision

The CLSI EP-5 protocol was used to investigate assay imprecision.¹¹ Aliquots were made from two patient samples and stored at minus 80 °C degrees until analysis (duplicate measurements twice a day for 20 days). The Afinion2 cannot utilize hemolyzed material so two fresh patient samples with an HbA1c value of approximately 48 mmol/mol and 75 mmol/mol were stored in the refrigerator for 14 days. Every day the samples were mixed and 200ul was taken from the original tube and put into a small cup for analysis twice a day in duplicate for 14 days. CVs were also calculated on the basis of the duplicates of the fresh patient samples in the EP-9 protocol.

Accuracy and Method Comparison

The CLSI EP-9 protocol was performed on two separate instruments from each manufacturer—the aim was to assess the quality of the instrument as a whole rather than just using different reagent lot numbers on one instrument. While reagent variability between lot numbers is a key factor in analytical performance, using two instruments will mimic the ‘between laboratory’ performance of these analyzers.

The CLSI EP-9 protocol was performed with 40 fresh patient samples with 2 instruments and the data were used to investigate the bias between the POC instruments and the 4 SRMPs as used in the prestudy (n = 40, 8 samples per day for 5 days, duplicate measurements).¹²

The data were also used to calculate performance against the NGSP certification criteria.⁷ To assess overall calibration and bias independently of the chosen SRMP, the results of the POC instruments in the EP-9 procedure were compared with the mean of the 4 SRMPs.

Medical decision point (MDP) analysis was performed at an HbA1c value of 48 and 75 mmol/mol (6.5% and 9.0% NGSP). When 2 methods are statistically identical, the 95% CI for each y MDP includes the corresponding x MDP. For example, 48 mmol/mol (6.5% NGSP), the diagnostic cut-off value for the diagnosis of diabetes falls within 46.0 mmol/mol (6.4% NGSP) to 49.5 mmol/mol (6.7% NGSP), the 95% CI around the calculated y, so both methods are statistically identical.

The bias at 48 mmol/mol and the CV at the same concentration in EP-5 were used to calculate sigma.

Analytical Performance Criteria

Sigma Metrics

Total allowable error (TAE) for HbA1c has been set by the IFCC Task Force on Implementation of HbA_1c standardization as a default of 5 mmol/mol (0.46% NGSP) at an HbA1c level of 50 mmol/mol (6.7% NGSP) which corresponds with a relative TAE of 10% ([5/50]*100%) in SI units (6.9% NGSP units ([0.46/6.7]*100%) with risk levels of 2σ for routine laboratories and 4σ for laboratories performing clinical trials.⁶

NGSP Manufacturer Certification Criteria

Thirty seven of 40 results need to be within 6% (relative) of an individual NGSP SRMP to pass certification.⁷

Enhanced Precision and Bias Criteria for a Full Evaluation Done at ERL

CV in EP-5 at 48 mmol/mol and 75 mmol/mol: ≤3.0% in SI units (≤2.0% in NGSP units)

Bias compared with the mean of at least 3 SRMPs at 48 and 75 mmol/mol (6.5% and 9.0% NGSP): ≤2.0 mmol/mol (≤0.2% NGSP)⁸

Statistical Significant Difference Between Instruments

The EP-9 results of both instruments have been used to test if the slope, intercept and MDPs were different (outside the 95% CI) between the 2 instruments. In other words, two instruments can be considered statistically identical if:

The slope is 1.00 (within 95% confidence)

The intercept is 0.00 (within 95% confidence)

The predicted Y MDPs are equal to the X MDPs (within 95% confidence)

Statistical Calculations

Calculations were performed using Microsoft® Excel 2010 (Microsoft Corporation). Statistical analyses were performed using Analyse-It® (Analyse-It Software) and EP Evaluator Release 9 (Data Innovations LLC).¹³

For the duplicates in the EP-9 protocol, CV was calculated with the following formula:

CVa = \frac{\sqrt{\frac{\sum {(Δ)}^{2}}{n}}}{\bar{x} \sqrt{2}} \times 100 %

where CV_a is the analytical CV, ∆ is the difference between duplicates, n is the number of duplicates, and $\bar{x}$ is the mean of the duplicates.

Sigma was calculated using the formula: σ = (TAE – Bias)/CV were the TAE was 10%.

Interference of Hb-variants

Twenty nonvariant samples (HbAA), 10 HbAS, 10 HbAC, 10 HbAD, 10 HbAE, 10 HbF, and 9 elevated A2 samples were analyzed on 3 different days. Due to a lack of cartridges with the same lot number for the Quo-Lab, we analyzed 20 HbAA, 9 HbAS, 9 HbAC, 9 HbAD, 9 HbAE, 5 HbF and 4 elevated A2 samples. Both the normal and Hb-variant samples were stored at -80°C until analysis. Specific variants were identified using cation-exchange HPLC (Menarini HA8180V, Diabetes Mode) and confirmed with capillary electrophoresis (Sebia Capillarys 2 Flex Piercing, Hemoglobin program). Percentage HbF (3.2, 4.6, 6.2, 6.9, 8.6, 11.0, 13.0, 16.5, 18.0 and 34.0%) was determined using the Sebia Capillarys 2 Flex Piercing Hemoglobin program. HbA1c values for samples with Hb variants were assigned using IFCC calibrated boronate affinity HPLC (Premier Hb9210). For samples with increased HbF, HbA1c values were assigned using IFCC calibrated cation-exchange HPLC (Menarini HA8180V, Diabetes Mode).

As a guide, one could say that the investigated Hb variant can be considered as not causing a clinically relevant interference if the results of the Hb variant fall within a defined scatter line of ±10% (IFCC units) of the regression line derived from the comparison of the test instrument and the IFCC assigned values of the nonvariant samples (HbAA). While this is a guide rather than an absolute, by graphing this relationship it is a simple way to identify patterns of interference.¹⁴

Results

Imprecision Studies

The imprecision results of the EP-5 protocol and calculated from the duplicates of the samples in EP-9 are detailed in Table 1. The Afinion2 and the Quo-Lab passed the criteria of having a CV ≤ 3% (≤ at 2.0% NGSP) at 48 and 75 mmol/mol (6.5% and 9.0% NGSP) and calculated from the duplicates in EP-9. The CV of the HbA1c 501 at 48 mmol/mol (6.5% NGSP) was just above the criteria (3.4% IFCC SI units [2.1% NGSP]) but passed it at an HbA1c value of 75 mmol/mol (9.0% NGSP) and from the duplicates in EP-9. The A1Care failed the criteria at both HbA1c values in EP-5 and calculated from the duplicates in EP-9.

Table 1.

Imprecision Results Based on EP-5 and on the Duplicates in EP-9.

	CV (%) SI units	CV(%) NGSP units
Afinion2	1.7 (44 mmol/mol)^b	1.2 (6.2%)^b
	1.1 (74 mmol/mol)^b	0.9 (9.0%)^b
Instrument 1^a	1.7	0.9
Instrument 2^a	1.7	0.9
A1Care	6.2 (47 mmol/mol)	4.1 (6.4%)
	4.1 (71 mmol/mol)	2.9 (8.7%)
Instrument 1^a	4.9	3.5
Instrument 2^a	5.0	3.5
HbA1c 501	3.4 (46 mmol/mol)	2.1 (6.3%)
	2.7 (72 mmol/mol)	1.7 (8.7%)
Instrument 1^a	2.1	1.5
Instrument 2^a	2.5	1.7
Quo-Lab	2.4 (46 mmol/mol)	1.6 (6.4%)
	2.4 (71 mmol/mol)	1.8 (8.6%)
Instrument 1^a	1.5	1.1
Instrument 2^a	1.9	1.3

Based on duplicates in EP-9.

Based on 14 days instead of 20.

Method Comparisons

Figure 1A-D shows the EP-9 results of the 4 POC instruments compared to the mean of the 4 SRMPs with 2 different instruments. Table 2 shows the NGSP certification pass/fail with respect to the individual SRMPs using the results of the EP-9 protocol. The bias at 48 (6.5% NGSP) and 75 mmol/mol (9.0% NGSP) of all 4 POC instruments was ≤2 mmol/mol (≤0.2% NGSP) compared to the mean of the 4 SRMPs except for HbA1c 501 instrument 2 at 75 mmol/mol (9.0% NGSP) (bias was 2.4 mmol/mol (0.2% NGSP) (Table 3). However, instrument 2 of the A1Care gave lower results in the low area and higher results in the high area resulting in a mean bias of <2 mmol/mol (<0.2% NGSP) (Figure 1D).

Figure 1.

HbA_1c results in SI units for two different instruments from (A) Afinion2, (B) Quo-Lab, (C) HbA1c 501, and (D) A1Care POC instruments compared to the mean HbA1c results from 4 SRMPs.

Table 2.

EP-9 Results in NGSP Units and Calculations of NGSP Certification Criteria.

Deming regression lines	Instrument 1	Bias	SEE	Out ±6% SRM	NGSP criteria	Instrument 2	Bias	SEE	Out ±6% SRM	NGSP criteria
Afinion2 (Y) vs Premier (X)	Y = 0.94X + 0.43	0.01	0.09	0	Pass	Y = 0.94X + 0.43	−0.01	0.09	0	Pass
vs Abbott (X)	Y = 0.95X + 0.38	0.01	0.15	1	Pass	Y = 0.95X + 0.35	0.01	0.14	1	Pass
vs Tina-quant (X)	Y = 0.96X + 0.29	0.03	0.16	1	Pass	Y = 0.98X + 0.22	0.04	0.14	1	Pass
vs Tosoh G8 (X)	Y = 0.95X + 0.27	−0.06	0.18	0	Pass	Y = 0.95X + 0.28	−0.05	0.19	1	Pass
A1Care (Y) vs Premier (X)	Y = 0.99X + 0.04	−0.07	0.34	9	Fail	Y = 1.07X – 0.56	−0.09	0.39	10	Fail
vs Abbott (X)	Y = 0.99X + 0.05	−0.04	0.28	5	Fail	Y = 1.08X – 0.66	−0.08	0.33	7	Fail
vs Tina-quant (X)	Y = 1.00X – 0.00	−0.02	0.30	8	Fail	Y = 1.08X – 0.63	−0.06	0.31	7	Fail
vs Tosoh G8 (X)	Y = 0.99X – 0.06	−0.10	0.33	8	Fail	Y = 1.08X – 0.72	−0.13	0.34	9	Fail
HbA1c 501 (Y) vs Premier (X)	Y = 0.97X + 0.10	−0.13	0.14	0	Pass	Y = 0.96X + 0.13^a	−0.15	0.13	0	Pass
vs Abbott (X)	Y = 0.97X + 0.08	−0.11	0.21	1	Pass	Y = 0.96X + 0.14^a	−0.14	0.18	1	Pass
vs Tina-quant (X)	Y = 0.99X – 0.06	−0.08	0.22	2	Pass	Y = 0.98X + 0.05^a	−0.11	0.18	2	Pass
vs Tosoh G8 (X)	Y = 0.98X – 0.02	−0.17	0.23	1	Pass	Y = 0.96X + 0.11^a	−0.18	0.22	4	Fail
Quo-Lab(Y) vs Premier (X)	Y = 0.95X + 0.31	−0.04	0.14	0	Pass	Y = 1.00X – 0.01	−0.01	0.15	0	Pass
vs Abbott (X)	Y = 0.97X + 0.24	−0.01	0.15	1	Pass	Y = 1.01X – 0.04	0.02	0.16	0	Pass
vs Tina-quant (X)	Y = 0.98X + 0.14	0.01	0.15	1	Pass	Y = 1.02X – 0.13	0.04	0.16	0	Pass
vs Tosoh G8 (X)	Y = 0.97X + 0.15	−0.08	0.20	3	Pass	Y = 1.01X – 0.14	−0.05	0.22	2	Pass

Shaded row means same measurement principle as investigated POC method.

Based on 35 samples instead of 40.

Table 3.

Medical Decision Point Analysis at 48 mmol/mol and 75 mmol/mol Compared to the Mean of the 4 SRMPs.

	Instrument 1		Instrument 2
Afinion2	47.9 [47.6, 48.2]	73.7 [73.2, 74.2]*	48.0 [47.7, 48.4]	73.8 [73.3, 74.3]*
Quo-Lab	47.8 [47.4, 48.2]	74.1 [73.6, 74.7]*	47.9 [47.5, 48.4]	75.4 [74.8, 76.0]
HbA1c 501	46.6 [46.1, 47.1]*	73.3 [72.5, 74.0]*	46.5 [46.0, 46.9]*	72.6 [71.9, 73.4]*
A1Care	47.5 [46.5, 48.4]	74.6 [73.2, 76.0]	46.4 [45.5, 47.3]*	76.3 [75.0, 77.7]

95% CI in brackets.

Statistically significant difference.

Statistical Significant Difference Between Instruments

There was a statistically significant difference in IFCC SI units between the 2 instruments for the Quo-Lab (Y = 1.04X [95% CI: 1.02, 1.06] −1.9 [95% CI: −3.0, −0.8]) and the A1Care (Y = 1.10X [95% CI: 1.05, 1.16] −5.8 [95% CI: −8.9, −2.7]) and no significant difference between the 2 instruments for the Afinion2 (Y = 1.00X [95% CI: 0.98, 1.01] + 0.3 [95% CI: −0.5, 1.1]) and the HbA1c 501 (Y = 1.00X [95% CI: 0.98, 1.03] − 0.3 [95% CI: −1.8, 1.3]).

Interference of Hb-variants

The 3 investigated instruments showed no consistent clinically significant interference from HbAS, HbAC, HbAD, HbAE, elevated A2 (Figures 2A-2C). There were some outliers, in particular for the A1Care (Figure 2C) however these may have been due to the high CV observed with this instrument. Table 4 shows the mean relative difference for the different Hb-variants except for HbF as the mean relative difference cannot be calculated since the interference is level-dependent (%HbF). In each case it is clear that high levels of HbF (>34% HbF ) appear to interfere with measurement of HbA1c however exact thresholds cannot be determined from this dataset. Results were corrected for bias based on the bias found in the nonvariant samples (HbAA) when calculating the mean relative difference (Table 4).

Figure 2.

Interference of common Hb-variants of Quo-Lab (A), HbA1c 501 (B), and A1Care (C).

Table 4.

Mean Relative Difference (%) of the Common Hb-variants (n = ±10 per Variant) Compared to the Assigned Value.

	HbAS	HbAC	HbAD	HbAE	Elevated A2
Quo-Lab	−4.3	2.2	−2.5	3.8	0.9
HbA1c 501	−7.8	6.3	−6.7	4.3	4.7
A1Care	4.2	2.7	1.9	0.5	−0.6

Analytical Performance Criteria

Figure 3 shows the combined results of EP-5 and EP-9 in sigma metrics at 48 mmol/mol (6.5% NGSP) for instrument 1. Sigma for the Afinion2 was 5.8, Quo-Lab 4.0, HbA1c 501 2.1, and A1Care 1.4.

Figure 3.

Sigma metrics results for the Afinion2 (A ), Quo-Test (B), HbA1c 501 (C), and A1Care (D) based on the CV in EP-5 at 48 mmol/mol and bias at 48 mmol/mol compared to the mean of 4 SRMPs.

Table 5 summarizes the results for each of the different analytical performance criteria. The Afinon2 and the Quo-Lab passed all criteria. The HbA1c 501 passed most of the criteria but not all and the A1Care failed most of the criteria.

Table 5.

Overview of Passing/Failing of Different Criteria.

	Afinion2		Quo-Lab		HbA1c 501		A1Care
Criteria	Instr. 1	Instr. 2	Instr. 1	Instr. 2	Instr. 1	Instr. 2	Instr. 1	Instr. 2
IFCC Task Force on HbA1c Standardization≥ 2 sigma at 50 mmol/mol	Pass	NT	Pass	NT	Pass	NT	Fail	NT
NGSP manufacturer
Premier Hb9210	Pass	Pass	Pass	Pass	Pass	Pass	Fail	Fail
Roche TQ	Pass	Pass	Pass	Pass	Pass	Pass	Fail	Fail
Tosoh G8	Pass	Pass	Pass	Pass	Pass	Fail	Fail	Fail
Abbott Enzymatic	Pass	Pass	Pass	Pass	Pass	Pass	Fail	Fail
Enhanced precision and bias criteria (see methods section for details)
CV ≤ 3% at 48 mmol/mol	Pass	NT	Pass	NT	Fail	NT	Fail	NT
CV ≤ 3% at 75 mmol/mol	Pass	NT	Pass	NT	Pass	NT	Fail	NT
Bias ≤ 2 mmol/mol at 48 mmol/mol	Pass	Pass	Pass	Pass	Pass	Pass	Pass	Pass
Bias ≤ 2 mmol/mol at 75 mmol/mol	Pass	Pass	Pass	Pass	Pass	Fail	Pass	Pass
No Interference of Hb-variants	NT	NA	Pass	NA	Pass	NA	Pass	NA

NA, not applicable; NT, not tested.

Discussion

In 2009 we investigated 8 different HbA1c POC instruments and four years later we investigated again 7 different POC instruments.^15,16 The analytical performance of HbA1c POC instruments investigated in 2013 improved considerably compared to the analytical performance of HbA1c POC instruments in 2009. The poor results of the first study along with greater collaboration with the IFCC and NGSP and sequential tightening of the NGSP certification and CAP EQA criteria may have acted as a “wakeup call” for some manufacturers to improve their methods and drive forward quality improvement. The impact of having poor quality performance highlighted in a peer-reviewed publication can have significant repercussions for manufacturers and may lead to reluctance or resistance to engaging in similar detailed quality studies. However these independent investigations are necessary to provide health professionals and their patients with confidence in the analytical equipment they are using. Some manufacturers may choose to withdraw their method from the market if they failed to improve the analytical performance and some improved their methods by working together with the ERL. The publication in 2009 showed that some methods (Quo-Test, Quo-Lab, and InnovaStar) had problems with IFCC frozen reference material.¹⁶ This frozen reference material was not commutable with these methods. After the publication the manufacturer of the Quo-test and Quo-Lab contacted the ERL and currently every week samples with assigned values are send to the manufacturer which are used by the manufacturer to calibrate or check the new produced cartridges. In this study we evaluated the Quo-Lab again and the results showed that the Quo-Lab met all criteria and that there is hardly any bias between the Quo-Lab and the mean of the 4 SRMPs (Table 5). Also the Quo-Test (not evaluated in this study) which is from the same manufacturer, has no bias anymore compared to the mean of 3 SRMPs.¹⁷ The Afinion2 met all criteria and precision has improved compared to previous studies. In the past the CVs obtained with controls were not in line with the CVs calculated from the duplicates in EP-9.^15,16 The controls gave lower CVs than the CVs calculated from patient blood. For this reason we used in this study fresh patient blood for 14 days. The CVs in this study were all ≤1.7% in SI units (≤1.2 NGSP) (Table 1) while the CV in the 2014 study calculated from the duplicates in EP-9 was 3.0% in SI units (2.0% NGSP), demonstrating an improvement in quality since the previous study.¹⁶ As frozen or hemolyzed material is not commutable with the Affinion2 system, samples with values assigned by an SRMP are sent to the company on a regular basis for calibration and/or check of the cartridges. This is likely to account for both methods (Afinion2 and Quo-Lab) showing excellent results within the method comparisons (Figures 1A and 1B).

The HbA1c 501 had a borderline analytical performance. The CV at an HbA1c value of 48 mmol/mol (6.5% NGSP) was high (3.4% in SI units, 2.1% in NGSP units) and the calibration of the instrument needs a small adjustment (+1.7 mmol/mol higher results (+0.2% NGSP). However, it passed the sigma metrics criteria for routine laboratory (σ was 2.1).

The A1Care did not meet any criteria mainly due to the high imprecision (CV was 6.2% [4.1% NGSP] at 47 mmol/mol [6.4% NGSP] and 4.1% [2.9% NGSP] at 71 mmol/mol [8.7% NGSP]). The instrument also had a high error rate (error rate of 6.3%). The A1Care gave numerous incorrect results which were too frequent to be considered outliers. The A1Care also failed the NGSP criteria compared with the 4 individual SRMPs for both instruments and there was a significant statistical difference between the 2 instruments, especially in the low area. This was quite a surprise as the A1Care has a NGSP manufacturer certificate obtained in January 2018.¹⁸ The outliers seen in our study may be as a result of a technical problem within the instruments or damage incurred during transport. A closer inspection of the results in the EP-9 study showed that one of the two results (duplicate measurement) was consistently in line with the reference method and one result was a complete outlier. This along with the high percentage of errors may be a sign of a technical problem with the instrument and serves to highlight the need to assess all aspects of the analytical process where possible. As the same reagent lot numbers were used on each instrument this data serves to demonstrate that it is not only reagent lot variability that can be a major contributing factor to poor performance and a holistic approach to assessing quality is needed. The poor precision of the method may mask errors in the bias/calibration as can be seen in the MPD analysis. The 95% CI around the MDPs of 48 and 75 mmol/mol (6.5% and 9.0% NGSP) was so large that there was no statistical significant difference between the mean of the 4 SRMPs and instrument 1 at 48 and 75 mmol/mol (6.5% and 9.0% NGSP) and instrument 2 at 75 mmol/mol (9.0% NGSP). This means that large differences between methods are more likely to be statistically significant when methods are precise, and conversely, the less precise a method is, the easier it is to be not statistically different to a reference method. Therefore it is necessary to look closely at both imprecision and bias when evaluating methods.

The 3 investigated POC instruments did not show clinically significant interference of common Hb-variants and ß-Thalassemia (Table 4). Samples with high percentages of HbF can only be accurately measured with methods that can separate HbF from total hemoglobin (generally cation exchange HPLC and capillary electrophoresis based methods) which is not the case with immunoassay, enzymatic and affinity based methods which may result in a falsely low HbA1c result.¹⁹ This also seems to apply to the 3 investigated POC instruments at higher levels of HbF. The Afinion2 was not tested for interference of common Hb-variants as this method cannot utilize frozen material. However, 2 studies in the past showed that there is no interference of HbAS, HbAC, HbAD and HbAE.^20,21 EQA data reveals the real analytical performance of HbA1c POC instruments as results are produced by end/intended users using different lot numbers and different instruments. A possible reason why the Afinion and the DCA Vantage have such a big market share in HbA1c POC could be because the manufacturers of these POC instruments can prove with EQA data that the analytical performance of these instruments is equal to laboratory based HbA1c methods and that it can be used for the diagnosis of diabetes.²² Participation in EQA schemes should be mandatory for users of POC instruments to assure quality.

Evaluation done at ERL is more independent than a certification done at the manufacturers site but still has its limitations. The limitations of our study was that this was a single center with one reagent lot number and an experienced technician. However, the novelty of the study is that two instruments were used from each manufacturer, with the same reagent lot numbers, effectively demonstrating how the instruments can perform between laboratories.

Conclusion

The analytical performance of POC instruments for HbA1c can be seen to be continually improving. However, there are still some instrument that do not perform to the desired level when different quality targets are applied. To improve access to health care and increase the efficacy of the service provided to patients, POC is likely to play an increasing role in health care delivery in the future. For this reason it is important to ensure that instruments meet the same quality criteria as main laboratory analyzers and that there is a continual drive for quality improvement. While robust evaluations such as those described in this article are invaluable in assisting decision making when choosing a device, enrolment in EQA schemes is also an integral part of monitoring the performance of instruments in their relevant clinical settings.

Footnotes

Acknowledgements

The authors would like to thank Agnes den Ouden for her contribution to technical and analytical aspects of this study.

Abbreviations

CLSI, Clinical and Laboratory Standards Institute; EQA, external quality assessment; ERL, European Reference Laboratory for Glycohemoglobin; FDA, Food and Drug Administration; IFCC, International Federation of Clinical Chemistry and Laboratory Medicine; IQA, internal quality control; MDP, medical decision point; NGSP, National Glycohemoglobin Standardization Program; POC, point of care; SRMP, secondary reference measurement procedures.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This work was funded by Abbott Diagnostics, EKF Diagnostics, HemoCue Diagnostics, and i-SENS Inc.

References

Schnell

Crocker

Weng

Impact of HbA1c testing at point of care on diabetes management. J Diabetes Sci Technol. 2017;3:611-617.

Motta

Shephard

MDS

Brink

Lawson

Rheeder

Point-of-care testing improves diabetes management in primary care clinic in South Africa. Prim Care Diabetes. 2017;3:248-253.

Patzer

Ardjomand

Gӧhring

et al . Implementation of HbA1c point of care testing in 3 German Medical Practices: impact on workflow and physician, staff, and patient satisfaction. J Diabetes Sci Technol. 2018;3:687-694.

NGSP website note CAP survey data. Available at: http://www.ngsp.org/CAP/CAP17c.pdf. Accessed April 2018.

Centers for Medicare & Medicaid Services. Available at: https://www.cms.gov/Regulations-and-Guidance/Legislation/CLIA/Certificate_of_-Waiver_Laboratory_Project.html. Accessed April 2018.

Weykamp

John

Gillery

et al . Investigation of 2 models to set and evaluate quality targets for Hb A1c: biological variation and sigma-metrics. Clin Chem. 2015;61:752-759.

NGSP manufacturer certification criteria. Available at: www.ngsp.org/critsumm.asp. Accessed April 2018.

Westgard. Available at: http://james.westgard.com/the_westgard_rules/2017/08/are-there-new-erna-lenters-westra-rules-for-hba1c.html. Accessed April 2018.

Approved laboratories of the IFCC network laboratories for HbA1c 2016. Available at: www.ifcchba1c.net/network/approved. Accessed April 2018.

10.

NGSP Network Laboratory Members. Available at: www.ngsp.org/network.asp. Accessed April 2018.

11.

Clinical and Laboratory Standards Institute. Evaluation of precision performance of clinical chemistry devices; approved guideline. NCCLS Document EP05-A. Wayne, PA: CLSI; 1992.

12.

Clinical and Laboratory Standards Institute. Method comparison and bias estimation using patient samples; approved guideline. CLSI Document EP09-A. Wayne, PA: CLSI; 1995.

13.

EP Evaluator Data Innovations. Available at: www.datainnovations.com/products/ep-evaluator. Accessed January 2017.

14.

Lenters-Westra

English

Evaluating new HbA1c methods for adoption by the IFCC and NGSP reference networks using international quality targets. Clin Chem Lab Med. 2017;55:1426-1434.

15.

Lenters-Westra

Slingerland

RJ.

Six of eight hemoglobin A1c point-of-care instruments do not meet the general accepted analytical performance criteria. Clin Chem. 2010;56:44-52.

16.

Lenters-Westra

Slingerland

RJ.

Three of seven hemoglobin A1c point-of-care instruments do not meet the generally accepted analytical performance criteria. Clin Chem. 2014;60:1062-1072.

17.

EKF Diagnostics. Available at: https://www.ekfdiagnostics.com/Quo-Test%20a1c%20Study%20reaffirms%20IFCC%20performance%20criteria. Accessed April 2018.

18.

NGSP. List of NGSP certified methods. Available at: http://www.ngsp.org/docs/methods.pdf. Accessed April 2018.

19.

Rohlfing

Conolly

England

et al . The effect of elevated fetal hemoglobin on hemoglobin A1c results: five common hemoglobin A1c methods compared with the IFCC reference method. Am J Pathol. 2008;129:811-814.

20.

Mongia

Little

Rohlfing

et al . Effects of hemoglobin C and S traits on fourteen commercial glycated hemoglobin assays. Amer J Clin Path. 2008;130:136-140.

21.

Little

Rohlfing

Hanson

et al . Effects of hemoglobin E and D traits on glycated hemoglobin (HbA1c) measurements by twenty-three methods. Clin Chem. 2008;54:1277-1282.

22.

Solvik

Roraas

Christensen

Sandberg

Diagnosing diabetes mellitus: performance of hemoglobin A1c point-of-care instruments in general practice offices. Clin Chem. 2013;59:1790-1801.