Abstract
Verification of new lot reagent’s suitability is necessary to ensure that results for patients’ samples are consistent before and after reagent lot changes. A typical procedure is to measure results of some patients’ samples along with quality control (QC) materials. In this study, the results of patients’ samples and QC materials in reagent lot changes were analysed. In addition, the opinion regarding QC target range adjustment along with reagent lot changes was proposed. Patients’ sample and QC material results of 360 reagent lot change events involving 61 analytes and eight instrument platforms were analysed. The between-lot differences for the patients’ samples (ΔP) and the QC materials (ΔQC) were tested by Mann–Whitney U tests. The size of the between-lot differences in the QC data was calculated as multiples of standard deviation (SD). The ΔP and ΔQC values only differed significantly in 7.8% of the reagent lot change events. This frequency was not affected by the assay principle or the QC material source. One SD was proposed for the cutoff for maintaining pre-existing target range after reagent lot change. While non-commutable QC material results were infrequent in the present study, our data confirmed that QC materials have limited usefulness when assessing new reagent lots. Also a 1 SD standard for establishing a new QC target range after reagent lot change event was proposed.
Introduction
It is essential that clinical laboratories achieve consistent and reliable test results. Tests performed using a modern automated clinical chemistry analyser involve the instrument, the calibrator and one or more reagents. Reagent lot changes occur relatively frequently because large amounts of reagents are used for short periods. Therefore, to ensure the consistency of the test results of a clinical laboratory, the results before and after reagent lot changes should be analysed at the statistical level. 1 For this reason, the inspection checklist of the College of American Pathologist (CAP) requires that when there is a reagent lot change, a certain number of patient specimens and 2–3 levels of quality control (QC) materials should be tested with the new and old reagent lots. 2 When differences in the results measured using new lots and old lots meet the acceptance criteria in each laboratory, the test results are considered as consistent irrespective of reagent lot changes. Although acceptance criteria vary from laboratory to laboratory, a new lot is generally considered to be acceptable when (i) the results of the patient specimens obtained with the new lot do not differ from those with the old lot 3 and (ii) the QC material values obtained with the new reagent lot fall within the predefined QC material target range of the old reagent lot. The validation process can be passed only when these both criteria are met.
However, there are some doubts about the usefulness and significance of the second criterion in lot-to-lot consistency evaluations because QC materials can show ‘non-commutability’, namely, the different numeric relationship between the test results with QC materials and those with patient specimens.1,4–13 This non-commutability of QC materials relates to a matrix effect that occurs during the production of QC materials and is due to the use of substances whose properties differ from those of similar substances in patient clinical specimens.6,14–16 However, despite the fact that QC materials cannot completely reflect the characteristics of patient specimens, they are used in the daily and routine internal QC practices of clinical laboratories. In this regard, the target ranges of QC materials need to be adjusted when the results of QC materials measured with new lot reagent are abruptly changed due to non-commutability. But there is no specific guideline as to the extent of alteration that demands the adjustment of QC material target ranges.
In this study, we estimated the extent of changes in QC materials’ results relative to patient specimens during reagent lot changes. In addition, we estimated the degree of changes in QC materials’ results caused by reagent lot changes and proposed the criteria of QC target range adjustment.
Materials and methods
The lists of manufacturers and models of the laboratory measurement procedures used and the manufacturers of the QC materials used for each measurement procedure.
AFP: alpha fetoprotein; BNP: B-type natriuretic peptide; CEA: carcinoembryonic antigen; CK: creatine kinase; CK-MB: creatine kinase MB; CRP: C-reactive protein; MPA: mycophenolic acid; NEFA: non-ester fatty acid; TSH: thyroid stimulating hormone.
In the laboratory, the new lot reagents are validated by testing 3–5 patients’ samples and 2–3 levels of QC materials with both old and new lots. The new lot is considered to be acceptable when the average Delta % of the patients’ sample results obtained with the new and old lots fall within ±10%. Delta % was calculated as follows: ([new lot result – old lot result] /old lot result) × 100. Also QC materials’ results are regarded as acceptable when the results tested with new lots fall within the predefined target range with old lot. Our laboratory usually establishes the target range as mean ± 2 SD by using the mean and standard deviation (SD) derived from the initial evaluation study.
All patient and QC sample tests with old and new reagent lots were conducted under similar conditions using identical equipment, reagents and manufacturer’s guides.
Statistical procedures
For all 360 reagent lot changes, the results of the patient specimens and QC with the old and new lots were natural log-transformed, and the differences between the new and old lots for each sample (ΔP) and quality control (ΔQC) were calculated. For each reagent lot change event, the differences between ΔP and ΔQC were tested for significance by using Mann–Whitney U testing because there were 3–5 ΔP and ΔQC, below 30, in each reagent lot change event. The software SPSS version 13.0 (SPSS, Inc., Chicago, IL, USA) was employed for this. P values below 0.05 were considered to be statistically significant.
Number of events with significant or non-significant differences between the results of QC materials and those of patients’ samples after changes in reagent lots for analytes grouped by category of analytical measurement procedure and source of QC material.
In total, there were 962 DQC results in the 360 reagent lot change events. For each of these 962 DQC results, the size of the DQC was calculated by dividing it by the SD used in the predefined criteria set for daily QC tasks and parallel testing. Thus, DQC was expressed as a multiple of SD. These multiples of SD results were then classified into five categories: <0.5, 0.5–1.0, 1.0–1.5, 1.5–2.0 and >2.0. The QC results in each corresponding category were assessed.
Results
As presented in Table 2, the ΔP and ΔQC differed significantly (P < 0.05) in 28 of the 360 reagent lot changes (7.8%). The 360 reagent lot changes involved 171 general chemistry tests and 189 immunoassays. The ΔP and ΔQC differed significantly in 14 (8.2%) and 14 (7.4%) of these test types, respectively. The two test types did not differ in terms of frequency of ΔP and ΔQC differences (P = 0.937). In addition, in 73 and 287 of the reagent lot change events, the QC material was provided by the method manufacturer and a third party, respectively. The ΔP and ΔQC differed significantly in 3 (4.1%) and 25 (8.7%) lot change events for each source of QC material. However, two sources of QC material did not significantly differ in the frequency of ΔP and ΔQC differences (P = 0.229).
Comparison of the trends of DQC and DP in reagent lot-to-lot change tests.
Number of events with QC results’ difference between two lots of reagents in multiple of SD.
Discussion
Ideally, when patients’ samples and QC materials are measured before and after reagent lot changes, the test results should be the same; this is often not the case. Instead, each laboratory defines acceptance criteria that signal whether the test results with a new reagent lot are consistent with the test results with the old reagent lot. The standard used in our laboratory is the new reagent lot was considered to be acceptable if (i) the same patient specimens tested with the new and old reagent lots returned values do not differ by more than ±10% (3), and (ii) the values obtained by testing the QC materials with the new lot reagent fell into a conventional predefined target range. We were able to confirm that reagent lot testing in our laboratory satisfied all acceptance criteria and met the requirements of CAP (data not shown). However, these criteria are the minimal requirement as practice guideline and insufficient as the standard of optimal laboratory practice.
In the present study, the differences in the values of patients’ samples (ΔP) and QC materials (ΔQC) after testing with the old and new reagent lots were determined. Only 7.8% of the 360 lot change events exhibited statistically significant differences between ΔP and ΔQC. By contrast, another study by Miller et al. 1 reported that 40.9% of lot change events were associated with statistically significant difference, even though similar clinical chemistry test items and instrument platforms were used in both studies. According to our lab policy, usually 3–5 patients’ samples are tested per each reagent lot change event for parallel test. Thus, we had to analyse the data using Mann–Whitney, a non-parametric statistical test, in the present study, while t-test with Satterthwaite adjustment was used in the previous study. 1 The disparity might be possibly attributed to the difference in statistical method. The present study also showed that the frequencies of significant differences between ΔP and ΔQC did not differ when the general chemistry tests were compared to the immunoassay tests. In addition, the source of QC material (third party versus the method manufacturer) did not significantly affect these frequencies. However, when QC materials from a third party were used, 8.7% events showed significant differences between ΔP and ΔQC, whereas the QC materials from the method manufacturer were associated with a relatively lower rate of such differences (4.1%). This phenomenon suggests the possibility that QC materials provided by method manufacturer have fewer matrix effects compared with those provided by third party.
The use of QC materials is inevitably compromised by intrinsic non-commutability problems. This non-commutability reflects the fact that the matrix of QC materials generated by the manufacturing process is different to that of patient specimens. As a result, questions have been asked about the usefulness of considering the lot-to-lot QC test data when determining whether a new reagent lot is acceptable. For example, our experience with non-commutability problems is generated by the use of non-human source (animal placenta) QC materials for an alkaline phosphatase assay. 17 Moreover, when performing new versus old lot tests for a lipase assay, we found that although the patient specimens rarely showed marked differences between the new and old lots, the values of the high level QC materials often exceeded the target range (unpublished data). Later, we found that this discrepancy was due to a matrix-related bias caused by the use of non-human (porcine-based) lipase in the QC materials. Thus, in our experience, QC material results obtained with reagent lot change are not 100% reliable and should only be considered as a reference. This is supported by the analysis in the present study of the trends of DQC and DP in the 332 lot change events that were not associated with significant differences between ΔP and ΔQC; DQC and Dp only followed the same trend in 66.3% of these events. Thus, even in the present study, the usefulness of the lot-to-lot QC material data in establishing the suitability of a new lot was limited.
Since QC materials are used in reagent lot change parallel tests in daily practice, it is important to adjust the target range of QC materials after reagent lot change to reduce the rates of false positive and false negative errors, as these errors eventually lead to inappropriate errors. Actually, applying too strict standards wastes resources (including reagents) and increases the test time. However, there are currently no guidelines about the degree of readjustment that will be associated with the lowest rates of false positive and false negative errors.
The extent of shift and the resulting false positive and false negative rates.
When the size of DQC in all 962 new versus old lot QC result pairs was assessed by calculating the multiple of SD using the SD employed in the predefined acceptable criteria, 71.0% of the QC results fell into multiple-of-SD categories below 1.0 (which is the theoretical standard that was suggested earlier). Thus, in 29% of reagent lot change events, the QC target range has to be readjusted when 1.0 is used as the upper limit of the target range.
In conclusion, this study confirmed that the QC material data in parallel tests performed during reagent lot-to-lot change are of limited usefulness in evaluating the suitability of a new lot. Also a new standard for establishing a new QC target range after reagent lot change event was proposed.
Footnotes
Conflict of interest
None declared.
Funding
This research received no specific grant from any funding agency in the public, commercial, or not-for-profit sectors.
Ethical approval
Asan Medical Center Institutional Review Board, Ethics reference number: (2013-0630).
Guarantor
WL.
Contributorship
M-CC, SYK and WL researched literature and conducted the study. M-CC, T-DJ, WL, SC and W-KM were involved in protocol development and data analysis. All authors reviewed and edited the manuscript and approved the final version of the manuscript.