Abstract
Background
It is vital that laboratories participate in External Quality Assurance (EQA) programmes, but results from such schemes do not necessarily help ensure assays are ‘in control’. We describe a series of experiments undertaken to explore apparent poor performance in an occult blood EQA scheme that manifested as a series of false-negatives. As a consequence of our laboratory misclassifying some EQA samples, we decided to design a simple sensitivity study that would reaffirm confidence in our testing procedures.
Methods
In order to verify analyst reliability and analytical sensitivity, a series of 10 faecal samples were supplemented with increasing amounts of whole blood (0–1.0 mg/Hb/g faeces) and these were then tested independently by four different operatives. EQA samples which our laboratory misclassified were shared with and tested by a second laboratory and results compared.
Results
The results demonstrated consistency in interpretation among the four ‘testers’ and also suggested that the sensitivity of the method is 0.7 rather than 0.6 mg/Hb/g faeces as claimed by the manufacturer. The second laboratory obtained identical results as our laboratory for the EQA samples which our laboratory had misclassified.
Conclusions
Occult blood EQA scheme results might not give a true reflection of a laboratory's ability to provide reliable results. We would encourage any laboratory observing relative poor performance to undertake local sensitivity studies and repeat testing of EQA samples if in any doubt.
Introduction
It has been established that around 35,000 people are newly diagnosed as having bowel cancer in the UK each year (male:female ratio 1.2:1.0), which means that approximately one in 20 people will develop it in their lifetime. After breast and lung cancers, it is the third most common cancer in the UK, and is the second leading cause of cancer deaths. More than 16,000 people die from bowel cancer each year. 1
As a consequence of this, bowel cancer screening programmes for both men and women are being introduced throughout the UK with the anticipation that such screening will detect cancer and enable relevant treatment to be given at the earliest opportunity. Pilot studies have shown that screening, using the faecal occult blood test, and appropriate intervention does improve outcomes. 2
The NHS Bowel Cancer Screening Programme began in 2006 and it is planned to have extended the programme to all areas of the UK by the end of 2009.
The screening test, faecal occult blood, has been used to detect trace amounts of blood in faeces since 1949, 3 and even though the test reagents have been improved over the decades, the test can be relatively non-specific and insensitive and is subject to human interpretation problems. 4 While newer, more specific and more sensitive immunoassays are becoming available, 5 these are more expensive so that the conventional colorimetric, eye readable, tests will remain in laboratories for quite some time to come.
It is vital that, where available, all laboratory and point of care assays should be registered on appropriate External Quality Assurance (EQA) programmes. There is such a programme for occult blood testing in the UK – the Yorkshire EQA Scheme (YEQAS) for faecal occult blood.
Our laboratory and one of our hospital wards have been registered on this scheme for many years and we both generally perform well on it. However, periodically we have observed apparent deterioration in performance and made a number of apparent misclassifications. These misclassification have typically been reported as ‘false-negatives’, suggesting that there is either a user/tester interpretational issue, the sensitivity of the test itself is inadequate or perhaps the EQA sample being of a simulated faecal matrix is unstable.
Although variability in the performance of the different methods on the EQA programme has been reported previously, 6 the major issue in this study related to false-positive rates rather than false-negative rates with the Hema-screen method yielding 2.3% and 0.7% false-positive and false-negative rates, respectively.
In contrast, an internal review of our own laboratories' EQA performance against the QA scheme's target responses (for the past 2 y) indicates a false-negative rate of 10.3%.
As a consequence of these problems, we decided to design a simple experiment to check user/tester interpretation reliability and at the same time confirm or refute the manufacturer's sensitivity claims.
We also decided to share EQA samples with a second laboratory that performed consistently well on the YEQAS scheme.
Materials and methods
Our faecal occult blood kit is the Hema-screen (Immunostics Inc, Sunset Avenue, Ocean, NJ, USA): a guaiac-based method, and was used throughout the study as per the manufacturer's instructions. The sensitivity of the method is claimed by the manufacturer to be 0.6 mg/Hb/g of faeces. However, an independent study has determined it to be 0.9 mg/Hb/g of faeces. 7
We prepared a series of 10 faecal samples in plain 25 mL sterilin universal containers with increasing amounts of haemoglobin, from 0.0 to 1.0 mg/Hb/g faeces, respectively, by supplementing an ‘occult blood-negative’ sample with blood from an EDTA whole-blood sample with known haemoglobin concentration. The samples were well mixed using disposable wooden spatulas so as to ensure homogeneity. ‘Target responses’ were then assigned based on the amounts of haemoglobin that had been added to each sample. The samples were then ‘mixed up’ so as to provide a ‘randomized’ order for testing purposes.
The samples were then tested independently by four operatives trained to analyse faecal occult blood samples: two experienced biomedical scientists, one trainee biomedical scientist and one nurse. Results were collated and are shown in Table 1.
Results for the supplemented faecal samples from four different operatives
In order to investigate the discrepancies we had periodically observed with results from the EQA samples, we decided to exchange some of these affected samples with a second laboratory that was registered on the EQA scheme who performed consistently well on the programme.
We sent two samples that we had tested, and reported (falsely) as being negative to the second laboratory. The samples contained 2.25 and 1.2 mg/Hb/g faeces: both concentrations that are well above the 0.6 mg/Hb/g sensitivity limit.
The second laboratory subsequently sent their two samples, from the same distribution and which they tested (correctly) as being positive, to us for analysis.
Results
The table shows the results of the supplemented faecal samples in the ‘randomized’ order they were tested in.
These results demonstrate consistency in interpretation among the four ‘testers’ and also suggested that the sensitivity of the method is 0.7 rather than 0.6 mg/Hb/g faeces.
The two samples containing 2.25 and 1.2 mg/Hb/g faeces that we sent to the second laboratory were also tested by them and found to be negative (therefore confirming our negative findings).
Their own two samples from this distribution that they reported, correctly, as being positive also tested positive in our laboratory, which therefore suggested the false-negative results were sample related.
Discussion
As can be seen, each tester's results compare favourably with the results of the target values with the exception of sample 3. The test is claimed by the manufacturer to be sensitive to 0.6 mg/Hb/g faeces, but all testers were unable to detect at this concentration. However, all testers consistently reported positive results at 0.7 mg/Hb/g of faeces and there was 100% consistency of results from all testers both above and below this sensitivity threshold. This was reassuring to us and maintained our confidence in the testing process, but a lower than expected occult blood detection limit will obviously have an impact on the tests sensitivity and specificity with respect to detecting colorectal cancer.
Moreover, a second laboratory also tested the same EQA samples that we had reported as falsely negative and they also got the same results as us. Their own samples, which they had tested as positive, were also found to be positive by our laboratory. We were encouraged by this consistency between the two testing sites, which therefore suggested a potential problem with the EQA samples themselves.
Since doing this work we have also done repeat testing of other EQA samples locally that were deemed to be ‘false-negatives’, and have demonstrated reproducibility.
The cause of these relatively high EQA false-negative rates is cause for concern but we have, as far as is practically possible to do so, ruled out operator misinterpretation as the possible cause. We have also verified that our local sensitivity is consistent with manufacturers' claims. Moreover, we have also shown that exchange of EQA samples with a second laboratory also shows reproducibility of results. We therefore speculate that the cause of our high false-negative rates is probably secondary to, as yet, unknown preanalytical factors – possibly relating to sample transport, delivery, storage or stability of the EQA material.
The EQA material is simulated faecal matter and is a mixture of wheat bran (not for human consumption), vegetable debris and a synthetic mucoid agent which is then ‘sterilized’ and supplemented with human whole blood.
The samples are sent from the EQA provider, in a frozen state, by courier and when received by us are immediately stored at −20°C in accordance with instructions and then thawed and tested as per the EQA scheme requirements. We will continue to participate in the EQA programme, but will in future do local verification studies if analytical performance is seen to ‘deteriorate’. The false-negative rates are of some concern and we are investigating further and have been in dialogue with the EQA provider, but so far, collectively, have been unable to fully explain these observations.
However, having done this work we have absolute confidence with our patient results, and we would encourage any other laboratories faced with these issues to undertake local internal quality control studies to verify their performance locally.
DECLARATIONS