Abstract
Control of
Detection of
Our objective was to compare the NVSL and IDEXX ELISAs with each other and a commercial AGID test kit (TECPAR) to evaluate their potential for use in disease detection on U.S. sheep operations. This information could be utilized by veterinarians, diagnosticians, and producers to determine which assay is best suited for their needs.
Between August 2015 and May 2016, sera were collected from 2,276 sheep from 18 operations in Wyoming. The University of Wyoming Institutional Animal Care and Use Committee (IACUC) approved the study and sampling method (protocol 200150622KS00178-01). Flocks varied in geographic location and size, and sheep varied in age, sex, and breed. 6 During sample collection, properly trained personnel drew 5–10 mL of blood via jugular venipuncture of healthy sheep using 10-mL blood collection tubes and needle holders (Vacutainer; Becton, Dickinson) with disposable 4-cm, 18–20-ga needles (Greiner Bio-One). Blood was allowed to clot before transportation back to the Wyoming State Veterinary Laboratory (Laramie, WY) on ice. Sera were separated from blood samples by centrifugation, decanted into labeled tubes, and stored at –20°C until serologic testing.
All samples were initially analyzed using the NVSL ELISA according to the manufacturer’s instructions. The diagnostic sensitivity and specificity are reported to be 96.3% and 99.6%, respectively,
7
and the NVSL-supplied antigen was a Triton X-100 extract of the washed REO 198 strain. Controls run in triplicate included high positive, low positive, and negative; sample sera were run in duplicate. Polystyrene microtiter plates (Nunc) were coated with an antigen dilution in carbonate buffer, incubated, and blocked with bovine serum albumin solution (KPL). Conjugated protein G–biotin from
Sera were then tested using the IDEXX ELISA according to the manufacturer’s instructions. The diagnostic sensitivity and specificity are reported to be 91.7% and 95.2%, respectively, 9 and the antigen used is an extract of the REO 198 strain prepared by IDEXX. Each kit contained 2 antigen-coated plates, positive and negative control sera, anti-ruminant IgG monoclonal antibody labeled with horseradish peroxidase as conjugate, tetramethylbenzidine (TMB) substrate, stop reagent, and wash solution. Controls and sample sera were run in duplicate. During the assay procedure, sera and conjugate were incubated at 37°C in a humidified chamber inside a digital display incubator (Boekel) for 60 min each. After each incubation, plates were washed 3 times with the wash solution, excess liquid emptied in the sink, and plates were tapped onto absorbent material. The TMB substrate incubation step was performed at room temperature for 15 min before adding the stop reagent. The plate was read at a wavelength of 450 nm on a microplate absorbance spectrophotometer (Tecan; BioTek Instruments), the average OD for each sample and control was calculated, and the average S/P ratio was used to determine sample status. Original S/P ratio cutoff values were as follows: > 50% = positive, ≤50% to ≥10% = suspect, and < 10% = negative. Suggested cutoff values of 30%, 45%, or 60% were also assessed. 9 The following equation was used to calculate the S/P ratio:
Finally, 295 sera were tested using the AGID test according to the manufacturer’s instructions. These 295 sera were chosen based on initial results from the NVSL ELISA. Of these samples, 20 were seropositive and 46 were indeterminate. The remaining 229 seronegative samples were chosen randomly. The diagnostic sensitivity and specificity of the AGID test kit is reported to be 70.1% and 100%, respectively.
14
Each kit contained 1 vial of lyophilized antigen prepared from soluble proteins and lipopolysaccharides extracted from the
Pairwise 2-tailed
Of 2,276 sera, 20 (0.88%) were seropositive by NVSL ELISA, and 25 (1.10%) by IDEXX ELISA, according to original cutoff values. Seven (0.31%) were seropositive on both ELISAs, and 2,049 (90.03%) were seronegative on both ELISAs. When comparing seroprevalence, there was not a statistically significant difference between the NVSL and IDEXX ELISAs (pairwise 2-tailed
Analysis of 2,276 sheep sera for
NVSL
Overall, there was poor agreement between different assay cutoffs utilized in this study, based on κ coefficients (Table 2). For the NVSL and IDEXX ELISAs, we first compared the original cutoff values, which included the indeterminate or suspect result. Then, we used multiple single cutoff values to compare ELISAs with each other and the AGID test. The cutoff values used for the NVSL ELISA included a lax and strict value (i.e., 0.75 and 0.40, respectively). The cutoff values used for the IDEXX ELISA included a lax, moderate, and strict value (i.e., 60%, 45%, and 30%, respectively). The lax cutoff values correspond with the higher threshold given the expected decrease in sensitivity caused by possible false-negative results. In contrast, the strict cutoff values correspond with the lower threshold given the expected increase in sensitivity caused by true-positive results. ELISA combinations that resulted in moderate κ coefficients included the NVSL and IDEXX ELISAs with strict cutoff values (0.40 and 30%, respectively; κ = 0.41) and lax cutoff values (0.75 and 60%, respectively; κ = 0.43).
Comparison of kappa coefficients and agreement for various assay combinations and cutoff values for
Numbers in parentheses are cutoff values.
Numbers in parentheses are 95% CIs.
When comparing results from each ELISA with results from the AGID test (
Results from the AGID test (
Sensitivity and specificity for each ELISA with various cutoff values using AGID as the reference test (
Numbers in parentheses are 95% CIs.
ROC analysis revealed the average ELISA result for AGID-positive and AGID-negative animals, the AUC value, and the optimal cutoff value with reported sensitivity and specificity for each ELISA (Table 4). For the NVSL ELISA, the mean S/P ratio for AGID-positive animals was 0.448 (95% CI: 0.043–1.77; median: 0.236) and the mean S/P ratio for AGID-negative animals was 0.272 (95% CI: 0.027–0.895; median: 0.151). The AUC value was 0.584 (95% CI: 0.452–0.716), which indicates low assay accuracy when compared to the AGID test. Assuming prevalence is 50%, the recommended cutoff value for the NVSL ELISA is 0.091, resulting in sensitivity of 85.7% and specificity of 31.8%. When using the strict cutoff value for the NVSL ELISA (0.40), the sensitivity and specificity were 23.8% and 77.4%, respectively. For the IDEXX ELISA, the mean S/P ratio for AGID-positive animals was 37.7% (95% CI: 9.76–97.9%; median: 24.0%) and the mean S/P ratio for AGID-negative animals was 17.4% (95% CI: < 0.001–56.5%; median: 15.5%). The AUC value was 0.704 (95% CI: 0.591–0.817), which indicates moderate assay accuracy when compared to the AGID test. Assuming prevalence is 50%, the recommended cutoff value for the IDEXX ELISA is 16.5%, which would result in a sensitivity of 81.0% and specificity of 53.6%. When using the moderate cutoff value for the IDEXX ELISA (45%), the sensitivity and specificity was 38.1% and 91.6%, respectively.
Summary of receiver-operating characteristic (ROC) analysis with the average ELISA result for AGID-positive and AGID-negative animals, area under the ROC curve (AUC) values, and optimal and suggested cutoff values with reported sensitivity (Se) and specificity (Sp) for each ELISA.
Numbers in parentheses are 95% CIs.
Our study demonstrated that the IDEXX ELISA detected
When we evaluated ELISA agreement, the highest percent agreement and κ coefficients were observed between the ELISAs with similar cutoff values. For example, the NVSL ELISA with its lax cutoff value of 0.75 and the IDEXX ELISA with its lax cutoff value of 60% yielded a κ value of 0.43. However, no pair of ELISAs resulted in a κ value equivalent to “substantial” agreement, which was unexpected. This suggests the transition from one ELISA to another may result in low intra-laboratory agreement. Specifically, a switch from the NVSL ELISA to the IDEXX ELISA with the original cutoff values would result in relatively low agreement. However, a switch from the NVSL ELISA to the IDEXX ELISA with new recommended cutoff values (i.e. 60%, 45%, or 30%) based on the purpose of testing or epidemiologic situation may be a better option. 9
The more important finding of our study resides in the agreement between the AGID and each ELISA with various cutoff values. The IDEXX ELISA with moderate and lax cutoff values and the AGID test resulted in the highest κ coefficients, suggesting that these cutoff values correctly identified the highest percent of AGID-positive and AGID-negative animals while taking into consideration the percent agreement based on random chance. Specifically, the IDEXX ELISA with the moderate cutoff value correctly identified 8 AGID-positive animals and 252 AGID-negative animals. The NVSL ELISA with a lax cutoff value correctly identified 4 AGID-positive animals and 258 AGID-negative ones. Based on these results, the IDEXX ELISA with the moderate cutoff value resulted in the fewest false-negative results (i.e., 13), while correctly identifying the highest number of true-positive results (i.e., 8) when the AGID is used as the reference test for this sample set. This moderate cutoff value is recommended in flocks that perform some serologic testing, but may possess seropositive or “suspect” animals, which are characteristics that apply to many Western flocks. 9
When we used the AGID test as a reference test to calculate sensitivity and specificity, the IDEXX ELISA with the moderate cutoff value had the highest combined sensitivity and specificity. When the cutoff value was relaxed, both the sensitivity and specificity of the IDEXX ELISA decreased, but remained higher than the NVSL ELISA. The lax cutoff value of the NVSL ELISA resulted in lower sensitivity (19.1% from 23.8%), but higher specificity (94.2% from 77.7%) compared to the strict cutoff value, suggesting many of the indeterminate results are truly seronegative, not seropositive. If the strict cutoff value is used for the NVSL ELISA, there will be a large number (e.g., 61 in our study) of false-positive results, which may lead to the culling of truly negative animals. However, if the overall objective and disease management strategy of a particular flock is to eliminate all animals with
When we looked at the optimal cutoff value for each ELISA, all recommended values were extremely stringent in order to increase diagnostic sensitivity. Although a cutoff value of 0.091 for the NVSL ELISA would result in higher sensitivity (85.7%), the specificity is only 31.8%. For the IDEXX ELISA, the recommended cutoff is 16.5%, resulting in a sensitivity and specificity of 81.0% and 53.6%, respectively. Given the low sensitivity values for both ELISAs, one concern is that false-negative results will likely be produced. False-negatives have implications for flocks that are trying to eliminate
Regarding potential bias and study flaws, sample handling and the decision to use the AGID test as a reference test may have influenced study outcomes. We included sera from 2,276 sheep on 18, non-random, producer-selected operations in Wyoming. Animals selected within the flocks often comprised the majority of a flock’s breeding rams and a convenient subset of breeding ewes, resulting in a sample population that is not entirely representative of breeding sheep in the state. We created additional sample bias when we chose the 295 samples for AGID testing based on the initial results from the NVSL ELISA. Additional serologic testing to rule out cross-reactions from other organisms, including the causative agent of caseous lymphadenitis,
Footnotes
Acknowledgements
We thank the Wyoming sheep producers who made this study possible.
Declaration of conflicting interests
The authors declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.
Funding
This work was supported by a Wyoming State Agriculture Producer Research Grant and Wyoming Agricultural Experiment Station funding provided through the USDA National Institute of Food and Agriculture, Hatch project 1008761.
