Ring test evaluation of the repeatability and reproducibility of a Porcine reproductive and respiratory syndrome virus oral fluid antibody enzyme-linked immunosorbent assay

Abstract

The precision of a Porcine reproductive and respiratory syndrome virus (PRRSV) oral fluid antibody enzyme-linked immunosorbent assay (ELISA) was evaluated by calculating reliability coefficients for assay repeatability (within laboratory) and assay reproducibility (between laboratories). Randomly ordered oral fluid samples of known (n = 39) and unknown (n = 224) PRRSV antibody status were tested in 12 diagnostic laboratories. Each laboratory tested the samples twice, first using an antibody ELISA kit and reagents provided to them (phase 1) and then using an ELISA kit and reagents configured in their respective laboratory (phase 2). Repeatability (within laboratory) reliability coefficients calculated using results from samples of known PRRSV antibody status ranged from 0.724 to 0.997 in phase 1 and from 0.953 to 0.998 in phase 2. Reproducibility (between laboratories) reliability coefficients were calculated for 3 conditions: case 1—samples of unknown status (n = 224); case 2—samples of known status (n = 39), and case 3—all samples (n = 263). Among the 3 cases, reliability coefficients ranged from 0.937 to 0.964 in phase 1 and from 0.922 to 0.935 in phase 2. For case 3, it was estimated that 96.67% of the total variation in phase 1 and 93.21% in phase 2 could be attributed to the oral fluid samples themselves. Overall, the PRRSV oral fluid antibody ELISA was highly repeatable and reproducible. The current study supports the routine use of this test in laboratories providing diagnostic service to pig producers.

Introduction

Oral fluid samples are increasingly used for the surveillance of pathogens in commercial swine operations in North America. For example, the specimen type “swine oral fluid” was entered into the Iowa State University Veterinary Diagnostic Laboratory (ISU VDL; Ames, Iowa) information management system in February 2010. During the remainder of 2010, 10,329 oral fluid specimens were received for testing. This number increased to 32,544 in 2011. In the first 3 months of 2012, the ISU VDL received 13,226 oral fluid samples. Thus, if the current level of submissions is maintained, the laboratory will receive >52,000 oral fluid samples in 2012. The majority of oral fluid samples received at the ISU VDL are tested by polymerase chain reaction (PCR)-based assays for Porcine reproductive and respiratory syndrome virus (PRRSV),^4,7,9 Influenza A virus, ¹ Porcine circovirus-2, ⁸ and other respiratory pathogens.

In 2012, the first antibody assay for swine oral fluids became available. ⁵ Although oral fluid antibody assays are already widely available in human diagnostic medicine for a variety of pathogens, ⁶ this is the first application in swine diagnostic medicine. Therefore, the objective of the current study was to evaluate the repeatability and reproducibility of the PRRSV oral fluid enzyme-linked immunosorbent assay (ELISA) ^a among 12 laboratories that offer diagnostic services to swine producers and veterinarians.

Materials and methods

Experimental design

Oral fluid samples (n = 263) were completely randomized and then distributed to the 12 laboratories participating in the ring trial. All participating laboratories had prior experience with ELISAs and 3 had previous experience with the PRRSV oral fluid ELISA. Each laboratory received 1 set of samples, one “pre-configured” commercial PRRSV ELISA kit ^a with pre-diluted reagents and controls, and a copy of the standard operating procedure for performing the assay. ³ Each laboratory tested the samples on the kit provided (phase 1) and then repeated the testing on a PRRSV ELISA kit ^a configured in their respective laboratory (phase 2). Results were evaluated for test repeatability (within laboratory) and reproducibility (between laboratories).

Oral fluid specimens

Oral fluid specimens tested (n = 263) included samples of unknown (n = 224) and known (n = 39) PRRSV antibody status. Oral fluid field specimens of unknown PRRSV antibody status were collected over a 2-month period from among diagnostic samples submitted to the ISU VDL for either PRRSV real-time reverse transcription polymerase chain reaction (real-time RT-PCR) or PRRSV antibody testing. The ISU VDL records indicated that specimens were collected on 42 different production sites located in 7 states (Colorado, Iowa, Illinois, Indiana, Missouri, North Carolina, and Ohio). Samples were included on the basis of sufficient sample volume and without knowledge of testing results in the ISU VDL. Each sample was assigned a random number, divided into 12 aliquots of 1.25 ml, and stored at −20°C.

Oral fluid samples of known PRRSV antibody status originated from pigs inoculated with a type 2 PRRSV modified-live virus vaccine ^b and collected on 0, 10, 15, 20, 28, 35, 41, 49, 56, 75, and 91 days post vaccination (DPV). The samples were used to create 13 samples that were run on every plate. Specifically, the day 0 sample was used as in-house negative control; the day 75 sample was used to create high (1:5), medium (1:10), and low (1:20) positive controls using the PRRSV ELISA kit ^a sample diluent. The remaining 9 samples (i.e., DPV 10, 15, 20, 28, 35, 41, 49, 56, and 91) were split into 3 aliquots and then placed within the sample set such that all 9 were run on every plate.

Oral fluid ELISA procedures

Oral fluid samples were tested on a commercial PRRSV ELISA using a protocol designed to detect PRRSV antibody in oral fluid.^3,5 In phase 1, all 12 laboratories used the same kit lot (40959-W721). The 11 laboratories that participated in phase 2 used 5 different kit lots: 5 used lot 40959-EG288, 3 used lot 40959-EG233, and 1 laboratory each used lots 40949-W721, 40949-W791, and 40949-X861. Phase 1 and phase 2 were completed before the data were analyzed.

To perform the test, oral fluid samples were diluted 1:2 in dilution plates using the diluent provided with the test kit. Two hundred fifty microliters of diluted oral fluid was then transferred to the 96-well PRRSV antigen-coated plates. Negative and positive kit controls were diluted 1:30 using the kit diluent and 100 µl transferred to the assigned wells on the plates (i.e., A1 and B1 for negative controls; C1 and D1 for positive controls). Thereafter, plates were incubated for 16 hr at 4°C. Three laboratories used refrigerated incubators; all other laboratories used standard refrigerators. At the end of the 16-hr incubation period, the plates were washed 3 times with 400 μl of 1× kit wash solution. To detect bound antibody, reagents were brought to room temperature and then 100 μl of a solution containing appropriately diluted horseradish peroxidase–conjugated anti-swine immunoglobulin G (IgG_FC) secondary antibody ^c was added to each well and incubated for 30 min at 22°C. Thereafter, plates were washed 3 times with wash solution, 100 µl of tetramethylbenzidine enzyme substrate solution was added to each well, and the plates incubated at 22°C for 15 min. After 15 min, 100 µl of kit stop solution was added to each well. The plates were read at 650 nm and measured as optical density (OD).

All laboratories provided results as both OD and sample-to-positive control (S/P) ratios. The S/P ratio was calculated as:

$$\text{S}/\text{P}=\frac{[\text{Sample A}\left(650\right)-\text{NC}]}{(\text{PC}-\text{NC})}$$

where NC and PC represented the mean OD of the 2 negative control wells and 2 positive control wells, respectively. The S/P ratios ≥ 0.40 were considered positive for PRRSV antibody. ⁵

Statistical analysis

The consistency of the PRRSV oral fluid ELISA was quantified by estimating reliability coefficients (ρ) for assay repeatability (within laboratory) and assay reproducibility (between laboratories) using the results from oral fluid samples of known (n = 39) and unknown (n = 224) PRRSV antibody status, respectively.^2,10

For calculation of repeatability reliability coefficients, the model and associated parameters were based on n samples with d repeated measurements of a continuous variable x and denoted by x _ij , the jth measurement made on the ith subject (for i = 1, . . . , n and j = 1, . . . , d):

$$x_{ij}={\tau }_i+{\beta }_j+e_{ij}$$

where τ _i is the sample effect, β _j is the plate effect, and e _ij is the measurement error. For within laboratory data, the d repeated measurements refer to results derived from repeated testing of the same samples by the same laboratory (i.e., the 13 samples tested on every plate). The equation for calculating assay repeatability reliability coefficients is given as:

$$\rho =\frac{{\sigma }_{\tau }^2}{{\sigma }_{\tau }^2+{\sigma }_{\beta }^2+{\sigma }_e^2}$$

where ${\sigma }_{\tau }^2$ is the variance due to the oral fluid sample, ${\sigma }_{\beta }^2$ is the variance due to the plate, and ${\sigma }_e^2$ is the variance due to measurement error.

Reproducibility reliability coefficients were estimated for 3 conditions (case 1: samples of unknown status, case 2: samples of known status, and case 3: all oral fluid samples) using equation (4):

$$x_{ijk}={\tau }_i+{\alpha }_k+{\gamma }_{jk}+{\omega }_{ik}+e_{ijk}$$

In equation (4), x _ijk is the measurement made on the ith subject in jth kit lot number by kth laboratory (for i = 1, . . . , n; j = 1, . . . , d; and k = 1, . . . , m), τ _i is the effect of the ith oral fluid, α _k is the effect of the kth laboratory, γ _jk is the effect of kit lot number jth within laboratory k (where kit lot is nested within laboratory), ω _ik is the interaction between oral fluid i and laboratory k, and e _ijk is the measurement error. The reproducibility reliability coefficient may be calculated as:

$$\rho =\frac{{\sigma }_{\tau }^2}{{\sigma }_{\tau }^2+{\sigma }_{\alpha }^2+{\sigma }_{\gamma }^2+{\sigma }_{\omega }^2+{\sigma }_e^2}$$

where the variance ${\sigma }_{\tau }^2$ is nominator and the total variance components (i.e., sum of variances of subject error, laboratory error, kit lot error, the interaction between oral fluid and laboratory and measurement error [denominator]).

Variance estimates for equations (2) and (4) were calculated from random effect models solved using commercial statistical software. ^d The S/P ratios within linear random effects models were analyzed using the GLIMMIX procedure. Oral fluid samples, laboratory, kit lot number, and interaction between oral fluid and laboratory were treated as random effects.

Results

A total of 3,156 PRRSV oral fluid ELISA results were produced by the 12 laboratories that participated in phase 1 (Fig. 1). Among these results, 1,581 (50.1%) were positive at a S/P cutoff of ≥0.40. In phase 2 (Fig. 2), the 11 participating laboratories produced 2,885 results with 1,438 positives. As illustrated in Figures 1 and 2, the dispersion of the results increased as the magnitude of the S/P response increased. The number of samples tested, mean ELISA S/P ratios, and S/P standard deviation are shown in Table 1 by laboratory. Among the samples of unknown PRRSV antibody status (n = 224), discordant results were produced in 7 samples in phase 1 and 5 samples in phase 2 (Table 2).

Figure 1.

Phase 1. Results from 224 samples tested in each of 12 laboratories. All laboratories using the same Porcine reproductive and respiratory syndrome virus oral fluid antibody enzyme-linked immunosorbent assay kit lot and reagents. S/P ratio = sample-to-positive control ratio.

Figure 2.

Phase 2. Results from 224 samples tested in each of 11 laboratories on a total of 5 Porcine reproductive and respiratory syndrome virus enzyme-linked immunosorbent assay kit lots using reagents prepared by respective laboratories. S/P ratio = sample-to-positive control ratio.

Table 1.

Summary of results used to estimate test variance components in the Porcine reproductive and respiratory syndrome virus (PRRSV) oral fluid enzyme-linked immunosorbent assay (ELISA) ring trial.*

		Phase 1 test results†				Phase 2 test results‡
Lab	No. of samples	Mean S/P	SD	No. positive	No. negative	Mean S/P	SD	No. positive	No. negative
1		2.47	0.58	36	3	2.51	0.57	36	3
	224¦	1.03	1.32	96	128	0.92	1.17	95	129
2	39	2.86	0.61	36	3	2.34	0.59	36	3
	224	0.98	1.32	95	129	0.83	1.06	95	129
3	39	2.53	0.63	36	3	3.10	0.66	36	3
	224	0.84	1.13	95	129	0.92	1.28	95	129
4	39	2.99	0.59	36	3	3.10	0.55	36	3
	224	1.14	1.50	95	129	1.17	1.52	98	126
5	39	3.07	0.58	36	3	2.68	0.60	33	2
	224	1.13	1.45	96	128	1.01	1.27	96	128
6#	39	2.96	0.66	37	2
	224	0.91	1.20	95	129
7	39	2.65	0.61	36	3	2.39	0.54	36	3
	224	0.94	1.18	95	129	0.73	0.93	95	129
8	39	2.74	0.64	36	3	2.52	0.62	36	3
	224	0.97	1.29	97	127	0.83	1.07	95	129
9	39	2.69	0.61	36	3	2.37	0.56	33	2
	224	0.93	1.25	96	128	0.78	1.06	95	129
10	39	2.49	0.60	36	3	2.19	0.66	36	3
	224	0.88	1.17	97	127	0.69	0.93	95	129
11	39	2.84	0.68	36	3	2.32	0.68	36	3
	224	0.87	1.18	95	129	0.68	0.92	94	130
12	39	2.54	0.60	36	3	2.33	0.65	36	3
	224	0.87	1.11	96	128	0.77	1.03	95	129

*

S/P = sample-to-positive control ratio; SD = standard deviation.

†

PRRSV ELISA kit (same lot number) and all reagents needed to test the samples were provided to each laboratory. Samples with S/P ≥0.40 were classified as positive. ⁵

‡

Laboratories provided their own PRRSV ELISA kit and prepared all reagents needed to test the samples. Note: four known-status samples in laboratories 5 and 9 were of insufficient volume to test in phase 2. Samples with S/P ≥0.40 were classified as positive. ⁵

§

S/P ratios from samples with known PRRSV antibody status.

¦

S/P ratios from samples with unknown PRRSV antibody status.

#

Laboratory did not participate in phase 2 testing.

Table 2.

Summary of Porcine reproductive and respiratory syndrome virus (PRRSV) oral fluid enzyme-linked immunosorbent assay (ELISA) discordant results among 224 oral fluid samples of unknown PRRSV antibody status (sample-to-positive control [S/P] cut-off ≥ 0.40).*

	Laboratories participating in the PRRSV oral fluid ELISA ring test (S/P ratios)
Sample random number	1	2	3	4	5	6	7	8	9	10	11	12
Phase 1†
4	−0.05	−0.03	0.00	−0.02	0.02	0.00	0.02	−0.03	−0.02	0.01	−0.04	1.21
16	0.05	−0.01	0.00	−0.01	0.01	0.04	0.04	0.03	0.00	0.72	0.00	0.01
62	0.07	−0.04	−0.01	0.00	0.02	0.02	0.04	0.00	0.07	0.96	0.00	0.02
112	−0.03	0.00	−0.02	−0.05	0.00	−0.08	0.03	0.51	−0.01	0.00	−0.06	−0.02
151	1.19	−0.03	−0.04	−0.06	0.00	−0.04	−0.01	0.08	−0.03	−0.02	−0.04	0.04
175	−0.03	−0.01	−0.04	−0.05	0.00	−0.04	0.02	−0.01	3.03	−0.02	−0.02	0.00
193	0.27	0.26	0.26	0.39	0.45	0.27	0.33	0.41	0.40	0.26	0.26	0.25
Phase 2‡
2	0.98	0.86	0.68	1.60	1.07	NA	0.83	0.96	0.80	0.61	NA	0.31
12	0.17	0.22	0.12	0.43	0.25	NA	0.20	0.19	0.14	0.14	0.12	0.15
53	−0.03	0.00	−0.02	0.51	0.01	NA	0.00	0.01	−0.07	0.01	−0.01	0.00
193	0.32	0.32	0.27	0.56	0.51	NA	0.30	0.28	0.30	0.17	0.20	0.22
231	0.81	0.51	0.51	0.69	0.70	NA	0.43	0.47	0.62	0.53	0.33	0.48

*

Boldface indicates discordant results. NA = not available.

†

PRRSV ELISA kit and all reagents required to test the samples were provided to each laboratory.

‡

Laboratories provided their own PRRSV ELISA kit and prepared all reagents required to test the samples.

Repeatability (within laboratory) reliability coefficients calculated from linear random effects models are shown in Table 3. The within laboratory agreement ranged from 0.942 to 0.997 in phase 1 and from 0.953 to 0.998 in phase 2. Laboratory 6 reported 1 false-positive result in phase 1, which resulted in a within laboratory reliability coefficient of 0.724 for that laboratory. No statistically significant difference in repeatability was detected for any laboratory when comparing phase 1 versus phase 2 reliability coefficients.

Table 3.

Porcine reproductive and respiratory syndrome virus (PRRSV) oral fluid antibody enzyme-linked immunosorbent assay (ELISA) reliability coefficients.*

	Phase 1†	Phase 2‡
Repeatability reliability coefficients
Laboratory 1	0.991	0.995
Laboratory 2	0.986	0.989
Laboratory 3	0.996	0.994
Laboratory 4	0.967	0.986
Laboratory 5	0.990	0.998
Laboratory 6	0.724	−
Laboratory 7	0.942	0.953
Laboratory 8	0.950	0.987
Laboratory 9	0.994	0.967
Laboratory 10	0.969	0.956
Laboratory 11	0.997	0.997
Laboratory 12	0.991	0.960
Reproducibility reliability coefficients
Case 1: samples of unknown status	0.964	0.935
Case 2: samples of known status	0.937	0.922
Case 3: all samples	0.957	0.932

*

Repeatability reliability coefficients were calculated using the data from 13 samples tested on each plate in each laboratory. Reproducibility reliability coefficients were calculated for 3 cases: all samples (n = 263), field samples of unknown PRRSV antibody status (n = 224), and samples of known PRRSV antibody status (n = 39). Reliability coefficients were interpreted from a previous study ¹¹ : excellent (>0.75), good (0.40–0.75), and poor (<0.4). – = no results provided.

†

PRRSV ELISA kit and all reagents required to test the samples were provided to each laboratory.

‡

Laboratories provided their own PRRSV ELISA kit and prepared all reagents required to test the samples.

Reproducibility (between laboratories) reliability coefficients were calculated using results from 224 samples of unknown status and 39 samples of known status tested in all laboratories. As shown in Table 3, this allowed for the calculation of reproducibility reliability coefficients for 3 conditions: case 1—samples of unknown status (n = 224); case 2—samples of known status (n = 39), and case 3—all samples (n = 263). Table 4 lists the variance components estimates used to calculate the reproducibility reliability coefficients for case 3 (i.e., oral fluid samples, laboratory, ELISA kit lot, ELISA plate, and measurement error). Among these sources of variation, oral fluid samples accounted for 95.67% of the total variation in phase 1 and 93.21% in phase 2.

Table 4.

Estimates of the variance components used to calculate reproducibility reliability coefficients for the Porcine reproductive and respiratory syndrome virus (PRRSV) oral fluid antibody enzyme-linked immunosorbent assay (ELISA).*

	Phase 1†		Phase 2‡
Variation by source	Variance	% of model variance	Variance	% of model variance
Oral fluid sample	1.768 (1.494–2.126)	95.67	1.403 (1.185–1.688)	93.21
Laboratory	0.007 (0.003–0.048)	0.39	0.022 (0.009–0.093)	1.43
ELISA kit lot-to-lot	NA	NA	0.007 (0.001–158.2)	0.60
ELISA plate (laboratory)	0.010 (0.006–0.023)	0.54	0.003 (0.002–0.008)	0.21
Measurement error	0.063 (0.060–0.066)	3.40	0.071 (0.067–0.075)	4.70

*

Variance components were estimated in a linear random effects model using the sample-to-positive control (S/P) ratio data from all oral fluid samples (n = 263). Numbers in parentheses are 95% confidence intervals. NA = not applicable because all laboratories used kits from the lot.

†

PRRSV ELISA kit and all reagents required to test the samples were provided to each laboratory.

‡

Laboratories provided their own PRRSV ELISA kit and prepared all reagents required to test the samples.

Discussion

A recent analysis of the PRRSV oral fluid ELISA reported test sensitivity of 94.7% (95% confidence interval [CI]: 92.4, 96.5) and specificity of 100% (95% CI: 99.0, 100.0). ⁵ Further evaluation of this ELISA was motivated by the fact that the protocol differs significantly from the protocol for the PRRSV serum antibody ELISA. That is, in contrast to the serum antibody assay, the PRRSV oral fluid ELISA uses a sample volume of 250 μl (100 μl for serum), a 16-hr incubation period at 4°C (30 min at 22°C for serum), and requires the operator to purchase and correctly prepare the appropriate concentration of anti-swine IgG_FC secondary antibody.^3,5 To fully evaluate the precision of the assay, the study was conducted in a ring test involving various North American laboratories that were expected to offer this test to swine clientele on a routine basis.

Visual analysis of the cumulative results (Fig. 1) showed greater dispersion of S/P ratios in phase 2 (i.e., laboratories tested the samples on a total of 5 ELISA kit lots using reagents they had prepared), as compared with phase 1 (i.e., samples were tested on a single kit lot with pre-diluted reagents). However, as shown in Table 2, relatively few discordant qualitative results were seen either in phase 1 (7 discordant samples) or phase 2 (5 discordant samples). With one exception, discordant results were single events. The one exception was a sample (#193) that was reported positive in 3 out of the 12 laboratories in phase 1 and in 2 out of the 11 laboratories in phase 2. The frequency of discordant results for this sample can be explained by the fact that its mean S/P value lies close to the positive cutoff (S/P ≥ 0.40). Thus, small differences in quantitative measurements produced divergent categorical results.

Reliability coefficients were used as quantitative summary measures of agreement. By definition, the repeatability reliability coefficient is a measure of intra-laboratory assay agreement, and the reproducibility reliability coefficient is a measure of inter-laboratory agreement. ¹² A previous study ¹¹ has suggested that coefficients >0.75 indicate excellent reliability. The limitation of this approach is that reliability coefficients reflect both the assay and the distribution of the analyte in the samples tested. That is, given the same assay, reliability coefficient estimates will be affected as the concentration and/or prevalence of the analyte in the population of samples tested varies. ²

Repeatability reliability coefficients were calculated using 13 samples tested on each ELISA plate in every laboratory (Table 3). Coefficients ranged from 0.724 to 0.997 (phase 1) and from 0.953 to 0.998 (phase 2). No statistically significant differences were detected in phase 1 versus phase 2 repeatability reliability coefficients in the 11 laboratories participating in both phases. Based on the reliability coefficient criteria described previously, ¹¹ the results were shown to be highly repeatable in the participating laboratories.

Reproducibility reliability coefficients were calculated for 3 cases (Table 3): samples of unknown PRRSV antibody status (case 1, n = 224), samples of known status (case 2, n = 39), and all samples tested in every laboratory, regardless of PRRSV antibody status (case 3, n = 263). Among the 3 cases, reliability coefficients ranged from 0.937 to 0.964 in phase 1 and from 0.922 to 0.935 in phase 2, indicating excellent reproducibility across laboratories. ¹¹ Based on overall samples results, the variance components estimated that 95.7% of the total variation in phase 1 and 93.2% in phase 2 could be attributed to the oral fluid samples themselves (Table 4), with the remainder of the variation allocated to laboratory, ELISA kit lot, ELISA plate (laboratory), and measurement error.

Cumulatively, the current study showed that a commercial serum antibody ELISA adapted to oral fluid samples was highly reproducible within and between laboratories responsible for swine health testing. The results support the routine use of this test in laboratories providing diagnostic service to pig producers. Thus, herd monitoring based on oral fluid antibody testing could be one part of a PRRSV control and/or elimination program. The successful ring trial of a commercial serum antibody ELISA to the oral fluid diagnostic suggests that antibody assays for other pathogens could also be modified to this approach.