Sage Journals: Discover world-class research

Abstract

Purpose

Following the analytical phase, the current practice of many hospital laboratories involves the manual verification of all test results followed by the production of the report. However, manual verification is a time-consuming and tedious process. In this paper, we provide a detailed description of how to design autoverification rules for thyroid function test profiles and sex hormones.

Materials and methods

We used DM2 (Data manager 2) to construct the algorithm and build the database for autoverification of thyroid function test profiles and sex hormones, with reference to Boolean logic, Auto 10-A and CLSI’88. The rules consist of checking quality control, instrument error flags, critical values, the analytical measurement range (AMR), the limit range, consistency check and delta check. Firstly, we established the rules in the DM2, collected clinical specimens for validation, then tested the rules in a ‘live’ environment.

Results

Agreement was achieved between manual verification by two senior laboratory personnel and verification using the autoverification rules in 99.78% of the cases. The total autoverification rate for all tests was 77.06%. Following implementation of the rules, the laboratory turnaround time (TAT) was reduced by 54.55% and staffing numbers fell from three to two whole time equivalents (WTE). Statistical analysis resulted in a kappa statistic of 0.99 (P < 0.001). Moreover, after implementing the autoverification rules, the error rate fell to 0.04%, indicating that errors were almost completely eliminated.

Conclusion

Implementing autoverification rules can reduce TAT, minimize the number of samples that require manual verification and allow for a reduction in staffing numbers. It also allows laboratory staff to devote more time and effort to the handling of problematic test results and contributing to improved patient care.

Keywords

Autoverification laboratory automation DM2 thyroid hormones sex hormones

Introduction

Currently, clinical laboratories are under continual pressure to increase their productivity, through the handling of larger workloads with fewer qualified staff. Following the parallel developments of increased laboratory automation and major developments in information technology (IT), many clinical laboratories in hospitals now utilize two-way communications between laboratory information system (LIS) and analytical instruments.^1,2 However, despite this, the speeds by which some test results are released remain compromised by the low efficiency of manual verification, which is performed by one or more senior member of staff, on a single analytical result or on a group of results, in order to ensure that no incorrect results are reported to the hospital information system (HIS) or to clinicians. Autoverification, a process by which clinical laboratory results are released without manual intervention or review,^3–9 can overcome this limitation.^3,10,11 The Autoverification of Clinical Laboratory Test Result Approved Guideline (AUTO 10-A) was issued by the American Clinical Laboratory Standards Institute (CLSI) in 2006.⁹ This guideline provides a basic framework to allow each clinical laboratory to design, implement and validate specific autoverification rules for specific tests according to a laboratory technologist’s demands. The use of such autoverification can enhance the efficiency of the laboratory, reduce budgetary costs and decrease turnaround time (TAT), as well as ensure quality¹² and enable laboratory staff to focus more on potentially problematic test results.⁴ Currently, many laboratories worldwide are exploring and implementing autoverification in several areas, including urine analysis,^13,14 haematology,^15,16 clinical biochemistry,^17–20 coagulation,^21,22 and clinical immunology.⁷ Although there is nearly a 20-year history of autoverification systems that can verify and validate clinical laboratory results and the Auto10-A guidelines have been around for nearly 10 years, there remains a lack of standardization, especially for the algorithms and verification limits,¹⁸ and it remains unclear how to build autoverification rules and parameters.^12,22

Although commercial autoverification systems, such as VALAB, are available, their processing algorithms and verifying rules are considered proprietary; therefore, they cannot be modified by the users.^10,23 Moreover, commercial software addresses only the most basic levels of autoverification, e.g. the reference interval, instrument alarm and internal quality control (IQC), and none of the commercially available programs can handle complex clinical data, such as testing logical relationships, clinical presentation and clinical history. As a result, clinical laboratories have built their own autoverification systems, such as DNSev™²⁴ and LabRespond.¹⁰ This can also be achieved by developing a within-laboratory autoverification system that would work as a part of the LIS or the middleware, which in our laboratory consists of Data Management 2 (DM2), and which was developed by Beckman Coulter.

Thyroid function and sex hormone tests are two examples of common clinically complex tests that are processed in the laboratory. Familiarity with the physiology and pathophysiology of the thyroid gland and the gonads is important to ensure appropriate requesting of the hormones they produce. A number of medications and diseases have been shown to influence the circulating concentrations of these hormones.^25,26 Circulating concentrations of sex hormones can also vary with the menstrual cycle and in patients undergoing treatment through assisted reproductive technology (ART). In those women receiving ART, circulating sex hormone concentrations differ from those found in normal or pregnant women,^27,28 presenting a challenge for the setting of any autoverification rules.^26,29–32 In this study, we present a detailed description of how to construct autoverification rules for thyroid function and sex hormone tests using the DM2 middleware, and describe their subsequent validation and evaluation in the clinic.

Materials and methods

Instrumentation

All equipment was provided by Beckman Coulter Inc., CA, USA (Power Processor Automated Sample Processing System, UniCel DxI 800 Immunoassay System, Prelink, Centrifuge, Specimen Stockyard, Outlet Rack System).

Methods

All tests were measured using chemiluminescent microparticle immunoassay (CMIA), on a Unicel DxI 800 immunoassay analyser (Beckman Coulter Inc., CA, USA). Thyroid function tests consisted of thyroid-stimulating hormone (TSH), total triiodothyronine (TT3), total thyroxine (TT4), free triiodothyronine (FT3) and free thyroxine (FT4).

Sex hormone tests consisted of: β-human chorionic gonadotropin (β-HCG), progesterone, testosterone, oestradiol, prolactin, follicle-stimulating hormone (FSH).

In addition, all internal quality control serum products were provided by Beckman Coulter Inc., CA, USA.

Software

The DM2 was obtained from Beckman Coulter Inc., CA, USA. The hospital information system (HIS) and Laboratory Information System (LIS) were both obtained from B-Soft Co., Ltd, Hangzhou, China.

Statistical analysis

Analysis of the data was performed using the SPSS statistics software package, version 20.0 for Microsoft Windows (SPSS Inc., Chicago, IL, USA). A P < 0.05 was considered significant.

A kappa test was used to evaluate the difference between autoverification and manual verification. The kappa coefficient represents the observed agreement above and beyond that due to chance. The strengths of any observed agreements were as follows: <0.20 bad; 0.20–0.40 common; 0.41–0.60 moderate; 0.61–0.80 strong and 0.81–1.00 very strong.

Methods

This study was conducted in the Clinical Chemistry Core Laboratory of the First Affiliated Hospital of Shantou University Medical College in cooperation with a medical expert from the Department of Reproductive Medicine, the DM2 vendor (Beckman Coulter, CA, USA) and the LIS vendor (B-Soft Co., Ltd, Hangzhou, China). Our hospital is a regional general academic tertiary care referral centre and university affiliated hospital with 67 clinical departments and 1806 beds. In this study, the DM2 was integral to the autoverification rules. We designed them by using Boolean logic according to the CLSI Auto10-A guidelines.⁹ We then simulated and validated their performance on the DM2 using historical patient data stored in the database. The details of the rules and the flowchart are shown in Figure 1. The flowchart shows that the rule processes consist of IQC, instrument error flags, critical value, instrument analytical range, limit range, consistency check and delta check.

Figure 1.

Flowchart of autoverification rules for thyroid function profiles and sex hormone tests.

Quality assurance

Our laboratory routinely uses IQC and takes part in external quality assessment. Daily IQC results were transmitted from the analyser to the DM2 and LIS and were evaluated using Levey-Jennings charts³³ and Westgard quality control multirules.^34,35 In addition, we also used the ‘moving average’ method as an additional quality assurance (QA) method to help ensure the quality of the results. This involves the collection of patient results over a period of 20 consecutive days. From these results, a mean result is determined, and a warning limit is calculated, which is the mean ±2SD, together with an action limit of the mean ± 3SD.³⁶ For example, in the case of FT4, over one period of 20 consecutive days, the warning limits (mean ± 2SD) were 10.72 and 13.96 pmol/L.

Instrument error flags

The instrument will give alerts when there are problems with the reagents, barcode, samples or mechanical failure, e.g. in the event of reagent crystallization or sample clots forming.

Critical value

The critical values or medical decision levels were determined locally and were based upon those described by Statland et al.³⁷ The critical values used in our hospital were as follows: β-HCG > 180,000.00 IU/L, oestradiol > 9177.50 pmol/L, LH > 40.00 IU/L, FSH >70.00 IU/L, testosterone < 1.04 nmol/L, prolactin >2120.00 mIU/L, FT3 > 40.00 pmol/L, FT3 < 1.50 pmol/L, FT4 > 22.00 pmol/L, FT4 < 3.50 pmol/L, TT3 >10.00 nmol/L, TT3 < 0.50 nmol/L, TT4 > 380.00 nmol/L, TT4 < 6.50 nmol/L, TSH > 80.00 mIU/L, and TSH < 0.02 mIU/L. Results that were outside the range of the critical value required verification by a technologist, and those within the range passed and continued to the limit check and delta check.

Instrument analytical range

The analytical measurement ranges were: β-HCG, 0.50–1000.00 IU/L (1000.00–200,000.00 IU/L following dilution); progesterone, 0.32–127.20 nmol/L; testosterone, 0.35–55.50 nmol/L; oestradiol, 73.00–17,621.00 pmol/L; prolactin, 5.30–4240.00 mIU/L; FSH, 0.20–200.00 IU/L; LH, 0.20–250.00 IU/L; TSH, 0.01–100.00 mIU/L; T4, 6.40–386.00 nmol/L; FT4, 3.20–77.20 pmol/L; T3, 0.20–12.30 nmol/L; FT3, 1.40–46.00 pmol/L. Results out with the analytical range generated a warning flag and required sample dilution prior to reanalysis.

Limit range

The limit range is the rule that screening test results must not be outside the critical values. It serves as a filter to verify whether the results are within the limit range specified for the analyte in the DM2. Fraser et al.³⁸ reported that conventional reference intervals are not usually ideal for autoverification strategies due to factors relating to biological variation, making conventional population-based reference intervals of little value. We discussed the limit range with local physicians, and the acceptable range for the limit check for all 12 tests was based upon the 95% confidence interval determined from the distribution of historical patient results obtained between October 2013 and October 2014 (Table 1).

Table 1.

Limit ranges of all the 12 tests.

Test	Limit range	Test	Limit range	Test	Limit range
TSH	0.01–12.15	FT4	6.43–34.15	Testosterone	0.35–15.34
TT3	0.54–3.91	Progesterone	0.41–110.25	Oestradiol	73.42–9704.66
TT4	46.35–222.17	HCG	0.50–171,512.00	Prolactin	77.59–1327.97
FT3	2.66–10.86	LH	0.45–46.88	FSH	1.62–68.37

FT3: free triiodothyronine; TT3: total triiodothyronine; TT4: total thyroxine; TSH: thyroid-stimulating hormone; FSH: follicle-stimulating hormone; LH: luteinizing hormone; FT4: free thyroxine; HCG: human chorionic gonadotropin.

Consistency check

Given the nature of acute illness and the fact that clinical test results fluctuate frequently, it was very difficult to perform a consistency rule check. Only some tests had a consistency check established based on clinical and practical diagnostic criteria. The hypothalamic pituitary-gonadal (HPG) and hypothalamic-pituitary thyroid (HPT) axes are negative feedback control systems. Therefore, there are some necessary relationships between the circulating concentrations of the hormones, e.g. elevated T4 or T3 concentrations associated with a decreased TSH concentration, in patients with toxic diffuse goitre. It was necessary that any consistency rule took account of the interrelationships of all of the hormones, and consequently, if any results violated the consistency check, test results were unable to be sent to the HIS. For example, if the test result for TSH was lower than 0.34 mIU/L, with a FT3 < 6.00 pmol/L or a TT3 < 2.73 pmol/L and FT4 < 14.40 pmol/L or TT4 < 157.40 pmol/L; or TSH > 0.34 mIU/L and FT3 > 6.00 pmol/L or TT3 > 2.73 pmol/L, the report was intercepted as a manual verification (MV) report and could not be sent to the HIS.

Delta check

The delta check compares current test results with previous results to identify those results that differ by more than a defined amount.^39,40 Using the delta check rule enables the identification of changes in the test results beyond their expected variation, e.g. due to changes in a patient’s clinical status or due to preanalytical errors.^41,42 We therefore believe that all test results that have previous test values must be verified by a delta check.⁴⁰ The parameters that have previously been used include the delta difference, rate difference, rate percent change, cumulative incremental weighted index^43–45 and delta percent change.⁴⁶ In this study, we chose the rate percent changes, which were calculated as the current value subtracted from the previous value and then divided by the previous value. We chose a time interval as less than seven days based on discussions of the biological and pathological functions of the thyroid gland and gonads with physicians in the Department of Reproductive Medicine. In addition, we suggest that if the rate of change of the concentrations of the hormones was within the set range in the previous seven days, the result could pass the delta check. The set ranges were: ±50%/7 days for TSH, TT3, FT3, TT4, progesterone, LH, testosterone, prolactin and FSH; ±70%/7 days for FT4 and oestradiol and ±80%/7 days for HCG.

Patient diagnostic information

All relevant data that could affect the interpretation of the tests results were collected according to the CLSI Auto-10 A guidelines.⁹ Patient-specific information and preanalytical variables included: gender; age; pregnancy status; menstrual cycle status; clinical presentation; thyroid history; pituitary and gonad disorders; drug history; presumptive diagnosis and other medical conditions that may affect the concentrations of the thyroid or gonad hormones, e.g. history of a vesicular mole, choriocarcinoma. Having patient information can be important in the interpretation of these test results. Because of this and because patient information cannot be sent to the DM2, we used both the LIS and DM2 to enable autoverification to be implemented safely. For example, if the primary diagnosis was hyperthyroidism, but test results of thyroid function were all within the reference intervals, the report would be handled by technologists and would not be sent to the HIS (Table 2).

Table 2.

Patient information associated with tests report results.

Number	Key diagnosis words	Tests results that stop autoverification
1	Hyperthyroidism	TSH > 4.85 mIU/L or FT3 < 3.60 pmol/L or TT3 < 1.12 nmol/L or FT4 < 7.86 pmol/L or TT4 < 91.90 nmol/L
2	Hypothyroidism	TSH < 0.51 mIU/L or FT3 > 5.70 pmol/L or TT3 > 2.41 nmol/L or FT4 > 14.41 pmol/L or TT4 > 167.80 nmol/L
3	Subacute thyroiditis	TSH > 4.85 mIU/L or FT3 < 3.60 pmol/L or TT3 < 1.12 nmol/L or FT4 < 7.86 pmol/L or TT4 < 91.90 nmol/L
4	Autoimmune thyroiditis	TSH < 0.51 mIU/L or FT4 > 14.41 pmol/L or TT4 > 167.80 nmol/L
5	Hyperprolactinemia	Prolactin < 530.00 mIU/L
6	Polycystic ovarian syndrome	FSH > 8 IU/L or LH < 10 IU/L or oestradiol < 99.12 pmol/L
7	Premature ovarian failure	FSH < 40 IU/L or LH < 40 IU/L or oestradiol > 100 pmol/L or progesterone > 2 nmol/L
8	Precocious puberty	LH < 5 IU/L or oestradiol < 275.33 pmol/L or testosterone < 0.52 nmol/L
9	Hydatidiform mole	β-HCG < 100,000 IU/L
10	Ectopic pregnancy	Progesterone > 79.50 nmol/L or β-HCG < 5 IU/L
11	Prolactinoma	Prolactin < 173.50 mIU/L
12	Sheehan's syndrome	FSH > 8 IU/L or LH > 10 IU/L or prolactin > 566.46 mIU/L or TSH > 4.85 IU/L or FT3 > 5.70 pmol/L or TT3 > 2.41 nmol/L or FT4 > 14.41 pmol/L or TT4 > 167.80 nmol/L
13	Ovarian hyperstimulation syndrome	Oestradiol < 3671.00 pmol/L

FSH: follicle-stimulating hormone; LH: luteinizing hormone; TSH: thyroid-stimulating hormone; FT3: free triiodothyronine; TT3: total triiodothyronine; FT4: free thyroxine; TT4: β-HCG: β-human chorionic gonadotropin.

The DM2 was installed in our Clinical Chemistry Core Laboratory, so we could use it to enable autoverification of the test results. The rules were written in computer language with reference to the literature;⁴⁷ the results that passed the autoverification rules were marked in green, and those that failed were marked in red to achieve autoverification in the LIS or DM2. The results that passed all the autoverification rules could be sent directly to the HIS. In our laboratory, the report for autoverified results contained ‘autoverification’ in place of the verifier’s signature as well as the programmed comment, if any. Cases that failed the autoverification rules were red flagged on the computer screen, indicating that the laboratory technologist needed to manually verify the results. In such cases, the verifier’s name was recorded on the reports and saved within the LIS. The details of autoverification rules and their actions are given in Table 3.

Table 3.

Autoverification rules and actions.

Rules	Action(s)
Quality control	Stop, identify reasons, and correct if QC unmet; conversely, pass
Instrument error flags	Stop and investigate if indicated; conversely, pass
Critical value	Stop and manually verify or recheck if exceeded, then call doctors; conversely, pass
Result above AMR	If manual protocol, perform dilution and manually verify; certain analytes have automated dilution protocol (Dil-HCG); once completed, proceed to other rules
Result below AMR	Stop and analyse if indicated; conversely, pass
Limit range	Stop and manually verify; conversely, pass
Consistency check	Stop and manually verify or recheck if violated, then communicate with doctors;
	conversely, pass
Delta check	If violated, do not autoverify; conversely, autoverify

AMR: analytical measurement range.

Validation methods

To validate whether the autoverification rules and their settings in the DM2 were able to meet our requirements and be implemented, we used electronically simulated cases, special samples and historical data, and the entire validation process was based on the recommendations in the CLSI Auto10-A.⁹

Electronically simulated case validation

Electronically simulated cases were used to verify that the programmed autoverification rules followed the expected logic and achieved the anticipated outcome. One case for each rule was previously programmed. The total number of cases was 1063. A total of 538 (50.6%) cases passed autoverification and 525 (49.4%) failed. The electronically simulated cases that passed the autoverification rules were reviewed in order to ensure that they matched the expected outcome.

Special sample validation

Special samples included the abnormal proficiency test samples and more than 50 patient samples, containing low and high concentrations of each analyte. Most of the test results fell outside the acceptable range of the critical value, instrument analysis and limit check. These tests results were chosen to verify the efficiency of the autoverification rules and the reliability of the reports. The validation results demonstrated that there were no errors in the autoverification rule use, which indicated that the reports generated using the rules were accurate.

Validation using historical data

The DM2 went live in the Clinical Chemistry Core Laboratory in 2013. All clinical chemistry test results of patients, from October 2013 to the present, were stored in the DM2 computer, which allowed us to validate the rules by using this historical data. The total number of cases, from October 2013 to December 2014, with thyroid and gonad hormones analysis in the DM2 were 47,448 and 23,983, respectively. All cases had been manually verified and revised by skilled laboratory specialists in the LIS. These cases were used to represent the same distribution and type of cases that are received by the laboratory, and they provided an estimate of the quality of autoverification. This validation process which uses many test results should uncover problems with both the autoverification rules and electronic equipment that occurred infrequently prior to autoverification. We had to ensure that those test results reported by autoverification were consistent with the clinical status of the patients. The use of historical data indicated that the rate of autoverification was 57.56%. In addition, the system and rules ran well and did not demonstrate any error flags.

Results

According to the CLSI Auto10-A Guideline,⁹ autoverification rules must be validated by using actual patient results before go-live. To confirm whether the autoverification rules were reliable, we established the rules in the DM2, and collected and assessed patient test results, of which there were 77400, produced during the calendar year 2015. The validation was performed by requesting that two senior technologists, who specialized in thyroid and gonadal disease and who worked in cooperation in the Clinical Chemistry Core Laboratory, verify the results in triplicate.

This study considered all factors, including the instrumentation, reagents, standard serum samples, specimen status, laboratory temperature and correlation of the hormones. Instrument flags appeared 45 times during validation of the rules. All IQC results were within limits, as judged by the Levey-Jennings quality control chart³³ and Westgard multi-rules.^34,35 The results of actual patient validation were as follows:

Table 4.

Passing rate of whole items.

Date	Status	Jan. 01	Mar. 01	May. 01	Jul. 01	Sep. 01	Nov. 01
The number of requisition sheets		210	220	209	232	208	242
Quality control	Pass^a	12	12	12	12	12	12
	Fail^a	0	0	0	0	0	0
Instrument error flags		4	6	5	4	3	3
Critical value	Yes	29	26	23	18	15	17
	No	177	188	181	210	190	222
Out of AMRs	Yes	3	4	5	3	6	4
	No	203	210	199	225	199	235
Limit range	Yes	122	134	130	148	137	155
	No	52	50	46	59	47	63
Consistence rule check	Pass	113	127	124	142	129	148
	Fail	9	7	6	6	8	7
Delta check	Pass	92	113	115	129	120	138
	Fail	21	14	9	13	9	10
No. of previous data		13	15	12	15	16	11
Number of autoverifications		95	116	117	130	122	142
Autoverification passing rate (%)		45.23	52.73	55.98	56.03	58.65	58.68
Number of manual interventions		115	104	92	102	86	100
Manual intervention rate (%)		54.77	47.27	44.02	43.97	41.35	41.32
Error rate (%)		0.48	0	0	0.43	0	0
Average autoverification passing rate (%)				77.06
Average rate of manual verification (%)				22.94
Average error rate (%)				0.04

AMRs: analytical measurement ranges.

QC results for thyroid function profiles and sex hormone tests. Bold numerals represent the number of results that passed the autoverification rules.

The autoverification rate for all thyroid function profiles and sex hormone tests ranged from 65.35% to 77.06% (Table 4 ). The autoverification rate for thyroid function profiles and sex hormone tests implemented in the Clinical Chemistry Core Laboratory increased over the period of 12 months from 65.57% to the current overall rate of 86.78% and from 52.78% to 63.70%, respectively.

A high percentage of results with critical values were autoverified. The average rates of results exceeding the critical value, limit range and delta check were 7.21%, 25.20% and 10.97%, respectively.

The reported laboratory TAT was reduced by 54.55% (from 66 to 30 min), and the time interval from completion of analysis to verification was reduced by 61.80%. The staffing numbers were reduced from three to two whole time equivalents (WTE).

Evaluation of the benefits

We used the TAT (from the time of sample receipt by the laboratory staff to the time of result verification on the LIS) and risk of human verification error to evaluate the advantages of implementing autoverification for the calendar year 2015. Following the implementation of autoverification, the number of samples that required manual verification decreased by 77.06% (from 77,400 to 59644, Table 4). Furthermore, the TAT fell by 54.55% (from 66 min to 30 min), findings, which are consistent with previous reports that described the use of autoverification rules as a way of reducing TAT. This approach also reduces the heavy workload in clinical laboratories.^4,20 In addition, our study demonstrates the additional advantage of improved consistency in the verification process over that undertaken by different clinical medical technologists, since the rules were based on the same standardized processes and all results were therefore treated in the same consistent manner. Before implementation, manually verified errors occurred at an average of approximately four results or fewer per day. In contrast, after implementing the rules, errors were almost eliminated in a ‘live’ clinical environment, using the observations of two senior technologists as a reference standard. The results of the kappa test for the inter-observer degree of agreement, indicated that there was a close agreement between results from professional reviewers and the results obtained following autoverification (Table 5).

Table 5.

Degree of agreement between the autoverification rules and two senior technicians.

		Autoverification rules		Agreement	Disagreement	Kappa value	Kappa approximate significance
Reviews		Pass	Fail	Agreement	Disagreement	Kappa value	Kappa approximate significance
Two senior	Pass	61484	123	99.78%	0.22%	0.99^a	P = 0.00^b
technicians	Fail	49	15744	(77,230)	(170)	0.99^a	P = 0.00^b

Degree of agreement: <0.2 poor; 0.2–0.4 fair; 0.41–0.6 moderate; 0.61–0.8 good; 0.81–1.0 very good. ^bHighly significant difference, indicating the agreement between the autoverification rules and two senior technicians was not caused by accident.

Discussion

Autoverification is a process of using computer-based rules to undertake the initial validation of test results without manual intervention. Data that fall outside the set rules should be reviewed by the laboratory technologists and those that pass the set rules can be directly released by the computer.^6,7,9,20,48 Although the autoverification of clinical tests results is an essential tool with which to increase the accuracy and efficiency in clinical laboratories,^6,48 there is relatively scant information on its practical applications within a clinical laboratory setting.^18,20 In this paper, we described the establishment and application of autoverification in a busy clinical chemistry core laboratory.

Thyroid function and sex hormone tests were chosen for the initial evaluation of the rules because they are tests that have a complex, but defined relationship with other system functions. Furthermore, they have clearly defined reference intervals and well-defined cut-off values for certain clinical abnormalities, and they represent the main bulk (85%) of hormone cases in our laboratory.

As shown in Figure 1, the autoverification rules utilize the following parameters: (1) IQC check; (2) initial screening rules check (including instrument error flags, critical value and out of analytical measurement range); (3) limit range; (4) consistency check; (5) delta check. The IQC check confirms whether each item falls within the acceptable range before the autoverification rules start; otherwise, the verification procedure will be halted according to the preconditions for autoverification.⁹ The IQC system should be integrated with the autoverification rules, e.g. when IQC failure occurs, the autoverification rules of the given test items will be automatically stopped to enable a qualified person to analyse the reasons for the failure and correct the problems. Then, once IQC is within limits, the autoverification rules would be restarted.

The critical value is important for enabling physicians to make a diagnosis. To the best of our knowledge, it is unusual to find published data on the critical values for autoverification. Our process allows the autoverification of critical value to proceed as long as no other rules are violated. In addition, critical values still need to be communicated to the clinical locations from which the requests originated. However, the communication of these results by the laboratory is facilitated by clinicians often having access to results, e.g. following their display on ward-based terminals, that have undergone autoverification, prior to any telephone call being made.

Because test results fluctuate substantially during acute illness, it was difficult to perform a consistency check on each item, and only portions of the test items were confirmed based on practical and clinical diagnostic criteria. By using the consistency rule check, we would avoid most of the base errors. However, the delta check could help us quickly find changes in a patient’s condition and identify any preanalytical errors.

We chose the total agreement between the two professional reviewers as a benchmark to maximize patient safety. A similar method was applied in the VALAB system.¹¹ There was a significant degree of consistency between our autoverification rules and the manual verification undertaken by two senior technicians (kappa value = 0.99, P < 0.001), and consistency was observed in 99.78% of cases. There were only five cases when the two reviewers decided to stop autoverification due to any inconsistency. These cases involved thyroid function tests in which the related rules stated that if the TSH is <0.01 mIU/L, TT4 is within limit range, and TT3 is above the limit range, one should autoverify and comment (possible T3 toxicosis). A high TSH alone was found in 29 cases; 34 cases had a T4 assay within normal limits and another 24 cases had both T3 and T4 within normal limits with a high TSH. In the previously mentioned 87 cases, the TSH values were below a five-fold increase in the upper reference limit (URL), and no clinical data were provided to support the diagnosis. According to previous research,²⁹ we did not verify those cases because the TSH value did not diagnose thyroid disease with certainty. However, they were verified manually by the reviewers. Therefore, despite these discrepancies, we did not believe it necessary to change any area of the autoverification rules to achieve the maximum possible patient safety. The biggest risk involved in using the rules is that some test results may be released without proper review or editing. The interpretation of sex hormone results is considered by some to be more complicated than the interpretation of thyroid function tests, and the hormone concentrations can change due to various factors, e.g. the menstrual cycle, ART.^27,28 Therefore, we believe that caution is needed in the interpretation of test results for sex hormones.

Most publications describe autoverification rules as an approach to shortening the TAT in reporting test results, through reduction in manual intervention, which in turn can facilitate staffing reductions in clinical laboratories.^4,20 In our laboratory, the implementation of the rules resulted in the staffing WTE numbers in the Clinical Chemistry Core Laboratory falling from 8 to 6. Furthermore, because results that undergo autoverification are released immediately, the TAT of patient reports was dramatically reduced in comparison to those obtained previously with manual verification.

Our experience has suggested that the TAT is the parameter that is the most affected by the introduction of autoverification rules. However, this improvement was reduced when comparing the average TAT values for the year. Our reduction in the TAT was 54.55%. This fall in improvement was due to the simultaneous increase in the average daily workload for hormone analysis in the clinical chemistry core laboratory from 51,367 samples to 77,400 samples (a 150.68% increase). However, we believe that the TAT will further improve following the full utilization of all the established programmed rules. McFadden⁴⁹ indicated that with the application of autoverification rules, there could be up to a 44% savings in time and labour capacity for the laboratory staff. We found that the autoverification rate obtained using historical patient data was 77.06%; this is close to that of the VALAB system which has a mean autoverification success of approximately 50% to 90%.¹¹ However, VALAB was applied on a wider range of tests. Our success rates are lower than that of the DNSev™ system, which shows a verification rate of approximately 80%.²⁴ These differences could be explained by differences in the studied group of tests, patient presentations and laboratory equipment used. In our study, the autoverification rates of sex hormone single tests were lower. For example, the autoverification rates of oestradiol and LH were 76.65% and 72.97%, respectively. We initially wanted to include menstrual status and pregnancy in a limit range. However, due to time constraints, this was not possible and it is our aim to develop this area in the future when time permits.

Because autoverification rules are relatively new to laboratory test result reporting, the scientific literature on the subject is limited. In this study, we found that the greatest benefit following the implementation of the rules is in the consistency of the test results, as previously observed.^4,7,20 In contrast, despite the advantages of autoverification, potential disadvantages could include an increase in false-negative results and the reporting of erroneous results due to analytical interference, such as when a partial or tiny clot is not detected by the instrument. Mechanical errors of this kind, although few, remain unavoidable. Our study also has some limitations. The limited connectivity between the LIS and HIS resulted in the disabling of any rules associated with any patient clinical information or drug history, which prevented us from taking full advantage of the programmed rules. Although we could make a connection with the keywords of the diagnosis, this work-around approach is insufficient. For the time being, we can only overcome this drawback by registering the previously established comprehensive patient sheets.

In conclusion, we developed and implemented a rule-based autoverification, according to Boolean logic and the guidelines of ‘Design of Algorithms’ in the Auto-10A document that utilizes clinical tests data and other parameters. The autoverification rate of the rules was comparable with that obtained using commercially available software for other systems. The rules that we designed can shorten the TAT, reduce manual data entry, and decrease the probability of errors associated with human review, as well as, minimize the number of samples requiring technologist intervention. However, there was a small sample size in the current study and the implementation time was short (only three months). In the future, we will continue implementing autoverification rules for thyroid function profiles and sex hormones, and add other tests to construct a more complete expert system.

Footnotes

Acknowledgements

We thank Yongxin Qiu, from Beckman Coulter Inc., for computer and DM2 technical support and Linli Fang and Yingqiu Zu, the senior technicians in the Clinical Chemistry Core Laboratory, for result verification. Additionally, we thank Dr. Lin, a professor of Shantou University Medical College (SUMC), for his review of the manuscript.

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) received no financial support for the research, authorship, and/or publication of this article.

Ethical approval

Not applicable.

Guarantor

JL and YC.

Contributorship

All authors reviewed and edited the manuscript and approved the final version of the manuscript. Due care has been taken to ensure the integrity of the work.

References

Guidi

Lippi

. Laboratory medicine in the 2000s: programmed death or rebirth? Clin Chem Lab Med 2006; 44: 913–917.

Panteghini

. The future of laboratory medicine understanding the new pressures. Clin Biochem 2004; 25: 207–215.

Valdlgulie

Rogari

Pilippe

. VALAB: expert system for validation of biochemical data. Clin Chem 1992; 38: 83–87.

Torke

NBL

Nguyen

Perri

et al.

Process improvement and operational efficiency through test result autoverification. Clin Chem 2005; 51: 2406–2408.

Crolla

Westgard

. Evaluation of rule-based autoverification protocols. Clin Leadersh Manag Rev 2003; 17: 268–272.

Guidi

Poli

Bassi

et al.

Development and implementation of an automatic system for verification, validation and delivery of laboratory test results. Clin Chem Lab Med 2009; 47: 1355–1360.

Cheng

Yang

et al.

Development and implementation of autoverification rules for ELISA results of HBV serological markers. J Lab Autom 2016; 21: 642–651.

College of American Pathologists. Laboratory Accreditation Program, Laboratory General Checklist. Commission on Laboratory Accreditation, College of American Pathologists, 2013.

CLSI. AUTO10-A. Wayne, PA: CLSI, 2006.

10.

Oosterhuis

Ulenkate

Goldschmidt

. Evaluation of LabRespond, a new automated validation system for clinical laboratory test results. Clin Chem 2000; 46: 1811–1817.

11.

Prost

Rogari

. How autoverification through the expert system VALAB can make your laboratory more efficient. Accred Qual Assur 2002; 7: 480–487.

12.

Jones

. A strategic informatics approach to autoverification. Clin Lab Med 2013; 33: 161–181.

13.

. Autoverification of macroscopic urinalysis. Lab Med 1999; 33: 56–60.

14.

Stankovic

DiLauri

. Quality improvements in the preanalytical phase: focus on urine specimen workflow. Clin Lab Med 2008; 28: 339–350.

15.

Smith

NRD

Watman

. Hematology autovalidation system. Lab Hem 1999; 5: 52–55.

16.

Butch

. Automation in the transfusion service. Immunohematology 2008; 24: 86–92.

17.

Shih

Chang

Tien

et al.

Building and validating an autoverification system in the clinical chemistry laboratory. Lab Med 2011; 42: 668–673.

18.

Gómez-Rioja

Alvarez

Ventura

et al.

Current status of verification practices in clinical biochemistry in Spain. Clin Chem Lab Med 2013; 51: 1739–1746.

19.

Sediq

Abdel-Azeez

. Designing an autoverification system in Zagazig University Hospitals Laboratories: preliminary evaluation on thyroid function profile. Ann Saudi Med 2014; 34: 427–432.

20.

Krasowski

Davis

Drees

et al.

Autoverification in a core clinical chemistry laboratory at an academic medical center. J Pathol Inform 2014; 5: 1–17.

21.

Davis

. Rules-based system for autoverification of routine coagulation testing. Lab Hem 2008; 30: 58–147.

22.

Zhao

Yang

Zheng

et al.

Building and evaluating the autoverification of coagulation items in the laboratory information system. Clin Lab 2014; 60: 143–150.

23.

Fuentes-Arderiu

Castineiras-Lacambra

Panadero-Garcia

. Evaluation of the VALAB expert system. Eur J Clin Chem Clin Biochem 1997; 35: 711–714.

24.

Dorizzi

Caruso

Meneghelli

et al.

The DNSev™ expert system in the auto-verification of tumour markers and hormones results. Accred Qual Assur 2006; 11: 303–307.

25.

Roef

Taes

Toye

et al.

Heredity and lifestyle in the determination of between-subject variation in thyroid hormone levels in euthyroid men. Eur J Endocrinol 2013; 169: 835–844.

26.

Yang

Choi

. Efficacy and safety of metformin or oral contraceptives, or both in polycystic ovary syndrome. Ther Clin Risk Manage 2015; 11: 1345–1353.

27.

Ortega-Hrepich

Polyzps

Anckaert

et al.

The effect of ovarian puncture on the endocrine profile of PCOS patients who undergo IVM. Reprod Biol Endocrinol 2014; 12: 1–6.

28.

El-Talatini

Taylor

Konje

. Fluctuation in anandamide levels from ovulation to early pregnancy in in-vitro fertilization-embryo transfer women, and its hormonal regulation. Hum Reprod 2009; 24: 1989–1998.

29.

Demers

LM, S. C

. Tietz textbook of clinical chemistry and molecular diagnostics, 4th ed. St Louis: Elsevier Saunders, 2006, pp. 2053–2095.

30.

Vadiveloo

Donnan

Murphy

et al.

Age- and gender-specific TSH reference intervals in people with no obvious thyroid disease in Tayside, Scotland: the Thyroid Epidemiology, Audit, and Research Study (TEARS). J Clin Endocrinol Metab 2013; 98: 1147–1153.

31.

Zhang

Post

Cheng

et al.

Thyroid hormones and electrocardiographic parameters: findings from the third national health and nutrition examination survey. PloS One 2013; 8: e59489–e59489.

32.

Melmed

Casanueva

Hoffman

et al.

Diagnosis and treatment of hyperprolactinemia: an Endocrine Society clinical practice guideline. J Clin Endocrinol Metab 2011; 96: 273–288.

33.

Levey

Jennings

. The use of control charts in the clinical laboratory. Am J Clin Pathol 1950; 20: 1059–1066.

34.

Westgard

Barry

Hunt

et al.

A multi-rule Shewhart chart for quality control in clinical chemistry. Clin Chem 1981; 27: 493–501.

35.

Westgard

. Internal quality control: planning and implementation strategies. Ann Clin Biochem 2003; 40: 593–611.

36.

Hayashi

Ichihara

Kanakura

et al.

A new quality control method based on a moving average of “Latent Reference Values” selected from patients, daily test results. Rinsho Byori 2004; 52: 204–211.

37.

Statland

. Clinical decision levels for lab tests, Ann Arbor, MI: Medical Economics Books, 2008.

38.

Fraser

Stevenson

Kennedy

IMG

. Biological variation data are necessary prerequisites for objective autoverification of clinical laboratory data. Accred Qual Assur 2002; 7: 455–460.

39.

Garner

Lewington

Barth

. Detection of patients with acute kidney injury by the clinical laboratory using rises in serum creatinine: comparison of proposed definitions and a laboratory delta check. Ann Clin Biochem 2012; 49: 59–62.

40.

Ovens

Naugler

. How useful are delta checks in the 21 century? A stochastic-dynamic model of specimen mix-up and detection. J Pathol Inform 2012; 3: 5–5.

41.

Strathmann

Baird

Hoffman

. Simulations of delta check rule performance to detect specimen mislabeling using historical laboratory data. Clin Chim Acta 2011; 412: 1973–1977.

42.

Chakraborty

Bhattacharya

Das

et al.

Delta check: a must in the management of hyponatremia. Clin Chem Lab Med 2013; 51: e161–e162.

43.

Yamashita

Ichihara

Miyamoto

. A novel weighted cumulative delta-check method for highly sensitive detection of specimen mix-up in the clinical laboratory. Clin Chem Lab Med 2013; 51: 781–789.

44.

Lok

. Chronic hepatitis B. N Engl J Med 2002; 346: 1682–1683.

45.

Torbenson

Thomas

. Occult hepatitis B. Lancet Infect Dis 2002; 2: 479–486.

46.

Tran

Cembrowski

Lee

et al.

Application of 3-D Delta check graphs to HbA1c quality control and HbA1c utilization. Am J Clin Pathol 2008; 130: 292–298.

47.

Duco

. Autoverification in a laboratory information system. Lab Med 2002; 33: 21–25.

48.

Torke

Boral

Nguyen

et al.

Process improvement and operational efficiency through test result autoverification. Clin Chem 2005; 51: 2404–2406.

49.

S, M., McFadden

. Evaluation of autoverification system which used in our laboratory. Adv Admin Lab 2006; 15: 64–64.

Designing and evaluating autoverification rules for thyroid function profiles and sex hormone tests

Abstract

Purpose

Materials and methods

Results

Conclusion

Keywords

Introduction

Materials and methods

Instrumentation

Methods

Software

Statistical analysis

Methods

Quality assurance

Instrument error flags

Critical value

Instrument analytical range

Limit range

Consistency check

Delta check

Patient diagnostic information

Validation methods

Electronically simulated case validation

Special sample validation

Validation using historical data

Results

Evaluation of the benefits

Discussion

Footnotes

Acknowledgements

Declaration of conflicting interests

Funding

Ethical approval

Guarantor

Contributorship

References