The accuracy of real-time procedure coding by theatre nurses: A comparison with the central national system

Abstract

Accurate and timely clinical data are required for clinical and organisational purposes and is especially important for patient management, audit of surgical performance and the electronic health record. The recent introduction of computerised theatre management systems has enabled real-time (point-of-care) operative procedure coding by clinical staff. However the accuracy of these data is unknown. The aim of this Scottish study was to compare the accuracy of theatre nurses’ real-time coding on the local theatre management system with the central Scottish Morbidity Record (SMR01). Paired procedural codes were recorded, qualitatively graded for precision and compared (n = 1038). In this study, real-time, point-of-care coding by theatre nurses resulted in significant coding errors compared with the central SMR01 database. Improved collaboration between full-time coders and clinical staff using computerised decision support systems is suggested.

Keywords

clinical coding computerised medical records systems data collection operative surgical procedures

Introduction

All healthcare organisations require accurate and timely clinical data as a resource for clinical and organisational purposes, including audit, epidemiology, finance and commissioning. Clinical coding is carried out using one of the many international classifications of disease. Examples include ‘Current Procedural Terminology’ (USA), the Australian Classification of Health interventions (Australia) and OPCS4 (Office of Population, Censuses and Surveys Classification of Surgical Operations and Procedures (fourth revision) which is used in the UK.¹ As well as different systems, there are also wide variations between countries in the support given to their health informatics systems, and the training and qualifications of their coders. In the USA, for example, there is a strong informatics career infrastructure which is geared towards financial and reimbursement services. Coding accuracy depends on a large number of factors, such as the use of credentialled coders, educational and career opportunities, and quality feedback checks. Recent efforts to improve coding efficiency, accuracy and timeliness include the development of computerised decision support systems and the promotion of ‘real time’ coding at the time and point of care.

In Scotland, all National Health Service (NHS) surgical procedures are coded using OPCS4 and this contributes to the national Scottish Morbidity Record (SMR01) database.^2,3 The data are collected by full-time, trained clinical coders (non-clinicians) working in the Health Records department. Coding is performed using the hospital discharge summaries, with a case note review only if difficulties are encountered. Quality assurance of the coding and accreditation of Health Boards is carried out nationally ‘every few years’ by the Information Services Division (ISD) and aims for a minimum of 90% accuracy.

However, in 2006 the Scottish Government National Theatres Project Report⁴ recognised that theatre data were not an accurate representation of activity and that quality improvement was required to improve completeness, reliability, accuracy, validity and ownership. The aim was to use accurate and timely data to drive organisational change and improve health provision.^5,6 The same document stressed the need to move towards using more ‘real-time’ data collection as an effective management resource.

One such initiative was ‘The Surgical Profiles Project’. This initiative used surgical hospital discharge data to provide a profile of surgical care both at health-board level and for the individual surgeon.⁷ However, project reviews found that the lead time to data collection was often too long, out of date and lagged behind organisational change. Delays often occurred, for example, as a result of late submission of data by the clinicians to the hospital coders (there is now a six-week target for submission). Such initiatives, and unprecedented public access to data granted by the Freedom of Information (Scotland) Act 2002,⁸ have added to the concerns within the clinical community about data accuracy.

In theory, at least, incorporating clinical coding into the daily workflow pattern (real-time coding) could reap major benefits and clinical teams may indeed be ideally placed to produce accurate timely data, if provided with sufficient knowledge and skills. In the UK, this is recognised by the Royal College of Nursing which advocates that nurses should be at least be aware of coding—recognising this as a potential new nursing role.⁹

The opportunity to do this study arose in our local hospital when a new, computerised surgical theatre management system was introduced (Operating Room Management Information System: ORMIS (iSoft, Banbury, England). This system incorporated mandatory procedural ‘real-time’ coding at the point of care and became part of the nursing workflow pattern. The nursing staff received a short period of training (usually a half-day session) in coding. They were shown how to enter codes and how to access the computerised search engine for OPCS4 codes. No change in coding was made at hospital level and the same surgical procedures were coded in parallel by the full-time trained clinical coders using hospital discharge summaries.

Aims and objectives

The aim of the study was to compare the accuracy and quality of theatre procedural data collected at point-of-care (‘real-time’) with the SMR01 data collected by the full-time hospital coders. Previous studies have shown major discrepancies between coding performed by clinicians for small local databases and that of nationally collected data;^10–12 some have highlighted the financial implications.¹³ This study aims to examine the accuracy of mandatory, real-time procedural coding using a computerised theatre management system when it is incorporated into the routine workflow of nursing care.

Methodology

This was a retrospective cohort study in which data were collected on all procedures carried out in two theatres (in a 10-theatre complex) for the six month period July 2007–December 2007.

One of theatres was used for elective orthopaedic work, the other was a general theatre used principally for general surgical emergencies (urgent and semi-urgent unscheduled care). These theatres were chosen to represent a cross-section of elective and emergency work.

Procedural coding was carried out by theatre nurses at the time of point-of-care (‘real-time’) using the theatre management system (ORMIS). The same surgical procedures were coded in parallel by the full-time, trained clinical coders using hospital discharge summaries. The two computer systems were linked for patient identification and demographics but were not interoperable for coding purposes. All data were examined retrospectively after discharge summary coding was performed. Neither nurses nor fulltime coders were aware of the study.

For the given study group of patients, the following OPCS4 coding results were compared.

Theatre nurses’ ‘real-time’ code of procedure(ORMIS).

Discharge code by the full-time, trained coders (SMR01; national data).

Information Services provided the following information for the sample by searching the relevant databases:

patient identifier(s);

date of procedure;

theatre nurses’ OPCS code recorded on ORMIS and associated text description of the OPCS4 code;

all SMR01 validated codes (OPCS4) and associated text description of the OPCS4 code.

ORMIS coding allowed up to two codes with associated free text. SMR01 data had up to four primary codes. It is unlikely this difference impacted on the final result as it was rare to have more than two procedures during any one theatre visit (and the associated free text was available for scrutiny). SMR01 data often included procedures carried outwith the theatre complex during a single patient inpatient stay e.g. chemotherapy administered in the wards.

Initial analysis of the data provided:

the total number of patient procedures carried out within the study group;

the number matching and non-matching paired codes (ORMIS vs SMR01).

Each of the paired procedural codes (ORMIS vs SMR01) were compared and graded according to their similarity and clinical acceptability by two of the authors (one clinician and one professional coder). Initially, these were carried out independently and then in collaboration. For the purposes of the comparison, the SMR data were taken as the gold standard and assumed to be accurate.

A five-point grading scale was developed to compare the procedural information provided by each of the paired codes (ORMIS vs. SMR01) (Table 1).

Table 1.

Grading: Degree of match of paired codes (ORMIS and SMR01)

Grade	Description
A	Exact match
B	Basically similar but SMR is more specific than ORMIS
C	Basically similar but ORMIS is more specific than SMR
D	ORMIS offers complementary (alternatively correct) or additional information to SMR
E	ORMIS does not match SMR

Early on in the project it became clear that there were many discrepancies in coding (ORMIS vs SMR01), some of which were clinically understandable.

Pragmatically, in order to manage the data in a systematic fashion, the authors developed subjective ‘rules for comparison’ to handle common, but understandable, discrepancies associated with inter-rater variability (see Table 3 for examples.) Grading would not have been possible without these additional qualitative and subjective rules. However, these rules may have affected granularity and introduced bias of a qualitative nature.

Additional rules for comparison of paired codes

A code was regarded as more specific if it was:

more specific or precise for clinical purpose;

more specific or precise for clinical or management audit purposes;

more specific or precise for theatre-management purposes (equipment, manpower or ‘operation time’ sensitive information).

Site of operation was regarded as important

Laterality (left/right) was not regarded as relevant for this study.

Frequently applied rules

Hip replacements

If there was a discrepancy in the use of cement/not cement, SMR01 was regarded as correct and more specific (code B).

Arthroscopies

‘Therapeutic’ was regarded as more specific than ‘Diagnostic’. ‘Irrigation’ was regarded as a ‘Therapeutic’ procedure.

Appendectomies

Emergency and non-emergency terms were disregarded. Normal and abnormal terms were disregarded.

Breast surgery

Drainage of a breast lesion was not regarded the same as excision of a breast lesion.

Ectopic pregnancies

Where the ORMIS code referred to a pregnancy (often associated with an ectopic pregnancy) but the SMR01 code did not, this was graded as ‘D’ (offering complementary or additional information).

Data handling

All real-time (ORMIS) and SMR01 coding was to the same system (OPCS4).

A total of 1049 procedures were recorded in ORMIS for the two theatres during the sample time (July 2007–Dec 2007). This was deemed correct, as it is mandatory for all theatre procedures to be entered into ORMIS. This represented 29% of the theatre complex non-obstetric workload.

Six hundred and sixty-six of these ORMIS procedures were paired with the SMR01 data with relative ease matching Community Health Index (CHI) numbers and date of procedure. A second and more time-consuming computer search was required to trace the missing cases. This difficulty arose because the date of the procedure (ORMIS) did not match the date recorded in SMR01 (usually the discharge date).

A particular difficulty arose with 31 cases where a change in hospital computer systems led to a change in the unique patient identification numbers (uptake of CHI numbers). A further search on patient name, date of birth and address was required to trace these cases. In 11 (0.1%) of these cases, SMR did not record a procedural code for the sample time and these cases were excluded from the totals. This may have been as a result of mistakes in the data search or may have been a result of omissions in the SMR01 database. The total remaining number of paired procedures, therefore, was 1038.

Results

Table 2 summarises the results for all paired operative procedures.

Table 2.

Results for all paired operative procedures

Grade	Description of code	Number of cases	Percentage of cases
A	Exact match	424	40.8
B	Basically similar but SMR was more specific than ORMIS	365	35.2
C	Basically similar but ORMIS was more specific than SMR	25	2.4
D	ORMIS offers complementary (alternatively correct) or additional information to SMR	79	7.6
E	ORMIS did not match SMR	145	14
Total		1038	100

Table 3.

Examples of discrepancies in each grade

Grade	ORMIS code	Description	SMR01 code	Description
A	H01.9	Unspecified emergency excision of appendix	H02.9	Unspecified other excision of appendix
B	W87.9	Unspecified diagnostic endoscopic examination of knee joint	W82.2	Endoscopic resection of semilunar cartilage
C	H33.5	Rectosigmoidectomy, and closure of rectal stump and exteriorisation of bowel	H11.8	Other specified other excision of colon
D	L91.2	Insertion of central venous catheter	X72.9	Unspecified delivery of chemotherapy for neoplasm
E	H15.1	Loop colostomy	M29.2	Endoscopic insertion of tubal prosthesis into ureter

Adopting our ‘rules for coding’ and using the SMR data as the ‘gold standard’, we found the ORMIS data:

accurate in 43.2% of cases (Groups A and C);

quite accurate but less precise in 35.2% of cases;

offered alternative or additional information in 7.6% of cases;

completely inaccurate in 14% of cases.

Table 3 lists an example for each grade. The examples B, C and D demonstrate differences in precision which, although clinically understandable, may not be acceptable for an electronic health record.

Discussion

There is a need for accurate and timely information and this is recognised by politicians, clinicians, managers and the general public alike. Studies performed internationally confirm that accurate and timely data are required for clinical management, resource management and financial purposes. In the UK, for example, Burns et al. focus on the accuracy of data for administrative use; in the USA, Mitchell et al. focus on financial impact; and, in Australia, many studies focus on the training, knowledge and resources of those who input data.^14–16

It is a stated aim of the Scottish Government to use available computerised clinical data as a resource to guide organisational change with the aim of improving patient care.⁷ Accurate, timely coding is to be a critical instrument in facilitating this process.

This study highlights the difficulties and traps inherent in clinical coding; coding is unable to record the complexity, granularity and variability of information which can be transferred by verbal communication or free text.¹⁷ In comparing the ORMIS codes with the SMR01 data, it became necessary to adopt a qualitative approach using ‘rules’ for comparison which would allow us to grade the coded information. This was necessary to compensate for inter-rater variability and for a complex system of clinical coding where minor variations in coding may be clinically acceptable for the same procedure. This qualitative approach may also have reduced granularity.

In this study, real-time point-of-care coding by nursing staff was compared with the SMR01 database, the ‘gold standard’ for the purposes of this study. The qualitative system developed to grade the accuracy of information showed that the real-time coding recorded the correct code in only 43.2% of cases. Clinically ‘similar’ information was entered in a further 37.6% of cases. The SMR01 data provided more specific or precise information in 35.2% of cases. In 14% of cases, the theatre coding was completely different to the SMR01 codes. This is of some concern as it significantly below the currently accepted level of accuracy (>90%) required by Information Services Division (ISD) for SMR01 procedural coding.¹⁶ The results show a major discrepancy between the real-time coding of nurses and that of the full-time hospital coders. Reliance on this real-time data, for example, as part of the electronic health record, could have a serious adverse impact on patient care and safety. Furthermore, reliance on aggregate real-time data could be detrimental to organisational policy planning, and the surgical profiles of quality, safety and effectiveness would be flawed.

In theory, it should be possible, using modern computer systems, to obtain accurate, timely data using real-time point-of-care coding. This study showed that discrepancies in data accuracy arose when this workflow model was adopted and there may be several reasons for this. The causes of the discrepancies were not part of this study but are likely to be related to many factors, including lack of education, training, feedback quality control systems and lack of resource support for the nurses who were working in a very busy theatre environment. The computerised search facility to obtain codes in ORMIS was, however, considered poor and the time available for the nursing staff to do coding was limited during busy operating lists (personal communication). In addition, with the exception of patient demographics, there was no computer link between the hospital patient-management system (PMS) and ORMIS. The hospital’s full-time coders had no access to the theatre information system and there was no communication or organisational link between the full-time coders and the theatre staff. The net result was that procedural coding was duplicated—separated by time, location and administration.

SMR01 undergoes centralised quality assurance (by full time non-clinical coders) with a claimed accuracy of >90%, but studies by doctors have challenged this.

Milburn compared a locally held clinical database with the central data and only found an 84% ‘acceptable clinical match’ with the central SMR data.¹⁰

Raza et al. also compared their local (senior clinician led) vascular surgery database with central SMR01 data. They found that 14% of major operations carried out during the year had not been recorded on the SMR01 database and that the complexity of operations carried out was not fully reflected in the centrally recorded data.¹¹

Colville compared the coding of plastic surgery operations by theatre nursing staff and concluded that clinical coding in theatre was so poor it should stop. He also found that the coding of the full time coders was 25–33% inaccurate owing to the complexity of coding.¹⁸

The high discrepancy rates in coding in these studies may reflect an increase in inter-rater variability or a difference in granularity between the doctors and the full-time coders. However, these studies by senior doctors do seem to highlight the need to improve coding in order to meet their clinical needs; perhaps through organisational change using the current professionally-trained coders or perhaps by engaging clinicians, such as doctors and nurses, in the process. Haliasos, for example, found that the accuracy of neurosurgical coding could be improved by engaging surgeons in the process.¹⁹

OPCS4 coding takes time and can be complex. Despite the aforementioned studies, central SMR01 data are still considered to have a minimum 90% accuracy. In our study, we compare point-of-care coding by theatre nurses (a new workflow model) with the coding by the full-time, non-clinical coders (the SMR01 database). Our results pertain to the use of ORMIS within our theatre workflow and may therefore not be generalisable. ORMIS was generally regarded by the nurses as having a poor search facility for finding codes and this may have increased error rates. However, it is known that computer-assisted coding systems have been used successfully elsewhere to increase the productivity of coding clerical staff.²⁰

Our study indicates caution in moving away from our model of using trained professional coders using hospital discharge summaries. The difficulty with this traditional model, however, is the inbuilt delay and the ‘distancing’ of coding from the clinical interface. The challenge is to link the organisation’s clerical coding structure, workflow and expertise with the clinical knowledge, workflow, and expertise. Further research is required and collaboration between hospital coders and clinical staff, together with interoperable computer systems may be indicated. Quality assurance feedback is of the utmost importance, however, no matter which workflow model is adopted.

Conclusion

In this study, real-time, point-of-care coding by theatre nurses resulted in significant coding errors compared with the central SMR01 database. Coding is highly specialised work and this study confirms the difficulty in cross training clinical staff, such as nurses, and the difficulties encountered when coding is embedded in the workflow of nurses in a busy work environment. This study suggests full-time, professional, clinical coders and quality assurance systems are still required. However, there remains a need to research and develop new systems for accurate and timely coding which equally meet the needs of the organisation and the clinician.

Footnotes

Acknowledgements

We would like to acknowledge the help of the theatres nurses, Margaret Anne McAlpine and Dr Jagannathan.

There are no competing interests and no financial support was provided.

Part of the dataset was used within an MSc Thesis in Healthcare Informatics (RCSED/University of Bath)

References

Office of Population, Censuses and Surveys. Classification of Surgical Operations and Procedures—4th rev. (OPCS-4),. http://www.connectingforhealth.nhs.uk/systemsandservices/data/clinicalcoding/classifications/opcs_4. (2008, accessed 16 October 2008).

Scottish Public Health Observatory. Overview of key data sources: Hospital discharges, http://www.scotpho.org.uk/web/site/home/resources/OverviewofKeyDataSources/Nationaldataschemes/SMR01.asp . (2008, accessed 16 October 2008).

Scottish Executive. ISD. Health and Social Care Data Dictionary. SMR01—General/Acute Inpatient and Daycase, http://www.datadictionaryadmin.scot.nhs.uk/isddd/9065.html . (2008, accessed 16 October 2008).

The Scottish Government. National Theatres Project Final Report: National Benchmarking Project Report 1, http://www.scotland.gov.uk/Publications/2006/11/24135440/1 (2006, accessed 16 July 2011).

Scottish Government. eHealth Strategy 2008–2011, http://www.ehealth.scot.nhs.uk/ (2008, accessed 16 July 2011).

Scottish Executive. Health Department. NHS Scotland eHealth Strategy. The Nursing, Midwifery and Allied Health Professions contribution to realising benefits of the National eHealth Programme, http://www.ehealthnurses.org.uk/pdf/NMAHP%20eHealth%20Action%20Plan.pdf (2007, accessed 13 June 2011).

NHS Quality Improvement Scotland. Using data to improve surgical care ~ the surgical profiles project, http://www.indicators.scot.nhs.uk/Surg_prof_summary_130907.pdf (2007, accessed 16 July 2011).

Office of Public Sector Information. Freedom of Information (Scotland) Act 2002, http://www.opsi.gov.uk/legislation/scotland/acts2002/asp_20020013_en_1 (2002, accessed 10 October 2011).

Royal College of Nursing. Nurses’ Business. An introduction to costing and coding healthcare, http://www.rcn.org.uk/__data/assets/pdf_file/0009/79758/003187.pdf (2007, accessed 18 July 2011).

10.

Milburn

. The accuracy of clinical data: a comparison between central and local data collection. Surgeon 2007; 5(5): 275.

11.

Raza

Holdsworth

Mccollum

. Accuracy of the recording of operative events by the Scottish Morbidity Record 1 (SMR1) for a teaching hospital vascular unit. J R Coll Surg Edinb 1999; 44(2): 96–98.

12.

Arthur

Nair

. Increasing the accuracy of operative coding. Ann R Coll Surg Engl 2004; 86(3): 210–212.

13.

Britton

Chambers

Ashmore

. Orthopaedic procedure coding. Does accuracy matter? Int J Health Care Qual Assur 2009; 22(4): 441–447.

14.

Burns

Rigby

Mamidanna

Bottle

Aylin

Ziprin

. Systematic review of discharge coding accuracy. J Public Health 2011; 2011 (Advance access): 1–11.

15.

Spillane

Krieser

Dalton

Heinrich

Babl

. Limitations to diagnostic coding accuracy in emergency departments: Implications for research and audits of care Emerg Med Australas 2010; 22(1): 91–92.

16.

King

Sharp

Lipsky

. Accuracy of CPT evaluation and management coding by family physicians. JABFP 2001; 14(3): 184–189.

17.

Rector

. Clinical terminology: why is it so hard? Methods Inf Med 1999; 38(4): 239–252.

18.

Colville

. Coding plastic surgery operations: an audit of performance using OPCS-4. Br J Plast Surg 2000; 53(5): 420.

19.

Haliasos

Rezajooi

O’neill

Van Dellen

Hudovsky

Nouraei

. Financial and clinical governance implications of clinical coding accuracy in neurosurgery: A multidisciplinary audit. Br J Neurosurg 2010; 24(2): 191–195.

20.

Terry

. Is computer-assisted coding ready for inpatient use? Early indications are that it can improve productivity without sacrificing accuracy. Healthcare Inf 2010; 27: 22–24.