Preliminary appraisal of the reliability and validity of the Colorado State University Feline Acute Pain Scale

Abstract

Objectives

The objective of this study was to determine the inter-rater reliability and convergent validity of the Colorado State University Feline Acute Pain Scale (CSU-FAPS) in a preliminary appraisal of its performance in a clinical teaching setting.

Methods

Sixty-eight female cats were assessed for pain after ovariohysterectomy. A cohort of 21 cats was examined independently by four raters (two board-certified anesthesiologists and two anesthesia residents) with the CSU-FAPS, and intra-class correlation coefficient (ICC) was used to determine inter-rater reliability. Weighted Cohen’s kappa was used to determine inter-rater reliability centered on the ‘need to reassess analgesic plan’ (dichotomous scale). A separate cohort of 47 cats was evaluated independently by two raters (one board-certified anesthesiologist and one veterinary small animal rotating intern) using the CSU-FAPS and the Glasgow Composite Measure Pain Scale (CMPS-Feline), and Spearman rank-order correlation was determined to assess convergent validity. Reliability was interpreted using Altman’s classification as very good, good, moderate, fair and poor. Validity was considered adequate if correlation coefficients were between 0.4 and 0.8.

Results

The ICC was 0.61 for anesthesiologists and 0.67 for residents, indicating good reliability. Weighted Cohen’s kappa was 0.79 for anesthesiologists and 0.44 for residents, indicating moderate to good reliability. The Spearman rank correlation indicated a statistically significant (P = 0.0003) positive correlation (0.31; 95% confidence interval 0.14–0.46) between the CSU-FAPS and the CMPS-Feline.

Conclusions and relevance

The CSU-FAPS showed moderate-to-good inter-rater reliability when used by veterinarians to assess pain level or need to reassess analgesic plan after ovariohysterectomy in cats. The validity fell short of current guidelines for correlation coefficients and further refinement and testing are warranted to improve its performance.

Introduction

The importance of accurate pain recognition and appropriate treatment cannot be overstated. Uncontrolled pain is detrimental to patient quality of life, as it causes undue suffering and has physiologic effects, such as increases in cortisol and catecholamines, which predispose to tissue edema, ischemia, hyperglycemia, infection and impaired wound healing. ¹ Despite these well-established negative physiologic consequences, and the ethical obligation of alleviating animal pain and suffering, pain assessment and management in veterinary medicine have begun to receive more attention only in recent years. ²

Various studies over the years have analyzed the attitudes of veterinarians towards perioperative analgesia and their use of various analgesic agents and techniques in dogs and cats. ³ Although these studies have been conducted in different countries and seem to show an increase in the usage of analgesics over the past 20 years,^4–9 the harmful practice of withholding analgesics remains a problem. ³ Recognizing the need to assist practitioners to elevate the standard of care in this area, several veterinary organizations have produced guidelines for pain management, including the American Animal Hospital Association (AAHA) and American Association of Feline Practitioners (AAFP) pain management guidelines and the World Small Animal Veterinary Association guidelines for recognition, assessment and treatment of pain.^10,11

In order to alleviate pain in veterinary patients, pain must first be recognized. In cats, pain recognition is challenging, contributing to the historical undertreatment of pain in cats vs dogs undergoing comparable surgical procedures.^4,5,8 Pain behaviors in cats are often subtle, especially in the hospital setting, where cats may mask signs of pain and illness. Behavioral signs of acute pain in cats may include depression, immobility, remaining in a hunched position with the head down, guarding behaviors, attempting to escape, squinting of the eyes, hissing, growling, and resistance to palpation or handling.^10,12,13 These clinical signs of pain appear very similar to signs of distress and may be overlooked, with the assumption that the cat is simply nervous or fearful in the clinic setting.

Validated pain scoring systems are available for equine, canine and murine patients, but a recent systematic review identified a current lack of adequate, versatile and validated feline pain assessment tools. ¹⁴ Nevertheless, great strides have been made in feline pain assessment in recent years. The UNESP-Botucatu Multidimensional Composite Pain Scale has been validated in cats undergoing ovariohysterectomy.^15,16 The Glasgow Composite Measure Pain Scale (CMPS-Feline) has recently been validated for cats and incorporates evaluation of the patient’s facial expression, decreasing the misclassification of painful vs non-painful cats compared with the earlier version of the tool, which lacked the embedded facial scale.^17–19 The development of pain assessment tools for use in feline patients is of utmost importance, and further work in this area is still greatly needed.

The most recent AAHA/AAFP Pain Management Guidelines strongly encourage the use of pain-scoring tools in clinical practice but acknowledge that there is no gold standard for the assessment of pain in companion animals. ¹⁰ The guidelines specifically reference the Colorado State University Feline Acute Pain Scale (CSU-FAPS) and it is used in clinical practice. Even though experts in the field developed the scale, it was not formally developed following psychometric principles underlying health assessment scale development and evaluation, nor has it undergone any sort of post-development evaluation. While the use of pain-scoring systems, as well as the education of veterinary students and continuing education of veterinarians about their use, might prevent analgesic withholding, ³ use of unreliable tools could preclude such benefits. In the current study, we sought to assess the inter-rater reliability and convergent validity of the CSU-FAPS in a preliminary appraisal of its performance in a clinical/teaching setting. The goal was not to perform an all-encompassing test and validation of the scale (ie, content and face validity, reliability, sensitivity and specificity). We hypothesized that the scale would show good inter-rater reliability when used by veterinarians to assess pain level or need to reassess analgesic plan after ovariohysterectomy in cats.

Materials and methods

Animals

Sixty-eight intact female cats submitted to ovariohysterectomy via a standard midline celiotomy as part of the University of Minnesota College of Veterinary Medicine student surgery and anesthesia practice laboratory were used in this study. No restrictions were placed on the breed, age or weight of the cats. The University of Minnesota Institutional Animal Care and Use Committee approved the laboratory protocol.

Anesthetic protocol

Third-year veterinary students performed anesthesia under the guidance of board-certified veterinary anesthesiologists, anesthesia residents and technical staff. Cats were premedicated intramuscularly with dexmedetomidine (0.005 mg/kg [Dexdomitor; Pfizer Animal Health]), hydromorphone (0.1 mg/kg [hydromorphone hydrochloride; Baxter Healthcare]) and ketamine (3 mg/kg [Vedco]). Anesthesia was induced with either propofol (4 mg/kg [Propofol; Abbott Laboratories]) or propofol and ketamine (2 mg/kg each) administered via a cephalic vein catheter to effect, to allow opening of the mouth and intubation of the trachea with an appropriately sized, cuffed endotracheal tube. Anesthesia was maintained with isoflurane delivered with a calibrated vaporizer in >95% oxygen via a Jackson-Rees breathing system.

During anesthesia, monitored physiologic variables included palpebral reflex, eye position, jaw tone, heart rate, respiratory rate, blood pressure via Doppler ultrasonography, arterial oxyhemoglobin saturation via pulse oximetry and body temperature. Lactated Ringer’s solution (5 ml/kg/h [Abbott Laboratories]) was infused intravenously during anesthesia. Additional intravenous doses of hydromorphone (0.05 mg/kg) were administered intraoperatively as needed when signs of autonomic response to nociception were observed (tachycardia, hypertension, tachypnea), or postoperatively prior to the cat being returned to its cage if behaviors such as agitation, aggression, vocalization, attempts to escape and/or aversive response to gentle abdominal palpation were expressed. Postoperatively, cats remained under the direct observation of the students and instructors until normothermia was achieved (temperature 37–39°C) before being returned to their cages.

Inter-rater reliability

In a cohort of 21 cats, two board-certified anesthesiologists (referred to as ‘anesthesiologists’) and two residents in veterinary anesthesiology (referred to as ‘residents’) independently evaluated each cat approximately 6 h after they had returned to their cages following anesthesia and surgery. This time point represented the final assessment of the cats prior to being left in the ward overnight. Owing to the variable surgical times of the different student groups, earlier postoperative pain assessments were not completed by all raters. Raters were familiar with using the scale and provided with a printed copy (see appendix in the supplementary material) for each cat. To ensure consistency, raters also received verbal and written instructions on how to use the scale.

Accordingly, raters were instructed to first assess the cat visually from a distance, prior to opening the cage, interacting with the cat and palpating the flanks and ventral abdomen. Raters were instructed to check off behaviors that were observed, to indicate parts of the body where cats were assessed to be painful by drawing on the diagram on the sheet, to circle the body tension they ascribed to the cat and, finally, to indicate a single, overall numeric pain score (0–4) for the cat. The quartermarks on the pain scale along the left border of the page were used for assigning the scores, such that the scale was interpreted as ordinal rather than continuous. Pain scores for all raters were performed over the course of approximately 30 mins with each individual performing his or her evaluation independently.

Validity

The concurrent validation approach was employed to assess the validity of the CSU-FAPS, with the CMPS-Feline used as the criterion measure. ²⁰ In a separate cohort of 47 cats, two raters (one veterinary rotating intern, one veterinary anesthesiologist) independently evaluated each cat in the morning and evening of days 1 and 2, and in the morning of day 3 postoperatively. One rater used only the CSU-FAPS and the other used only the CMPS-Feline for all evaluations. Both raters were familiar with the content and use of the scales.

Statistical analyses and data interpretation

Statistical analyses were performed with commercially available software GraphPad Prism version 5.0f for Mac and Microsoft Excel for MAC 2011 version 14.7.2. The inter-rater reliability was evaluated using the intra-class correlation coefficient (ICC) involving a two-way random effect model with 95% confidence intervals (CIs). Inter-rater reliability for the anesthesiologist pair and for the resident pair, centered on the ‘need to reassess analgesic plan’ (dichotomous scale), was determined using weighted Cohen’s kappa. The results of inter-rater reliability were interpreted using Altman’s classification, ²¹ in which values ranging from 0.81–1.0, 0.61–0.80, 0.41–0.60, 0.20–0.40 and <0.2 are considered very good, good, moderate, fair and poor, respectively. The data set used to assess convergent validity was not normally distributed according to the D’Agostino and Pearson normality test. Therefore, convergent validity was assessed by calculating Spearman rank-order correlation between the scores obtained with the CMPS-Feline and the CSU-FAPS. Consistent with current recommendations, ²⁰ it was determined a priori that correlations ranging from 0.4–0.8 would be acceptable.

Results

Inter-rater reliability

The cats in this group ranged from 0.3–4 years of age (median 1 year) and from 1.71–5.12 kg (mean weight 3.3 kg). Overall, the lowest and highest assigned pain scores were 0 and 3.25, respectively. The ICC was 0.61 (95% CI 0.25–0.82) between anesthesiologists and 0.67 (95% CI 0.35–0.85) between residents, indicating good inter-rater reliability. The weighted Cohen’s kappa for the dichotomous scale was 0.79 (95% CI 0.5–1.0) between anesthesiologists and 0.44 (95% CI 0.1–0.8) between residents, indicating good and moderate inter-rater reliability, respectively. The SD of the differences between the anesthesiologists’ and residents’ scores was 0.56 (95% CI −0.24 to 1.94).

Validity

These cats ranged from 0.3–6 years of age (median 1 year) and from 1.5–4.6 kg (mean weight 2.9 kg). In one cat, the spleen suffered iatrogenic trauma during surgery and was subsequently removed. This cat was included in the analysis. Pain scores ranged from 0–4 for the CMPS-Feline and from 0–2.5 for the CSU-FAPS. The Spearman rank correlation indicated a statistically significant (P = 0.0003) positive correlation (0.31; 95% CI 0.14–0.46) between the CMPS-Feline and CSU-FAPS. None of the scores met the cut-off for rescue analgesia according to the CMPS-Feline, whereas five scores met the cut-off to ‘reassess analgesic plan’ according to the CSU-FAPS.

Discussion

This study assessed the inter-rater reliability and validity of the CSU-FAPS as this information has not been published despite the scale being recommended for clinical use. ¹¹ Results showed a moderate-to-good level of reliability for both anesthesiologists and residents. This reliability is lower than that obtained with the Botucatu-MCPS, which was good to very good among blinded observers as assessed by video recordings. ¹⁵ No information could be found regarding inter-rater reliability for the CMPS-Feline, which has been validated recently,^17–19 precluding comparison with the current results using the CSU-FAPS. Compared with the more detailed Botucatu-MCPS, the CSU-FAPS is a simpler and more concise tool, which may make it more apt for use in clinical practice. ¹¹ The greater simplicity might result in greater variability. The finding that the CSU-FAPS yielded a much wider ICC 95% CI than those obtained in the Botucatu-MCPS ¹⁵ supports this assumption.

The inter-rater reliability of the CSU-FAPS did not appear to depend on level of anesthesia training, at least among observers with some degree of advanced training in anesthesia. Variability related to different levels of anesthesia training was noted during validation testing of the Botucatu-MCPS. ¹⁵ No information could be found regarding influence of training level on reliability in the recently validated CMPS-Feline.^17–19 One possible explanation as to why training level might influence the performance of a pain assessment tool is that the rater’s knowledge base and experience might compensate for gaps in the tool. Indeed, education on pain management in the veterinary curriculum and continuing education to veterinarians remains an unmet need. ³ In the current study, all raters had formal training on feline pain assessment, and thus the findings may not be directly extrapolated to raters with no or minimal training.

While inter-rater reliability assesses consistency, validity refers to how well a pain scale correlates with other measures of the same construct or related variables. ²⁰ In the current study, we employed a concurrent validation approach as an accepted method to assess validity. ²⁰ As there is currently no gold standard for pain assessment in cats, the validated CMPS-Feline^17–19 was used as the criterion measure. Scores were assigned independently by two different individuals in an effort to avoid bias that can artificially inflate correlation coefficients if the two methods were used by the same individual. ²⁰ Therefore, lower correlations are to be expected, and current guidelines are that correlations coefficients should be between 0.4 and 0.8. ²⁰ We found that the Spearman rank correlation (0.31) fell below this guideline. There are at least three possible explanations for this finding.

First, the reliability of one or the other scale may be unacceptably low. This is not the case for the CSU-FAPS because it had moderate-to-good inter-rater reliability. We were not able to find published information regarding the inter-rater reliability of the CMPS-Feline.

The second possible explanation is that the two scales measure different phenomena. This may be the case as there were five instances of possible need for rescue analgesia identified by the CSU-FAPS vs none with the CMPS-Feline. It is preferable to give rescue analgesics that may be unwarranted in some instances, particularly in otherwise healthy patients, rather than withholding them when they are actually needed. ³ Thus, it could be argued that the CSU-FAPS showed clinically acceptable validity in the context of this study, even though its correlation coefficient with the CMPS-Feline was below current guidelines.

The third possible explanation is that the difference in experience level between the veterinary anesthesiologist using the CSU-FAPS and the rotating intern using the CMPS-Feline could have resulted in systematic differences between the two observers. It is possible that the correlation would have been higher if the two observers had been similarly experienced, or if the scale used by each observer was randomly assigned.

Weaknesses of this study include a relatively small sample size and the single pain etiology studied (ie, pain post-ovariohysterectomy). Most cats in this study had a mild-to-moderate level of pain; the scale’s performance in cases of more severe pain is undetermined. A larger sample size could have provided stronger confidence in our results. How the CSU-FAPS performs with regard to the evaluation of other painful clinical and surgical conditions is currently unknown and requires further investigation. No attempt was made to assess the responsiveness of the scale to changes in pain level, as would be expected with analgesic administration, and this represents another area for further study.

Conclusions

The CSU-FAPS was simple to use, showing moderate-to-good inter-rater reliability when used by veterinarians with advanced training in anesthesia to assess pain in cats submitted to ovariohysterectomy. The convergent reliability of the scale fell short of current guidelines for correlation coefficients. The wide inter-rater reliability CIs suggest that refinement of the scale could be beneficial to increase consistency of results. In summary, the CSU-FAPS would benefit from additional refinement and testing prior to being recommended for use in clinical practice.

Supplementary Material

Supplementary Material, JFM_777506_supplementary_material – Preliminary appraisal of the reliability and validity of the Colorado State University Feline Acute Pain Scale

Appendix. Colorado State University Feline Acute Pain Scale (CSU-FAPS) form provided to observers for pain assessment.