Testing the Healthcare Complaints Analysis Tool in a Specialist Pediatric Hospital to Assess Potential Utility for Organizational Learning from Complaints

Abstract

The importance of improving the analysis of complaints about poor healthcare experiences is recognized but often not addressed. We explored the utility of the Healthcare Complaints Assessment Tool for analyzing written complaints in a specialist pediatric hospital receiving relatively few, but complex, complaints. The scoring framework could be applied but, in contrast to previous work, the complexity of complaints was considerably greater. Responses to the complainant were also scored and we identified instances of both higher and lower levels of severity than identified by the complainant, suggesting a novel application of the tool which will be important for institutional learning.

Background

Healthcare complaints from patients and/or carers can be key indicators of problems in patient care and provide a valuable, diverse source of data for improving the quality and safety of services and patient/family experience through the implementation of evidence-based interventions (1–3). Complainants frequently invest significant time and effort into raising concerns and describing the impact of what they perceive to have gone “wrong” (4) and a driver for making a complaint is often the desire to know this will not happen to others (5). Information garnered from complaints can contribute to improvements in patient engagement, experience, and outcomes as well as clinical effectiveness. However, change can only happen if there is firstly an opportunity for patients and their carers to provide feedback on their experiences and secondly if the health care system can analyze, respond to, and use that feedback (6).

Whilst the importance of improving the analysis of complaints from patients and/or carers about poor healthcare experiences has been identified as an urgent priority for service providers (7) it is often not addressed. The Healthcare Complaints Analysis Tool (HCAT) (8–10) was developed to facilitate extraction of data from formal complaints, thereby enhancing organizational listening, analysis and aggregation of complaints data to improve service monitoring and institutional learning. However, HCAT has been predominantly applied in adult settings with less complex complaints (8,10,11). One recent study reviewed physician-related patient complaints in a Canadian specialist pediatric hospital and categorized them using HCAT (12). The majority of complaints were in the clinical and/or relationship domains but the severity or harm associated with the complaints were not reported, which are some of the unique aspects of HCAT. Our primary aim, therefore, was to explore the utility of HCAT in a specialist pediatric hospital where the number of formal complaints is relatively small but the complaints are often extremely complex. A secondary aim was to explore the utility of HCAT to score the organization's responses to complainants.

Methods

Ten written complaints (10% of those received) from a range of clinical services, all of which were complex and had required detailed, lengthy responses, were randomly selected (using a random number generator) from all formal complaints received during the previous 12 months and identifying information removed by the complaints team. Using HCAT and accompanying guidance, complaints were independently scored by the three authors, all of whom had experience of health-care complaints resolution (GO/CW) and/or working clinically with children with complex health conditions (JW/GO). Each complaint was individually reviewed and discussed to assess whether the scoring framework was applicable and to identify any challenges with extracting the information. The written response provided by the organization to the complainant was similarly scored, using the same HCAT scoring framework, and Wilcoxon tests were used to compare pre- and post-investigation scores for total problem score, maximum harm, and maximum severity. Inter-rater reliability was compared using the Kappa statistic and any differences in scores were resolved by consensus. Scores were compared with published data (8) using unpaired t-tests. The time taken to score each complaint and corresponding response was recorded. Ethical approval was not required.

Tool

HCAT is a valid and reliable analytical tool with a coding taxonomy. The three domains (clinical, management, relationship problems) are broken down into seven conceptually distinct problem categories (clinical comprises quality and safety; management comprises institutional processes and environment; and relationship problems incorporate listening, communication, and respect and patient rights). Each problem category has three levels of severity and five possible levels of harm and problems are allocated to at least one stage of care reflecting each stage of the patient journey (8–10).

Setting

Our inner-city specialist pediatric hospital has 383 beds (including 44 intensive care beds) and treats children from 0 to 18 years with rare and complex chronic health conditions, with >255,000 patient visits per year. Our complaints investigation process is outlined in Supplemental Figure 1.

Results

All elements of the HCAT scoring framework could be applied to the complaints and agreement between raters was almost perfect (>85% for all elements). In contrast to previous work using HCAT, the complexity of our formal complaints was significantly greater and an average of 7.1 ± 4.65 problems (range: 2-16) per complaint were identified (compared with 1.92 ± 0.26; t = 3.52; P = .007) (1). Fifty-five percent (n = 39) of all problems were in the domain of relationship problems, particularly in the category of communication (n = 26; 36%) (Table 1). The highest levels of severity and overall harm were in the categories of safety and communication.

Table 1.

Domains, Problem Categories, and Number of Problems (%) in Each Category.

Domain	Problem category	Number of problems in each category (%)
Clinical	Quality	7 (10)
	Safety	13 (18)
Management	Institutional processes	10 (14)
	Environment	2 (3)
Relationship problems	Listening	9 (13)
	Communication	26 (36)
	Respect and patient rights	4 (6)

Due to the complexity of some situations about which complaints were being made and the lack of understanding of some complainants about what had actually happened or reasons for a particular course of action, responses were often far more detailed than might be anticipated. Scoring responses highlighted instances where the response suggested more problems and higher levels of severity than identified by the complainant, as well as instances where a seemingly serious problem in a complaint was not supported by the evidence. Although overall differences in complaint and response scores were not significant for total problem score (Z = 1.866; P = .062), maximum harm (Z = 0.973; P = .330) or maximum severity (Z = 1.342; P = .180), scores for complaints and responses differed in one or more aspect for 8/10 cases (Table 2/Supplemental Table 1). The median time to score each complaint and corresponding response together was 50 (range: 30-60) minutes.

Table 2.

Number of Problems, Maximum Severity and Maximum Harm for Each Complaint and Corresponding Response and Comparisons of These Parameters Between the Complaint and Response.

Complaint number (1)	Total problems—complaint (2)	Maximum harm—complaint (3)	Maximum severity—complaint (4)	Total problems—response (5)	Maximum harm—response (6)	Maximum severity—response (7)	Change in number of problems from complaint to response (8)	Change in total harm from complaint to response (9)	Change in total severity from complaint to response (10)	Change in number of problems (11)	Change in total harm score (12)	Change in total severity score(13)
1	16	4	3	11	2	2	↓	↓	↓	−5	−2	−1
2	2	0	3	2	0	3	→	→	→	0	0	0
3	10	4	2	5	4	3	↓	→	↑	−5	0	+ 1
4	2	0	1	3	2	1	↑	↑	→	+ 1	+ 2	0
5	5	1	2	2	0	2	↓	↓	→	−3	−1	0
6	13	3	3	3	1	2	↓	↓	↓	−10	−2	−1
7	8	5	3	2	3	2	↓	↓	↓	−6	−2	−1
8	5	2	2	5	3	2	→	↑	→	0	+ 1	0
9	6	1	1	6	1	1	→	→	→	0	0	0
10	4	1	2	5	1	1	↑	→	↓	+ 1	0	−1

Each complaint (column 1) was individually scored and the total number of problems identified (2), together with the maximum harm (3) and maximum severity (4) score assigned to any individual problem. Following the investigation of the complaint, the response sent to the family was scored using the same framework and the total number of problems in the response was identified (5), together with the maximum harm (6) and maximum severity (7) assigned to any individual problem. Comparison of the scoring of the complaint and corresponding response indicated whether there were the same (→), fewer (↓), or more (↑) problems identified in the response (8) and whether the maximum harm score (9) and maximum severity score (10) were the same, less, or greater in the response. Columns 11-13 quantify the difference in number of problems, maximum harm, and maximum severity comparing the response with the complaint, with negative numbers indicating fewer problems and positive numbers denoting more problems identified in the response compared with the complaint.

Discussion

We have reported initial work assessing the feasibility and acceptability of using HCAT, previously used in adult settings, to score complex complaints in a pediatric setting. Our results indicated that, although complaints received by our organization are complex compared with many other organizations (8,10), using the HCAT scoring taxonomy was feasible. The time to score the complaint and response supports the routine use of HCAT as we receive fewer than 100 formal complaints a year. All elements of each complaint and response could be categorized according to the HCAT typology and scores indicating severity and harm could be allocated to each identified problem. Inter-rater reliability was high and, anecdotally, learning how to use HCAT was not onerous. In contrast to a previous application of HCAT across 59 studies (8), representing 88,069 adult patient complaints, problems in relationships with healthcare staff was the domain in which we identified the greatest number of complaints (45% of problems vs 29% in adult settings), particularly related to communication. Differences may be due to the chronicity, complexity, and medical acuity of our patients’ conditions/treatment and the associated stress for parents. Conversely, there were relatively few complaints about the management of the organization (identified in 35% of adult complaints). Our findings did, however, resonate with those of another pediatric study using HCAT (12) in which problems related to relationships featured in 60% of complaints.

Our use of HCAT for scoring the organization's response is a novel application of the tool, not previously described, and it illuminated some of the challenges in dealing with complex medical issues. Scores for the complaint and response differed in 8/10 of cases and this, coupled with the high proportion of communication issues, is particularly salient for institutional learning. Healthcare professionals perceive that patient complaints damage staff and affect the staff–patient relationship (13) but scoring responses as well as complaints may offer a way of increasing transparency in the process of complaints management, thereby addressing negative perceptions about how complaints are handled (14).

If patient complaints are to be effectively managed and used for quality improvement we have to be able to analyze, respond to, and act upon them (6). This initial work suggests HCAT is a useful, practical, and feasible tool for enhancing analysis of complex complaints received by a specialist pediatric hospital. Through our Patient Advice and Liaison Service we also receive less complex concerns which are dealt with on an informal basis and we plan to use HCAT to score these contacts. Further work is now required to implement training using HCAT, including assessment of inter-rater reliability, and this is currently being developed. Using HCAT to score individual complaints also offers an opportunity for immediacy, which may help to stimulate timely reflective learning with relevant staff and promote change. Furthermore, HCAT should make it easier to identify recurrent themes in complaints received by individual teams and/or across the organization and enable interventions to be implemented to address them. For the potential of HCAT to be fully realized, however, we also need to further develop and embed a robust process for ensuring timely organizational learning through using HCAT and continuous improvement at team, division, and board level (15).

Limitations

Despite the promise of this initial work for scoring complex complaints and responses in a pediatric setting, it is important to acknowledge that HCAT has only been applied to ten cases and scoring has been carried out by individuals with an understanding of complaints and the process of conducting complaints investigations. At this early stage we do not know whether use of HCAT will be scalable within our organization or generalizable to other pediatric organizations. We also do not know whether use of HCAT will facilitate enhanced organization learning from complaints, which is one of our primary objectives as we develop this work further.

Supplemental Material

sj-docx-1-jpx-10.1177_23743735221106594 - Supplemental material for Testing the Healthcare Complaints Analysis Tool in a Specialist Pediatric Hospital to Assess Potential Utility for Organizational Learning from Complaints

Supplemental material, sj-docx-1-jpx-10.1177_23743735221106594 for Testing the Healthcare Complaints Analysis Tool in a Specialist Pediatric Hospital to Assess Potential Utility for Organizational Learning from Complaints by Jo Wray, Claire Williams and Geralyn Oldham in Journal of Patient Experience

Supplemental Material

sj-docx-2-jpx-10.1177_23743735221106594 - Supplemental material for Testing the Healthcare Complaints Analysis Tool in a Specialist Pediatric Hospital to Assess Potential Utility for Organizational Learning from Complaints

Supplemental material, sj-docx-2-jpx-10.1177_23743735221106594 for Testing the Healthcare Complaints Analysis Tool in a Specialist Pediatric Hospital to Assess Potential Utility for Organizational Learning from Complaints by Jo Wray, Claire Williams and Geralyn Oldham in Journal of Patient Experience