Quality Review in Psychiatry

Abstract

This position paper has been substantially revised by the Canadian Psychiatric Association’s Professional Standards and Practice Committee and approved for republication by the CPA’s Board of Directors on August 31, 2016. The original position paper¹ was developed by the Professional Standards and Practice Council and approved by the Board of Directors on April 9, 1994.

Introduction

The original CPA position paper on this topic was published in 1994. ¹ At the time, the profession was threatened by psychiatric services increasingly coming under review by provincial health plans regarding their appropriateness for reimbursement. Correspondingly, the 1994 paper tackled this issue (utilization review) as well as the related topic of standards and practice review. The latter reviews typically were conducted under the auspices of provincial regulatory bodies or health care institutions. The central thesis of the 1994 paper was that only psychiatrists were properly suited to be engaged in the conduct of these forms of reviews, essentially promoting the principle of peer review. Although many of the points raised in the original paper are still valid more than 20 years later, the health care system has evolved considerably. Consequently, there is a need to address more contemporary issues, many of which did not exist to the same extent in the 1990s, when considering the involvement of psychiatrists in assessing the quality of clinical care.

Although the utilization review for reimbursement eligibility is still relevant, it is less of a priority than it was in the 1990s, and a spate of new realities confronts today’s psychiatrists in Canada. More psychiatrists now practise in collaborative care settings with family physicians or community mental health agencies. Even though hospital-based psychiatrists practised within multidisciplinary teams in the 1990s, today, there is much more emphasis on team-based care, including the delegation of medical duties. This has contributed to a shift from focusing on the individual expertise of the provider, to a focus on the broader contributors to patient safety and the system of care. In this time of disseminated yet overlapping responsibilities, the Canadian Medical Protective Association (CMPA) and others speak to the current complexities of assigning individual attribution of responsibility. Physician Achievement Review (PAR) programs exist in several provinces, incorporating patient and non-physician feedback. Clinical practice reviews have also changed, with less emphasis on peer opinion and more on performance measures, the latter facilitated by the evolution of practice guidelines, electronic information management, and decision support, which were not part of the 1990s Zeitgeist. The previous position paper well pre-dated the Royal College of Physicians and Surgeons of Canada’s Maintenance of Certification (MOC) program, which, now in its second iteration, places more emphasis on practice audit and self-assessment. Similarly, the Royal College’s Practice Eligibility Route (PER) to certification for specialists is an evolving, new program that relies on practice assessment. Likewise, the provincial regulatory licensing bodies have increasingly come to rely on successful clinical practice evaluation by their peers to meet the provincial colleges’ dual mandate to protect the public and respond to human resource shortages.

A point made in the 1994 position paper was that theoretically, attempts to enhance quality could lead to either a decrease or increase in costs, essentially these were orthogonal processes. Moreover, excessive focus on cost control would likely lead to reductions in the quality of services provided and outcomes achieved. ¹ Although the latter statement is likely still true, there is a growing appreciation that variation in health care spending does not necessarily reflect health outcomes. ² Other provinces are following Ontario’s lead with its enactment in 2010 of The Excellent Care for All Act, and, more recently, its formula change in hospital funding, with a movement away from block funding toward activity-based funding with performance measures. Ontario recently initiated various Pay-For-Performance (P4P) programs, which entail using payment mechanisms linked to the achievement of specific targets. Thus far, there is a paucity of evidence to support their widespread use; ² though, it is likely we shall see an evolution of such programs. As such, the current position taken by the CPA is that quality of services can be maintained or even improved with cost controls in place so long as quality of care issues are explicitly protected in these endeavours.

Some brief comments were made on education-related issues in the 1994 position paper; however, much has changed with enhanced teaching in evidence-informed practice, the improved quality of training programs, and the evolved publication and knowledge dissemination avenues in this area. Considering these developments, several new sections are included in this updated version, reflecting contemporary thoughts on these and other changes. Previous sections from the 1994 paper, which are not addressed in the present position paper, still reflect the CPA’s position (e.g., confidentiality, appeal processes, legal issues).

In this paper, quality review is defined as a review process carried out by psychiatrists, possibly in conjunction with others, where the main goal is to assess the qualitative aspects and appropriateness of clinical services performed by other psychiatrists, either alone or in team-based care. Traditionally, this type of “standards and practice review” has occurred with provincial licensing authorities (Colleges), as they are mandated to ensure that physicians under their jurisdiction meet competent, safe standards of practice. ^3,4

As stated in the original 1994 position paper, “Medicine, and particularly psychiatry are not exact sciences. Psychiatric diagnoses are made largely on the basis of the history taken with few “hard” or pathognomonic observable signs, or objective indicators of the diagnosis (radiologic and laboratory tests, etc.) available.” Notwithstanding the significant discoveries and scientific advances over the past 20 years, this remains much the case today. Even the recently released DSM-5 ⁵ continues to rely predominantly on symptomatic expressions of disorders for their varied diagnostic criteria. Whereas the 1994 paper also stated, “There are also multiple legitimate therapies of different type, length and cost, for the same diagnosis,” real progress has been made since then in evidence-based or, more properly, evidence-informed medicine and psychiatry. Consequently, much of contemporary quality review is guided by evidence-informed psychiatry based on clinical practice guidelines, which, in turn, come from meta-analyses and systematic reviews of the literature. ⁶

The Evolution of Quality Review in the Current Clinical Context

Early quality strategies within the organizational environment tended to focus on individual cases, and were reactive to critical events. ⁷ In the 1980s, the rediscovery of continuous quality improvement within the manufacturing industry by Deming and Juran began influencing quality management approaches within the health care sector. ⁸ Quality improvement efforts shifted from tactics focused on the individual, such as training and individual behaviour change, to team-based improvements. In addition to the increased emphasis on evidence-informed care and the previously mentioned use of clinical guidelines and care paths, there became a greater emphasis on the use of quality measures: “quantified indicators of care processes or outcomes believed to reflect the quality of care delivered.” ⁹ Donabedian introduced a framework for quality measures in 1966, which has typically guided the development of quality measures within health care. He outlined the importance of considering structural measures (characteristics of patients, providers, and the organization), process measures (the technical and interprofessional aspects of care), and outcome measures (typically focused on clinical and functional outcomes, quality of life measures, and satisfaction with care). However, the number and variety of quality measures and reporting systems within health care has tended to cause confusion and irritation to many providers. ¹⁰

Attempts to streamline reporting, and focus on value (as a function of quality, cost and patient satisfaction) have led to the adoption of a “balanced scorecard” approach, originally developed within the business sector in 1996 by Kaplan and Norton. The balanced scorecard tries to bring together data on cost, utilization, and quality into a single framework. ¹¹ Early physician responses to quality measures in general, and to the balanced scorecard specifically, have been mixed ^10,12 and this is often related to the choice of indicators, and whether they are valid, reliable, and useful in the medical context.

Several studies have outlined attempts to develop scorecards within “behavioural health care” settings, including community mental health centres, ¹³ and hospital settings ^8,11 in the US. Recommendations have emphasized the importance of strong physician leadership and engagement; ¹⁴ ensuring that clinical data and the resulting analysis are trustworthy, valid, and implementable; and the importance of creating a learning and improvement culture and avoiding a culture of blame. ¹⁵ The Institute of Medicine report “To Err is Human” ¹⁶ was a catalyst for change within health care, with a growing recognition of the role of complexity and system design as causes of error within the health care sector. James Reason (1995) emphasized the importance of considering both team and organizational factors in the design of safety systems, and brought knowledge from human factors engineering from the aviation and other high-risk industries to health care. ¹⁷ Reason also emphasized the importance of avoiding a culture of blame, as this leads to a non-reporting of adverse events and near misses.

A “just culture” is described as a fair and supportive system whereby the reasons for clinical outcomes and events are not pre-judged, blame is avoided, and the focus of analysis is on system failures. ^{18 –20} Within this framework, there is a systematic approach for designing and improving work environments that minimizes the possibility of human error and the potential impact when such error occurs. A just culture does not imply an absence of accountability; rather, negligence, intentional rule violations, and reckless conduct are not tolerated and are dealt with through disciplinary processes. Therefore, physician leaders who play a role in annual performance reviews, or who otherwise ensure accountability and are responsible for disciplinary matters, should not be reviewers or committee members involved in quality improvement reviews involving the psychiatrists who report to them. ²¹ The CMPA published a handbook in 2009: “Learning from adverse events: Fostering a just culture of safety in Canadian hospitals and health care institutions,” ¹⁸ which outlines the structures, policies, and procedures that contribute to the creation of a just culture and appropriate quality reviews. They recommend that each organization have a quality improvement committee that receives reports of adverse events and near misses; identifies opportunities for improvement using tools, such as root cause analysis (retrospective) and failure mode and effect analysis (prospective); and conducts appropriate quality improvement reviews, where the focus is on the system issues.

Although the balanced scorecard approach focuses on multiple dimensions, incorporating the use of guidelines can facilitate the review itself. Guidelines allow for standardization of the quality review across different reviewers, and transparency with regards to what is being evaluated. ²² They can also enhance quality of care if distributed widely to the practitioners involved. ²³ In the context of quality review, there are 2 types of guidelines that are relevant: The first is the clinical practice guidelines, ^{24 –27} which guide the delivery of high-quality clinical care. Reviewers should be familiar with these in their work. However, while the optimal or ideal standard is the one that must be aspired to, it cannot always also be the minimally expected standard. Consequently, in setting practice guidelines intended for use in peer review, the “optimal” standard should be a consideration in the development of the “acceptable” standard, with the latter guidelines generally giving more latitude than the former.

Within each Canadian province or territory, legislation exists that protects quality improvement reviews with regards to the opinions discussed, and the documentation surrounding such proceedings. Typically, such legislation does not protect the facts of the case or the summary recommendations, which are important to share with patients, their families, and other health care providers so learning can occur. The legal protection of quality improvement reviews helps encourage meaningful participation by providers, and therefore the ability to concentrate on improving future care rather than defending action. Psychiatrists with clinical expertise should be integral members of the group conducting the quality improvement review for mental health care in a multidisciplinary setting. Moreover, for the quality review to be credible, the reviewers must be respected, knowledgeable, and fair. Reviewers must also keep in check any biases they may hold in the review process, including affective and cognitive ones, which can be subtle. Especially germane in retrospective reviewing is outcome bias: Sometimes referred to as hindsight bias, outcome bias occurs when considering a known, deleterious outcome, the degree to which the occurrence seemed predictable—and thereby preventable—is exaggerated. ^28,29

Quality Reviews and Practice Assessment

There has been a growing focus on the multidimensional contributors to patient safety and the system of care; yet, there continues to also be an interest in the use of individual practice performance as it relates to quality. Two main areas of focus are described below:

360-Degree Physician Performance Assessments

Initially implemented by the provincial medical regulatory college in Alberta, ³⁰ and followed by the respective colleges in Nova Scotia, ³¹ and Manitoba, as well as being trialled in Ontario through the Council of Academic Hospitals of Ontario (CAHO), ³² there has been a growth in the use of 360-degree physician performance assessments to improve the quality of physician-provided care. ³³ Typically, these assessments involve peers, non-physician co-workers, as well as a sampling of patients, who all provide feedback to the physician (and possibly others) on their impression of the physician’s knowledge, skills, and behavioural characteristics, such as interpersonal skills and professional behaviour. Currently, these assessments are formative, confidential, and mainly used by the individual physician as feedback and for quality improvement.

Practice Eligibility Route (PER) to Certification

Over the past few years, the Royal College of Physicians and Surgeons of Canada (RCPSC) has been developing the PER to certification as an alternative to the traditional approval of residency training and subsequent examinations. ³⁴ There are 3 components to PER: the first two are an approved 24-month involvement with the MOC program, and a credentials review. The final component is further divided into either Route A, where the candidate challenges the existing RCPSC examinations, or Route B, a highly structured, practice-based examination/assessment to determine if the candidate meets all the competencies deemed essential for practice as a specialist in the discipline. Like the PAR process described above, multisource feedback is collected before either Route A or B is undertaken.

Psychiatry is the first discipline to offer Route B, which is very much a peer review assessment of quality of care. Two peer assessors, jointly appointed by the RCPSC and the individual’s medical regulatory authority (provincial college), conduct the in-practice assessment, which takes approximately 1.5 days. This assessment is comparable in scope and difficulty to the standard examination route, and is comprises a review of record keeping, discussion, a review of cases from the charts, an observed consultation, standardized cases, and structured oral questions, amongst other elements.

Educational Aspects of Quality Review

To instill a culture of continued quality improvement among psychiatrists, it is vital to teach methods for quality improvement in psychiatry training programs. Psychiatry residents can use a clinical audit as a learning opportunity to develop a number of skills defined by CanMEDS; e.g. becoming aware of the evidence-based guidelines (Medical Expert), and communication and negotiation skills (Communicator); engaging in interprofessional collaboration (Collaborator); getting acquainted with the health care structure and administrative systems (Leader); and advocating for improved quality of care for patients (Medical Expert, Leader, and Health Advocate). ³⁵ For the success of an audit, it is pivotal to make available effective training, dedicated support staff, protected time, and an environment where a clinical audit is a priority of the administrative leadership. ³⁶

Much of the work pertaining to quality improvement in postgraduate training and continuing professional development has been formally instituted in the UK. One of the learning outcomes within the Royal College of Psychiatrists’ core curriculum for psychiatric trainees is the ability to conduct and complete a clinical audit. ³⁷ To guide trainees to conduct efficient and fruitful audits, the Royal College of Psychiatrists (RCPsych) recently published a book, “101 Recipes for Audit in Psychiatry.” This book summarizes the audit projects in numerous clinical areas that were successfully completed by experts, and the results were useful in informing practice and designing services. ³⁸

Quality improvement projects are typically conducted to inform local services; but these can also be conducted at a national level to allow benchmarking with other services and national standards. ³⁹ The RCPsych has taken the lead in conducting national audits in England and Wales; this has included the use of psychological therapies for people suffering from anxiety and depression; the care of individuals with dementia in general hospitals; the prescription of antipsychotics and conservation of physical health in individuals suffering from schizophrenia. ³⁹

Both the General Medical Council and the RCPsych recommend that audits be considered an integral part of the revalidation process for doctors and psychiatrists. ^40,41 The revalidation guidance produced by the RCPsych (2012) recommends psychiatrists undertake at least 2 audits per 5-year revalidation period.

The RCPSC have set similar recommendations for quality improvement activities for the maintenance of certification in Canada. ⁴² The MOC framework specifically defines Section 2 credits for “Systems Learning” (including activities that stimulate learning through contributions to practice standards, patient safety, quality of care) and Section 3 credits for “Performance Assessment” (including activities that provide data with feedback to individual physicians, groups, or interprofessional health teams related to personal or collective performance across a broad range of professional practice domains). The RCPSC recently updated the MOC framework, and for all MOC cycles beginning January 1, 2014, a minimum of 25 credits is required in each of sections 1, 2, and 3 of the MOC program during the 5-year MOC cycle. The reason for this update, as quoted by the RCPSC, is “The CPD research literature has clearly demonstrated that physician’s self-assessment compared to external measures of performance is inaccurate, and assessment strategies that provide data with feedback have a higher likelihood of changing performance and improving patient outcomes compared to other forms of continuing professional development. Finally, assessment of competence and performance in practice is an increasing expectation of provincial medical regulatory authorities and the public.” ⁴²

The MOC program defined by the American Board of Psychiatry and Neurology (ABPN) ⁴³ also requires participation in sanctioned self-assessment performance measures and development of quality improvement programs based on personal clinical practice. “The goal is for diplomates to reflect on their personal knowledge and performance and commit to a process of improvement and re-evaluation of performance measures over a specified time frame that will ultimately lead to improved care for their patients.” ⁴³

Because there are many similarities between a research and an audit project, residents who do not have any background in research can use quality improvement or audit projects as preliminary experience. In recent years, opportunities have been made available to publish results of quality improvement projects in journals, which are entirely devoted to the publication of research in health care quality. ^{44 –46}

For optimum patient care, it is essential that psychiatrists develop evidence-informed practice, skills, and behaviour throughout their career. Therefore, it is imperative for the psychiatric residency candidates to inculcate evidence-based medicine during their training. This can be achieved through evidence-based journal clubs or regular discussions of “best evidence” during clinical rotations. ^47,48 The knowledge and skills required for practising evidence-based medicine should be comprehensively tested by the RCPSC examining boards. The syllabus for evidence-based psychiatry is now clearly defined by the RCPsych, ⁴⁹ with an opportunity for Canadian and other professional bodies to follow suit.

Conclusion and Recommendations

The CPA’s position is that quality of service can be maintained or even improved with cost controls in place, so long as quality of care issues are explicitly protected in the endeavours. Over the last 20 years, quality improvement has shifted from focusing on finding error and the individual care provider, to focusing on the system, human factor engineering, and improved design; yet, there remains challenges in how to improve quality and patient safety within the clinical context.

Areas of future focus should include understanding the role of communication within teams, ⁵⁰ and the application of complexity science, to create high reliability organizations in the health care sector.

Organizations should develop a clear process by which clinical care is reviewed from a perspective of quality of care. This should include a quality improvement committee that receives reports of adverse events and near misses, identifies opportunities for improvement using tools that allow for both retrospective and prospective review, and is grounded in evidence-based standards and guidelines. Such reviews should occur within a “just culture” framework, which emphasizes the contribution of system failures and the importance of continuous improvement.

Psychiatrists with clinical expertise in the field of interest must be integral members of members of the group conducting the review. Ideally, they should be viewed as peers rather than as physician leaders to ensure that there is a clear distinction between quality review and performance management.

At the level of the individual psychiatrist, the CPA supports the development of a significant focus on self-assessment through educational opportunities, such as practice audit and personal learning plans.

Residency training programs across Canada should consider the completion of a quality improvement project as a useful adjunct in training.

Educators should consider testing knowledge and skills in quality review in residency training programs.