The Decline of Screening: Psychiatry is Just Not that Simple

Clinical Predictors of Discordance Between Screening Tests and Psychiatric Assessment for Depressive and Anxiety Disorders Among Patients Being Evaluated for Seizure Disorders

Holper S, Foster E, Lloyd M, Rayner G, Rychkova M, Ali R, Winton-Brown TT, Velakoulis D, O'Brien TJ, Kwan P, Malpas CB. Epilepsia. 2021;62:1170-1183.

Objective

This study was undertaken to identify factors that predict discordance between the screening instruments Neurological Disorders Depression Inventory for Epilepsy (NDDI-E) and Generalized Anxiety Disorder scale (GAD-7), and diagnoses made by qualified psychiatrists among patients with seizure disorders. Importantly, this is not a validation study; rather, it investigates clinicodemographic predictors of discordance between screening tests and psychiatric assessment.

Methods

Adult patients admitted for inpatient video-electroencephalographic monitoring completed 8 psychometric instruments, including the NDDI-E and GAD-7, and psychiatric assessment. Patients were grouped according to agreement between the screening instrument and psychiatrists' diagnoses. Screening was “discordant” if the outcome differed from the psychiatrist’s diagnosis, including both false positive and false negative results. Bayesian statistical analyses were used to identify factors associated with discordance.

Results

A total of 411 patients met inclusion criteria; mean age was 39.6 years, and 55.5% (n = 228) were female. Depression screening was discordant in 33% of cases (n = 136/411), driven by false positives (n = 76/136, 56%) rather than false negatives (n = 60/136, 44%). Likewise, anxiety screening was discordant in one third of cases (n = 121/411, 29%) due to false positives (n = 60/121, 50%) and false negatives (n = 61/121, 50%). Seven clinical factors were predictive of discordant screening for both depression and anxiety: greater dissociative symptoms, greater patient-reported adverse events, subjective cognitive impairment, negative affect, detachment, disinhibition, and psychoticism. When the analyses were restricted to only patients with psychogenic nonepileptic seizures (PNES) or epilepsy, the rate of discordant depression screening was higher in the PNES group (n = 29, 47%) compared to the epilepsy group (n = 70, 30%, Bayes factor for the alternative hypothesis = 4.65).

Significance

Patients with seizure disorders who self-report a variety of psychiatric and other symptoms should be evaluated more thoroughly for depression and anxiety, regardless of screening test results, especially if they have PNES and not epilepsy. Clinical assessment by a qualified psychiatrist remains essential in diagnosing depressive and anxiety disorders among such patients.

Commentary

Physicians are curious sorts of people, geared towards puzzle and problem solving in high stakes life and death environments. Not surprisingly, they are trained and ingrained to come up with answers to questions even when they are particularly vexing or distressing. Ambiguity is uncomfortable for physicians, especially for neurologists, for that is a field where even if treatments are not forthcoming, precise clarification and characterization are expected outcomes of clinical encounters.

The problem-solving itself in epileptology has dramatically changed in recent years. Attention to comorbidity and to the notion of comprehensive care have become conventional wisdom. Today, few epileptologists, albeit skewed towards the younger demographic, ignore psychiatric symptoms that may be associated with epilepsy syndromes. The idea that neurologists are capable of managing psychiatric symptoms is openly encouraged, a notion that would have been dubious even a decade ago.^1,2

Yet, significant challenges still exist in finding sensible and discrete heuristics when diagnosing and treating comorbidity. The use of rating scales to help elucidate psychiatric symptoms in epilepsy is a natural offshoot of current clinical practice. Modern medicine depends upon data, whether it be lab data, EEG data, or rating scale data, all of which ideally reflects well defined symptoms. As such, it is no accident that epileptologists seek a data-based approach for accurately identifying psychiatric illness associated with seizure disorder. Holper and colleagues ³ have attempted just that, in a sophisticated study assessing the efficacy of rating scales for identifying depression or anxiety comorbid with epilepsy.

The study itself is solidly designed, using a wide variety of questionnaire measures, including the very well accepted Neurological Disorders Depression Inventory for Epilepsy (NDDI-E) and Generalized Anxiety Disorder scale (GAD-7) to identify depression and anxiety respectively. The sample is well defined, from a tertiary care epilepsy monitoring unit in Australia. The rating scale results are compared to diagnostic findings from direct psychiatric evaluations. The intuitive and hopeful hypothesis is that rating scales could effectively serve as evaluative tools for detecting psychiatric comorbidity in epilepsy. Such an approach would be groundbreaking, proving to efficiently and effectively provide comprehensive care to persons with epilepsy. Unfortunately, that is not what happened.

The rating scales, even these high-quality, well accepted, commonly used rating scales, were discordant with the psychiatrists’ findings one third of the time, with the discrepancies nearly equally divided into false positives and false negatives. Several factors were associated with the discordance, such as patient experience of adverse events, dissociation, or cognitive impairment. However, the ultimate result was that for a substantial percentage of patients, either the rating scales were wrong or the psychiatric assessments were wrong. The rating scales were even worse for persons with psychogenic non epileptic seizures (PNES), nearly 50% discordant. Either way, screening failure to this degree is very disappointing, and raises serious questions about the utility of rating scales in this population. How did it happen that these solidly validated and reliable rating scales have proven to be so practically ineffective?

Perhaps, the problem started long ago, with the birth of psychiatry itself. Back when Jean Martin Charcot, a neurologist, was inducing psychogenic non epileptic episodes, the novel and mysterious idea of internal psychic conflicts challenged existing norms. The notion of an inner mental state had previously been the domain of philosophers. But the emergent scientific understanding of the mind was powerfully seductive while at the same time raising questions about how to interpret and quantify such states of mind. Freud’s titillating and intoxicating views of psychiatry and good old-fashioned demagoguery became the prevailing paradigm. The idea that psychic phenomena drove our being was enticing, not only for its multi-dimensional view of humanity, but also for the implication that improvement in mental states was possible.

Of course, demagoguery always declines, sometimes replaced by new practitioners, but often replaced by new sensibilities. As medicine progressed in terms of practical measurement and algorithmic treatment, psychiatry too developed a more logical data-based approach. The outcome was the development of the diagnostic and statistical manual (DSM). It was a vast improvement and comforting to consider that clear categorical criteria could explain mental illness. The DSM showed that identifying a certain number of categorical symptom criteria would tidily convert into a diagnosis and even a prescription, often conveniently marketed by the same journal publishing the research papers.

The development of questionnaire-based screening tools was a natural development in psychiatry. Screening was a convenient way for non-psychiatrists and psychiatrists alike to gain confidence that their potentially biased or flawed assessments could be corroborated or even validated. It would be straightforward to screen vulnerable populations and then to develop comprehensive treatment protocols. However, Holper et al have proven that it is not so simple.

The fault is probably not with the rating scales. The scales are high quality, and for depression in epilepsy, there are none better than the NDDI-E. It is hard to blame the psychiatrists doing the clinical evaluations for the discordance. The researchers themselves seem to accept those assessments as gold standards. Perhaps the fault lies with the heavy onus placed upon seeking discrete and simplified data in psychiatry. That may be the ultimate lesson from Holper et al. The fact that the psychiatric rating scales were wrong more than one third of the time, suggests that this approach is markedly flawed. As appealing as it may be, rating scales cannot be considered acceptable as a standard for psychiatric screening in this population, and quite possibly, they should not be used at all.

Physicians are compelled to seek order from disorder and to make logical treatment plans. Elegant physiologic explanations are pleasing and even euphoria inducing for any hardcore neuroscientist. Although psychiatric evaluation and psychiatrists themselves may be biased and flawed, the rating scales appear to be worse. Despite the field’s attempts to operationalize, psychiatry is just not so orderly. Psychiatric comorbidity, and by logical extension, epilepsy itself, just may not be that simple.