Sage Journals: Discover world-class research

Abstract

Objectives:

Optimal depression screening necessitates measurement tools that are valid across varied populations and in the presence of comorbidities.

Methods:

This study assessed the test properties of two versions of the Center for Epidemiologic Studies Depression scale against psychiatric diagnoses established by the Mini International Neuropsychiatric Interview among a clinical sample of US Veterans deployed during Operations Enduring Freedom, Iraqi Freedom, and New Dawn. Participants (N = 359) recruited from two Department of Veterans Affairs hospitals completed a clinical interview, structured diagnostic interview, and self-reported measures.

Results:

Based on diagnostic interview and the Diagnostic and Statistical Manual of Mental Disorders 4th Edition criteria, 29.5% of the sample met diagnostic criteria for major depressive disorder and 26.5% met diagnostic criteria for post-traumatic stress disorder. Both Center for Epidemiologic Studies Depression-20 and Center for Epidemiologic Studies Depression-10 scales performed well and almost identically against the Mini International Neuropsychiatric Interview-major depressive disorder in identifying Veterans with major depressive disorder (Center for Epidemiologic Studies Depression-20 area under the Receiver Operating Characteristic curve 91%; Center for Epidemiologic Studies Depression-10 area under the ROC curve 90%). Overall, higher cut points for the Center for Epidemiologic Studies Depression scales performed better in correctly identifying true positives and true negatives for major depressive disorder (Center for Epidemiologic Studies Depression-20 cut point 18+ sensitivity 92% specificity 72%; Center for Epidemiologic Studies Depression-10 cut point 10+ sensitivity 92% specificity 69%).

Conclusions:

The specificity of the Center for Epidemiologic Studies Depression scales was poor among Veterans with co-occurring post-traumatic stress disorder (13% and 16%). Veterans with post-traumatic stress disorder who have a positive depression screen should have a more thorough assessment of mental health symptoms and comorbidities, rather than immediate diagnosis of and treatment for depression.

Keywords

Depression major depressive disorder post-traumatic stress disorder psychometrics Veterans

Introduction

Major depressive disorder (MDD) is a common mental health problem affecting nearly one out of every six adults in the United States.¹ This condition is associated with significant disability, symptom burden, and health care costs.^1,2 Identifying depression in community and primary care settings allows for targeting interventions,³ but screening tools for depression may not be equally valid across clinical populations, or in the context of medical or other psychiatric comorbidities.

Veterans have much higher levels of depression relative to the general US population,⁴ including younger cohorts of Veterans involved in the Iraq and Afghanistan conflicts.⁵ Veterans are also at higher risk for other debilitating mental health conditions, including post-traumatic stress disorder (PTSD) and traumatic brain injury.^6–8 Adequately assessing health needs of returning service members is important both for providing clinical care and allocating resources.

The accurate identification of MDD may pose significant challenges because it co-occurs and overlaps with other mental health conditions. A number of widely used scales may accurately identify depressive symptoms, although they may not correlate entirely with a clinical diagnosis. The Center for Epidemiologic Studies Depression scale (CES-D) has wide applicability and utility in identifying depressive symptoms in people with traumatic injury and disability in VA health care settings^9,10 and has consistently demonstrated excellent screening properties for detecting depression outcomes.¹¹ In addition, full and shortened versions of the CES-D scale have been used to assess depression outcomes, with the shortened scale demonstrating comparable test properties to the full scale when used as a screening tool to identify MDD.^12–14

There is a high degree of comorbidity and symptom overlap between PTSD and MDD among Veterans.^15–17 Patients with depression alone and patients with PTSD and depression together exhibit very different disease severities, symptomatologies, and progression. Further, patients diagnosed with both MDD and PTSD experience delayed response to depression treatment.¹⁶ The presence of PTSD may thus alter the properties of depression screening tests.

The objective of this study is to evaluate the validity of the CES-D (the full 20-item and shortened 10-item scales) at various cutoff points for a population of Veterans using a gold standard established by the Mini International Neuropsychiatric Interview (MINI-MDD). In order to assess the validity of the full and shortened CES-D scales, we examined the sensitivity, specificity, and receiver operating characteristic (ROC) curves. In addition, we examined these test properties for Veterans with PTSD and without PTSD.

Methodology

This was a cross-sectional study of a cohort of returned Veterans from Operations Enduring Freedom (OEF), Iraqi Freedom (OIF), and New Dawn (OND) who were either receiving care or had registered for care at VA health care facilities. Recruitment details of the study population are described elsewhere.⁸

Study sites

Data were drawn from two (one northern and one southeastern) of the five VA facilities with regional inpatient Polytrauma Rehabilitation Centers (PRCs). Both facilities are large, tertiary care hospitals that provide a broad range of comprehensive medical and mental health care, and both maintain a comprehensive list (OEF/OIF/OND registry) of current or former service members who have applied for VA services in their region. Full details of the parent study protocols, informed consent procedures, and Institutional Review Board approvals are published elsewhere.⁸ Informed consent procedures were approved by the Institutional Review Boards and Research and Development Committees for each of the two VA facilities (one northern and one southeastern) that conducted participant enrollment and data collection. As secondary data analysis of existing, de-identified clinical information, the current study was exempt from human subject review by the Institutional Review Board of Oregon Health & Science University.

Eligibility and recruitment

Eligible participants were deployed during OEF/OIF/OND between October 2001 through September 2010, able to read and write in English, and able to complete study forms, scales, and provide informed consent. Eligible individuals with moderate to high traumatic brain injury and who met criteria for minimal to moderate cognitive impairment¹⁸ required clearance by an attending physician in order to participate.

Measures

Electronic health record data

Descriptive information was extracted from electronic health records, including demographic information, physical and psychiatric diagnoses, and health history.

MINI diagnostic interviews

Psychiatric diagnoses were obtained via the MINI.¹⁹ The MINI is a brief, validated, structured clinical interview designed to provide reliable Axis I psychiatric diagnoses in accordance with the Diagnostic and Statistical Manual of Mental Disorders (DSM-IV) for various psychiatric disorders.²⁰ The MINI battery of clinical interviews administered to the study population yields diagnoses for a number of psychiatric conditions, including MDD and PTSD. Each MINI measure was dichotomized as having met criteria for having the condition or not, using the established cut points.¹⁹

Depressive symptoms

Depressive symptomatology was assessed using the full, self-reported 20-item Center for Epidemiologic Studies-Depression Scale (CESD-20) as well as the shortened 10-item CES-D (CESD-10) (see Appendix 1).^12,21 Total scores range from 0 to 60 for the CESD-20 scale and from 0 to 30 for the CESD-10, with higher scores indicating greater depressive symptoms. Previously, while the norm for the cut point of the CESD-20 has been 16 points, some authors have argued for cut points at 20 or above.^22–24 We assessed various cut points: 16+ and 18+ for the CESD-20 and 8+ and 10+ for the CESD-10.^12,13,25

Statistical analyses

We assessed the validity of the CESD-20 and CESD-10 against MDD, as defined by the MINI diagnostic interview. We assessed the sensitivity and specificity of the CESD-10 and the CESD-20 against the MINI-MDD. More specific properties of the CES-D scales were examined with ROC curves. The area under the ROC curve is interpreted as the probability that the result of the diagnostic test of a randomly selected individual with MDD will be greater than the result of the same diagnostic test from a randomly selected healthy individual. This approach also can be used to delineate cut points for the scales. The greater the area under the ROC curve the better the diagnostic test performs. We used the cut-offs of 0.9–1 to indicate excellent accuracy versus the MINI, 0.8–0.9 to indicate good accuracy, and 0.7–0.8 to indicate fair accuracy.²⁶ Because PTSD is a highly significant and frequently co-occurring condition for this population, we examined whether the performance of the CES-D was similar among Veterans with and without PTSD using the MINI-PTSD diagnostic assessment. We used Stata version 13.1 to conduct all analyses.

Results

This study included 359 Veterans, 91% of whom were male. The mean age was 35 years. The sample demographic characteristics are summarized in Table 1. Most participants were married or living with a partner (59%) and the vast majority were non-Hispanic white (77%). Few medical comorbidities were assessed in this population. 56% were experiencing clinically significant pain (assessed by a rating of usual pain intensity over the last week of >3 on the Numeric Rating Scale, ranging from 0 to 10) and 16% reported postconcussional disorder (assessed using DSM-IV diagnostic criteria). The MINI psychiatric interviews identified sample population prevalence of 29.5% for MDD, 26.5% for PTSD, 28.7% for agoraphobia, 23.1% for hypomania or mania, 18.1% for panic disorder, and 13.9% for alcohol dependence. Approximately half of the study population scored above the lower cut points on the CES-D depressive symptoms scales. The mean score for depressive symptoms was 18.5 for the CESD-20; for the CESD-10 the mean was 10.2.

Table 1.

Descriptive results of a sample of n = 359 US Veterans of Operations Enduring Freedom (OEF) Iraqi Freedom (OIF) New Dawn (OND).

Characteristic	N (%)	Mean (SD)	Range
Men	327 (91)
Age		35.1 (9.7)	20–66
Marital status
Married	188 (52)
Co-habitating	24 (6.7)
Divorced/separated	60 (16.7)
Never married	87 (24.2)
Years education (n = 356)		14.5 (2.0)	12–22
Race
Non-Hispanic white	278 (77.4)
Non-Hispanic black	34 (9.5)
Hispanic	38 (10.6)
Asian	6 (1.7)
Other	3 (0.8)
Clinically Significant Pain^a	201 (56)
Post-concussional disorder	58 (16.2)
Met criteria:
MINI-MDD	106 (29.5)
MINI-PTSD	95 (26.5)
MINI-Dysthymia	21 (6.6)
MINI-Panic disorder	65 (18.1)
MINI-ETOH dependence	50 (13.9)
MINI-Psychotic conditions	18 (5)
MINI-Agoraphobia	103 (28.7)
MINI-Hypomania or mania	83 (23.1)
MINI-Social phobia	34 (9.5)
CESD-20		18.5 (12.9)	0–56
Positive for depressive symptoms (16+ points)	182 (50.7)
Positive for depressive symptoms (18+ points)	169 (47.1)
CESD-10		10.2 (7.2)	0–30
Positive for depressive symptoms (8+ points)	205 (57.1)
Positive for depressive symptoms (10+ points)	175 (48.7)

SD: standard deviation; MINI-MDD: Mini International Neuropsychiatric Interview Diagnosis for Major Depressive Disorder; MINI-PTSD: Mini International Neuropsychiatric Interview Diagnosis for Post-Traumatic Stress Disorder; CES-D: Center for Epidemiologic Studies Depression Scale.

Clinically Significant Pain was defined as “usual” pain intensity over the last week rated as >3 on the Numeric Rating Scale (range 0–10).

Both the CESD-20 and CESD-10 performed well against the MINI-MDD in identifying MDD at the various cut points examined (Table 2; Figure 1). The test properties of the CESD-20 were comparable for the 16+ cut point (sensitivity 95%, specificity 68%, area under ROC curve 91%) and the 18+ cut point (sensitivity 92%, specificity 72%, area under ROC curve 91%). For the CESD-10, the test properties were also fairly similar for the 8+ cut point (sensitivity 97%, specificity 60%, area under the ROC curve 90%) compared to the 10+ cut point (sensitivity 92%, specificity 69%, area under ROC curve 90%). Figure 1 illustrates the excellent test properties of the full and shortened CES-D scales against the MINI-MDD, and the very similar performance of the two CES-D scales.

Table 2.

Descriptive information for condition overlap between MINI-MDD and MINI-PTSD (met criteria) among US Veterans of OEF/OIF/OND.

Condition overlap for MINI-MDD and MINI-PTSD (met criteria), n (percent of total)
	MINI-MDD Positive	MINI-MDD Negative
MINI-PTSD Positive	64 (18%)	31 (9%)
MINI-PTSD Negative	42 (12%)	222 (62%)
CESD-20 and CESD-10 scores based on MINI-MDD and MINI-PTSD categories, mean(sd)
	MINI-MDD Positive	MINI-MDD Negative
MINI-PTSD Positive	34.0 (8.8)/18.6 (4.6)	25.4 (8.7)/14.1 (5.3)
MINI-PTSD Negative	28.3 (10.2)/15.8 (5.7)	11.1 (8.3)/6.2 (4.8)

MINI-MDD: Mini International Neuropsychiatric Interview Diagnosis for Major Depressive Disorder; MINI-PTSD: Mini International Neuropsychiatric Interview Diagnosis for Post-Traumatic Stress Disorder; SD: standard deviation; CES-D: Center for Epidemiologic Studies Depression Scale.

Figure 1.

Receiver Operating Characteristic (ROC) curves for MINI-major depressive disorder and CES-D.

The stratified analyses by Veterans with and without PTSD showed differences in the performance of the CES-D against the MINI-MDD. We examined both CES-D scales at the higher cut points because of the comparable test properties and higher specificities demonstrated by the CESD-20 at the 18+ cut point and the CESD-10 at the 10+ cut point. Both scales were more accurate in identifying MDD for Veterans without PTSD (for CESD-20: sensitivity 81%, specificity 80%, area under ROC curve 90%; for CESD-10: sensitivity 81%, specificity 77%, area under ROC curve 89%) (Table 3; bottom of Figure 2). For Veterans with PTSD, however, the specificity was quite low for both the CESD-20 (sensitivity 98%, 13% specificity, area under ROC curve 75%) and CESD-10 (sensitivity 98%, specificity 16%, area under ROC curve 73%) (Top of Figure 2; Table 4).

Table 3.

Validity of Center for Epidemiologic Studies Depression Scale (CES-D) Versions compared to the mini international neuropsychiatric interview among US Veterans of OEF/OIF/OND.

	Major Depressive Disorder using Mini International Neuropsychiatric Interview
CESD-20 (cut point 16+)
Sensitivity	95.28%
Specificity	67.98%
Area under ROC curve	0.91
CESD-20 (cut point 18+)
Sensitivity	91.51%
Specificity	71.54%
Area under ROC curve	0.91
CESD-10 (cut point 8+)
Sensitivity	97.17%
Specificity	59.68%
Area under ROC curve	0.90
CESD-10 (cut point 10+)
Sensitivity	91.51%
Specificity	69.17%
Area under ROC curve	0.90

CESD: Center for Epidemiologic Studies Depression Scale; ROC: Receiver Operating Characteristic.

Figure 2.

Receiver Operating Characteristic (ROC) curves for MINI-MDD and CES-D for Veterans with (n = 95) and without (n=264) PTSD.

Table 4.

Differences in validity by diagnostic subgroups among US Veterans of OEF/OIF/OND.

	Major depression using Mini International Neuropsychiatric Interview
CESD-20 (cut point 18+)
Veterans with PTSD	N = 95
Sensitivity	98.4%
Specificity	12.9%
Area under ROC curve	0.75
Veterans without PTSD	N = 264
Sensitivity	81.0%
Specificity	79.7%
Area under ROC curve	0.90
CESD-10 (cut point 10+)
Veterans with PTSD	N = 95
Sensitivity	98.4%
Specificity	16.1%
Area under ROC curve	0.73
Veterans without PTSD	N = 264
Sensitivity	81.0%
Specificity	76.6%
Area under ROC curve	0.89

PTSD: Post-Traumatic Stress Disorder; CESD: Center for Epidemiologic Studies Depression Scale; ROC: Receiver Operating Characteristic.

Discussion

Using data based on face-to-face clinical interviews and a validated clinical assessment tool as the gold standard, we found that both the CESD-20 and CESD-10 demonstrated excellent test properties in classifying MDD. We found only small differences in test properties between the CESD-20 and the CESD-10 against the MINI-MDD in any of our analyses. There was no evidence that the longer CESD-20 scale provided any meaningful benefits. We found moderate support for using the higher diagnostic cut points on both scales. Our results confirm the utility of using the CES-D scale, and the shortened CESD-10 battery of questions in particular, to screen Veterans for MDD.

Importantly, we found that the test properties of the CES-D are different for Veterans with and without PTSD. The CES-D scales performed better at identifying MDD for Veterans without co-occurring PTSD. For those with co-occurring PTSD, the specificity was quite low (13% and 16%) at the cut points we used. These findings could argue for a higher MDD cut point on the CES-D scale for Veterans with PTSD, but doing so would diminish the sensitivity of the tests (as seen in the upper graphs in Figure 2). Instead, we recommend that any Veteran with PTSD who has a positive depression screen should have a thorough assessment of depressive symptoms and comorbidities prior to diagnosis or treatment.

PTSD shows heterogeneous symptoms, and other research has found substantial symptom overlap between PTSD and MDD. Several key symptoms, especially numbness and dysphoria, appear to explain this overlap.¹⁵ Our findings are generally consistent with research suggesting that these two conditions may influence each other in significant ways.^16,17 Our results do not resolve whether the two are part of a single general mood disorder, as has been proposed.²⁷ In all, 18% of our study sample had co-occurring MDD and PTSD, which was considerably higher than the 9% found in a primary care VA population¹⁵ but a large number had only one of the conditions. Additional research can identify areas of overlap and differentiation, as well as the consequences (such as differences in longitudinal course) of various manifestations of PTSD and MDD.

It is likely that the PHQ-9, which the VA has adopted for use during clinical encounters,²⁸ operates similarly to the CES-D in terms of test properties.²⁹ However, there has been limited research about the test properties of the PHQ-9 among broader Veteran populations, or about its properties among Veterans with co-occurring PTSD. Further examination of how the CES-D and PHQ-9 operate similarly or differently is warranted, given utility and applicability of each in population-based and clinical settings, respectively.²⁹

There are several advantages to using screening survey measures. First, self-reported survey tools can quickly ascertain information from respondents at-large. As a result, health care systems such as the VA could feasibly distribute the CES-D questionnaire widely to difficult-to-reach populations. Second, clinical interviews such as the MINI-MDD require face-to-face assessments of patients conducted by trained professionals. In contrast, the CES-D need not be administered face-to-face, and data collection can be attained through various modes—phone-assisted interviews, mailed paper questionnaires, or via Internet access.³⁰ Measures used in self-reported surveys can be standardized for comparisons to other populations. Our study encourages the use of the CESD-10 scale, with a cut point of 10, but with the caveat that the test has low specificity among Veterans with co-occurring PTSD. It is possible that screening for specific symptoms of PTSD and depression (such as numbness and dysphoria)¹⁵ could promote more targeted mental health interventions, since the group in this overlap seems to experience the most severe symptoms.

Some aspects of our study may limit its clinical relevance. While the CES-D indices may be useful in identifying Veterans with likely MDD, they do not assess either the severity of depressive symptoms, or comorbid mental health diagnoses such as anxiety disorders, PTSD, and substance abuse, which are important factors for clinical care. It will be important in future research to clarify the degree and consequences of overlap between comorbid mental conditions²⁷ as well as with co-occurring medical conditions. Initial screening with the CESD-10 would require follow-up with additional clinical history, or another assessment tool, especially in the setting of PTSD. However, if the goal is to maximize identification, assessment, and treatment of Veterans with MDD, the CESD-10 may be a particularly useful and viable screening tool for Veteran populations at large.

Conclusion

This study confirmed the predictive accuracy of the CESD-20 and CESD-10 in identifying Veterans with MDD. The CESD-20 offered no advantages. However, the specificity of the CES-D for identifying MDD was low among Veterans with co-occurring PTSD, which encourages more detailed mental health assessment in that group.

Footnotes

Appendix 1

The original CES-D contains 20 items. The list below shows the original CESD-20 item notations, the corresponding numbers for the CESD-10 in parentheses. Positive items are in italics. Scoring is a simple sum (after reversing scores for the positive items).

During the past week:

Declaration of conflicting interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Ethics approval

Ethical approval for this study was waived by Oregon Health & Science University’s Institutional Review Board because analyses were performed on de-identified secondary data.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Support for this article was provided in part by the Department of Veterans Affairs Health Services Research & Development grant SDR-07-047 (ME Clark), by Department of Defense Award Grant (W81XWH-11-2-0189), and the Sierra Pacific Mental Illness Research, Education and Clinical Center (MIRECC). Dr. Quiñones is supported by the American Diabetes Association (ADA 7-13-CD-08). Dr. Thielke is supported by the National Institute of Mental Health (K23 MH093591). All authors are each supported by their respective Intuitions.

Informed consent

Informed consent was not sought for the present study because the study involved secondary data analysis of existing data.

References

Kessler

Berglund

Demler

. The epidemiology of major depressive disorder: results from the National Comorbidity Survey Replication (NCS-R). JAMA 2003; 289: 3095–3105.

Sullivan

Neale

Kendler

KS.

Genetic epidemiology of major depression: review and meta-analysis. Am J Psychiatry 2000; 157: 1552–1562.

Coventry

Hudson

Kontopantelis

. Characteristics of effective collaborative care for treatment of depression: a systematic review and meta-regression of 74 randomised controlled trials. PLoS ONE 2014; 9: e108114.

Zivin

Kim

McCarthy

. Suicide mortality among individuals receiving treatment for depression in the Veterans Affairs health system: associations with patient and treatment setting characteristics. Am J Public Health 2007; 97: 2193–2198.

Seal

Metzler

Gima

. Trends and risk factors for mental health diagnoses among Iraq and Afghanistan veterans using Department of Veterans Affairs health care, 2002–2008. Am J Public Health 2009; 99: 1651–1658.

Lew

Tun

Cifu

DX.

Prevalence of chronic pain, posttraumatic stress disorder, and persistent postconcussive symptoms in OIF/OEF veterans: polytrauma clinical triad. J Rehabil Res Dev 2009; 46: 697–702.

Morissette

Woodward

Kimbrel

. Deployment-related TBI, persistent postconcussive symptoms, PTSD, and depression in OEF/OIF veterans. Rehabil Psychol 2011; 56: 340–350.

Phillips

Clark

Gironda

. Pain and Emotional Comorbidities among Two Groups of Iraq and Afghanistan Era Veterans. J Rehabil Res Dev, in press.

Pickard

Dalal

Bushnell

DM.

A comparison of depressive symptoms in stroke and primary care: applying Rasch models to evaluate the center for epidemiologic studies-depression scale. Value Health 2006; 9: 59–64.

10.

Kilbourne

Justice

Rollman

. Clinical importance of HIV and depressive symptoms among veterans with HIV infection. J Gen Intern Med 2002; 17: 512–520.

11.

Watson

Pignone

MP.

Screening accuracy for late-life depression in primary care: a systematic review. J Fam Pract 2003; 52: 956–964.

12.

Andresen

Malmgren

Carter

. Screening for depression in well older adults: evaluation of a short form of the CES-D. Am J Prev Med 1994; 10: 77–84.

13.

Andresen

Byers

Friary

. Performance of the 10-item Center for Epidemiologic Studies Depression scale for caregiving research. SAGE Open Med 2013; 1: 2050312113514576.

14.

Irwin

Artin

Oxman

MN.

Screening for depression in the older adult: criterion validity of the 10-item Center for Epidemiological Studies Depression Scale (CES-D). Arch Intern Med 1999; 159: 1701–1704.

15.

Gros

Price

Magruder

. Symptom overlap in posttraumatic stress disorder and major depression. Psychiatry Res 2012; 196: 267–270.

16.

Campbell

Felker

Liu

. Prevalence of depression-PTSD comorbidity: implications for clinical practice guidelines and primary care-based interventions. J Gen Intern Med 2007; 22: 711–718.

17.

Franklin

Zimmerman

Posttraumatic stress disorder and major depressive disorder: investigating the role of overlapping symptoms in diagnostic comorbidity. J Nerv Ment Dis 2001; 189: 548–551.

18.

Hagen

Malkmus

Durham

Rehabilitation of the head injured adult: comprehensive physical management. Downey, CA: Professional Staff Association of Rancho Los Amigos Hospital, 1979, pp. 87–89.

19.

Sheehan

Lecrubier

Sheehan

. The Mini-International Neuropsychiatric Interview (MINI): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. J Clin Psychiatry 1998; 59: 22–33.

20.

American Psychiatric Association (APA). Diagnostic and statistical manual-text revision (DSM-IV-TRim, 2000). Washington, DC: APA, 2000.

21.

Radloff

LS.

The CES-D scale: a self-report depression scale for research in the general population. Appl Psych Meas 1977; 1: 385–401.

22.

Himmelfarb

Murrell

SA.

Reliability and validity of five mental health scales in older persons. J Gerontol 1983; 38: 333–339.

23.

Murrell

Himmelfarb

Wright

Prevalence of depression and its correlates in older adults. Am J Epidemiol 1983; 117: 173–185.

24.

Julian

Gregorich

Tonner

. Using the Center for Epidemiologic Studies Depression Scale to screen for depression in systemic lupus erythematosus. Arthritis Care Res 2011; 63: 884–890.

25.

Sakakibara

Miller

Orenczuk

. A systematic review of depression and anxiety measures used with individuals with spinal cord injury. Spinal Cord 2009; 47: 841–851.

26.

Fawcett

ROC graphs: notes and practical considerations for researchers. Mach Learn 2004; 31: 1–38.

27.

Najavits

Capezza

NM.

Depression and posttraumatic stress disorder comorbidity. In: Richards

O’Hara

(eds) The Oxford handbook of depression and comorbidity. New York: Oxford University Press, 2014, pp. 100–110.

28.

Kroenke

Spitzer

Williams

JB.

The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 2001; 16: 606–613.

29.

Smarr

Keefer

AL.

Measures of depression and depressive symptoms: Beck Depression Inventory-II (BDI-II), Center for Epidemiologic Studies Depression Scale (CES-D), Geriatric Depression Scale (GDS), Hospital Anxiety and Depression Scale (HADS), and Patient Health Questionnaire-9 (PHQ-9). Arthritis Care Res 2011; 63: S454–S466.

30.

Houston

Cooper

. Screening the public for depression through the Internet. Psychiatr Serv 2001; 52: 362–367.

Validity of Center for Epidemiologic Studies Depression (CES-D) scale in a sample of Iraq and Afghanistan Veterans