The authors report on the development and calibration of item banks for depression, anxiety, and anger as part of the Patient-Reported Outcomes Measurement Information System (PROMIS®). Comprehensive literature searches yielded an initial bank of 1,404 items from 305 instruments. After qualitative item analysis (including focus groups and cognitive interviewing), 168 items (56 for each construct) were written in a first person, past tense format with a 7-day time frame and five response options reflecting frequency. The calibration sample included nearly 15,000 respondents. Final banks of 28, 29, and 29 items were calibrated for depression, anxiety, and anger, respectively, using item response theory. Test information curves showed that the PROMIS item banks provided more information than conventional measures in a range of severity from approximately −1 to +3 standard deviations (with higher scores indicating greater distress). Short forms consisting of seven to eight items provided information comparable to legacy measures containing more items.
American Psychiatric Association. (2000). Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, Text Revision. Washington, DC: Author .
2.
Beck, A.T., Epstein, N., Brown, G., & Steer, R.A. ( 1988). An inventory for measuring clinical anxiety: Psychometric properties. Journal of Consulting and Clinical Psychology , 56, 893-897.
3.
Bejar, I.I. ( 1977). An application of the continuous response level model to personality measurement. Applied Psychological Measurement , 1, 509-521.
4.
Bernstein, I.H., Rush, A.J., Thomas, C.J., Woo, A., & Trivedi, M.H. ( 2006). Item response analysis of the Inventory of Depressive Symptomatology. Neuropsychiatric Disease and Treatment, 2, 557-564.
5.
Berzon, R., Patrick, D., Guyatt, G., & Conley, J.M. ( 1994). Intellectual property considerations in the development and use of HRQL measures for clinical trial research. Quality of Life Research, 3, 273-277.
6.
Bjorner, J.B., Chang, C.H., Thissen, D., & Reeve, B.B. ( 2007). Developing tailored instruments: Item banking and computerized adaptive assessment. Quality of Life Research, 16, 95-108.
7.
Bjorner, J.B., Smith, K.J., Orlando, M., Stone, C.A., Thissen, D., & Xiaowa, S. ( 2006). IRTFIT: A macro for item fit and local dependence tests under IRT models [Computer software]. Chapel Hill, NC: L. L. Thurstone Psychometric Laboratory.
8.
Bock, R.D. ( 1997). A brief history of item response theory. Educational Measurement: Issues and Practice, 16, 21-33.
9.
Bode, R.K., Lai, J.S., Cella, D., & Heinemann, A.W. (2003). Issues in the development of an item bank. Archives of Physical Medicine and Rehabilitation , 84(Suppl.2), S52-S60.
10.
Brod, M., Tesler, L.E., & Christensen, T.L. (2009). Qualitative research and content validity: Developing best practices based on science and experience . Quality of Life Research, 18, 1263-1278.
11.
Brown, T.A., Chorpita, B.F., & Barlow, D.H. ( 1998). Structural relationships among dimensions of the DSM-IV anxiety and mood disorders and dimensions of negative affect, positive affect, and autonomic arousal. Journal of Abnormal Psychology, 107, 179-192.
12.
Buss, A.H., & Perry, M. ( 1992). The aggression questionnaire. Journal of Personality and Social Psychology, 59, 73-81.
13.
Buysse, D.J., Moul, D.E., Germain, A., Yu, L., Stover, A.M., Dodds, N.E., . . . Pilkonis, P. A. (2010). Development and validation of patient-reported outcome measures for sleep disturbance and sleep-related impairments. Sleep, 33, 781-792.
14.
Carmody, T.J., Rush, A.J., Bernstein, I., Warden, D., Brannan, S., Burnham, D., . . . Trivedi, M.H. ( 2006). The Montgomery Asberg and the Hamilton ratings of depression: A comparison of measures. European Neuropsychopharmacology , 16, 601-611.
15.
Cella, D., Gershon, R., Lai, J.-S., & Choi, S. ( 2007). The future of outcomes measurement: Item banking, tailored short forms, and computerized adaptive assessment. Quality of Life Research, 16, 133-144.
16.
Cella, D., Riley, W., Stone, A., Rothrock, N., Reeve, B., Yount, S., . . . Hays, R.D. ( 2010). The Patient-Reported Outcomes Measurement Information System (PROMIS) developed and tested its first wave of adult self-reported health outcome item banks: 2005-2008. Journal of Clinical Epidemiology, 63, 1179-1194.
17.
Cella, D., Yount, S., Rothrock, N., Gershon, R., Cook, K., Reeve, B., . . . Matthias, R. ( 2007). The Patient-Reported Outcomes Measurement Information System (PROMIS): Progress of an NIH Roadmap cooperative group during its first two years. Medical Care, 45(Suppl.1), S3-S11.
18.
Chang, C.-H., Cella, D., Clarke, S., Heinemann, A., Von Roenn, J.H., & Harvey, R. ( 2003). Should symptoms be scaled for intensity, frequency, or both?Palliative & Supportive Care, 1,51-60.
19.
Chang, C.-H., & Reeve, B.B. ( 2005). Item response theory and its applications to patient-reported outcomes measurement. Evaluation & the Health Professions , 28, 264-282.
20.
Choi, S.W., Reise, S.P., Pilkonis, P.A., Hays, R.D., & Cella, D. ( 2010). Efficiency of static and computer adaptive short forms compared to full length measures of depressive symptoms. Quality of Life Research, 19, 125-136.
Chorpita, B.F., Albano, A.M., & Barlow, D.H. ( 1998). The structure of negative emotions in a clinical sample of children and adolescents. Journal of Abnormal Psychology , 107, 74-85.
23.
Clark, D.C., vonAmmon Cavanaugh, S., & Gibbons, R.D. ( 1983). The core symptoms of depression in medical and psychiatric patients. Journal of Nervous and Mental Disease, 171, 705-713.
24.
Clark, L.A., & Watson, D. ( 1991). Tripartite model of anxiety and depression: Psychometric evidence and taxonomic implications. Journal of Abnormal Psychology , 100, 316-336.
25.
Cox, D.E., & Harrison, D.W. (2008). Models of anger: Contributions from psychophysiology, neuropsychology, and the cognitive behavioral perspective . Brain Structure & Function, 212, 371-385.
26.
de Jong-Gierveld, J., & Kamphuis, F. ( 1985). The development of a Rasch-type loneliness scale. Applied Psychological Measurement, 9, 289-299.
27.
Derogatis, L.R. ( 1983). SCL-90-R: Administration, scoring and procedures manual II. Towson, MD: Clinical Psychometric Research.
28.
DeWalt, D.A., Rothrock, N., Yount, S., & Stone, A.A. ( 2007). Evaluation of item candidates: The PROMIS qualitative item review. Medical Care, 45(Suppl. 1), S12-S21.
29.
Fliege, H., Becker, J., Walter, O.B., Bjorner, J.B., Klapp, B.F., & Rose, M. ( 2005). Development of a computer-adaptive test for depression (D-CAT). Quality of Life Research, 14, 2277-2291.
30.
Food and Drug Administration. (2009). Guidance for industry: Patient-reported outcome measures: Use in medical product development to support labeling claims. Retrieved from http://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM193282.pdf
31.
Forbey, J.D., & Ben-Porath, Y.S. (2007). Computerized adaptive personality testing: A review and illustration with the MMPI-2 computerized adaptive version . Psychological Assessment, 19, 14-24.
32.
Forgays, D.K., Spielberger, C.D., Ottaway, S.A., & Forgays, D.G. ( 1998). Factor structure of the State-Trait Anger Expression Inventory for middle-aged men and women. Assessment, 5, 141-155.
33.
Fries, J.F., Cella, D., Rose, M., Krishnan, E., & Bruce, B. ( 2009). Progress in assessing physical function in arthritis: PROMIS short forms and computerized adaptive testing. Journal of Rheumatology, 36, 2061-2066.
34.
Gardner, W., Kelleher, K.J. & Pajer, K.A. ( 2002). Multidimensional adaptive testing for mental health problems in primary care. Medical Care, 40, 812-823.
35.
Gardner, W., Shear, K., Kelleher, K.J., Pajer, K.A., Mammen, O., Buysse, D., & Frank, E. ( 2004). Computerized adaptive measurement of depression: A simulation study. BMC Psychiatry, 4, 13.
36.
Gershon, R.C., Rothrock, N.E., Hanrahan, R., Jansky, L.J., Harniss, M., & Riley, W. ( 2010). Development of a clinical outcomes research survey application: Assessment Center. Quality of Life Research, 19, 677-685.
37.
Gibbons, R.D., Bock, D., Hedeker, D., Weiss, D.J., Segawa, E., Bhaumik, R., . . . Stover, A. ( 2007). Full-information item bifactor analysis of graded response data. Applied Psychological Measurement, 31, 4-19.
38.
Gibbons, R.D., Clark, D.C., vonAmmon Cavanaugh, S., & Davis, J.M. ( 1985). Application of modern psychometric theory in psychiatric research. Journal of Psychiatric Research, 19, 43-55.
Gibbons, R.D., Weiss, D.J., Kupfer, D.J., Frank, E., Fagiolini, A., Grochocinski, V.J., . . . Immekus, J.C. ( 2008). Using computerized adaptive testing to reduce the burden of mental health assessment. Psychiatric Services, 59, 361-368.
41.
Gollwitzer, M., Eid, M., & Jurgensen, R. ( 2005). Response styles in the assessment of anger expression . Psychological Assessment, 17, 56-69.
42.
Gray, J.A. ( 1987). The psychology of fear and stress (2nd ed.). Cambridge, England: Cambridge University Press.
43.
Hawthorne, G., Mouthaan, J., Forbes, D., & Novaco, R.W. ( 2006). Response categories and anger measurement: Do fewer categories result in poorer measurement? Development of the DAR5. Social Psychiatry & Psychiatric Epidemiology, 41, 164-172.
44.
Hays, R.D., Bjorner, J., Revicki, D.A., Spritzer, K., & Cella, D. ( 2009). Development of physical and mental health summary scores from the Patient-Reported Outcomes Measurement Information System (PROMIS) global items. Quality of Life Research, 18, 873-880.
45.
Kelly, M.A.R., Morse, J.Q., Stover, A., Hofkens, T., Huisman, E., Shulman, S., . . . Pilkonis, P.A. (2011). Describing depression: Congruence between patient experiences and clinical assessments. British Journal of Clinical Psychology, 50, 46-66.
46.
Kendel, F., Wirtz, M., Dunkel, A., Lehmkuhl, E., Hetzer, R., & Regitz-Zagrosek, V. (2010). Screening for depression: Rasch analysis of the dimensional structure of the PHQ-9 and the HADS-D. Journal of Affective Disorders, 122, 241-246.
47.
Kim, Y., Pilkonis, P.A., Frank, E., Thase, M.E., & Reynolds, C.F. , III. (2002). Differential functioning of the Beck Depression Inventory in late-life patients: Use of item response theory. Psychology and Aging, 17, 379-391.
48.
Kingsbury, G.G., & Weiss, D.J. ( 1980). An alternate-forms reliability and concurrent validity comparison of Bayesian adaptive and conventional ability tests (Research Report 80-5). Minneapolis, MN: University of Minnesota.
49.
Kingsbury, G.G., & Weiss, D.J. ( 1983). A comparison of IRTbased adaptive mastery testing and a sequential mastery testing procedure. In D. J. Weiss (Ed.), New horizons in testing: Latent trait theory and computerized adaptive testing (pp. 257-283). New York, NY : Academic Press.
50.
Klem, M.L., Saghafi, E., Abromitis, R., Stover, A., Dew, M.A., & Pilkonis, P.A. (2009). Building PROMIS item banks: Librarians as co-investigators. Quality of Life Research, 18, 881-888.
51.
Krueger, R.F. ( 1999). The structure of common mental disorders. Archives of General Psychiatry, 56, 921-926.
52.
Krueger, R.F., & Finger, M.S. ( 2001). Using item response theory to understand comorbidity among anxiety and unipolar mood disorders. Psychological Assessment , 13, 140-151.
53.
Krueger, R.F., McGue, M., & Iacono, W.G. ( 2001). The higher-order structure of common DSM mental disorders: Internalization, externalization, and their connections to personality. Personality and Individual Differences, 30, 1245-1259.
54.
Lara, D.G., Pinto, O., Akiskal, K., & Akiskal, H.S. ( 2006). Toward an integrative model of the spectrum of mood, behavioral, and personality disorders based on fear and anger traits: 1. Clinical implications. Journal of Affective Disorders, 94, 67-87.
55.
Liu, H., Cella, D., Gershon, R., Shen, J., Morales, L.S., Riley, W., & Hays, R.D. ( 2010). Representativeness of the Patient-Reported Outcomes Measurement Information System Inter-net panel. Journal of Clinical Epidemiology , 63, 1169-1178.
56.
Loevinger, J. ( 1948). The technique of homogenous tests compared with some aspects of scale analysis and factor analysis. Psychological Bulletin, 45, 507-530.
57.
Lord, F.M. ( 1980). Applications of item response theory to practical testing problems. Hillsdale, NJ: Lawrence Erlbaum.
58.
Martin, R., Wan, C., David, J.P., Wegner, E., Olson, B.D., & Watson, D. ( 2000). Style of anger expression: Relation to expressivity, personality, and health. Personality and Social Psychology Bulletin, 25, 1196-1207.
59.
Martin, R., Watson, D., & Wan, C.K. ( 2000). A three-factor model of trait anger: Dimensions of affect, behavior, and cognition. Journal of Personality, 68, 869-897.
60.
McBride, J.R., & Martin, J.R. ( 1983). Reliability and validity of adaptive ability tests in a military setting. In D. J. Weiss (Ed.), New horizons in testing: Latent trait theory and computerized adaptive testing (pp. 223-236). New York, NY: Academic Press.
61.
McHugh, R.K., & Behar, E. ( 2009). Readability of self-report measures of depression and anxiety. Journal of Consulting and Clinical Psychology, 77, 1100-1112.
62.
Meijer, R.R., & Baneke, J.J. ( 2004). Analyzing psychopathology items: A case for nonparametric item response theory modeling. Psychological Methods, 9, 354-368.
63.
MetaMetrics, Inc. (2008). What does the Lexile measure mean?Durham, NC: Author.
64.
Mokkink, L.B., Terwee, C.B., Stratford, P.W., Alonso, J., Patrick, D.L., Riphagen, I., . . . de Vet, H.C.W. (2009). Evaluation of the methodological quality of systematic reviews of health status measurement instruments. Quality of Life Research, 18, 313-333.
65.
Molenaar, I.W., & Sijtsma, K. ( 2000). User’s manual for MSP5 for Windows. Groningen, Netherlands: IEC ProGAMMA.
66.
Muhr, T., & Freise, S. ( 2004). User’s manual for ATLAS.ti 5.0 (2nd ed.). Berlin, Germany: Scientific Software Development.
67.
Mui, A.C., Burnette, D., & Chen, L.M. ( 2001). Cross-cultural assessment of geriatric depression: A review of the CES-D and GDS. Journal of Mental Health and Aging, 7, 137-164.
Orlando, M., & Thissen, D. ( 2003). Further investigation of the performance of S-X2: An item fit index for use with dichotomous item response theory models. Applied Psychological Measurement, 27, 289-298.
70.
Pallant, J.F., & Tennant, A. ( 2007). An introduction to the Rasch measurement model: An example using the Hospital Anxiety and Depression Scale (HADS). British Journal of Clinical Psychology, 46, 1-18.
71.
Pedersen, R.D., Pallay, A.G., & Rudolph, R.L. ( 2002). Can improvement in well-being and functioning be distinguished from depression improvement in antidepressant clinical trials? Quality of Life Research, 11, 9-17.
72.
Polimetrix, Inc. (2006). Scientific sampling for online research. Palo Alto, CA: Author.
73.
Quilty, L.C., Zhang, K.A., & Bagby, M.R. ( 2010). The latent symptom structure of the Beck Depression Inventory-II in outpatients with major depression. Psychological Assessment , 22, 603-608.
74.
Radloff, L.S. ( 1977). A self-report depression scale for research in the general population. Applied Psychological Measurement, 1, 385-401.
75.
Ramsay, J.O. ( 2000). TestGraph: A program for the graphical analysis of multiple choice test and questionnaire data [Computer software and manual]. Montreal, Quebec, Canada: McGill University.
76.
Reeve, B.B., Hays, R.D., Bjorner, J.B., Cook, K.F., Crane, P.K., Teresi, J.A., . . . Cella, D. ( 2007). Psychometric evaluation and calibration of health-related quality of life item banks: Plans for the Patient-Reported Outcomes Measurement Information System (PROMIS). Medical Care, 45(Suppl.1), S22-S31.
77.
Reise, S.P., Moore, M.T., & Haviland, M.G. (IN PRESS). Applying item response theory models to psychological data. In K. F. Geisinger (Ed.), Handbook of testing and assessment in psychology . Washington, DC: American Psychological Association.
78.
Reise, S.P., Morizot, J., & Hays, R.D. ( 2007). The role of the bifactor model in resolving dimensionality issues in health outcomes measures. Quality of Life Research , 16, 19-31.
79.
Reise, S.P., & Waller, N.G. ( 2009). Item response theory and clinical measurement. Annual Review of Clinical Psychology, 5, 25-46.
80.
Revicki, D.A., Chen, W.-H., Harnam, N., Cook, K., Amtmann, D., Callahan, L.F., . . . Keefe, F.J. (2009). Development and psychometric analysis of the PROMIS pain behavior item bankPain, 146, 158-169.
81.
Revicki, D.A., & Schwartz, C.E. (2009). Intellectual property rights and good research practice. Quality of Life Research, 18, 1279-1280.
82.
Revicki, D.A., & Sloan, J. ( 2007). Practical and philosophical issues surrounding a national item bank: If we build it will they come? Quality of Life Research, 16, 167-174.
83.
Riley, W.T., Rothrock, N., Bruce, B., Christodolou, C., Cook, K., Hahn, E.A., & Cella, D. ( 2010). Patient-Reported Outcomes Measurement Information System (PROMIS) domain names and definitions revisions: Further assessment of content validity in IRT-derived item banks. Quality of Life Research , 19, 1311-1321.
84.
Roberson-Nay, R., Strong, D.R., Nay, W.T., Beidel, D.C., & Turner, S.M. ( 2007). Development of an abbreviated Social Phobia and Anxiety Inventory (SPAI) using item response theory: The SPAI-23. Psychological Assessment, 19, 133-145.
85.
Roth, W.M., & Roychoudhury, A. (1991). Nonmetric multidimensional item analysis in the construction of an anxiety attitude survey. Educational and Psychological Measurement, 51, 931-942.
86.
Rothrock, N.E., Hays, R.D., Spritzer, K., Yount, S.E., Riley, W., & Cella, D. ( 2010). Relative to the general US population, chronic diseases are associated with poorer health-related quality of life as measured by the Patient-Reported Outcomes Measurement Information System (PROMIS). Journal of Clinical Epidemiology, 63, 1195-1204.
87.
Santor, D.A., Gregus, M., & Welch, A. ( 2006). Eight decades of measurement in depression. Measurement, 4, 135-155.
88.
Shafer, A.B. ( 2006). Meta-analysis of the factor structures of four depression questionnaires: Beck, CES-D, Hamilton, and Zung. Journal of Clinical Psychology, 62, 123-146.
89.
Sijtsma, K., Meijer, R.R., & van der Ark, L.A. (2011). Mokken Scale Analysis as time goes by: An update for scaling practitioners. Personality and Individual Differences, 50, 31-37.
90.
Simms, L.J., Gros, D.F., Watson, D., & O’Hara, M.W. (2008). Parsing the general and specific components of depression and anxiety with bifactor modeling. Depression and Anxiety, 25, E34-E46.
91.
Steinberg, L., & Thissen, D. ( 1996). Use of item response theory and the testlet concept in the measurement of psychopathology. Psychological Methods , 1, 81-97.
92.
Stone, C.A. ( 1992). Recovery of marginal maximum likelihood estimates in the two-parameter logistic response model: An evaluation of MULTILOG. Applied Psychological Measurement, 16, 1-16.
93.
Teresi, J.A., Ocepek-Welikson, K., Kleinman, M., Eimicke, J.P., Crane, P.K., Jones, R.N., . . . Cella, D. (2009). Analysis of differential item functioning in the depression item bank from the Patient Reported Outcome Measurement Information System (PROMIS): An item response theory approach. Psychology Science Quarterly, 51, 148-180.
94.
Thissen, D., Chen, W.-H., & Bock, R.D. ( 2003). MULTILOG (version 7) [Computer software]. Lincolnwood, IL: Scientific Software International .
95.
Thissen, D., & Steinberg, L. ( 1983). An item response theory for personality and attitude scales: Item analysis using restricted factor analysis. Applied Psychological Measurement, 7, 211-226.
96.
Thissen, D., Steinberg, L., & Wainer, H. ( 1993). Detection of differential item functioning using the parameters of item response models. In P. W. Holland & H. Wainer (Eds.), Differential item functioning (pp. 67-113). Hillsdale, NJ: Lawrence Erlbaum.
97.
Vancleef, L.M.G., Vlaeyen, J.W.S., & Peters, M.L. ( 2009). Dimensional and componential structure of a hierarchical organization of pain-related anxiety constructs. Psychological Assessment, 21, 340-351.
98.
Waller, N.G., & Reise, S.P. ( 1989). Computerized adaptive personality assessment: An illustration with the Absorption scale. Journal of Personality and Social Psychology, 57, 1051-8.
99.
Walter, O.B., Becker, J., Bjorner, J.B., Fliege, H., Klapp, B.F., & Rose, M. ( 2007). Development and evaluation of a computer adaptive test for anxiety (Anxiety-CAT). Quality of Life Research, 16, 143-155.
100.
Ware, J.E. ( 2003). Conceptualization and measurement of health-related quality of life: Comments on an evolving field. Archives of Physical Medicine and Rehabilitation, 84 (Suppl. 2), S43-S51.
101.
Watson, D., Clark, L.A., Weber, K., Assenheimer, J.A., Strauss, M.E., & McCormick, R.A. (1995a). Testing a tripartite model: I. Evaluating the convergent and discriminant validity of anxiety and depression symptoms. Journal of Abnormal Psychology, 104, 3-14.
102.
Watson, D., Clark, L.A., Weber, K., Assenheimer, J.S., Strauss, M.E., & McCormick, R.A. (1995b). Testing a tripartite model: II. Exploring the symptom structure of anxiety and depression in student, adult, and patient samples. Journal of Abnormal Psychology , 104, 15-25.
103.
Weiss, D.J. ( 1985). Adaptive testing by computer. Journal of Consulting and Clinical Psychology, 53, 774-789.
104.
Weiss, D.J. ( 2004). Computerized adaptive testing for effective and efficient measurement in counseling and education. Measurement and Evaluation in Counseling and Development, 37, 70-84.
105.
Willis, G.B. ( 2005). Cognitive interviewing: A tool for improving questionnaire design. Thousand Oaks, CA: SAGE .
106.
Winkielman, P., Knäuper, B., & Schwarz, N. ( 1998). Looking back at anger: Reference periods change the interpretation of emotion frequency questions. Journal of Personality and Social Psychology, 75, 719-728.
107.
Woods, C.M. ( 2006). Ramsay-curve item response theory (RC-IRT) to detect and correct for nonnormal latent variables. Psychological Methods, 11, 253-270.
108.
World Health Organization. (2007). Constitution of the World Health Organization: Basic documents (46th ed.). Geneva, Switzerland: Author.
109.
Zinbarg, R.E., & Barlow, D.H. ( 1996). Structure of anxiety and the anxiety disorders: A hierarchical model. Journal of Abnormal Psychology, 105, 181-193.
110.
Zumbo, B.D. ( 1999). A handbook on the theory and methods of differential item functioning (DIF): Logistic regression modeling as a unitary framework for binary and Likert-type (ordinal) item scores. Ottawa, Ontario, Canada: Directorate of Human Resources Research and Evaluation, Department of National Defense.
