The authors examined data (N = 34,108) on the differential reliability and validity of facet scales from the NEO Inventories. They evaluated the extent to which (a) psychometric properties of facet scales are generalizable across ages, cultures, and methods of measurement, and, (b) validity criteria are associated with different forms of reliability. Composite estimates of facet scale stability, heritability, and cross-observer validity were broadly generalizable. Two estimates of retest reliability were independent predictors of the three validity criteria; none of three estimates of internal consistency was. Available evidence suggests the same pattern of results for other personality inventories. Internal consistency of scales can be useful as a check on data quality but appears to be of limited utility for evaluating the potential validity of developed scales, and it should not be used as a substitute for retest reliability. Further research on the nature and determinants of retest reliability is needed.
Allik, J., Laidra, K., Realo, A., & Pullmann, H. ( 2004). Personality development from 12 to 18 years of age: Changes in mean levels and structures of traits. European Journal of Personality, 18, 445-462.
2.
American Education Research Association, American Psychological Association, & National Council on Measurement in Education. ( 1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association .
3.
Ando, J., Suzuki, A., Yamagata, S., Kijima, N., Maekawa, H., Ono, Y., & Jang, K.L. ( 2004). Genetic and environmental structure of Cloninger’s temperament and character dimensions. Journal of Personality Disorders, 18, 379-393.
4.
Ashton, S.G., & Goldberg, L.R. (1973). In response to Jackson’s challenge: The comparative validity of personality scales constructed by the external (empirical) strategy and scales developed intuitively by experts, novices, and laymen. Journal of Research in Personality, 7, 1-20.
5.
Beatty, M.J., Heisel, A.D., Hall, A.E., Levine, T.R., & La France, B.H. (2002). What can we learn from the study of twins about genetic and environmental influences on interpersonal affiliation, aggressiveness, and social anxiety? A meta-analytic study. Communication Monographs, 69, 1-18.
6.
Bleidorn, W., Kandler, C., Riemann, R., & Angleitner, A. (2009). Patterns and sources of adult personality development: Growth curve analyses of the NEO-PI-R scales in a longitudinal twin study. Journal of Personality and Social Psychology, 97, 142-155.
7.
Borkenau, P., & Liebler, A. ( 1995). Observable attributes as manifestations and cues of personality and intelligence. Journal of Personality, 63, 1-25.
8.
Bouchard, T.J., Jr. , McGue, M., Hur, Y.-M., & Horn, J.M. (1998). A genetic and environmental analysis of the California Psychological Inventory using adult twins reared apart and together. European Journal of Personality, 12, 307-320.
9.
Boyle, G.J. ( 1991). Does item homogeneity indicate internal consistency or item redundancy in psychometric scales? Personality and Individual Differences, 12, 291-294.
10.
Carey, G., Goldsmith, H.H., Tellegen, A., & Gottesman, I.I. (1978). Genetics and personality inventories: The limits of replication with twin data. Behavior Genetics, 8, 299-313.
Cattell, R.B., Eber, H.W., & Tatsuoka, M.M. (1970). The handbook for the Sixteen Personality Factor Questionnaire. Champaign, IL: Institute for Personality and Ability Testing.
13.
Cheung, F., Cheung, S.F., Zhang, J., Leung, K., Leong, F., & Yeh, K.-H. (2008). Relevance of Openness as a personality dimension in Chinese culture: Aspects of its cultural relevance . Journal of Cross-Cultural Psychology, 39, 81-108.
14.
Chmielewski, M., & Watson, D. ( 2009). What is being assessed and why it matters: The impact of transient error on trait research. Journal of Personality and Social Psychology, 97, 186-202.
15.
Churchill, G.A., Jr. , & Peter, J.P. (1984). Research design effects on the reliability of rating scales: A meta-analysis. Journal of Marketing Research, 21, 360-375.
16.
Cloninger, C.R., Przybeck, T.R., Svrakic, D.M., & Wetzel, R.D. ( 1994). The Temperament and Character Inventory (TCI): A guide to its development and use. St. Louis, MO: Author.
17.
Comrey, A.L. ( 1970). Manual for the Comrey Personality Scales. San Diego, CA: EdITS.
18.
Conley, J.J. ( 1984). The hierarchy of consistency: A review and model of longitudinal findings on adult individual differences in intelligence, personality, and self-opinion. Personality and Individual Differences, 5, 11-26.
19.
Conn, S. R., & Rieke, M. L. (Eds.). (1994). 16PF Fifth Edition technical manual. Champaign, IL: Institute for Personality and Ability Testing.
20.
Costa, P.T., Jr., Bagby, R.M., Herbst, J.H., & McCrae, R.R. (2005). Personality self-reports are concurrently reliable and valid during acute depressive episodes. Journal of Affective Disorders, 89, 45-55.
21.
Costa, P.T., Jr., Herbst, J.H., McCrae, R.R., & Siegler, I.C. ( 2000). Personality at midlife: Stability, intrinsic maturation, and response to life events. Assessment, 7, 365-378.
22.
Costa, P.T., Jr., & McCrae, R.R. (1978). Objective personality assessment . In M. Storandt, I. C. Siegler, & M. F. Elias (Eds.), The clinical psychology of aging (pp. 119-143). New York, NY : Plenum.
Costa, P.T., Jr., & McCrae, R.R. (1988). Personality in adulthood: A six-year longitudinal study of self-reports and spouse ratings on the NEO Personality Inventory. Journal of Personality and Social Psychology , 54, 853-863.
Costa, P.T., Jr., McCrae, R.R., & Dye, D.A. ( 1991). Facet scales for Agreeableness and Conscientiousness: A revision of the NEO Personality Inventory. Personality and Individual Differences, 12, 887-898.
27.
Cronbach, L.J. ( 1951). Coefficient alpha and the internal structure of tests . Psychometrika, 16, 297-334.
28.
Cronbach, L.J., Nageswari, R., & Gleser, G.C. ( 1963). Theory of generalizability: A liberalization of reliability theory. British Journal of Statistical Psychology, 16, 137-163.
29.
Cronbach, L.J., & Shavelson, R.J. (2004). My current thoughts on coefficient alpha and successor procedures. Educational and Psychological Measurement, 64, 391-418.
30.
Dai, X.-Y., & Wu, Y.-Q. ( 2005). A study on NEO-PI-R used in 16-20 years old people [in Chinese]. Chinese Journal of Clinical Psychology, 13, 14-18.
31.
DeShon, R.P. ( 1998). A cautionary note on measurement error corrections in structural equation models. Psychological Methods, 3, 412-423.
32.
Eichler, W.C., & Kurtz, J.E. ( 2008, March). Coefficient alpha and the convergent validity of multi-item personality scales. Poster presented at the annual meeting of the Society of Personality Assessment, New Orleans, LA.
33.
Fekken, G.C., Holden, R.R., Jackson, D.N., & Guthrie, G.M. ( 1987). An evaluation of the validity of the Personality Research Form with Filipino university students. International Journal of Psychology, 22, 399-407.
34.
Fleeson, W. ( 2001). Toward a structure- and process-integrated view of personality: Traits as density distributions of states. Journal of Personality and Social Psychology, 80, 1011-1027.
35.
Gottesman, I.I. ( 1966). Genetic variance in adaptive personality traits. Journal of Child Psychology and Psychiatry, 7, 199-208.
Gough, H.G., & Heilbrun, A.B. , Jr. (1983). Adjective Check List manual. Palo Alto, CA: Consulting Psychologists Press.
38.
Grucza, R.A., & Goldberg, L.R. (2007). The comparative validity of 11 modern personality inventories: Behavioral acts, informant reports, and clinical indicators. Journal of Personality Assessment , 89, 167-187.
39.
Guadiano, B.A. ( 2006). Is symptomatic improvement in clinical trials of cognitive-behavior therapy for psychosis clinically significant? Journal of Psychiatric Practice, 12, 11-23.
40.
Guilford, J.S., Zimmerman, W.S., & Guilford, J.P. (1976). The Guilford-Zimmerman Temperament Survey Handbook: Twenty-five years of research and application . San Diego, CA: EdITS.
41.
Harris, J.A., Vernon, P.A., & Jang, K.L. ( 1995). A possible genetic basis of accuracy in personality perception . Personality and Individual Differences, 18, 791-792.
42.
Heise, D.R. ( 1969). Separating reliability and stability in test- retest correlation. American Sociological Review, 34, 93-101.
43.
Helson, R., & Moane, G. ( 1987). Personality change in women from college to midlife. Journal of Personality and Social Psychology, 53, 176-186.
44.
Horn, J.M., Plomin, R., & Rosenman, R. ( 1976). Heritability of personality traits in adult male twins . Behavior Genetics, 6, 17-30.
45.
Jackson, D.N. ( 1975). The relative validity of scales prepared by naive item writers and those based on empirical methods of personality scale construction . Educational and Psychological Measurement, 35, 361-370.
46.
Jackson, D.N. ( 1984). Personality Research Form manual (3rd ed.). Port Huron, MI: Research Psychologists Press.
47.
Jacobson, N.S., Wilson, L., & Tupper, C. ( 1988). The clinical significance of treatment gains resulting from exposure-based intervention for agoraphobia: A reanalysis of outcome data. Behavior Therapy, 19, 539-554.
48.
Jang, K.L., McCrae, R.R., Angleitner, A., Riemann, R., & Livesley, W.J. (1998). Heritability of facet-level traits in a cross-cultural twin sample: Support for a hierarchical model of personality. Journal of Personality and Social Psychology , 74, 1556-1565.
49.
John, O.P., & Robins, R.W. ( 1993). Determinants of interjudge agreement on personality traits: The Big Five domains, observability, evaluativeness, and the unique perspective of the self. Journal of Personality, 61, 521-551.
50.
John, O.P., & Soto, C.J. ( 2007). The importance of being valid: Reliability and the process of construct validation. In R. W. Robins , R. C. Fraley, & R. F. Krueger (Eds.), Handbook of research methods in personality psychology (pp. 461-494). New York, NY : Guilford.
51.
Johnson, A.M., Vernon, P.A., Harris, J.A., & Jang, K.L. ( 2004). A behavior genetic investigation of the relationship between leadership and personality. Twin Research, 7, 27-32.
52.
Kurtz, J.E., Lee, P.A., & Sherker, J.L. ( 1999). Internal and temporal reliability estimates for informant ratings of personality using the NEO-PI-R and IAS. Assessment , 6, 103-113.
53.
Kurtz, J.E., & Parrish, C.L. ( 2001). Semantic response consistency and protocol validity in structured personality assessment: The case of the NEO-PI-R. Journal of Personality Assessment, 76, 315-332.
54.
Löckenhoff, C.E., Terracciano, A., Bienvenu, O.J., Particiu, N.S., Nestadt, G., McCrae, R.R., et al. (2008). Ethnicity, education, and the temporal stability of personality traits in the East Baltimore Epidemiologic Catchment Area study. Journal of Research in Personality, 42, 577-598.
55.
Loehlin, J.C. ( 1992). Genes and environment in personality development. Newbury Park, CA: Sage.
56.
Loevinger, J. ( 1954). The attenuation paradox in test theory. Psychological Bulletin, 51, 493-504.
57.
Markon, K.E., Krueger, R.F., & Watson, D. ( 2005). Delineating the structure of normal and abnormal personality: An integrative hierarchical approach. Journal of Personality and Social Psychology, 88, 139-157.
58.
Marsella, A.J., Dubanoski, J., Hamada, W.C., & Morse, H. ( 2000). The measurement of personality across cultures: Historical, conceptual, and methodological issues and considerations. American Behavioral Scientist, 44, 41-62.
59.
Martin, T.A., Costa, P.T., Jr., Oryol, V.E., Rukavishnikov, A.A., & Senin, I.G. ( 2002). Applications of the Russian NEO-PI-R. In R. R. McCrae & J. Allik (Eds.), The five-factor model of personality across cultures (pp. 253-269). New York, NY: Kluwer/ Plenum.
60.
Matthews, G. ( 1989). The factor structure of the 16PF: Twelve primary and three secondary factors. Personality and Individual Differences , 10, 931-940.
61.
McCrae, R.R. ( 1982). Consensual validation of personality traits: Evidence from self-reports and ratings. Journal of Personality and Social Psychology, 43, 293-303.
62.
McCrae, R.R., & Costa, P.T., Jr. (1992). Discriminant validity of NEO-PI-R facets. Educational and Psychological Measurement, 52, 229-237.
63.
McCrae, R.R., & Costa, P.T., Jr. (2008). The five-factor theory of personality . In O. P. John, R. W. Robins, & L. A. Pervin (Eds.), Handbook of personality: Theory and research (3rd ed., pp. 157-180). New York, NY : Guilford.
64.
McCrae, R.R., & Costa, P.T., Jr. (in press). Professional manual for the NEO Inventories: NEO-PI-3, NEO-PI-R, and NEO-FFI-3. Odessa, FL: Psychological Assessment Resources.
65.
McCrae, R.R., Costa, P.T., Jr., Lima, M.P., Simões, A., Ostendorf, F., Angleitner, A., et al. (1999). Age differences in personality across the adult life span: Parallels in five cultures. Developmental Psychology, 35, 466-477.
66.
McCrae, R.R., Costa, P.T., Jr., & Martin, T.A. ( 2005). The NEO-PI-3: A more readable Revised NEO Personality Inventory. Journal of Personality Assessment, 84, 261-270.
67.
McCrae, R.R., Costa, P.T., Jr., Martin, T.A., Oryol, V.E., Rukavishnikov, A.A., Senin, I.G., et al. (2004). Consensual validation of personality traits across cultures. Journal of Research in Personality, 38, 179-201.
68.
McCrae, R.R., Martin, T.A., & Costa, P.T., Jr. (2005). Age trends and age norms for the NEO Personality Inventory-3 in adolescents and adults. Assessment , 12, 363-373.
69.
McCrae, R.R., Martin, T.A., Hřebíčková, M., Urbánek , T., Boomsma, D.I., Willemsen, G., & Costa, P.T., Jr. (2008). Personality trait similarity between spouses in four cultures. Journal of Personality, 76, 1137-1163.
70.
McCrae, R.R., Stone, S.V., Fagan, P.J., & Costa, P.T., Jr. (1998). Identifying causes of disagreement between self-reports and spouse ratings of personality. Journal of Personality, 66, 285-313.
71.
McCrae, R.R., Terracciano, A., & 78 Members of the Personality Profiles of Cultures Project. (2005). Universal features of personality traits from the observer’s perspective: Data from 50 cultures. Journal of Personality and Social Psychology, 88, 547-561.
72.
McCrae, R.R., Terracciano, A., & 79 Members of the Personality Profiles of Cultures Project. (2005). Personality profiles of cultures: Aggregate personality traits. Journal of Personality and Social Psychology, 89, 407-425.
73.
McCrae, R.R., Yik, M.S.M., Trapnell, P.D., Bond, M.H., & Paulhus, D.L. ( 1998). Interpreting personality profiles across cultures: Bilingual, acculturation, and peer rating studies of Chinese undergraduates. Journal of Personality and Social Psychology, 74, 1041-1055.
74.
McDonald, R.P. ( 1999). Test theory: A unified treatment. Mahwah, NJ: Erlbaum.
75.
Millon, T. ( 1994). Millon Index of Personality Styles manual. San Antonio, TX: Psychological Corporation.
76.
Nichols, R.C. ( 1966). The resemblance of twins in personality and interests . National Merit Scholarship Corporation Research Reports , 2, 1-23.
77.
Nichols, R.C. ( 1967). Personality change and college. American Educational Research Journal, 4, 173-190.
78.
Nunnally, J.C., & Bernstein, I. ( 1994). Psychometric theory (3rd ed.). New York, NY: McGraw-Hill.
79.
Ostendorf, F., & Angleitner, A. (2004). NEO-Persönlichkeitsinventar, revidierte Form, NEO-PI-R nach Costa und McCrae [Revised NEO Personality Inventory, NEO-PI-R of Costa and McCrae]. Göttingen, Germany: Hogrefe.
80.
Peter, J.P., & Churchill, G.A. (1986). Relationships among research design choices and psychometric properties of rating scales: A meta-analysis . Journal of Marketing Research, 23, 1-10.
81.
Piedmont, R.L. ( 1994). Validation of the NEO-PI-R observer form for college students: Toward a paradigm for studying personality development. Assessment, 1, 259-268.
82.
Piedmont, R.L. ( 2001). Cracking the plaster cast: Big Five personality change during intensive outpatient counseling. Journal of Research in Personality, 35, 500-520.
83.
Piedmont, R.L., Bain, E., McCrae, R.R., & Costa, P.T., Jr. (2002). The applicability of the five-factor model in a sub-Saharan culture: The NEO-PI-R in Shona. In R. R. McCrae & J. Allik (Eds.), The five-factor model of personality across cultures (pp. 155-173). New York, NY: Kluwer/Plenum.
84.
Pilia, G., Chen, W.-M., Scuteri, A., Orrú, M., Albai, G., Deo, M., et al. (2006). Heritability of cardiovascular and personality traits in 6,148 Sardinians. PLoS Genetics, 2, 1207-1223.
85.
Riemann, R., Angleitner, A., & Strelau, J. ( 1997). Genetic and environmental influences on personality: A study of twins reared together using the self- and peer report NEO-FFI scales . Journal of Personality, 65, 449-475.
86.
Roberts, B.W., & DelVecchio, W.F. (2000). The rank-order consistency of personality traits from childhood to old age: A quantitative review of longitudinal studies. Psychological Bulletin, 126, 3-25.
87.
Rushton, J.P., Fulker, D.W., Neale, M.C., Nias, D.K.B., & Eysenck, H.J. ( 1986). Altruism and aggression: The heritability of individual differences. Journal of Personality and Social Psychology , 50, 1192-1198.
88.
Scarr, S. ( 1966). The origins of individual differences in adjective check list scores. Journal of Consulting Psychology, 30, 354-357.
89.
Schinka, J.A., & Borum, R. ( 1994). Readability of normal personality inventories. Journal of Personality Assessment, 62, 95-101.
90.
Schmidt, F.L., Le, H., & Ilies, R. ( 2003). Beyond alpha: An empirical examination of the effects of different sources of measurement error on reliability estimates for measures of individual differences constructs. Psychological Methods , 8, 206-224.
91.
Schmitt, N. ( 1996). Uses and abuses of coefficient alpha. Psychological Assessment, 8, 350-353.
92.
Sijtsma, K. ( 2009). On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika, 74, 107-120.
93.
Snell, M.N., Mallinckrodt, B., Hill, R.D., & Lambert, M.J. ( 2001). Predicting counseling center clients’ response to counseling: A 1-year follow-up. Journal of Counseling Psychology , 48, 463-473.
94.
Streiner, D.L. ( 2003). Starting at the beginning: An introduction to coefficient alpha and internal consistency. Journal of Personality Assessment , 80, 99-103.
95.
Sung, M.S., Kim, J.H., Yang, E., Abrams, K.Y., & Lyoo, I.K. ( 2002). Reliability and validity of the Korean version of the Temperament and Character Inventory. Comprehensive Psychiatry , 43, 236-243.
96.
Terracciano, A., Costa, P.T., Jr., & McCrae, R.R. (2006). Personality plasticity after age 30. Personality and Social Psychology Bulletin, 32, 999-1009.
97.
Vernon, P.A., Jang, K.L., Harris, J.A., & McCarthy, J.M. (1997). Environmental predictors of personality differences: A twin and sibling study. Journal of Personality and Social Psychology, 72, 177-183.
98.
Vul, E., & Pashler, H. ( 2008). Measuring the crowd within: Probabilistic representations within individuals. Psychological Science, 19, 645-647.
99.
Watson, D. ( 2004). Stability versus change, dependability versus error: Issues in the assessment of personality over time. Journal of Research in Personality, 38, 319-350.
100.
Yamagata, S., Ando, J., Ostendorf, F., Angleitner, A., Riemann, R., Spinath, F., et al. (2006, September). Cross-cultural differences in heritability of personality traits: Using behavioral genetics to study culture . Paper presented at the 4th CEFOM/21 International Symposium, Tokyo, Japan.
101.
Yang, J., McCrae, R.R., Costa, P.T., Jr., Dai, X., Yao, S., Cai, T., & Gao, B. ( 1999). Cross-cultural personality assessment in psychiatric populations: The NEO-PI-R in the People’s Republic of China. Psychological Assessment, 11, 359-368.
