The Psychometric Latent Agreement Model (PLAM) is proposed for estimating the subpopulation membership of individuals (e.g., satisfactory performers vs. unsatisfactory performers) at discrete levels of multiple latent trait variables. A binary latent Type variable is introduced to take account of the possibility that, for a given set of observed variables, the latent group memberships of some individuals are indeterminate. The latent Type variable allows for separating individuals who can reliably be assigned to satisfactory versus unsatisfactory performers classes from those individuals whose ratings do not contain the necessary information to make the class assignment possible for a particular set of rating items. Agreements among discrete latent trait variables are also estimated. The PLAM was illustrated with two examples using real data on behavioral rating measures. One example involved ratings of two behavioral constructs by a single rater type, whereas the other involved ratings of one construct by three rater types. Implications were presented for using behavioral ratings to determine the subpopulation membership, such as qualified versus unqualified groupings in hiring decisions and pass versus fail groupings in performance evaluations.
Achenbach, T.M. ( 1991). Manual for the Child Behavior Checklist/4-18 and 1991 profile. Burlington, VT: University of Vermont, Department of Psychiatry.
2.
Achenbach, T.M., McConaughy, S.H., & Howell, C.T. ( 1987). Child/adolescent behavioral and emotional problems: Implications of cross-informant correlations for situational specificity. Psychological Bulletin, 101, 213-232.
3.
Achenbach, T.M., & Rescorla, L.A. (2001). Manual for the ASEBA school-age forms & profiles. Burlington, VT: University of Vermont, Center for Children, Youth, and Families.
4.
Agresti, A. ( 1988). A model for agreement between ratings on an ordinal scale . Biometrics, 44, 539-548.
5.
Agresti, A. ( 1989). An agreement model with kappa as parameter. Statistics & Probability Letters, 7, 271-273.
6.
Agresti, A. ( 2002). Categorical data analysis (2nd ed.). New York, NY: Wiley.
7.
Agresti, A., & Lang, J.B. ( 1993). Quasi-symmetric latent class models, with application to rater agreement. Biometrics, 49, 131-139.
8.
Aickin, M. ( 1990). Maximum likelihood estimation of agreement in the constant predictive probability model and its relation to Cohen’s kappa. Biometrics, 46, 293-302.
9.
Akaike, H. ( 1987). Factor analysis and AIC. Psychometrika , 52, 317-332.
10.
Bartholomew, D.J., & Knott, M. ( 1999). Latent variable models and factor analysis. London, UK: Arnold.
11.
Becker, M.P. ( 1989). Using association models to analyse association data: Two examples. Statistics in Medicine, 8, 1199-2107.
12.
Bergan, J.R., Schwarz, R.D., & Reddy, L.A. ( 1999). Latent structure analysis of classification errors in screening and clinical diagnosis: An alternative to classification analysis . Applied Psychological Measurement, 23, 69-86.
13.
Brett, J.F., & Atwater, L.E. ( 2001). 360° feedback: Accuracy, reactions, and perceptions of usefulness. Journal of Applied Psychology, 86, 930-942.
14.
Campbell, D.T., & Fiske, D.W. ( 1959). Convergent and discriminant validation by the multitrait-multimethod matrix. Psychological Bulletin, 56, 81-105.
15.
Cheung, G.W. ( 2010). Introducing the latent congruence model for improving the assessment of similarity, agreement, and fit in organizational research . Organizational Research Methods, 12, 6-33.
16.
Cohen, A., Doveh, E., & Nahum-Shani, I. (2010). Testing agreement with multi-item scales with the indices rWG(J) and ADM(J). Organizational Research Methods, 12, 148, 164.
17.
Collins, L.M., & Wugalter, S.E. (1992). Latent class models for stage-sequential dynamic latent variables. Multivariate Behavioral Research, 27, 131-157.
18.
Croon, M. ( 2002). Ordering the classes. In J. A. Hagenaars & A. L. McCutcheon (Eds.), Applied latent class analysis (pp. 137-162). Cambridge, UK: Cambridge University Press.
19.
Darroch, J.N., & McCloud, P.I. ( 1986). Category distinguishability and observer agreement. Australian Journal of Statistics, 28, 371-388.
20.
Dayton, C.M. (May, 2006). π*: The two-point mixture index of model fit. Paper presented at the University of Maryland CILVR conference ‘‘Mixture Models in Latent Variable Research,’’College Park, MD.
21.
Dayton, C.M., & Macready, G.B. (1976). A probabilistic model for validation of behavioral hierarchies. Psychometrika, 41, 189-204.
22.
Dayton, C.M., & Macready, G.B. (1980). A scaling model with response errors and intrinsically unscalable respondents. Psychometrika , 45, 343-356.
23.
Dillon, W.R., & Mulani, N. ( 1984). A probabilistic latent class model for assessing inter-judge reliability. Multivariate Behavioral Research, 19, 438-458.
24.
Dumenci, L. (July, 2005). A three-latent class variable model of attention problems . Poster presented at the International Meeting of the Psychometric Society, Tilburg, the Netherlands.
25.
Edwards, M.R., & Ewen, A.J. ( 1996). How to manage performance and pay with 360-degree feedback . Compensation and Benefit Review, 28, 41-46.
26.
Eid, M., Langeheine, R., & Diener, E. ( 2003). Comparing typological structures across cultures by multigroup latent class analysis: A premier. Journal of Cross-Cultural Psychology, 34, 195-210.
27.
Flaherty, B.P. ( 2002). Assessing reliability of categorical substance use measures with latent class analysis. Drug and Alcohol Dependence, 68, S7-S20.
28.
Funderburgh, S.A., & Levy, P. ( 1997). The influence of individual and contextual variables on 360-degree feedback system attitudes. Groups and Organizational Management, 22, 210-235.
29.
Gitomer, D.H., & Yamamoto, K. ( 1991). Performance modeling that integrates latent trait and class theory. Journal of Educational Measurement, 28, 173-189.
30.
Glarus, A.G., & Kline, R.B. ( 1988). Understanding the accuracy of tests with cutting scores: The sensitivity, specificity, and predictive value model. Journal of Clinical Psychology, 44, 1013-1023.
31.
Goodman, L.A. ( 1975). A new model for scaling response patterns: An application of the quasi-independence concept. Journal of the American Statistical Association, 70, 755-768.
32.
Goodman, L.A. ( 1979). Simple models for the analysis of association in cross-classifications having ordered categories. Journal of American Statistical Association, 74, 537-552.
33.
Guggenmoos-Holzmann, I. ( 1996). The meaning of kappa: Probabilistic concepts of reliability and validity revisited. Journal of Clinical Epidemiology, 49, 775-782.
34.
Guggenmoos-Holzmann, I., & Vonk, R. ( 1998). Kappa-like indices of observer agreement viewed from a latent class perspective. Statistics in Medicine, 17, 797-812.
35.
Hollingshead, A.B. ( 1975). Four factor scale of social status. Unpublished paper . New Haven, CT: Yale University, Department of Sociology.
36.
Lawler, E.E. III. ( 1967). The multitrait-multirater approach to measuring managerial job performance. Journal of Applied Psychology, 51, 369-381.
Pasisz, D.J., & Hurtz, G.M. ( 2010). Testing for between-group differences in within-group interrater agreement. Organizational Research Methods, 12, 590-613.
43.
Pinada, D.A., Lopera, F., Palacio, J.D., Ramirez, D., & Henao, G.C. ( 2003). Prevalence estimations of attention-deficit/hyperactivity disorder: Differential diagnoses and comorbidities in a Colombian sample. International Journal of Neuroscience, 113, 49-71.
44.
Rindskopf, D., & Rindskopf, W. ( 1986). The value of latent class analysis in medical diagnosis . Statistics in Medicine, 5, 21-27.
45.
Rupp, A.A., Templin, J., & Henson, R.A. ( 2010). Diagnostic measurement: Theory, method, and application . New York, NY: The Guilford Press .
46.
Schuster, C. ( 2006). Latent-class analysis approaches to determining the reliability of nominal classifications: A comparison between the response-error and the target-type approach. In C. S. Bergeman & S. M. Boker (Eds.), Methodological issues in aging research (pp. 165-184). Mahwah, NJ: Lawrence Erlbaum Associates.
47.
Schuster, C., & Smith, D.A. ( 2002). Indexing systematic rater agreement with a latent-class model. Psychological Methods, 7, 384-395.
48.
Schuster, C., & Von Eye, A. (2001). Models for ordinal agreement . Biomedical Journal, 7, 795-808.
49.
Schwarz, G. ( 1978). Estimating the dimension of a model. Annals of Statistics, 6, 461-464.
50.
Sclove, L.S. ( 1987). Application of model-selection criteria to some problems in multivariate analysis. Psychometrika, 52, 333-343.
