Situational judgment tests have gained popularity in educational and psychological measurement and are widely used in personnel assessment. A situational judgment item presents a hypothetical scenario and a list of actions, and the individuals are asked to select their most likely action for that scenario. Because actions have no explicit order, the item generates nominal responses consisting of the actions selected by the individuals. This article shows how to factor-analyze the nominal responses originated from such a test, including the estimation of the number of latent factors and a factor invariance analysis in a multiple group design. The method consists of applying the MNCM, a multidimensional extension of the nominal categories model by Bock. The article includes the results of two studies: (1) a simulation study about Type-I error rate, statistical power, and recovery of the parameters in a multigroup factorial invariance design and (2) a real data example using responses to a situational judgment test measuring gender stereotypes to illustrate the approach. Results suggest the use of the Akaike information criterion, Bayesian information criterion, and corrected Bayesian information criterion indices to guide the selection of the number of factors with nominal responses. All the analyses are conducted using the computer program Mplus. The code is included as Supplemental Material (available online) for the readers so that they can adapt it to their own purposes.
BenjaminiY.HochbergY. (1995). Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society: Series B (Methodological), 57(1), 289-300. https://doi.org/10.1111/1467-9868.00353
BockR. D. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37(1), 29-51. https://doi.org/10.1007/BF02291411
4.
BockR. D. (1975). Multivariate statistical methods in behavioral research. McGraw-Hill.
5.
BockR. D.AitkinM. (1981). Marginal maximum likelihood estimation of item parameters: Application of an EM algorithm. Psychometrika, 46(4), 443-459. https://doi.org/10.1007/BF02293801
6.
BockR. D.GibbonsR. (2010). Factor analysis of categorical item responses. In NeringM. L.OstiniR. (Eds.), Handbook of polytomous item response theory models: Developments and applications (pp. 43-75). Taylor & Francis.
7.
BrowneM. W. (2001). An overview of analytic rotation in exploratory factor analysis. Multivariate Behavioral Research, 36(1), 111-150. https://doi.org/10.1207/S15327906MBR3601\_05
8.
BurnhamK. P.AndersonD. R. (2004). Multimodel inference: understanding AIC and BIC in model selection. Sociological Methods & Research, 33(2), 261-304. https://doi.org/10.1177/0049124104268644
9.
ChenQ.LuoW.PalardyG. J.GlamanR.McEnturffA. (2017). The efficacy of common fit indices for enumerating classes in growth mixture models when nested data structure is ignored: A Monte Carlo study. Sage Open, 7(1), 1-19. https://doi.org/10.1177/2158244017700459
10.
DziakJ. J.CoffmanD. L.LanzaS. T.LiR.JermiinL. S. (2020). Sensitivity and specificity of information criteria. Briefings in Bioinformatics, 21(2), 553-565. https://doi.org/10.1093/bib/bbz016
11.
EroshevaE. A.CurtisS. M. (2017). Dealing with reflection invariance in Bayesian factor analysis. Psychometrika, 82(2), 295-307. https://doi.org/10.1007/s11336-017-9564-y
12.
FahrmeirL.TutzG. (1991). Multivariate statistical modelling based on generalized linear models. Springer.
13.
FalkC. F.CaiL. (2016). A flexible full-information approach to the modeling of response styles. Psychological Methods, 21(3), 328-347. https://doi.org/10.1037/met0000059
14.
FalkC. F.JuU. (2020). Estimation of response styles using the multidimensional nominal response model: A tutorial and comparison with sum scores. Frontiers in Psychology, 11, 72. https://doi.org/10.3389/fpsyg.2020.00072
15.
FloydF. J.WidamanK. F. (1995). Factor analysis in the development and refinement of clinical assessment instruments. Psychological Assessment, 7(3), 286-299. https://doi.org/10.1037/1040-3590.7.3.286
16.
GewekeJ.ZhouG. (1996). Measuring the pricing error of the arbitrage pricing theory. Review of Financial Studies, 9(2), 557-587. https://doi.org/10.1093/rfs/9.2.557
17.
HamiltonM. C. (1988). Using masculine generics: Does generic he increase male bias in the user’s imagery?Sex Roles, 19(11-12), 785-799. https://doi.org/10.1007/BF00288993
18.
JöreskogK. G.GoldbergerA. S. (1975). Estimation of a model with multiple indicators and multiple causes of a single latent variable. Journal of the American Statistical Association, 70(351a), 631-639. https://doi.org/10.1080/01621459.1975.10482485
19.
KaufmannC.BohnerG. (2014). Masculine generics and gender-aware alternatives in Spanish. IFFOnZeit, 4(3), 8-17.
20.
LievensF.PeetersH.SchollaertE. (2008). Situational judgment tests: A review of recent research. Personnel Review, 37(4), 426-441. https://doi.org/10.1108/00483480810877598
21.
López-SáezM.MoralesJ. F.LisbonaA. (2008). Evolution of gender stereotypes in Spain: Traits and roles. Spanish Journal of Psychology, 11(2), 609-617. https://doi.org/10.1017/S1138741600004613
22.
LuceR. D. (1959). Individual choice behavior. Wiley.
23.
McDanielM. A.NguyenN. (2001). Situational judgment tests: A review of practice and constructs assessed. International Journal of Selection and Assessment, 9(12), 103-113. https://doi.org/10.1111/1468-2389.00167
24.
McFaddenD. (1974). Conditional logit analysis of qualitative choice behavior. In ZarembkaP. (Ed.), Frontiers of econometrics (pp. 105-142). Academic Press.
25.
MerrittR. D.KokC. J. (1995). Attribution of gender to a gender-unspecified individual: An evaluation of the people = male hypothesis. Sex Roles, 33(3-4), 145-147. https://doi.org/10.1007/BF01544608
26.
MurakiE. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16(2), 159-176. https://doi.org/10.1177/014662169201600206
NissenU. K. (2013). Is Spanish becoming more gender fair? a historical perspective on the interpretation of gender-specific and gender-neutral expressions. Linguistik Online, 58(1). https://doi.org/10.13092/lo.58.241
29.
PloyhartR. E.WardA.-K. (2013). Situational judgment measures. In GeisingerK. F.BrackenB. A.CarlsonJ. F.HansenJ.-I. C.KuncelN. R.ReiseS. P.RodriguezM. C. (Eds.), APA handbook of testing and assessment in psychology: Vol. 1. Test theory and testing and assessment in industrial and organizational psychology (pp. 551-564). American Psychological Association. https://doi.org/10.1037/14047-030
RajuS.LaffitteL.ByrneB. (2002). Measurement equivalence: A comparison of methods based on confirmatory factor analysis and item response theory. Journal of Applied Psychology, 87(3), 517-529. https://doi.org/10.1037//0021-9010.87.3.517
32.
RevueltaJ. (2014). Multidimensional item response model for nominal variables. Applied Psychological Measurement, 38(7), 549-562. https://doi.org/10.1177/0146621614536272
RevueltaJ.XiménezC. (2017). Bayesian dimensionality assessment for the multidimensional nominal response model. Frontiers in Psychology, 8, 961. https://doi.org/10.3389/fpsyg.2017.00961
35.
ScloveS. L. (1987). Application of model-selection criteria to some problems in multivariate analysis. Psychometrika, 52(3), 333-343. https://doi.org/10.1007/BF02294360
36.
SharmaS.GangopadhyayM.AustinE.MandalM. (2013). Development and validation of a situational judgment test of emotional intelligence. International Journal of Selection and Assessment, 21(1), 57-73. https://doi.org/10.1111/ijsa.12017
Stan Development Team. (2018). RStan: The R interface to Stan. R package version 2.17.3. http://mc-stan.org
39.
TakaneY.de LeeuwJ. (1987). On the relationship between item response theory and factor analysis of discretized variables. Psychometrika, 52(3), 393-408. https://doi.org/10.1007/BF02294363
40.
TeinJ. Y.CoxeS.ChamH. (2013). Statistical power to detect the correct number of classes in latent profile analysis. Structural Equation Modeling, 20(4), 640-657. https://doi.org/10.1080/10705511.2013.824781
41.
ThissenD.CaiL. (2016). Nominal categories models. In van der LindenW. J. (Ed.), Handbook of item response theory: Vol. 1. Models (pp. 51-73). Chapman & Hall/CRC Press.
42.
ThissenD.CaiL.BockR. D. (2010). The nominal categories item response model. In NeringM. L.OstiniR. (Eds.), Handbook of polytomous item response theory models: Developments and applications (pp. 155-184). Taylor & Francis.
43.
VandenbergR. J.LanceC. E. (2000). A review and synthesis of the measurement invariance literature: Suggestions, practices, and recommendations for organizational research. Organizational Research Methods, 3(1), 4-70. https://doi.org/10.1177/109442810031002
44.
VriezeS. (2012). Model selection and psychological theory: A discussion of the differences between the Akaike information criterion (AIC) and the Bayesian information criterion (BIC). Psychological Methods, 17(2), 228-243. https://doi.org/10.1037/a0027127
45.
ZuJ.KyllonenP. C. (2018). Nominal response model is useful for scoring multiple-choice situational judgment tests. Organizational Research Methods, 23(2), 342-366. https://doi.org/10.1177/1094428118812669
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
