A multiple-step procedure is outlined that can be used for examining the latent structure of behavior measurement instruments in complex empirical settings. The method permits one to study their latent structure after assessing the need to account for clustering effects and the necessity of its examination within individual levels of fixed factors, such as gender or group membership of substantive relevance. The approach is readily applicable with binary or binary-scored items using popular and widely available software. The described procedure is illustrated with empirical data from a student behavior screening instrument.
AgrestiA. (2002). Categorical data analysis. Wiley.
2.
AgrestiA. (2018). Statistical methods for the social sciences. Pearson.
3.
AlkherainejK.AlebrahimS. (2019). Preliminary psychometric findings on the Arabic Systematic Screening for Behavior Disorders. Assessment for Effective Intervention, 44(3), 184–196.
4.
BenjaminiY.HochbergY. (1995). Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society, Series B, 57, 289–300.
5.
BenjaminiY.YekutieliD. (2001). The control of the false discovery rate in multiple testing under dependency. Annals of Statistics, 29, 1165–1188.
6.
CaldarellaP.YoungE. L.RichardsonM. J.YoungB. J.YoungK. R. (2008). Validation of the systematic screening for behavior disorders in middle and junior high school. Journal of Emotional and Behavioral Disorders, 16, 105–117.
7.
CrockerL.AlginaA. (2006). Introduction to classical and modern test theory. Prentice Hall.
8.
DemidenkoE. (2007). Sample size determination for logistic regression revisited. Statistics in Medicine, 26, 3385–3397.
9.
GeiserC. (2013). Data analysis with Mplus. Guilford.
10.
GoldsteinH. (2011). Multilevel modeling. Arnold.
11.
HarrellF. E. (2015). Regression modeling strategies. With applications to linear models, logistic and ordinal regression, and survival analysis (2nd ed.). Springer.
12.
HuL.-T.BentlerP. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling, 61, 55.
13.
KumleL.VoL.-H.DraschkowD. (2021). Estimating power in generalized linear mixed models: An open introduction and tutorial in R. Behavior Research Methods, 53, 2528–2543.
14.
MacCallumR. C.BrowneM. W.SugawaraH. M. (1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1, 130–149.
15.
McDonaldR. P. (1999). Test theory. A unified treatment. Erlbaum.
Rabe-HeskethS.SkrondalA. (2022). Multilevel and longitudinal modeling using Stata. Stata Press.
20.
RaudenbushS. W.BrykA. S. (2002). Hierarchical linear models. Sage.
21.
RaykovT. (2023). Psychometric scale evaluation using structural equation modeling and latent variable modeling. In HoyleR. (Ed.), Handbook of structural equation modeling (2nd ed., pp. 462–480). Guilford Press.
22.
RaykovT.LichtenbergP. A.PaulsonD. (2012). Examining the missing completely at random mechanism in incomplete data sets: A multiple testing approach. Structural Equation Modeling, 19, 399–408.
23.
RaykovT.MarcoulidesG. A. (2011). Introduction to psychometric theory. Taylor & Francis.
24.
RaykovT.MarcoulidesG. A. (2015). Intra-class correlation coefficients in hierarchical design studies with discrete response variables: A note on a direct interval estimation procedure. Educational and Psychological Measurement, 75, 1063–1070.
25.
RaykovT.MarcoulidesG. A. (2016). Scale reliability evaluation under multiple assumption violations. Structural Equation Modeling, 23, 302–313.
26.
RaykovT.MarcoulidesG. A.GablerS.LeeY. (2017). Testing criterion correlations with scale component measurement errors using latent variable modeling. Structural Equation Modeling, 24, 468–474.
27.
ReckaseM. D. (2009). Multidimensional item response theory. Springer.
28.
SneijdersT. A. B.BoskerR. J. (2012). Multilevel analysis. An introduction to basic and advanced multilevel modeling. Sage.
29.
SnijdersT. A. B. (2005). Power and sample size in multilevel linear models. In EverittB. S.HowellD. (Eds.), Encyclopedia of Statistics in Behavioral Science. Wiley.
WassermanL. W. (2004). All of statistics. A concise course in statistical inference. Springer.
33.
YoungE. L.SabbahH. Y.YoungB. J.ReiserM. L.RichardsonM. J. (2010). Gender differences and similarities in a screening process for emotional and behavioral risks in secondary schools. Journal of Emotional and Behavioral Disorders, 18, 225–235.
