Examining the accuracy of orthogonal latent mean comparisons in unbalanced conditions

Abstract

Researchers understand that conducting numerous pairwise comparisons between group means increases the Type I error rate, prompting the use of planned contrasts like orthogonal contrast sets. Implicit to orthogonal contrast sets is the principal assumption that groups are balanced in size. Further, when dealing with complex variables like latent constructs, specialized modeling is necessary. Understanding how violating the assumptions of orthogonal contrasts, specifically under conditions of sample imbalance, can help identify variability in parameter recovery. This study examines the effect of sample size imbalance and modeling approach on the accuracy of latent group mean difference estimates when using orthogonal contrasts. Monte Carlo simulations compared the Multiple Indicators Multiple Causes (MIMIC) and re-parameterized multigroup confirmatory factor analysis models while manipulating sample sizes, group proportions, and effect size. Results suggest declining parameter recovery as group imbalance increased, particularly in small samples, with some estimates falling below acceptable thresholds for power, Type I error, and bias. The MIMIC model consistently produced more accurate estimates, though is replete with implicit measurement assumptions that are seldom tested. These findings suggest that researchers using orthogonal contrasts when comparing groups on a latent variable continuum must (a) be aware of examined group’s sample size proportions and the impact of group size inequalities on estimate accuracy, and (b) carefully consider the costs and benefits of the latent variable modeling approach, including how the model addresses measurement non-invariance.