Random Responding Detection in Two Alternative Forced Choice Tests: l z vs. Optimal Appropriateness Measurement

Abstract

Multidimensional forced choice (MFC) test formats are commonly used as an alternative to traditional rating scale formats to reduce aberrant responding, especially faking in high-stakes settings. However, MFC remains susceptible to random responding, particularly in low-stakes settings where respondents may be insufficiently motivated and in high-stakes settings where some assessments may be viewed as less consequential. To ensure the validity of inferences drawn from MFC data, effective methods for detecting random responding are needed. This research contributes to the MFC literature on aberrant responding detection by evaluating the effectiveness of the item response theory (IRT)-based person fit statistic l _z for detecting random responding in multi-unidimensional pairwise preference (MUPP)-based MFC tests, using optimal appropriateness measurement (OAM) as a theoretical benchmark. Two simulation studies were conducted. Study 1 compared l _z with OAM, and Study 2 examined l _z in a broader simulation design. Results showed that (1) higher proportions of randomly answered items, longer tests, and the use of empirical critical values were associated with greater detection power for l _z , (2) the proportion of aberrant respondents did not affect l _z performance, and (3) OAM outperformed l _z only when the random responding model was correctly specified, a condition that can be realized in simulation but may not hold in applied testing contexts. Overall, the findings support the use of l _z with empirical critical values as a practical method for detecting random responding in MUPP-based MFC tests.