Does Rating Time Predict Illusory Halo? A Mixture Rasch Facets Analysis of Halo Effects in Onscreen Assessments

Abstract

Halo effects refer to a persistent rater error posing a significant threat to the validity and fairness of assessments involving human raters. This research introduces a novel approach to analyzing halo effects by considering rating times recorded automatically in technology-based, online, or onscreen assessments as an additional data source. For this purpose, we propose the mixture Rasch facets model for halo with rating time. Utilizing Bayesian parameter estimation methods, we applied the model to a real dataset from a large-scale Chinese writing assessment. We found that rating time predicted illusory halo, such that longer rating times were associated with a greater likelihood of observing halo effects. Compared to traditional models, considering rating time as an additional variable resulted in better data–model fit and preserved the integrity of the latent scale, maintaining the informative value each rating criterion conveyed regarding the examinees’ performances. A simulation study confirmed the model’s superior parameter recovery under different levels of impact rating time had on the likelihood of halo error. Findings suggest that integrating rating time into measurement models can enhance the detection of illusory halo effects, enrich research on the psychological mechanisms underlying these effects, and improve the overall psychometric quality of assessments. The discussion focuses on implications for model development and future research into rater effects.