Careless Responding May Reduce Data Quality in Self-Report Questionnaire Research: Evidence From Adult Temperament Samples From Two Cultures

Abstract

With the increasing publication of self-report online studies, concerns are growing about the quality of the data collected through these methods. This study focused on response bias, a major threat to data quality, by analyzing data from a real-world study on adult temperament conducted in two different countries. The sample included 1,497 participants aged 18–80 years from the United States (n = 598) and Lithuania (n = 899). The primary objectives were to determine the prevalence of response bias and to evaluate its impact on psychometric outcomes. Indicators of biased responding included patterns suggestive of potentially careless responding (e.g., invariant and random response patterns) and those flagged by internal validity checks or clinical controls (e.g., social desirability and ratings-perception discrepancies). Results indicated that the inclusion of data reflecting potentially careless responding reduced internal consistency and distorted factor structure, whereas its exclusion improved these psychometric indicators. In contrast, with regard to clinical controls, removing flagged data resulted in a decline in psychometric quality. Additionally, higher rates of careless responding were observed in the sample subjected to forced answering. These findings highlight the importance of mitigating response bias in online self-report research and raise broader questions about the integrity of data in existing survey-based datasets. By jointly evaluating careless responding and clinical threats in real-world, cross-national samples, this study extends prior work and demonstrates the applied value of post-hoc screening for improving psychometric quality.