Attention Promises Fail to Improve Accuracy on Online Attention Checks in College Sample: A Test Across IRI and IMC Formats

Abstract

This study tested whether a brief pre-survey “attention promise” improves attention-check performance and whether accuracy differs between Instructed Response Items (IRIs) and an Instructional Manipulation Check (IMC). Undergraduate participants (N = 249) were randomly assigned to an attention-promise or control condition and completed two IRIs and one IMC embedded within personality, self-control, and attention measures. Accuracy was re-expressed as proportion correct (0–1). A mixed ANOVA (Check Type: IRI vs. IMC; Attention Promise: present vs. absent) showed a large main effect of check type (IRI > IMC), with no main effect of the attention promise and no interaction. A trial-level mixed-effects logistic regression with a random intercept for participants converged on the same pattern (substantially lower odds of passing the IMC; no effects of the promise). As an external validity check, internal consistency (Cronbach’s α) for the multi-item scales was generally higher among participants who passed all attention checks than among those who failed at least one, suggesting that attention-check performance tracks broader response quality. Overall, accuracy was substantially lower on the IMC than on the IRIs, and a simple pre-survey commitment did not meaningfully improve attentiveness in this undergraduate sample.

Keywords

attention check promise instructed response item instructional manipulation check

Get full access to this article

View all access options for this article.

References

Abbey

J. D.

Meloy

M. G.

(2017). Attention by design: Using attention checks to detect inattentive respondents and improve data quality. Journal of Operations Management, 53–56(1), 63–70. https://doi.org/10.1016/j.jom.2017.06.001

Derryberry

Reed

(2002). Anxiety-related attentional biases and their regulation by attentional control. Journal of Abnormal Psychology, 111(2), 225–236. https://doi.org/10.1037//0021-843x.111.2.225

Festinger

Katz

(Eds.), (1953). Research methods in the behavioral sciences (p. xi–660). Dryden Press.

Geisen

(2022). Using attention checks in your surveys may harm data quality. Qualtrics. https://www.qualtrics.com/blog/attention-checks-and-data-quality/

Gummer

Roßmann

Silber

(2021). Using instructed response items as attention checks in web surveys: Properties and implementation. Sociological Methods & Research, 50(1), 238–264. https://doi.org/10.1177/0049124118769083

Hauser

D. J.

Schwarz

(2015). It’s a trap! Instructional manipulation checks prompt systematic thinking on “tricky” tasks. Sage Open, 5(2), Article 2158244015584617. https://doi.org/10.1177/2158244015584617

John

O. P.

Donahue

E. M.

Kentle

R. L.

(1991). Big Five Inventory (BFI) [Measurement instrument]. https://doi.org/10.1037/t07550-000

Kam

C. C. S.

Chan

G. H.

(2018). Examination of the validity of instructed response items in identifying careless respondents. Personality and Individual Differences, 129, 83–87. https://doi.org/10.1016/j.paid.2018.03.022

Kincaid

J. P.

Fishburne

R. P.

Jr. Rogers

R. L.

Chissom

B. S.

(1975). Derivation of new readability formulas (Automated readability index, fog count, and Flesch reading ease formula) for Navy enlisted personnel (Research Branch Report 8-75). Naval Technical Training Command.

10.

Kung

F. Y. H.

Kwok

Brown

D. J.

(2018). Are attention check questions a threat to scale validity? Applied Psychology, 67(2), 264–283. https://doi.org/10.1111/apps.12108

11.

Morren

Paas

L. J.

(2020). Short and long instructional manipulation checks: What do they measure? International Journal of Public Opinion Research, 32(4), 790–800. https://doi.org/10.1093/ijpor/edz046

12.

Muszyński

(2023). Attention checks and how to use them: Review and practical recommendations. ASK: Research & Methods, 32(1), 3–38. https://doi.org/10.18061/ask.v32i1.0001

13.

Oppenheimer

D. M.

Meyvis

Davidenko

(2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867–872. https://doi.org/10.1016/j.jesp.2009.03.009

14.

Silber

Roßmann

Gummer

(2022). The issue of noncompliance in attention check questions: False positives in instructed response items. Field Methods, 34(4), 346–360. https://doi.org/10.1177/1525822X221115830

15.

Tangney

J. P.

Baumeister

R. F.

Boone

A. L.

(2004). High self-control predicts good adjustment, less pathology, better grades, and interpersonal success. Journal of Personality, 72(2), 271–324. https://doi.org/10.1111/j.0022-3506.2004.00263.x

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.19 MB