Comparing respondents who passed versus failed an Instructional Manipulation Check: A case study about support for climate change policies

Abstract

The presence of satisficers among survey respondents threatens survey data quality. To identify such respondents, Oppenheimer et al. developed the Instructional Manipulation Check (IMC), which has been used as a tool to exclude observations from the analyses. However, this practice has raised concerns regarding its effects on the external validity and the substantive conclusions of studies excluding respondents who fail an IMC. Thus, more research on the differences between respondents who pass versus fail an IMC regarding sociodemographic and attitudinal variables is needed. This study compares respondents who passed versus failed an IMC both for descriptive and causal analyses based on structural equation modeling (SEM) using data from an online survey implemented in Spain in 2019. These data were analyzed by Rubio Juan and Revilla without taking into account the results of the IMC. We find that those who passed the IMC do differ significantly from those who failed for two sociodemographic and five attitudinal variables, out of 18 variables compared. Moreover, in terms of substantive conclusions, differences between those who passed and failed the IMC vary depending on the specific variables under study.

Keywords

adaptation policies climate change policies instructional manipulation check (IMC)mitigation policies structural equation modeling (SEM)

Get full access to this article

View all access options for this article.

References

Anduiza

Galais

(2016). Answering without reading: IMCs and strong satisficing in online surveys. International Journal of Public Opinion Research, 29(3), 497–519. https://doi.org/10.1093/ijpor/edw007

Berinsky

A. J.

Margolis

M. F.

Sances

M. W.

(2014). Separating the shirkers from the workers? Making sure respondents pay attention on self-administered surveys. American Journal of Political Science, 58(3), 739–753. https://doi.org/10.1111/ajps.12081

Bosch

O. J.

Revilla

Paura

(2018). Do millennials differ in terms of survey participation? International Journal of Market Research, 61(4), 359–365. https://doi.org/10.1177/1470785318815567

De Castellarnau

Saris

W. E.

(2014). A simple way to correct for measurement errors. European Social Survey Education Net (ESS EduNet). http://essedunet.nsd.uib.no/cms/topics/measurement/

Gummer

Roßmann

Silber

(2018). Using instructed response items as attention checks in web surveys: Properties and implementation. Sociological Methods & Research, 50, 238–264. https://doi.org/10.1177/0049124118769083

Jöreskog

K. G.

Sörbom

(1996). LISREL 8: User’s reference guide. Scientific Software International.

Krosnick

J. A.

(1991). Response strategies for coping with the cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5(3), 213–236. https://doi.org/10.1002/acp.2350050305

Liu

Wronski

(2018). Trap questions in online surveys: Results from three web survey experiments. International Journal of Market Research, 60(1), 32–49. https://doi.org/10.1177/1470785317744856

Morren

Paas

L. J.

(2020). Short and long instructional manipulation checks: What do they measure? International Journal of Public Opinion Research, 32, 790–800. https://doi.org/10.1093/ijpor/edz046

10.

Oppenheimer

D. M.

Meyvis

Davidenko

(2009). Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology, 45(4), 867–872. https://doi.org/10.1016/j.jesp.2009.03.009

11.

Paas

L. J.

Morren

(2018). Please do not answer if you are reading this: Respondent attention in online panels. Marketing Letters, 29, 13–21. https://doi.org/10.1007/s11002-018-9448-7

12.

Rubio Juan

Revilla

(2020). Support for mitigation and adaptation climate change policies: Effects of five attitudinal factors [Working paper]. http://hdl.handle.net/10230/44338

13.

Saris

W. E.

Gallhofer

I. N.

(2014). Design, evaluation, and analysis of questionnaires for survey research. John Wiley.

14.

Saris

W. E.

Revilla

(2016). Correction for measurement errors in survey research: necessary and possible. Social Indicators Research, 127(3), 1005–1020. https://doi.org/10.1007/s11205-015-1002-x

15.

Saris

W. E.

Satorra

Van der Veld

W. M.

(2009). Testing structural equation models or detection of misspecifications? Structural Equation Modeling, 16(4), 561–582.

16.

StataCorp. (2015). Stata statistical software: Release 14.

17.

Van der Veld

W. M.

Saris

W. E.

Satorra

(2008). JRule 3.0: User’s guide. http://www.vanderveld.nl/JRule

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.34 MB