Handle with Care: Implementation of the List Experiment and Crosswise Model in a Large-scale Survey on Academic Misconduct

Abstract

This research analyzes the effectiveness of the list experiment and crosswise model in measuring self-plagiarism and data manipulation. Both methods were implemented in a large-scale survey of academics on social norms and academic misconduct. As the results lend little confidence about the effectiveness of the methods, researchers are best advised to avoid them or, at best, to handle them with care.

Get full access to this article

View all access options for this article.

References

Aronow

P. M.

Coppock

Crawford

F. W.

Green

D. P.

. 2015. Combining list experiment and direct question estimates of sensitive behavior prevalence. Journal of Survey Statistics and Methodology 3:43–66.

Blair

Imai

. 2012. Statistical analysis of list experiments. Political Analysis 20:47–77.

Coffman

K. B.

Coffman

L. C.

Ericson

K. M. M.

. 2017. The size of the LGBT population and the magnitude of antigay sentiment are substantially underestimated. Management Science 63:3168–86.

Coutts

Jann

. 2011. Sensitive questions in online surveys: Experimental results for the randomized response technique (RRT) and the unmatched count technique (UCT). Sociological Methods & Research 40:169–93.

Diekmann

2012. Making use of Benford’s Law for the randomized response technique. Sociological Methods & Research 41:325–34.

Droitcour

Caspar

R. A.

Hubbard

M. L.

Parsley

T. L.

Visscher

Ezzati

T. M.

. 1991. The item count technique as a method of indirect questioning: A review of its development and a case study application. In Measurement errors in surveys, eds. Biemer

P. P.

Groves

R. M.

Lyberg

L. E.

Mathiowetz

N. A.

Sudman

, 185–210. New York: John Wiley & Sons.

Fanelli

2009. How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PloS One 4:e5738.

Gelman

2014. Thinking of doing a list experiment? Here’s a list of reasons why you should think again. In Statistical modeling, causal inference, and social science. https://statmodeling.stat.columbia.edu/2014/04/23/thinking--list--experiment--heres--list--reasons--think/ (accessed December 10, 2020).

Gilens

Sniderman

P. M.

Kuklinski

J. H.

. 1998. Affirmative action and the politics of realignment. British Journal of Political Science 28:159–83.

10.

Glynn

A. N.

2013. What can we learn with statistical truth serum? Design and analysis of the list experiment. Public Opinion Quarterly 77 (S1): 159–72.

11.

Hinsley

Keane

John

F. A. St.

Ibbett

Nuno

. 2019. Asking sensitive questions using the unmatched count technique: Applications and guidelines for conservation. Methods in Ecology and Evolution 10:308–19.

12.

Hoffmann

de Puiseau

B. W.

Schmidt

A. F.

Musch

. 2017. On the comprehensibility and perceived privacy protection of indirect questioning techniques. Behavior Research Methods 49:1470–83.

13.

Hoffmann

Musch

. 2016. Assessing the validity of two indirect questioning techniques: A stochastic lie detector versus the crosswise model. Behavior Research Methods 48:1032–46.

14.

Höglinger

2016. Revealing the truth? Validating the randomized response technique for surveying sensitive topics. PhD thesis, ETH Zurich, Zurich.

15.

Höglinger

Diekmann

. 2017. Uncovering a blind spot in sensitive question research: False positives undermine the Crosswise-Model RRT. Political Analysis 25:131–37.

16.

Höglinger

Jann

. 2018. More is not always better: An experimental individual-level validation of the randomized response technique and the crosswise model. PLoS One 13:e0201770.

17.

Höglinger

Jann

Diekmann

. 2016. Sensitive questions in online surveys: An experimental evaluation of different implementations of the randomized response technique and the crosswise model. Survey Research Methods 10:171–87.

18.

Holbrook

A. L.

Krosnick

J. A.

. 2010. Measuring voter turnout by using the randomized response technique: Evidence calling into question the methods validity. Public Opinion Quarterly 74:328–43.

19.

Jann

Jerke

Krumpal

. 2012. Asking sensitive questions using the crosswise model: An experimental survey measuring plagiarism. Public Opinion Quarterly 76:32–49.

20.

Jerke

Johann

Rauhut

Thomas

. 2019. Too sophisticated even for highly educated survey respondents? A qualitative assessment of indirect question formats for sensitive questions. Survey Research Methods 13:319–51.

21.

Johann

Thomas

. 2017. Testing the properties of the crosswise model in reducing social desirability bias on attitudes towards Muslims: A validation study. Survey Methods: Insights from the Field. https://surveyinsights.org/?p=8887 (accessed June 28, 2020).

22.

Kiewiet de Jonge

C. P.

2015. Who lies about electoral gifts? Experimental evidence from Latin America. Public Opinion Quarterly 79:710–39.

23.

Korndörfer

Krumpal

Schmukle

S. C.

. 2014. Measuring and explaining tax evasion: Improving self-reports using the crosswise model. Journal of Economic Psychology 45:18–32.

24.

Krumpal

2013. Determinants of social desirability bias in sensitive surveys: A literature review. Quality & Quantity 47:2025–47.

25.

Krumpal

Jann

Auspurg

von Hermanni

. 2015. Asking sensitive questions: A critical account of the randomized response technique and related methods. In Improving survey methods: Lessons from recent research, eds. Engel

Jann

Lynn

Scherpenzeel

Sturgis

, 122–36. New York: Routledge.

26.

Landsheer

J. A.

Van Der Heijden

Van Gils

. 1999. Trust and understanding, two psychological aspects of randomized response. Quality and Quantity 33:1–12.

27.

Van den Noortgate

. 2019. A meta-analysis of the relative effectiveness of the item count technique compared to direct questioning. Sociological Methods & Research, Online First:1–40.

28.

Miller

J. D.

1984. A new survey technique for studying deviant behavior. PhD thesis, Department of Sociology, George Washington University, Washington, DC.

29.

Rauhut

Johann

Jerke

Rathmann

Velicu

. 2020. The Zurich Survey of Academics: Methods, design, and data. University of Zürich.

30.

Roberts

D. L.

John

F. A. S.

. 2014. Estimating the prevalence of researcher misconduct: A study of UK academics within biological sciences. PeerJ 2:e562.

31.

Schnell

Thomas

. Forthcoming. A meta-analysis of studies on the performance of the crosswise model. Sociological Methods and Research.

32.

Thomas

Johann

Kritzinger

Plescia

Zeglovits

. 2016. Estimating sensitive behavior: The ICT and high-incidence electoral behavior. International Journal of Public Opinion Research 29:157–71.

33.

Tourangeau

Rips

L. J.

Rasinski

. 2000. The psychology of the survey response. Cambridge: Cambridge University Press.

34.

Tourangeau

Yan

. 2007. Sensitive questions in surveys. Psychological Bulletin 133:859–83.

35.

Tsuchiya

Hirai

Ono

. 2007. A study of the properties of the item count technique. Public Opinion Quarterly 71:253–72.

36.

Umesh

U. N.

Peterson

R. A.

. 1991. A critical evaluation of the randomized response method applications, validation, and research agenda. Sociological Methods & Research 20:104–38.

37.

Walzenbach

Hinz

. 2019. Pouring water into wine: Revisiting the advantages of the crosswise model for asking sensitive questions. Survey Methods: Insights from the Field, 1–16. doi:10.13094/SMIF-2019-00002

38.

Wolter

2012. Heikle Fragen in interviews: Eine theoretische und empirische Validierung der Randomized Response-Technik [Sensitive questions in interviews: A theoretical and empirical validation of the randomized response technique]. Wiesbaden, Germany: VS Verlag.

39.

Wolter

Laier

. 2014. The effectiveness of the item count technique in eliciting valid answers to sensitive questions. An evaluation in the context of self-reported delinquency. Survey Research Methods 8:153–68.

40.

J.-W.

Tian

G.-L.

Tang

M.-L.

. 2008. Two new models for survey sampling with sensitive characteristic: Design and analysis. Metrika 67:251–63.