Sage Journals: Discover world-class research

Abstract

Cheating is a serious threat in unproctored ability assessment, irrespective of countermeasures taken, anticipated consequences (high vs. low stakes), and test modality (paper-pencil vs. computer-based). In the present study, we examined the power of (a) self-report-based indicators (i.e., Honesty-Humility and Overclaiming scales), (b) test data (i.e., performance with extremely difficult items), and (c) para data (i.e., reaction times, switching between browser tabs) to predict participants’ cheating behavior. To this end, 315 participants worked on a knowledge test in an unproctored online assessment and subsequently in a proctored lab assessment. We used multiple regression analysis and an extended latent change score model to assess the potential of the different indicators to predict cheating. In summary, test data and para data performed best, while traditional self-report-based indicators were not predictive. We discuss the findings with respect to unproctored testing in general and provide practical advice on cheating detection in online ability assessments.

Keywords

test taking cheating honesty para data declarative knowledge

Get full access to this article

View all access options for this article.

References

Ashton

M. C.

Lee

(2007). Empirical, theoretical, and practical advantages of the HEXACO model of personality structure. Personality and Social Psychology Review, 11(2), 150-166. https://doi.org/10.1177/1088868306294907

Ashton

M.C.

Lee

(2008). The HEXACO model of personality structure and the importance of the H factor. Social and Personality Psychology Compass, 2(5), 1952-1962. https://doi.org/10.1111/j.1751-9004.2008.00134.x

Ashton

M. C.

Lee

De Vries

R. E.

(2014). The HEXACO honesty-humility, agreeableness, and emotionality factors: A review of research and theory. Personality and Social Psychology Review, 18(2), 139-152. https://doi.org/10.1177/1088868314523838

Aust

Diedenhofen

Ullrich

Musch

(2012). Seriousness checks are useful to improve data validity in online research. Behavior Research Methods, 45, 527-535. https://doi.org/10.3758/s13428-012-0265-2

Bensch

Paulhus

D. L.

Stankov

Ziegler

(2019). Teasing apart overclaiming, overconfidence, and socially desirable responding. Assessment, 26(3), 351-363. https://doi.org/10.1177/1073191117700268

Bing

M. N.

Kluemper

Kristl Davison

Taylor

Novicevic

(2011). Overclaiming as a measure of faking. Organizational Behavior and Human Decision Processes, 116(1), 148-162. https://doi.org/10.1016/j.obhdp.2011.05.006

Bloemers

Oud

Dam

K. van

. (2016). Cheating on unproctored internet intelligence tests: Strategies and effects. Personnel Assessment and Decisions, 2(1), 21-29. https://doi.org/10.25035/pad.2016.003

Boubekki

Kröhne

Goldhammer

Schreiber

Brefeld

(2016). Data-driven analyses of electronic text books. In Michaelis

Piatkowski

Stolpe

(Eds.), Solving large scale learning tasks. Challenges and algorithms (pp. 362-376). Springer. https://doi.org/10.1007/978-3-319-41706-6_20

Bressan

Rosseel

Lombardi

(2018). The effect of faking on the correlation between two ordinal variables: Some population and Monte Carlo results. Frontiers in Psychology, 9, 1876. https://doi.org/10.3389/fpsyg.2018.01876

10.

Buchanan

E. M.

Scofield

J. E.

(2018). Methods to detect low quality data and its implication for psychological research. Behavior Research Methods, 50, 2586-2596. https://doi.org/10.3758/s13428-018-1035-6

11.

Butcher

J. N.

Graham

J. R.

Ben-Porath

Y. S.

Tellegen

Dahstrom

W. G.

Kaemmer

(2001). MMPI-2: Manual for administration and scoring. University of Minnesota Press.

12.

Carpenter

R. W.

Wycoff

A. M.

Trull

T. J.

(2016). Ambulatory assessment: New adventures in characterizing dynamic processes. Assessment, 23(4), 414-424. https://doi.org/10.1177/1073191116632341.

13.

Cattell

R. B.

Eber

H. W.

Tatsuoka

M. M.

(1970). Handbook for the Sixteen Personality Factor Questionnaire. Institute for Personality and Ability Testing.

14.

Chatterjee

Price

(1991). Regression diagnostics. Wiley.

15.

Cicek

G. J.

(1999). Cheating on tests: How to do it, detect it, and prevent it. Lawrence Erlbaum.

16.

Condon

D. M.

Revelle

(2014). The international cognitive ability resource: Development and initial validation of a public-domain measure. Intelligence, 43, 52-64. https://doi.org/10.1016/j.intell.2014.01.004

17.

Couper

M. P.

(2005). Technology trends in survey data collection. Social Science Computer Review, 23(4), 486-501. https://doi.org/10.1177/0894439305278972

18.

Davy

J. A.

Kincaid

J. F.

Smith

K. J.

Trawick

M. A.

(2007). An examination of the role of attitudinal characteristics and motivation on the cheating behavior of business students. Ethics & Behavior, 17(3), 281-302. https://doi.org/10.1080/10508420701519304

19.

Diedenhofen

Musch

(2017). PageFocus: Using paradata to detect and prevent cheating on online achievement tests. Behavior Research Methods, 49, 1444-1459. https://doi.org/10.3758/s13428-016-0800-7

20.

Drasgow

Nye

C. D.

Guo

Tay

(2009). Cheating on proctored tests: The other side of the unproctored debate. Industrial and Organizational Psychology, 2(1), 46-48. https://doi.org/10.1111/j.1754-9434.2008.01106.x

21.

Dunlop

P. D.

Bourdage

J. S.

de Vries

R. E.

Hilbig

B. E.

Zettler

Ludeke

S. G.

(2017). Openness to (reporting) experiences that one never had: Overclaiming as an outcome of the knowledge accumulated through a proclivity for cognitive and aesthetic exploration. Journal of Personality and Social Psychology, 113(5), 810-834. https://doi.org/10.1037/pspp0000110

22.

Fell

C. B.

König

C. J.

Jung

Sorg

Ziegler

(2019). Are country level prevalences of rule violations associated with knowledge overclaiming among students? International Journal of Psychology, 54(1), 17-22. https://doi.org/10.1002/ijop.12441

23.

Freund

P. A.

Holling

(2011). Who wants to take an intelligence test? Personality and achievement motivation in the context of ability testing. Personality and Individual Differences, 50(5), 723-728. https://doi.org/10.1016/j.paid.2010.12.025

24.

Freund

P. A.

Kuhn

J.-T.

Holling

(2011). Measuring current achievement motivation with the QCM: Short form development and investigation of measurement invariance. Personality and Individual Differences, 51(5), 629-634. https://doi.org/10.1016/j.paid.2011.05.033

25.

Gao

Shi

Maydeu-Olivares

(2020). Estimating the maximum likelihood root mean square error of approximation (RMSEA) with non-normal data: A Monte-Carlo study. Structural Equation Modeling: A Multidisciplinary Journal, 27(2), 192-201. https://doi.org/10.1080/10705511.2019.1637741

26.

Geiger

Olderbak

Sauter

Wilhelm

(2018). The “g” in faking: Doublethink the validity of personality self-report measures for applicant selection. Frontiers in Psychology, 9, Article 2153. https://doi.org/10.3389/fpsyg.2018.02153

27.

Gerbasi

M. E.

Prentice

D. A.

(2013). The self- and other-interest inventory. Journal of Personality and Social Psychology, 105(3), 495-514. https://doi.org/10.1037/a0033483

28.

Gnambs

Kaspar

(2017). Socially desirable responding in web-based questionnaires: A meta-analytic review of the candor hypothesis. Assessment, 24(6), 746-762. https://doi.org/10.1177/1073191115624547

29.

Goldhammer

Klein Entink

R. H.

(2011). Speed of reasoning and its relation to reasoning ability. Intelligence, 39(2-3), 108-119. https://doi.org/10.1016/j.intell.2011.02.001

30.

Guo

Drasgow

(2010). Identifying cheating on unproctored internet tests: The Z-test and the likelihood ratio test. International Journal of Selection and Assessment, 18(4), 351-364. https://doi.org/10.1111/j.1468-2389.2010.00518.x

31.

Hair

J. F.

Jr. Anderson

R. E.

Tatham

R. L.

Black

W. C.

(1995). Multivariate data analysis (3rd ed.). Macmillan.

32.

Harari

G. M.

Lane

N. D.

Wang

Crosier

B. S.

Campbell

A. T.

Gosling

S. D.

(2016). Using smartphones to collect behavioral data in psychological science: Opportunities, practical considerations, and challenges. Perspectives on Psychological Science, 11(6), 838-854. https://doi.org/10.1177/1745691616650285

33.

Hartung

Weiss

Wilhelm

(2017). Individual differences in performance on comprehension and knowledge tests with and without passages and questions. Learning and Individual Differences, 56, 143-150. https://doi.org/10.1016/j.lindif.2016.11.001

34.

Hathaway

S. R.

McKinley

J. C.

(1943). The Minnesota Multiphasic Personality Inventory. University of Minnesota Press.

35.

Hays

R. D.

Liu

Kapteyn

(2015). Use of Internet panels to conduct surveys. Behavior Research Methods, 47, 685-690. https://doi.org/10.3758/s13428-015-0617-9

36.

Heck

D. W.

Thielmann

Moshagen

Hilbig

B. E.

(2018). Who lies? A large-scale reanalysis linking basic personality traits to unethical decision making. Judgment and Decision Making, 13(4), 356-371.

37.

Hilbig

B. E.

Moshagen

Zettler

(2015). Truth will out: Linking personality, morality, and honesty through indirect questioning. Social Psychological and Personality Science, 6(2), 140-147. https://doi.org/10.1177/1948550614553640

38.

Hilbig

B. E.

Zettler

(2015). When the cat’s away, some mice will play: A basic trait account of dishonest behavior. Journal of Research in Personality, 57, 72-88. https://doi.org/10.1016/j.jrp.2015.04.003

39.

Hoffmann

Diedenhofen

Verschuere

Musch

(2015). A strong validation of the crosswise model using experimentally-induced cheating behavior. Experimental Psychology, 62(6), 403-414. https://doi.org/10.1027/1618-3169/a000304

40.

Holden

R. R.

Lambert

C. E.

(2015). Response latencies are alive and well for identifying fakers on a self-report personality inventory: A reconsideration of van Hooft and Born (2012). Behavior Research Methods, 47, 1436-1442. https://doi.org/10.3758/s13428-014-0524-5

41.

Hülür

Wilhelm

Schipolowski

(2011). Prediction of self-reported knowledge with over-claiming, fluid and crystallized intelligence and typical intellectual engagement. Learning and Individual Differences, 21(6), 742-746. https://doi.org/10.1016/j.lindif.2011.09.006

42.

Johnson

J. A.

(2001). Personality psychology: Methods. In Smelser

N. J.

Baltes

P. B.

(Eds.), International encyclopedia of the social and behavioral sciences (pp. 11313-11317). Pergamon.

43.

Karabatsos

(2003). Comparing the aberrant response detection performance of thirty-six person-fit statistics. Applied Measurement in Education, 16(4), 277-298. https://doi.org/10.1207/S15324818AME1604_2

44.

Karim

M. N.

Kaminsky

S. E.

Behrend

T. S.

(2014). Cheating, reactions, and performance in remotely proctored testing: An exploratory experimental study. Journal of Business and Psychology, 29, 555-572. https://doi.org/10.1007/s10869-014-9343-z

45.

Kieslich

P. J.

Henninger

(2017). Mousetrap: An integrated, open-source mouse-tracking package. Behavior Research Methods, 49, 1652-1667. https://doi.org/10.3758/s13428-017-0900-z

46.

Krantz

J. H.

Reips

U.-D.

(2017). The state of web-based research: A survey and call for inclusion in curricula. Behavior Research Methods, 49, 1621-1629. https://doi.org/10.3758/s13428-017-0882-x

47.

Kroehne

Goldhammer

(2018). How to conceptualize, represent, and analyze log data from technology-based assessments? A generic framework and an application to questionnaire items. Behaviormetrika, 45, 527-563. https://doi.org/10.1007/s41237-018-0063-y

48.

Lee

Ashton

M. C.

de Vries

R. E.

(2005). Predicting workplace delinquency and integrity with the HEXACO and five-factor models of personality structure. Human Performance, 18(2), 179-197. https://doi.org/10.1207/s15327043hup1802_4

49.

Lee

Ashton

M. C.

Wiltshire

Bourdage

J. S.

Visser

B. A.

Gallucci

(2013). Sex, power, and money: Prediction from the dark triad and honesty-humility. European Journal of Personality, 27(2), 169-184. https://doi.org/10.1002/per.1860

50.

Lievens

Burke

(2011). Dealing with the threats inherent in unproctored internet testing of cognitive ability: Results from a large-scale operational test program. Journal of Occupational and Organizational Psychology, 84(4), 817-824. https://doi.org/10.1348/096317910X522672

51.

MacCann

(2013). Instructed faking of the HEXACO reduces facet reliability and involves more Gc than Gf. Personality and Individual Differences, 55(7), 828-833. https://doi.org/10.1016/j.paid.2013.07.007

52.

Maricuțoiu

L. P.

Sârbescu

(2019). The relationship between faking and response latencies: A meta-analysis. European Journal of Psychological Assessment, 35(1), 3-13. https://doi.org/10.1027/1015-5759/a000361

53.

McArdle

J. J.

(2009). Latent variable modeling of differences and changes with longitudinal data. Annual Review of Psychology, 60(1), 577-605. https://doi.org/10.1146/annurev.psych.60.110707.163612

54.

McClelland

D. C.

Atkinson

J. W.

Clark

R. A.

Lowell

E. L.

(1953). The achievement motive. Appleton-Century-Crofts.

55.

McClintock

J. C.

(2016). Reduction in cheating following a forensic investigation on a statewide summative assessment. Applied Measurement in Education, 29(2), 132-143. https://doi.org/10.1080/08957347.2016.1138958

56.

Meijer

R. R.

(1996). Person-fit research: An introduction. Applied Measurement in Education, 9(1), 3-8. https://doi.org/10.1207/s15324818ame0901_2

57.

Moshagen

Hilbig

B. E.

(2017). The statistical analysis of cheating paradigms. Behavior Research Methods, 49, 724-732. https://doi.org/10.3758/s13428-016-0729-x

58.

Moshagen

Hilbig

B. E.

Zettler

(2014). Faktorenstruktur, psychometrische Eigenschaften und Messinvarianz der deutschsprachigen Version des 60-Item HEXACO Persönlichkeitsinventars [Factor structure, psychometric features and measurement invariance of the German version of the 60-item HEXACO personality inventory]. Diagnostica, 60(2), 86-97. https://doi.org/10.1026/0012-1924/a000112

59.

Moshagen

Hilbig

B. E.

Zettler

(2018). The dark core of personality. Psychological Review, 125(5), 656-688. https://doi.org/10.1037/rev0000111

60.

Moshagen

Musch

Erdfelder

(2012). A stochastic lie detector. Behavior Research Methods, 44, 222-231. https://doi.org/10.3758/s13428-011-0144-2

61.

Müller

Moshagen

(2019). True virtue, self-presentation, or both? A behavioral test of impression management and overclaiming. Psychological Assessment, 31(2), 181-191. https://doi.org/10.1037/pas0000657

62.

Murdock

T. B.

Anderman

E. M.

(2006). Motivational perspectives on student cheating: Toward an integrated model of academic dishonesty. Educational Psychologist, 41(3), 129-145. https://doi.org/10.1207/s15326985ep4103_1

63.

Naslund

J. A.

Marsch

L. A.

McHugo

G. J.

Bartels

S. J.

(2015). Emerging mHealth and eHealth interventions for serious mental illness: A review of the literature. Journal of Mental Health (Abingdon, England), 24(5), 321-332. https://doi.org/10.3109/09638237.2015.1019054

64.

Neter

Wassermann

Kutner

M. H.

(1989). Applied linear regression models. Irwin.

65.

Niessen

A. S. M.

Meijer

R. R.

Tendeiro

J. N.

(2016). Detecting careless respondents in web-based questionnaires: Which method to use? Journal of Research in Personality, 63(August), 1-11. https://doi.org/10.1016/j.jrp.2016.04.010

66.

Nosek

B. A.

Alter

Banks

G. C.

Borsboom

Bowman

S. D.

Breckler

S. J.

Buck

Chambers

C. D.

Chin

Christensen

Contestabile

Dafoe

Eich

Freese

Glennerster

Goroff

Green

D. P.

Hesse

Humphreys

Ishiyama

. . . Yarkoni

(2015). Promoting an open research culture. Science, 348(6242), 1420-1422. https://doi.org/10.1126/science.aab2374

67.

Nye

C. D.

B.-R.

Drasgow

Fine

(2008). Two-step testing in employee selection: Is score inflation a problem? International Journal of Selection and Assessment, 16(2), 112-120. https://doi.org/10.1111/j.1468-2389.2008.00416.x

68.

Olson

Parkhurst

(2013). Collecting paradata for measurement error evaluations. In Kreuter

(Ed.), Improving surveys with paradata (pp. 43-72). Wiley. https://doi.org/10.1002/9781118596869.ch3

69.

O’Neill

H. M.

Pfeiffer

C. A.

(2012). The impact of honour codes and perceptions of cheating on academic cheating behaviours, especially for MBA bound undergraduates. Accounting Education, 21(3), 231-245. https://doi.org/10.1080/09639284.2011.590012

70.

Pahor

Stavropoulos

Jaeggi

S. M.

Seitz

A. R.

(2018). Validation of a matrix reasoning task for mobile devices. Behavior Research Methods, 51, 2256-2267. https://doi.org/10.3758/s13428-018-1152-2

71.

Paulhus

D. L.

Harms

P. D.

Bruce

M. N.

Lysy

D. C.

(2003). The over-claiming technique: Measuring self-enhancement independent of ability. Journal of Personality and Social Psychology, 84(4), 890-904. https://doi.org/10.1037/0022-3514.84.4.890

72.

Phillips

D. L.

Clancy

K. J.

(1972). Some effects of “social desirabilty” in survey studies. American Journal of Psychology, 77(5), 921-940.

73.

Pulfrey

Butera

(2013). Why neoliberal values of self-enhancement lead to cheating in higher education: A motivational account. Psychological Science, 24(11), 2153-2162. https://doi.org/10.1177/0956797613487221

74.

R Core Team. (2018). R: A language and environment for statistical computing (Version 3.5.1). https://www.R-project.org/

75.

Roma

Mazza

Mammarella

Mantovani

Mandarelli

Ferracuti

(2019). Faking-good behavior in self-favorable scales of the MMPI-2: A study with time pressure. European Journal of Psychological Assessment. Advance online publication. https://doi.org/10.1027/1015-5759/a000511

76.

Rosseel

(2012). lavaan: An R package for structural equation modeling. Journal of Statistical Software, 48(2), 1-36. http://doi.org/10.18637/jss.v048.i02

77.

Rovai

A. P.

(2000). Online and traditional assessments: What is the difference? Internet and Higher Education, 3(3), 141-151. https://doi.org/10.1016/S1096-7516(01)00028-8

78.

Schroeders

Bucholtz

Formazin

Wilhelm

(2013). Modality specificity of comprehension abilities in the sciences. European Journal of Psychological Assessment, 29(1), 3-11. https://doi.org/10.1027/1015-5759/a000114

79.

Schroeders

Wilhelm

Schipolowski

(2010). Internet-based ability testing. In Gosling

S. D.

Johnson

J. A.

(Eds.), Advanced methods for conducting online behavioral research (pp. 131-148). American Psychological Association.

80.

Sliwinski

M. J.

Mogle

J. A.

Hyun

Munoz

Smyth

J. M.

Lipton

R. B.

(2018). Reliability and validity of ambulatory cognitive assessments. Assessment, 25(1), 14-30. https://doi.org/10.1177/1073191116643164

81.

Slobogin

(2005). Mental disorder as an exemption from the death penalty: The ABA-IRR task force recommendations. Catholic University Law Review, 54(4), 1133-1152.

82.

Sotaridona

L. S.

Meijer

R. R.

(2002). Statistical properties of the K-Index for detecting answer copying. Journal of Educational Measurement, 39(2), 115-132.

83.

Steger

Schroeders

Gnambs

(2020). A meta-analysis of test scores in proctored and unproctored ability assessments. European Journal of Psychological Assessment, 36(1), 174-184. https://doi.org/10.1027/1015-5759/a000494

84.

Steger

Schroeders

Wilhelm

(2019). On the dimensionality of crystallized intelligence: A smartphone-based assessment. Intelligence, 72, 76-85. https://doi.org/10.1016/j.intell.2018.12.002

85.

Steinmayr

Spinath

(2008). Sex differences in school achievement: What are the roles of personality and achievement motivation? European Journal of Personality, 22(3), 185-209. https://doi.org/10.1002/per.676

86.

Stieger

Lewetz

Reips

(2018). Can smartphones be used to bring computer-based tasks from the lab to the field? A mobile experience-sampling method study about the pace of life. Behavior Research Methods, 50, 2267-2275. https://doi.org/10.3758/s13428-017-0991-6

87.

Stowell

J. R.

Bennett

(2010). Effects of online testing on student exam performance and test anxiety. Journal of Educational Computing Research, 42(2), 161-171. https://doi.org/10.2190/EC.42.2.b

88.

Thielmann

Hilbig

B. E.

(2018). Daring dishonesty: On the role of sanctions for (un)ethical behavior. Journal of Experimental Social Psychology, 79, 71-77. https://doi.org/10.1016/j.jesp.2018.06.009

89.

Tippins

N. T.

Beaty

Drasgow

Gibson

W. M.

Pearlman

Segall

D. O.

Shepherd

(2006). Unproctored internet testing in employment settings. Personnel Psychology, 59(1), 189-225. https://doi.org/10.1111/j.1744-6570.2006.00909.x

90.

Vohs

K. D.

Schooler

J. W.

(2008). The value of believing in free will encouraging a belief in determinism increases cheating. Psychological Science, 19(1), 49-54. https://doi.org/10.1111/j.1467-9280.2008.02045.x

91.

Whitley

(1998). Factors associated with cheating among college students: A review. Research in Higher Education, 39, 235-274. https://doi.org/10.1023/A:1018724900565

92.

Wilhelm

McKnight

P. E.

(2002). Ability and achievement testing on the World Wide Web. In Batinic

Reips

U.-D.

Bosnjak

(Eds.), Online social sciences (pp. 167-193). Hogrefe & Huber.

93.

Wiltermuth

S. S.

(2011). Cheating more when the spoils are split. Organizational Behavior and Human Decision Processes, 115(2), 157-168. https://doi.org/10.1016/j.obhdp.2010.10.001

94.

Wright

A. G. C.

Zimmermann

(2019). Applied ambulatory assessment: Integrating idiographic and nomothetic principles of measurement. Psychological Assessment, 31(12), 1467-1480. https://doi.org/10.1037/pas0000685

95.

Yarkoni

(2012). Psychoinformatics: New horizons at the interface of the psychological and computing sciences. Current Directions in Psychological Science, 21(6), 391-397. https://doi.org/10.1177/0963721412457362

96.

Zettler

Lang

J. W. B.

Hülsheger

U. R.

Hilbig

B. E.

(2016). Dissociating indifferent, directional, and extreme responding in personality data: Applying the three-process model to self- and observer reports: Response processes in personality data. Journal of Personality, 84(4), 461-472. https://doi.org/10.1111/jopy.12172

97.

Zimmermann

Ritter

Masuhr

Jaeger

Spitzer

Woods

W. C.

Happel

Wright

A. G. C.

(2019). Integrating Structure and Dynamics in Personality Assessment: First Steps Toward the Development and Validation of a Personality Dynamics Diary. Psychological Assessment, 516–531. http://dx.doi.org/10.1037/pas0000625

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.48 MB

Caught in the Act: Predicting Cheating in Unproctored Knowledge Assessment

Abstract

Keywords

Get full access to this article

References

Supplementary Material