Identifying Invalid Responders in a Campus Climate Survey: Types,Impact on Data,and Best Indicators

Abstract

Self-report surveys that are online, lengthy, and contain sensitive material greatly increase the probability of invalid responding (IR) on the instrument. Most research to inform our identification of invalid responders have not been able to test their methodologies where all these conditions are present. This study systematically adopted 10 IR indicators based on direct, archival, and statistic strategies to identify IR providing answers on a lengthy survey collecting campus climate/violence information that college students (N = 6,995) accessed online. Exploratory factor analysis indicated two internal factors (i.e., careless and extreme responding) underlying these IR indicators. Latent class analysis identified 4.8% of the sample as being invalid responders. Compared with honest responders, invalid responders were significantly more likely to report forms of victimization and a greater negative impact from physical abuse or sexual assault. Of importance, mean scores on victimization scales were significantly higher for invalid responders, illustrating the potential for IR data to skew prevalence rates. IR indicators differentially identified honest and invalid responders. The findings of this study contribute to the systematic investigation of IR with college students completing online and lengthy surveys that address sensitive material.

Keywords

invalid responding survey measurement college surveys statistical strategies for invalid responding

Get full access to this article

View all access options for this article.

References

Baer

R. A.

Ballenger

Berry

D. T. R.

Wetter

M. W.

(1997). Detection of random responding on the MMPI-A. Journal of Personality Assessment, 68, 139–151.

Banyard

Cohn

Edwards

K. M.

Moynihan

M. M.

Walsh

Ward

(2013). Unwanted sexual experiences at UNH: 2012 Study and changes over time. University of New Hampshire. https://cola.unh.edu/sites/default/files/media/2018/10/unwanted-sexual-experiences-unh.pdf

Beach

D. A.

(1989). Identifying the random responder. The Journal of Psychology, 123(1), 101–103.

Berry

D. T. R.

Wetter

M. W.

Baer

R. A.

Larsen

Clark

Monroe

(1992). MMPI-2 random responding indices: Validation using a self-report methodology. Psychological Assessment, 4, 340–345.

Campbell

(2014). Johns Hopkins University campus sexual assault project. Johns Hopkins University.

Centers for Disease Control and Prevention. (1997). Youth risk behavior surveillance: National college health risk behavior survey—United States, 1995 (DHHS Publication No. SS-6). Epidemiology Program Office.

Cohen

(1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum.

Cornell

Klein

Konold

Huang

(2012). Effects of validity screening items on adolescent survey data. Psychological Assessment, 24, 21–35. https://doi.org/10.1037/a0024824

Curran

P. G.

(2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4–19. doi.org/10.1016/j.jesp.2015.07.006

10.

DeSimone

J. A.

Harms

P. D.

DeSimone

A. J.

(2015). Best practice recommendations for data screening. Journal of Organizational Behavior, 36(2), 171–181.

11.

Douglas

K. M.

McGarty

(2001). Identifiability and self-presentation: Computer-mediated communication and intergroup interaction. British Journal of Social Psychology, 40, 399–416.

12.

Dye

M. L.

Davis

K. E.

(2003). Stalking and psychological abuse: Common factors and relationship-specific characteristics. Violence and Victims, 18, 163–180.

13.

Fan

Miller

B. C.

Park

Winward

Christensen

Grotevant

Tai

(2006). An exploratory study about inaccuracy and invalidity in adolescent self-report surveys. Field Methods, 18(3), 223–244.

14.

Follingstad

D. R.

(2011). A measure of severe psychological abuse normed on a nationally representative sample of adults. Journal of Interpersonal Violence, 26(6), 1194–1214. https://doi.org/10.1177/0886260510368157

15.

Goldberg

L. R.

(1999). A broad-bandwidth, public domain, personality inventory measuring the lower-level facets of several five-factor models. In Mervielde

Dreary

Fruyt

F. D.

Ostendorf

(Eds.), Personality psychology in Europe (Vol. 7, pp. 7–28). Tilberg University Press.

16.

Gorsuch

R. L.

(1983). Factor analysis (2nd ed.). Lawrence Erlbaum.

17.

Guttman

(1954). Some necessary and sufficient conditions for common factor analysis. Psychometrika, 19, 149–161.

18.

Hamburger

M. E.

Basile

K. C.

Vivolo

A. M.

(2011). Measuring bullying victimization, perpetration, and bystander experiences: A compendium of assessment tools. Centers for Disease Control and Prevention, National Center for Injury Prevention and Control.

19.

Huang

Bowling

Liu

(2015). Detecting insufficient effort responding with an infrequency scale: Evaluating validity and participant reactions. Journal of Business & Psychology, 30(2), 299–311.

20.

Huang

J. L.

Curran

P. G.

Keeney

Poposki

E. M.

DeShon

R. P.

(2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business Psychology, 27, 99–114. https://doi.org/10.1007/s10869-011-9231-8

21.

IBM Corp. (2016). IBM SPSS statistics for Windows (Version 24.0).

22.

Johnson

J. A.

(2005). Ascertaining the validity of individual protocols from web-based personality inventories. Journal of Research in Personality, 39, 103–129.

23.

Kaiser

H. F.

(1960). The application of electronic computers to factor analysis. Educational and Psychological Measurement, 20, 141–151.

24.

Krebs

C. P.

Lindquist

C. H.

Warner

T. D.

Fisher

B. S.

Martin

S. L.

(2007). Campus sexual assault (CSA) study, Final report. National Institute of Justice. https://www.ncjrs.gov/pdffiles1/nij/grants/221153.pdf

25.

Lee

(2006). Privacy, publicity, and accountability of self-presentation in an on-line discussion group. Sociological Inquiry, 76, 1–22.

26.

Maniaci

M. R.

Rogge

R. D.

(2014). Caring about carelessness: Participant inattention and its effect on research. Journal of Research in Personality, 48, 61–83.

27.

Meade

A. W.

Craig

S. B.

(2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455. https://doi.org/10.1037/a0028085.

28.

Miller

McCauley

H. L.

(2013). Adolescent relationship abuse and reproductive and sexual coercion among teens. Current Opinion in Obstetrics & Gynecology, 25(5), 364–369.

29.

Muthén

L. K.

Muthén

B. O.

(1998–2017). Mplus user’s guide: Statistical analysis with Latent variables (7th ed.).

30.

Pett

M. A.

Lackey

N. R.

Sullivan

J. J.

(2003). Making sense of factor analysis: The use of factor analysis for instrument development in health care research. SAGE.

31.

Robinson

J. P.

Espelage

D. L.

(2011). Inequities in educational and psychological outcomes between LGBTQ and straight students in middle and high school. Educational Researcher, 41, 309–319. https://doi.org/10.3102/0013189X11422112

32.

Robinson-Cimpian

J. P.

(2014). Inaccurate estimation of disparities due to mischievous responders: Several suggestions to assess conclusions. Educational Researcher, 43(4), 171–185. https://doi.org/10.3102/0013189X14534297

33.

Rousseeuw

P. J.

Croux

(1992). Explicit scale estimators with high breakdown point. In Dodge

(Ed.), L1-statistical analysis and related methods (pp. 77–92). North-Holland.

34.

Saewyc

E. M.

Bauer

G. R.

Skay

C. L.

Bearinger

L. H.

Resnick

M. D.

Reis

Murphy

(2004). Measuring sexual orientation in adolescent health surveys: Evaluation of eight school-based surveys. Journal of Adolescent Health, 35(4), 345.E1–345.E15. https://doi.org/10.1016/j.jadohealth.2004.06.002

35.

Savin-Williams

Joyner

(2014). The dubious assessment of gay, lesbian, and bisexual adolescents of Add Health. Archives of Sexual Behavior, 43(3), 413–422.

36.

Schwarz

(1999). Self-reports: How the questions shape the answers. American Psychologist, 54(2), 93–105.

37.

Tabachnick

B. G.

Fidell

L. S.

(2013). Using multivariate statistics (6th ed.). Pearson.

38.

Tourangeau

Yan

(2007). Sensitive questions in surveys. Psychological Bulletin, 133, 859–883. https://doi.org/10.1037/0033-2909.133.5.859

39.

Turner

C. F.

Rogers

S. M.

Lindberg

L. D.

Pleck

J. H.

(1998). Adolescent sexual behavior, drug use, and violence: Increased reporting with computer survey technology. Science, 280(5365), 867–873.

40.

Upton

Cook

(1996). Understanding statistics (p. 55). Oxford University Press.

41.

U.S. Census Bureau. (2016). https://www.census.gov/quickfacts/fact/table/KY#viewtop

42.

Zwillinger

Kokoska

(2000). CRC standard probability and statistics tables and formulae (p. 15). CRC Press.