Sage Journals: Discover world-class research

Abstract

In the context of high-stakes tests, test takers who do not have enough time to complete a test rush toward the end and may engage in speeded behavior when tests do not penalize guessing. Using mathematical derivations and simulations, previous research showed that random guessing responses should attenuate interitem correlations, and therefore, decrease estimates of reliability. Meanwhile, other researchers showed that random guessing could in fact inflate reliability estimates using real data. We provide analytical derivations on how speededness could affect correlations between two dichotomous items in multiple ways, depending on the manifestation and prevalence of test speededness. Furthermore, we provide two simulation studies that evaluate the magnitude of impact of test speededness on interitem correlations and Cronbach’s alpha. We found that the impact of test speededness can vary between item pairs and that it depends on the manifestation of test speededness and item level characteristics. Furthermore, speeded responses will, in general, attenuate or not affect reliability estimates, depending on the prevalence of such responses and conceptual interpretation of speeded responses. Implications of the findings are discussed.

Keywords

speeded responses random responses careless responses reliability Cronbach’s alpha

Get full access to this article

View all access options for this article.

References

Attali

(2005). Reliability of speeded number-right multiple-choice tests. Applied Psychological Measurement, 29, 357-368. doi:10.1177/0146621605276676

Birnbaum

(1968). Some latent trait models and their use in inferring an examinee’s ability. In Lord

F. M.

Novick

M. R.

(Eds.), Statistical Theories of Mental Test Scores (pp. 397-479). Reading, MA: Addison-Wesley.

Bolt

D. M.

Cohen

A. S.

Wollack

J. A.

(2002). Item parameter estimation under conditions of test speededness: Application of a mixture Rasch Model with ordinal constraints. Journal of Educational Measurement, 39, 331-348. doi:10.1111/j.1745-3984.2002.tb01146.x

Clark

M. E.

Gironda

R. J.

Young

R. W.

(2003). Detection of back random responding: Effectiveness of MMPI-2 and Personality Assessment Inventory validity indices. Psychological Assessment, 15, 223-234. doi:10.1037/1040-3590.15.2.223

Crocker

Algina

(1986). Introduction to classical and modern test theory. In Crocker

Algina

(Eds.), Introduction to classical and modern test theory (p. 145). Fort Worth, TX: Harcourt Brace Jovanovich.

Flinn

Braham

Das Nair

(2015). How reliable are case formulations? A systematic literature review. British Journal of Clinical Psychology, 54, 266-290. doi:10.1111/bjc.12073

Huang

J. L.

Curran

P. G.

Keeney

Poposki

E. M.

DeShon

R. P.

(2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27, 99-114. doi: 10.1007/s10869-011-9231-8

Lord

Novick

(1968). Statistical Theories of Mental Test Scores. Reading, MA: Addison-Wesley.

Meade

A. W.

Craig

S. B.

(2012). Identifying careless responses in survey data. Psychological Methods, 17, 437-455. doi:10.1037/a0028085

10.

Niessen

A. S. M.

Meijer

R. R.

Tendeiro

J. N.

(2016). Detecting careless respondents in web-based questionnaires: Which method to use? Journal of Research in Personality, 63, 1-11. doi:10.1016/j.jrp.2016.04.010

11.

Rasch

(1960). Probabilistic models for some intelligence and attainment tests. Chicago, IL: The University of Chicago Press.

12.

Shao

Cheng

(2016). Detection of test speededness using change-point analysis. Psychometrika, 81, 1118-1141. doi:10.1007/s11336-015-9476-7

13.

Sinharay

(2016). The choice of the ability estimate with asymptotically correct standardized person-fit statistics. British Journal of Mathematical and Statistical Psychology, 69, 175-193. doi:10.1111/bmsp.12067

14.

Wang

Shang

(2018). A two-stage approach to differentiating normal and aberrant behavior in computer based testing. Psychometrika, 83, 223-254. doi:10.1007/s11336-016-9525-x

15.

Wise

S. L.

DeMars

C. E.

(2009). A clarification of the effects of rapid guessing on coefficient α: A note on Attali’s reliability of speeded number-right multiple-choice tests. Applied Psychological Measurement, 33, 488-490. doi:10.1177/0146621607304655

16.

Yamamoto

(1995). Estimating the effects of test length and test time on parameter estimation using the HYBRID model (ETS RR-95-02). Princeton, NJ: Educational Testing Service.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.58 MB