Quantifying the Distorting Effect of Rapid Guessing on Estimates of Coefficient Αlpha

Abstract

An underlying threat to the validity of reliability measures is the introduction of systematic variance in examinee scores from unintended constructs that differ from those assessed. One construct-irrelevant behavior that has gained increased attention in the literature is rapid guessing (RG), which occurs when examinees answer quickly with intentional disregard for item content. To examine the degree of distortion in coefficient alpha due to RG, this study compared alpha estimates between conditions in which simulees engaged in full solution (i.e., do not engage in RG) versus partial RG behavior. This was done by conducting a simulation study in which the percentage and ability characteristics of rapid responders as well as the percentage and pattern of RG were manipulated. After controlling for test length and difficulty, the average degree of distortion in estimates of coefficient alpha due to RG ranged from −.04 to .02 across 144 conditions. Although slight differences were noted between conditions differing in RG pattern and RG responder ability, the findings from this study suggest that estimates of coefficient alpha are largely robust to the presence of RG due to cognitive fatigue and a low perceived probability of success.

Keywords

Reliability coefficient alpha rapid guessing non-effortful responding test-taking motivation

Get full access to this article

View all access options for this article.

References

Abdelfattah

F. A.

(2007). Response latency effects on classical and item response theory parameters using different scoring procedures [Unpublished doctoral dissertation]. Ohio University.

American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (2014). Standards for educational and psychological testing (6th ed.). American Educational Research Association.

Attali

(2005). Reliability of speeded number-right multiple-choice tests. Applied Psychological Measurement, 29(5), 357–368.

Crocker

Algina

(1986). Introduction to classical and modern test theory. Holt, Rinehart and Winston.

Cronbach

L. J.

(1951). Coefficient α and the internal structure of tests. Psychometrika, 16(3), 297–334.

DeMars

C. E.

(2007). Changes in rapid-guessing behavior over a series of assessments. Educational Assessment, 12(1), 23–45.

DeMars

C. E.

Wise

S. L.

(2010). Can differential rapid-guessing behavior lead to differential item functioning? International Journal of Testing, 10(3), 207–229.

Grömping

(2006). Relative importance for linear regression in R: The package relaimpo. Journal of Statistical Software, 17(1), 1–27.

Hong

M. R.

Cheng

(2019). Clarifying the effect of test speededness. Applied Psychological Measurement, 43(8), 611–623.

10.

Hubley

A. M.

Zumbo

B. D.

(2017). Response processes in the context of validity: Setting the stage. In Zumbo

B. D.

Hubley

A. M.

(Eds.), Understanding and investigating response processes in validation research (pp. 1–12). Springer.

11.

Liu

O. L.

Rios

J. A.

Borden

(2015). The effects of motivational instruction on college students’ performance on low-stakes assessment. Educational Assessment, 20(2), 79–94.

12.

McNeish

(2018). Thanks coefficient α, we’ll take it from here. Psychological Methods, 23(3), 412–433.

13.

Mittelhaëuser

M. A.

Béguin

A. A.

Sijtsma

(2015). The effect of differential motivation on irt linking. Journal of Educational Measurement, 52(3), 339–358.

14.

Pastor

D. A.

Ong

T. Q.

Strickman

S. N.

(2019). Patterns of solution behavior across items in low-stakes assessments. Educational Assessment, 24(3), 189–212.

15.

Penk

Richter

(2017). Change in test-taking motivation and its relationship to test performance in low-stakes assessments. Educational Assessment, Evaluation and Accountability, 29(1), 55–79.

16.

Pintrich

P. R.

Schunk

D. H.

(2002). Motivation in education: Theory, research, and applications. Prentice Hall.

17.

Revelle

(2020). Package “psych” (package version 2.0.9) Retrieved from https://cran.r-project.org/web/packages/psych/psych.pdf

18.

Rios

J. A

. (2021a). Assessing the accuracy of parameter estimates in the presence of rapid guessing misclassifications. Educational and Psychological Measurement. Advanced online publication https://doi.org/10.1177/00131644211003640

19.

Rios

J. A

. (2021b). Is differential noneffortful responding associated with type I error in measurement invariance testing? Educational and Psychological Measurement, 81(5), 957–979. https://doi.org/10.1177/0013164421990429

20.

Rios

J. A.

Guo

(2020). Can culture be a salient predictor of test-taking engagement? An analysis of differential noneffortful responding on an international college-level assessment of critical thinking. Applied Measurement in Education, 33(4), 263–279.

21.

Rios

J. A.

Guo

Mao

Liu

O. L.

(2017). Evaluating the impact of noneffortful responses on aggregated scores: To filter unmotivated examinees or not? International Journal of Testing, 17(1), 74–104.

22.

Smith

J. K.

Given

L. M.

Julien

Ouellette

DeLong

(2013). Information literacy proficiency: Assessing the gap in high school students' readiness for undergraduate academic work. Library & Information Science Research, 35(2), 88–96.

23.

Soland

Jensen

Keys

T. D.

S. Z.

Wolk

(2019). Are test and academic disengagement related? Implications for measurement and practice. Educational Assessment, 24(2), 119–134.

24.

Steedle

J. T.

Hong

Cheng

(2019). The effects of inattentive responding on construct validity evidence when measuring social–emotional learning competencies. Educational Measurement: Issues and Practice, 38(2), 101–111.

25.

Sundre

D. L.

Wise

S. L.

(2003). “Motivation filtering”: An exploration of the impact of low examinee motivation on the psychometric quality of tests. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL.

26.

Wise

S. L.

(2006). An investigation of the differential effort received by items on a low-stakes computer-based test. Applied Measurement in Education, 19(2), 95–114.

27.

Wise

S. L.

(2015). Effort analysis: Individual score validation of achievement test data. Applied Measurement in Education, 28(3), 237–252.

28.

Wise

S. L.

(2017). Rapid‐guessing behavior: Its identification, interpretation, and implications. Educational Measurement: Issues and Practice, 36(4), 52–61.

29.

Wise

S. L.

DeMars

C. E.

(2005). Low examinee effort in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10(1), 1–17.

30.

Wise

S. L.

DeMars

C. E.

(2006). An application of item response time: The effort-moderated IRT model. Journal of Educational Measurement, 43(1), 19–38.

31.

Wise

S. L.

DeMars

C. E.

(2009). A clarification of the effects of rapid guessing on coefficient α: A note on Attali’s ‘‘Reliability of speeded number-right multiple-choice tests’’. Applied Psychological Measurement, 33(6), 488–490.

32.

Wise

S. L.

DeMars

C. E.

(2010). Examinee noneffort and the validity of program assessment results. Educational Assessment, 15(1), 27–41.

33.

Wise

S. L.

Kingsbury

G. G.

(2016). Modeling student test‐taking motivation in the context of an adaptive achievement test. Journal of Educational Measurement, 53(1), 86–105.

34.

Wise

S. L.

Kong

(2005). Response time effort: A new measure of examinee motivation in computer-based tests. Applied Measurement in Education, 18(2), 163–183.

35.

Wise

S. L.

Kuhfeld

M. R.

(2020). A cessation of measurement: Identifying test taker disengagement using response time. In Margolis

M. J.

Feinberg

R. A.

(Eds.), Integrating timing considerations to improve testing practices (1st ed., pp. 150–164). Routledge.

36.

Wise

S. L.

Pastor

D. A.

Kong

X. J.

(2009). Correlates of rapid-guessing behavior in low-stakes testing: Implications for test development and measurement practice. Applied Measurement in Education, 22(2), 185–205.

37.

Wise

V. L.

Wise

S. L.

Bhola

D. S.

(2006). The generalizability of motivation filtering in improving test score validity. Educational Assessment, 11(1), 65–83.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.38 MB