A Signal Detection Model for Multiple-Choice Exams

Abstract

A model for multiple-choice exams is developed from a signal-detection perspective. A correct alternative in a multiple-choice exam can be viewed as being a signal embedded in noise (incorrect alternatives). Examinees are assumed to have perceptions of the plausibility of each alternative, and the decision process is to choose the most plausible alternative. It is also assumed that each examinee either knows or does not know each item. These assumptions together lead to a signal detection choice model for multiple-choice exams. The model can be viewed, statistically, as a mixture extension, with random mixing, of the traditional choice model, or similarly, as a grade-of-membership extension. A version of the model with extreme value distributions is developed, in which case the model simplifies to a mixture multinomial logit model with random mixing. The approach is shown to offer measures of item discrimination and difficulty, along with information about the relative plausibility of each of the alternatives. The model, parameters, and measures derived from the parameters are compared to those obtained with several commonly used item response theory models. An application of the model to an educational data set is presented.

Keywords

signal detection theory item response theory choice models grade-of-membership nominal response model item difficulty item discrimination multiple choice exams

Get full access to this article

View all access options for this article.

References

Aitchison

Shen

S. M.

(1980). Logistic-normal distributions: Some properties and uses. Biometrika, 67, 261–272.

Akaike

(1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19, 716–723.

Birnbaum

(1968). Some latent trait models. In Lord

F. M.

Novick

M. R.

(Eds.), Statistical theories of mental test scores (pp. 397–424). Addison-Wesley.

Bock

R. D.

(1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37, 29–51.

Bock

R. D.

(1997). The nominal categories model. In van der Linden

W. J.

Hambleton

R. K.

(Eds.), Handbook of modern item response theory (pp. 33–49). Springer.

Chalmers

R. P.

(2012). MIRT: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48, 1–29. https://doi.org/10.18637/jss.v048.i06

de Ayala

R. J.

(2009). The theory and practice of item response theory. Guilford Press.

DeCarlo

L. T.

(2002). Signal detection theory with finite mixture distributions: Theoretical developments with applications to recognition memory. Psychological Review, 109, 710–721.

DeCarlo

L. T.

(2010). On the statistical and theoretical basis of signal detection theory and extensions: Unequal variance, random coefficient, and mixture models. Journal of Mathematical Psychology, 54, 304–313.

10.

DeCarlo

L. T.

(2011). On the analysis of fraction subtraction data: The DINA model, classification, latent class sizes, and the Q-matrix. Applied Psychological Measurement, 35, 8–26.

11.

DeCarlo

L. T.

(2012). On a signal detection approach to m-alternative forced choice with bias, with maximum likelihood and Bayesian approaches to estimation. Journal of Mathematical Psychology, 56, 196–207.

12.

DeCarlo

L. T.

(2019). Insights from reparameterized DINA and beyond. In von Davier

Lee

Y.-S.

(Eds.), Handbook of diagnostic classification models (pp. 223–243). Springer.

13.

DeCarlo

L. T.

(2020). An item response model for true-false exams based on signal detection theory. Applied Psychological Measurement, 44, 215–229.

14.

Erosheva

E. A.

(2002). Grade of membership and latent structure models with application to disability survey data [Doctoral dissertation]. Carnegie Mellon University.

15.

Fechner

(1966). Elements of psychophysics. Holt, Rinehart and Winston. (Original work published 1860)

16.

Green

D. M.

Swets

J. A.

(1988). Signal detection theory and psychophysics. Peninsula Publishing. (Reprinted from 1966)

17.

Gumbel

E. J.

(1958). Statistics of extremes. Columbia University Press.

18.

Hensher

D. A.

Roxe

J. M.

Greene

W. H.

(2015). Applied Choice Analysis (2nd ed.). Cambridge University Press.

19.

Luce

R. D.

(1959). Individual choice behavior: A theoretical analysis. John Wiley.

20.

Luce

R. D.

Suppes

(1965). Preference, utility, and subjective probability. In Luce

R. D.

Bush

R. R.

Galanter

(Eds.), Handbook of mathematical psychology (Vol. 3, pp. 249–410). John Wiley.

21.

Macmillan

N. A.

Creelman

C. D.

(2005). Detection theory: A user’s guide (2nd ed.). Lawrence Erlbaum.

22.

McFadden

(1974). Conditional logit analysis of qualitative choice behavior. In Zarembka

(Ed.), Frontiers in econometrics (pp. 105–142). John Wiley.

23.

McFadden

(2001). Economic choices. The American Economic Review, 91, 351–378.

24.

Nering

M. L.

Ostini

(2010). Handbook of polytomous item response theory models. Routledge.

25.

Schwarz

(1978). Estimating the dimension of a model. Annals of Statistics, 6, 461–464.

26.

Swets

J. A.

Tanner

W. P.

Jr. Birdsall

T. G.

(1961). Decision processes in perception. Psychological Review, 68, 301–340.

27.

Thissen

Steinberg

(1984). A response model for multiple choice items. Psychometrika, 49, 501–519.

28.

Thissen

Steinberg

(1997). A response model for multiple-choice items. In van der Linden

W. J.

Hambleton

R. K.

(Eds.), Handbook of modern item response theory (pp. 51–65). Springer.

29.

Thurstone

L. L.

(1927). A law of comparative judgment. Psychological Review, 34, 273–286.

30.

Train

K. E.

(2009). Discrete choice methods with simulation (2nd ed.). Cambridge University Press.

31.

van der Linden

W. J.

Klein Entink

R. H.

Fox

J.-P.

(2010). IRT parameter estimation with response times as collateral information. Applied Psychological Measurement, 34, 327–347.

32.

Vermunt

J. K.

Magidson

(2016). Technical guide for LatentGOLD 5.1: Basic, advanced, and syntax. Statistical Innovations.

33.

Wickens

T. D.

(2002). Elementary signal detection theory. Oxford University Press.

34.

Yellot

J. I.

Jr. (1977). The relationship between Luce’s choice axiom, Thurstone’s theory of comparative judgment, and the double exponential distribution. Journal of Mathematical Psychology, 15, 109–144.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.63 MB