Sage Journals: Discover world-class research

Abstract

A true–false exam can be viewed as being a signal detection task—the task is to detect whether or not an item is true (signal) or false (noise). In terms of signal detection theory (SDT), examinees can be viewed as performing the task by comparing the perceived plausibility of an item (a perceptual component) to a threshold that delineates true from false (a decision component). The resulting model is distinct yet is related to item response theory (IRT) models and grade of membership models, with the difference that SDT explicitly recognizes the role of examinees’ perceptions in determining their response to an item. SDT also views IRT concepts such as “difficulty” and “guessing” in a different light, in that both are viewed as reflecting the same aspect—item bias. An application to a true–false algebra exam is presented and the various models are compared.

Keywords

item response models signal detection theory grade of membership item difficulty item guessing item bias true–false exams

Get full access to this article

View all access options for this article.

References

Baker

F. B.

(1992). Item response theory: Parameter estimation techniques. New York, NY: Marcel Dekker.

Birnbaum

(1968). Some latent trait models. In Lord

F. M.

Novick

M. R.

(Eds.), Statistical theories of mental test scores (pp. 397-479). Reading, MA: Addison-Wesley.

de Ayala

R. J

. (2009). The theory and practice of item response theory. New York, NY: Guilford Press.

DeCarlo

L. T.

(1998). Signal detection theory and generalized linear models. Psychological Methods, 3, 186-205.

DeCarlo

L. T.

(2010). On the statistical and theoretical basis of signal detection theory and extensions: Unequal variance, random coefficient, and mixture models. Journal of Mathematical Psychology, 54, 304-313.

DeCarlo

L. T.

(2012). On a signal detection approach to m-alternative forced choice with bias, with maximum likelihood and Bayesian approaches to estimation. Journal of Mathematical Psychology, 56, 196-207.

Embretson

S. E.

Reise

S. P.

(2000). Item response theory for psychologists. Mahwah, NJ: Lawrence Erlbaum.

Erosheva

E. A.

(2002). Grade of membership and latent structure models with application to disability survey data (Doctoral dissertation), Carnegie Mellon University, Pittsburgh, PA, USA.

Erosheva

E. A.

(2005). Comparing latent structures of the grade of membership, Rasch, and latent class models. Psychometrika, 70, 619-628.

10.

Estes

W. K.

(2002). Traps in the route to models of memory and decision. Psychonomic Bulletin & Review, 9, 3-25.

11.

Fechner

(1860/1966). Elements of psychophysics. New York, NY: Holt, Rinehart and Winston.

12.

Gelman

Carlin

J. B.

Stern

H. S.

Dunson

D. B.

Vehtari

Rubin

D. B.

(2014). Bayesian data analysis (3rd ed.). New York, NY: Chapman & Hall.

13.

Goldstein

Wood

(1989). Five decades of item response modelling. British Journal of Mathematical and Statistical Psychology, 42, 139-167.

14.

Green

D. M.

Swets

J. A.

(1988). Signal detection theory and psychophysics. Los Altos, CA: Peninsula.

15.

Haberman

S. J.

(1995). Review of Statistical applications using fuzzy sets. Journal of the American Statistical Association, 90, 1131-1133.

16.

Hambleton

R. K.

Swaminathan

(1985). Item response theory: Principles and applications. Boston, MA: Kluwer-Nijhoff.

17.

Hambleton

R. R.

Swaminathan

Rogers

H. J.

(1991). Fundamentals of item response theory. Newbury Park, CA: SAGE.

18.

Lord

F. M.

(1965). A strong true-score theory, with applications. Psychometrika, 30, 239-270.

19.

Lord

F. M.

(1980). Applications of item response theory to practical testing problems. Mahwah, NJ: Lawrence Erlbaum.

20.

Lord

F. M.

Novick

M. R.

(1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.

21.

Macmillan

N. A.

Creelman

C. D.

(2005). Detection theory: A user’s guide (2nd ed.). Mahwah, NJ: Lawrence Erlbaum.

22.

Maris

Bechger

(2009). On interpreting the model parameters for the three parameter logistic model. Measurement, 7, 75-88.

23.

Penfield

R. D.

de la Torre

(2008, April). A new response model for multiple-choice items. Paper presented at the Annual Meeting of the National Council on Measurement in Education, New York, NY.

24.

Robitzsch

(2018). Package ‘sirt’: Supplementary item response theory models (Version 2.6-9). Retrieved from https://cran.r-project.org/web/packages/sirt/

25.

San Martín

del Pino

De Boeck

. (2006). IRT models for ability-based guessing. Applied Psychological Measurement, 30, 183-203.

26.

San Martin

Gonzalez

Tuerlinckx

. (2015). On the unidentifiability of the fixed-effects 3PL model. Psychometrika, 80, 450-467.

27.

Sinharay

(2006). Bayesian item fit analysis for unidimensional item response theory models. British Journal of Mathematical and Statistical Psychology, 59, 429-449.

28.

Sinharay

Johnson

M. S.

Stern

H. S.

(2006). Posterior predictive assessment of item response theory models. Applied Psychological Measurement, 30, 298-321.

29.

Tanner

M. A.

(1996). Tools for statistical inference: Methods for the exploration of posterior distributions and likelihood functions (3rd ed.). New York, NY: Springer.

30.

Thurstone

L. L.

(1927). A law of comparative judgment. Psychological Review, 34, 273-286.

31.

von Davier

. (2009). Is there need for the 3PL model? Guess what? Measurement, 7, 110-114.

32.

Wickens

T. D.

(2002). Elementary signal detection theory. New York, NY: Oxford University Press.

33.

(2016). A note on the identifiability of fixed-effect 3PL models. Psychometrika, 81, 1093-1097.

Supplementary Material

Please find the following supplemental material available below.

For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.

For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.

0.00 MB

0.60 MB

An Item Response Model for True–False Exams Based on Signal Detection Theory

Abstract

Keywords

Get full access to this article

References

Supplementary Material