Multiple dichotomous-scored items in second language testing: investigating the multiple true-false item type under norm-referenced conditions

Abstract

This study examined the multiple true-false (MTF) test format in second language testing by comparing multiple-choice (MCQ) and multiple true-false (MTF) test formats in two language areas of general English: vocabulary and reading. Two counter-balanced experimental designs - one for each language area - were examined in terms of the number of MCQ and MTF differentially responded to, MTF item dependency, reliability, and concurrent validity. The data were analysed by classical test theory (CTT) and Rasch analysis. The results showed a two- and three-fold increase in vocabulary and reading items answered, respectively. Participants responded to significantly more MTF items than MCQ, and further analysis revealed no item dependency for both language domains. Reliability increases were found in the reading tests. Item conversions did not alter the basic functioning of the MTF items, and common person equating plots demonstrated a steady relationship between MCQ and MTF person ability estimates.

Get full access to this article

View all access options for this article.

References

Albanese, M. A. and Sabers, D. L. 1978: Multiple response vs. multiple true-false scoring: A comparison of reliability and validity . Paper presented at the annual meeting of National Council on Measurement in Education, Toronto, ON.

Albanese, M. A. and Sabers, D. L. 1988: Multiple true-false items: a study of interitem correlations, scoring alternatives, and reliability estimation . Journal of Educational Measurement 25, 111-123 .

Andrews, K. M. 1989: Methods of estimating the reliability of scores from multiple true-false tests . Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco, CA.

Brown, J. D. 1988: Understanding research in second language learning. New York: Cambridge University Press .

Brown, J. D. 1996: Testing in language programs. Upper Saddle River, NJ; Prentice-Hall .

Brown, J. D. and Hudson, T. 2002: Criterion-referenced language testing. New York, NY: Cambridge University Press .

Bond, T. G. 1976/1995: BLOT-Bond’s logical operations test. Townsville, Queensland: James Cook University .

Bond, T. G. 1995: Piaget and measurement II: empirical validation of the Piagetian model . Archives de Psychologie 63, 155-185 .

Bond, T. G. and Fox, C. M. 2001: Applying the Rasch model: fundamental measurement in the human sciences. Mahwah, NJ: Lawrence Erlbaum Associates .

10.

Coombs, C. H. , Milholland, J. E. and Womer, F. B. 1956: The assessment of partial knowledge . Educational and Psychological Measurement 16, 13-37 .

11.

Cronbach, L. J. 1939: Note on the multiple true-false test exercise . Journal of Educational Psychology 30, 628-631 .

12.

Cronbach, L. J. 1941: An experimental comparison of the multiple true-false and multiple-choice tests . Journal of Educational Psychology 32, 533-543 .

13.

Frisbie, D. A. 1992: The multiple true-false item format: A status review . Educational Measurement: Issues and Practice 11, 21-26 .

14.

Frisbie, D. A. and Becker, D. F. 1991: An analysis of textbook advice about true-false tests . Applied Measurement in Education 4, 67-83 .

15.

Frisbie, D. A. and Druva, C. A. 1986: Estimating the reliability of multiple true-false tests . Journal of Educational Measurement 23, 99-105 .

16.

Frisbie, D. A. and Ory, J. C. 1980: Alternatives to the complex multiple choice item: an experimental comparison. Research Report 370. Office of Instructional Resources, University of Illinois at Urban-Champain, IL .

17.

Frisbie, D. A. and Sweeney, D. C. 1982: The relative merits of multiple true-false tests . Journal of Educational Measurement 19, 29-35 .

18.

Gross, L. J. 1982: Scoring multiple true/false tests . Evaluation and the Health Professions 5, 459-468 .

19.

Guilford, J. P. and Fruchter, B. 1978: Fundamental statistics in psychology and education. 6th edition. New York: McGraw-Hill .

20.

Haladyna, T. M. 1999: Developing and validating multiple-choice test items. 2nd edition. Mahwah, NJ: Lawrence Erlbaum .

21.

Hill, G. C. and Woods, G. T. 1974: Multiple true-false questions . Education in Chemistry 11, 86-87 .

22.

Hsu, T.-C. , Moss, P. A. and Khampalikit, C. 1984: The merits of multiple-answer items as evaluated by using six scoring formulas . Journal of Experimental Education 52, 152-158 .

23.

Kreiter, C. D. and Frisbie, D. A. 1989: Effectiveness of multiple true-false items . Applied Measurement in Education 2, 207-216 .

24.

Linacre, J. M. 1999: Explorations into local independence with T-Rasch Ponocny, I., Ponocny-Seliger, E . Rasch Measurement Transactions 13, 710-710 .

25.

McNamara, T. F. 1996: Measuring second language performance. New York: Longman .

26.

Muchinsky, P. M. 1996: The correction for attenuation . Educational and Psychological Measurement, 65-75 .

27.

Muo-Ying, L. 1992: The impact of alternative scoring methods for multiple true-false items. Unpublished doctoral dissertation, Stanford University, Stanford, CA.

28.

Ponocny, I. and Ponocny-Seliger, E. 1999: T-Rasch 1.0. Groningen: ProGAMMA .

29.

Scheidemann, N. V. 1933: Multiplying the possibilities of the multiple choice form of objective question . Journal of Applied Psychology 17, 337-340 .

30.

Suen, H. K. 1990: Principles of test theories. Hillsdale, NJ: Lawrence Erlbaum Associates .

31.

Tabachnick, C. H. and Fidell, S. 1996: Using multivariate statistics. 3rd edition. Harper Collins College Publishers: New York .

32.

Thorndike, E. L. and Lorge, I. 1944: The teacher’s word book of 30,000 words, Columbia University: Teachers College .

33.

Wright, B. D. 1998: Estimating measures for extreme scores . Rasch Measurement Transactions 12, 632-633 .

34.

Wright, B. D. and Stone, M. H. 1979: Best test design: Rasch measurement. Chicago, IL: Mesa Press .