Obtaining Information about Learners' Thinking Strategies from Wrong Answers on Multiple-Choice Tests

Abstract

An age-dependent answer-selection sequence was inferred from earlier studies (Powell, 1968, 1970, 1976 and 1977). These earlier studies suggested, from self-report, interview data, and distractors designed from logic errors, that content-independent thinking strategies (generalized procedures used to solve problems) were employed by students to select their answers to a multiple-choice test when they did not recognize the "correct" answer. The present study replicated the results of previous ones by capturing this developmental sequence using statistical rather than clinical means. The same multiple-choice reading-comprehension test (Gorham, 1956) used in some of these previous studies (Powell, 1968, 1976 and 1977) was administered twice to 2810 students aged 8- to 19+, with a five-month elapsed time between these two administrations. Using a statistical procedure based on the multinomial distribution to interpret single cells in the resulting contingency tables, this set of data was studied for evidence of this systematically-connected age-dependent sequence. It reappeared for all anticipated events within an error factor of less than five months.

Get full access to this article

View all access options for this article.

References

Bock, R. D. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37, 29-51.

Borasi, R. (1987). Exploring mathematics through the analysis of errors. For the Learning of Mathematics, 7, 2-8.

Fisher, K. M. and Lipson, J. I. (1986). Twenty questions about student errors. Journal of Research in Science Teaching, 23, 783-803.

Fouldes, G. A. and Forbes, A. R. (1965). Advanced Progressive Matrices: Sets I and II (Manual). London, UK: H. K. Lewis.

Fuchs, C. and Kenett, R. (1980). A test for detecting outlying cells in the multinomial distribution and two-way contingency tables. Journal of the Amenican Statistical Association, 75, 395-398.

Gates, A. I. and MacGinitie, W. H. (1969). Gates-MacGinitie Reading Test. New York: Teachers College, Columbia.

Gorham, D. R. (1956). Proverbs Test. Missoula, MT: Psychological Test Specialists.

Haberman, S. J. (1973). The analysis of residuals in cross-classification tables. Biometrics, 29, 205-220.

Haney, W. and Scott, L. (1980). Talking with children about tests: A pilot study of test item ambiguity. Cambridge, MA: National Consortium on Testing; Staff Circular No. 7.

10.

Hoffmann, B. (1962). The tyranny of testing. New York: Collier.

11.

Jacobs, P. 1. and Vanderventer, M. (1968). Information from wrong responses. Research Bulletin RB-68-25. Princeton, NJ: Educational Testing Service.

12.

Kansup, W. and Hakstian, A. R. (1975). A comparison of several methods of assessing partial knowledge in multiple-choice tests; I. Journal of Educational Measurement, 12, 219-230.

13.

Pea, R. D. , Soloway, E. , and Spohrer, J. C. (1987). The buggy path to the development of programming expertise. Focus on Learning Problems in Mathematics. 9, 5-30.

14.

Powell, J. C. (1987). Quelling the invisible monsters: A commentary on classroom testing procedures. Breakthrough, 8(3-4), 36-38.

15.

Powell, J. C. (1980, April). Patterns of change in answer selection. Paper presented to the National Council on Measurement in Education, Boston, MA.

16.

Powell, J. C. (1977, August). The developmental sequence of cognition as revealed by wrong answers. Alberta Journal of Educational Research, 23, 43-51.

17.

Powell, J. C. (1976). Evidence for a phase and stage developmental sequence derived from response patterns on multiple-choice tests. Paper presented at the meeting of the American Psychological Association. Washington, D. C. (ED1333372)

18.

Powell, J. C. (1970). An exploration into the derivation of achievement information from wrong answers given to multiple-choice tests. Unpublished Ph.D. Dissertation, University of Alberta.

19.

Powell, J. C. (1968). The interpretation of wrong answers from a multiple-choice test. EDUCATIONAL AND PSYCHOLOGICAL MEASUREMENT, 28, 403-412.

20.

Powell, J. C. and Isbister, A. G. (1974). A comparison between right and wrong answers on a multiple-choice test. Educational and Psychological Measurement, 34, 499-509.

21.

Powell, J. C. and Shklov, N. (1987, June). Determining the statistical significance of individual cells in contingency tables. Paper presented to the Psychometric Society. Montreal, PQ.

22.

Resnick, L. (1984). A developmental theory of number understanding. Pittsburgh, PA: Learning research and Development Center. (ED251328).

23.

Shklov, N. and Powell, J. C. (1988, June). Distribution bases for individual-cell significance tests in contingency tables: Binomial, multinomial hypergeometric and uniform. Paper presented to the Psychometric Society. Los Angeles, A.

24.

Tatsouka, K. K. (1989). Toward an integration of item-response theory and cognitive error diagnosis. In N. Frederiksen , R. Glaser , A. Lesgold and M. G. Shafto (Eds.). Diagnostic monitoring of skill and knowledge acquisition. Hillsdale, NJ: Erlbaum.

25.

Tatsouka, K. K. (1983). A probabilistic model for diagnostic misconceptions by a pattern classification approach. Urbana. IL: University of Illinois; Computer-Based Educational Research Laboratory. (ED243969).