Sage Journals: Discover world-class research

Abstract

Logical Test Design (LTD) is a technique for developing test items that evaluate procedural learning. This research investigated the ability of LTD to predict student performance in reading Roman numerals. The study (a) compared item structural properties identified by LTD with item size and expert judgment, in their ability to predict item difficulty, and (b) investigated the consistency of students' performance on item pairs varying on their LTD structural similarity. Two hundred and eleven sixth-grade students were tested on Roman numeral items varying on LTD-related and LTD-non-related variables. An LTD-related variable, item iterativity, the number of iterations through the algorithm for decoding a Roman numeral, was found to be the best predictor of item difficulty. Furthermore, item association, the proportion of students that consistently performed correctly or incorrectly on any two items, was greater for those items identified by LTD as structurally equivalent. However, that expert judgment, almost independently of iterativity, predicted difficulty about as well, suggested a cognitive component not captured by the model. Strengths and limitations of LTD for test development are discussed.

Get full access to this article

View all access options for this article.

References

Anderson, J. R. (1985). Cognitive psychology and its implications. New York: W. H. Freeman.

Ashcraft, M. H. (1989). Human memory and cognition. Glenview, Illinois: Scott, Foresman.

Bejar, I. I. (1985). Speculations on the future of test design. In S. E. Embretson (Ed.), Test design: Developments in psychology -and psychometrics. New York: Academic Press.

Bormouth, J. R. (1970). On the theory of achievement test items. Chicago, Illinois: University of Chicago Press.

Conoley, J. 0. and O'Neil, H. F., Jr. (1979). A primer for developing test items. In H. F. O'Neil, Jr. (Ed.), Procedures for instructional systems development. New York: Academic Press.

Ebel, R. (1972). Essentials of educational measurement. Englewood Cliffs, New Jersey: Prentice Hall.

Embretson, S. E. (Ed.) (1985a). Test design: Developments in psychology and psychometrics. New York: Academic Press.

Embretson, S. E. (1985b). Introduction to the problem of test design. In S. E. Embretson (Ed.), Test design: Developments in psychology and psychometrics. New York: Academic Press.

Farina, M. V. (1970). Flowcharting. Englewood Cliffs, New Jersey: Prentice-Hall.

10.

Gibbons, A. (1985). Algorithmic graph theory. Cambridge: Cambridge University Press.

11.

Gronlund, N. E. (1971). Measurement and evaluation in teaching. New York: Macmillan.

12.

Gronlund, N. E. (1988). How to construct achievement tests. Englewood Cliffs, New Jersey: Prentice-Hall.

13.

Guttman, L. (1969). Integration of test design and analysis. In Proceedings of the 1969 invitational conference on testing problems. Princeton, New Jersey: Educational Testing Service.

14.

Harary, F. (1969). Graph theory. Reading, Massachusetts: Addison-Wesley.

15.

Hively, W. (1974). Introduction to domain referenced testing. Educational Technology, 14, 5-9.

16.

Hively, W. , Patterson, H. L. , and Page, S. H. (1968). A "universe-defined" system of arithmetic achievement tests. Journal of Educational Measurement, 5(4), 275-290.

17.

Jonassen, D. H. , Hannum, W. H. , and Tessmer, M. (1989). Hand-book of task analysis procedures. New York: Praeger.

18.

Lord, F. M. and Novick, M. R. (1968). Statistical theories of mental test scores. Reading, Massachusetts: Addison-Wesley.

19.

Martin, J. and McClure, C. (1985). Diagramming techniques for analysts and programmers. Englewood Cliffs, New Jersey: Prentice-Hall.

20.

McHugh, J. A. (1990). Algorithmic graph theory. Englewood Cliffs, New Jersey: Prentice Hall.

21.

Millman, J. (1980). Computer-based item generation. In R. A. Berk (Ed.), Criterion-referenced measurement: the state of the art. Baltimore: The Johns Hopkins University Press.

22.

Nunally, J. C. (1964). Educational measurement and evaluation. New York: McGraw-Hill, 1964.

23.

Nunally, J. C. (1967). Psychometric theory. New York: McGraw-Hill.

24.

Osterlind, S. J. (1989). Constructing test items. Boston: Kluwer Academic Publisher.

25.

Payne, D. A. (1968). The specification and measurement of learning outcomes. Lexington, Massachusetts: Xerox.

26.

Priestley, M. (1982). Performance assessment in education and training: altemative techniques. Englewood Cliffs, New Jersey: Education Technology Publications.

27.

Roid, G. H. and Haladyna, T. M. (1978). A comparison of objective-based and modified-Bormouth item writing techniques. Educational and Psychological Measurement, 35, 19-28.

28.

Roid, G. H. and Haladyna, T. M. (1982). A technology for test-item writing. New York: Academic Press.

29.

Scandura, J. (1973). Structural learning, Vol. 1: Theory and research. New York: Gordon Breach and Associates.

30.

Schriber, T. J. (1969). Fundamentals of flowcharting. New York: Wiley.

31.

Solano-Flores, G. (1989). Estimación empÍrica de la capacidad predictiva de la t6cnica de diseño lógico de exámenes. Unpublished Master's dissertation. Mexico City, Mexico: Universidad Nacional Autónoma de Mexico.

32.

Solano-Flores, G. (1991). Diseño lógico de exámenes. Mexico: Trillas.

33.

Stern, N. B. (1975). Flowcharting: A tool for understanding computer logic. New York: Wiley.

34.

Sternberg, S. (1969). The discovery of processing stages: extensions of Donder's method. In W. G. Koster (Ed.), Attention and performance II. Acta Psychologica, 30, 276-315.

35.

Thorndike, R. L. and Hagen, E. (1969). Measurement and evaluation in psychology and education. New York: Wiley.

36.

Trakhtenbrot, B. A. (1963). Algorithms and automatic computing machines. Chicago: University of Chicago Press.

37.

Wheatley, D. M. and Unwin, A. W. (1972). The algorithm writer's guide. London: Longman.

38.

Wirth, N. (1976). Algorithms + data structures = programs. Engle-wood Cliffs, New Jersey: Prentice-Hall.

39.

Wood, R. (1977). Multiple choice: a state of the art report. Evaluation in Education, 1, 191-280.

Item Structural Properties as Predictors of Item Difficulty and Item Association

Abstract

Get full access to this article

References