Three open-ended response types—mathematical expression (ME), generating examples (GE), and graphical modeling (GM)—are described that could broaden the conception of mathematical problem solving used in computerized admissions tests. ME presents single-best-answer problems that call for an algebraic formalism, the correct rendition of which can take an infinite number of surface forms. GE presents loosely structured problems that can have many good answers taking the form of a value, letter pattern, expression, equation, or list. GM asks the examinee to represent a given situation by plotting points on a grid; these items can have a single best answer or multiple correct answers. For the three basic types, sample items are provided, the examinee interfaces and approaches to automated scoring are described, and research results are reported. It is illustrated how ME, GE, and GM can be combined to form extended constructed-response problems, and adescription is offered of how item classes might be used as a basis for creating production-ready scoring keys.
Get full access to this article
View all access options for this article.
References
1.
Bejar, I. I. (1995). From adaptive testing to automated scoring of architectural simulations. In E. L.Mancall & P. G. Bashook (Eds.), Assessing clinical reasoning: The oral examination and alternative methods(pp. 115–130). Evanston IL: American Board of Medical Specialties.
2.
Bennett, R. E. (1993). On the meanings of constructed response. In R. E. Bennett & W. C. Ward (Eds.), Construction vs. choice in cognitive measurement: Issues in constructed response, performance testing, and portfolio assessment(pp. 1–27). Hillsdale NJ: Erlbaum.
3.
Bennett, R. E. (1998). Speculations on the future of large-scale educational testing. Princeton NJ: Policy Information Center, Educational Testing Service. Available from: http://www.ets.org/research/pic/bennett.html.
4.
Bennett, R. E. , & Bejar, I. I. (1998). Validity and automated scoring: It’s not only the scoring. Educational Measurement: Issues and Practice, 17 (4), 9–17.
5.
Bennett, R. E. , Goodman, M., Hessinger, J., Ligget, J., Marshall, G., Kahn, H., & Zack, J. (1999). Using multimedia in large-scale computer-based testing programs. Computers in Human Behavior, 15, 283–294.
6.
Bennett, R. E. , Morley, M., Quardt, D., & Rock, D.A. (2000). Graphical modeling: A new response type for measuring the qualitative component ofmathematical reasoning. Applied Measurement in Education, 13,301–320.
7.
Bennett, R. E. , Morley, M., Quardt, D., Rock, D. A., Singley, M. K., Katz, I. R., & Nhouyvanisvong, A. (1999). Psychometric and cognitive functioning of an under-determined computer-based response type for quantitative reasoning. Journal of Educational Measurement, 36, 233–252.
8.
Bennett, R. E. , & Rock, D. A. (1995). Generalizability, validity, and examinee perceptions of a computer-delivered Formulating-Hypotheses test.Journal of Educational Measurement, 32, 19–36.
9.
Bennett, R. E. , Steffen, M., Singley, M. K., Morley, M., & Jacquemin, D. (1997). Evaluating an automatically scorable, open-ended response type for measuring mathematical reasoning in computeradaptive tests. Journal of Educational Measurement, 34, 163–177.
10.
Bradlow, E. T. , Wainer, H., & Wang, X. (1999). A Bayesian random effects model for testlets. Psychometrika, 64, 153–168.
11.
Braswell, J. , & Kupin, J. (1993). Item formats for assessment in mathematics. In R. E. Bennett & W. C. Ward (Eds.), Construction versus choice in cognitive measurement(pp. 167–182). Hillsdale NJ: Erlbaum.
12.
Breland, H. M. (1999). Exploration of an automated editing task as a GRE writing measure(Research Report No. 99-9). Princeton NJ: Educational Testing Service.
13.
Bridgeman, B. (1992). A comparison of quantitative questions in open-ended and multiple-choice formats. Journal of Educational Measurement, 29, 253–271.
14.
Burstein, J. , Braden-Harder, L., Chodorow, M., Hua, S., Kaplan, B., Kukich, K., Lu, C., Nolan, J., Rock, D., & Wolff, S. (1998). Computer analysis of essay content for automated score prediction(Research Report No. 98-15). Princeton NJ: Educational Testing Service.
15.
Chung, G. K. W. K. , O’Neil, H. F., & Herl, H. E.(1999). The use of computer-based collaborative knowledge mapping to measure team processes and outcomes. Computers in Human Behavior, 15, 463–494.
16.
Clyman, S. G. , Melnick, D. E., & Clauser, B. E. (1995). Computer-based case simulations. In E. L. Mancall & P. G. Bashook (Eds.), Assessing clinical reasoning: The oral examination and alternative methods(pp. 139–149). Evanston IL: American Board of Medical Specialties.
17.
Educational Testing Service . (1993). Tests at a glance: Praxis I: Academic skills assessments. Princeton NJ: Author.
18.
Educational Testing Service . (1999). TOEFL 1999-2000: Information bulletin for computer-based testing. Princeton NJ: Author.
19.
Educational Testing Service . (2000). GRE Graduate Record Examinations 2000-2001 information and registration bulletin(Brochure). Princeton NJ: Author.
20.
Enright, M. K. , Rock, D. A., & Bennett, R. E. (1999). Improving measurement for graduate admissions. Journal of Educational Measurement, 35, 250–267.
21.
Frederiksen, N. (1984). The real test bias: Influences of testing on teaching and learning. American Psychologist, 39, 193–202.
22.
Gallagher, A. , Bennett, R. E., & Cahalan, C. (in press). Detecting construct-irrelevant variance in an open-ended computerized mathematics task. Princeton NJ: Educational Testing Service.
23.
Glaser, R. (1991). Expertise and assessment. In M. C. Wittrock & E. L. Baker (Eds.), Testing and cognition(pp. 17–30). Englewood Cliffs NJ: Prentice-Hall.
24.
Graduate Management Admission Council . (2000). GMAT Graduate Management Admission Test: GMAT information bulletin(Brochure). Princeton NJ: Author.
25.
Herl, H. E. , O’Neil, H. F., Chung, G. K. W. K., & Schacter, J. (1999). Reliability and validity of a computer-based knowledge mapping system to measure content understanding. Computers in Human Behavior, 15, 315–333.
26.
Kenney, J. F. (1997). New testing methodologies for the Architect Registration Examination. CLEAR Exam Review, 8 (2), 23–28.
27.
Linn, M. C. , & Petersen, A. C. (1985). Emergence and characterization of sex differences in spatial ability: A meta-analysis. Child Development, 56, 1479–1498.
28.
Marshall, S. (1995). Schemas in problem solving. Cambridge, England: Cambridge University Press.
29.
National Council of Teachers of Mathematics . (1989). Curriculum and evaluation standards for school mathematics(p. 207–207). Reston VA: Author.
30.
National Council of Teachers of Mathematics . (2000). Principles and standards for school mathematics. Reston VA: Author.
31.
Nhouyvanisvong, A. , Katz, I. R., & Singley, M. K. (1997, March). Toward a unified model of problem solving in well-determined and under-determined algebra word problems. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
32.
Schacter, J. , Herl, H. E., Chung, G. K. W. K., Dennis, R. A., & O’Neil, Jr., H. F. (1999). Computerbased performance assessments: A solution to the narrow measurement and reporting of problemsolving. Computers in Human Behavior, 15, 403–418.
33.
Simon, H. (1978). Information-processing theory of human problem solving. In W. K. Estes (Ed.), Handbook of learning and cognitive processes (pp. 271–295). Hillsdale NJ: Erlbaum.
