The Relationship of Expert-System Scored Constrained Free-Response Items to Multiple-Choice and Open-Ended Items

Abstract

This study examined the relationship of an expert- system scored constrained free-response item (requir ing the student to debug a faulty computer program) to two other item types: (1) multiple-choice and (2) free- response (requiring production of a program). Confir matory factor analysis was used to test the fit of a three-factor model to these data and to compare the fit of the model to three alternatives. These models were fit using two random-half samples, one given a faulty program containing one bug and the other a program with three bugs. A single-factor model best fit the data for the sample taking the one-bug constrained free re sponse and a two-factor model fit the data somewhat better for the second sample. In addition, the factor intercorrelations showed this item type to be highly re lated to both the free-response and multiple-choice measures.

Keywords

Index terms: artificial intelligence con structed-response items expert-system scoring free- response items open-ended items.

Get full access to this article

View all access options for this article.

References

Ackerman, T.A. , & Smith, P.L. (1988). A comparison of the information provided by essay, multiple-choice, and free-response writing tests. Applied Psychological Measurement, 12, 117-128.

Bennett, R.E. (in press). Toward intelligent assessment: An integration of constructed-response testing, artificial intelligence, and model-based measurement. In N. Frederiksen , R. J. Mislevy , & I. Bejar (Eds.), Test theory for a new generation of tests. Hillsdale NJ : Erlbaum.

Birenbaum, M. , & Tatsuoka, K.K. (1987). Open-ended versus multiple-choice response formats—It does make a difference for diagnostic purposes. Applied Psychological Measurement, 11, 385-395.

Bleistein, C. , Maneckshana, B. , & McLean, D. (1988). Test analysis: College Board Advanced Placement Examination Computer Science 3JBP (SR-88-63. Princeton NJ: Educational Testing Service.

Braun, H.I. (1988). Understanding scoring reliability: Experiments in calibrating essay readers. Journal of Educational Statistics , 13, 1-18.

Braun, H.I. , Bennett, R.E. , Frye, D. , & Soloway, E. (in press). Scoring constructed-responses using expert systems. Journal of Educational Measurement.

College Board. (1988). Advanced Placement course description : Computer science. New York: Author.

Johnson, W.L. , & Soloway, E. (1985). PROUST: An automatic debugger for Pascal programs . Byte, 10(4), 179-190.

Jöreskog, K. , & Sörbom, D. (1986). PRELIS: A program for multivariate data screening and data summarization. Mooresville IN: Scientific Software, Inc.

10.

Jöreskog, K. , & Sörbom, D. (1988). LISREL 7: A guide to the program and applications . Chicago: SPSS, Inc.

11.

Loehlin, J.C. (1987). Latent variable models. Hillsdale NJ: Erlbaum.

12.

Lord, F.M. (1980). Applications of item response theory to practical testing problems. Hillsdale NJ: Erlbaum.

13.

Marsh, H.W. , & Hocevar, D. (1985). Application of confirmatory factor analysis to the study of self-concept : First and higher order factor models and their invariance across groups. Psychological Bulletin, 97, 562-582.

14.

Mazzeo, J. , & Bleistein, C. (1986). Test analysis: College Board Advanced Placement Examination Computer Science 3IBP (SR-86-105. Princeton NJ: Educational Testing Service.

15.

Mazzeo, J. , & Flesher, R. (1985). Test analysis: College Board Advanced Placement Examination Computer Science 3HBP (SR-85-180. Princeton NJ: Educational Testing Service.

16.

Sobel, M.E. , & Bohrnstedt, G.W. (1985). Use of null models in evaluating the fit of covariance structure models. In N. B. Tuma (Ed.), Sociological methodology (pp. 152-178). San Francisco CA: Jossey-Bass.

17.

Spohrer, J.C. (1989). MARCEL: A generate-test-and-debug (GTD) impasse/repair model of student programmers (CSD/RR No. 687. New Haven CT: Yale University, Department of Computer Science .

18.

Sternberg, R.J. (1980). Factor theories of intelligence are all right almost. Educational Researcher, 9, 6-18.

19.

Traub, R.E. , & Fisher, C.W. (1977). On the equivalence of constructed-response and multiple-choice tests. Applied Psychological Measurement, 1, 355-369.

20.

Tucker, L.R. , & Lewis, C. (1973). A reliability coefficient for maximum likelihood factor analysis. Psychometrika, 38, 1-10.

21.

Ward, W.C. (1982). A comparison of free-response and multiple-choice forms of verbal aptitude tests. Applied Psychological Measurement , 6, 1-11.

22.

Ward, W.C. , Frederiksen, N. , & Carlson, S.B. (1980). Construct validity of free-response and machine-scorable forms of a test. Journal of Educational Measurement, 17, 11-29.