Results obtained from computerized vocabulary tests were compared under conditions in which item review was permitted and not permitted. Comparisons between the review and no-review conditions yielded no statistically significant differences in number correct scores, ability estimates, or measurement error, but examinees in both conditions strongly desired review opportunities. Comparisons of answers before and after review within the review condition showed that only a small percentage of answers was changed (3.63%), that more answers were changed from wrong to right than from right to wrong (by a ratio of 2.25 to 1), that a large proportion of examinees (45%) changed answers to at least some questions, and that most examinees who changed answers improved their performance by doing so (by a ratio of 2.44 to 1). The results revealed that performance gains after review were greater for examinees of higher ability and that review was desired more by examinees with higher test anxiety. The major drawback to allowing review was a 35% increase in testing time.
Get full access to this article
View all access options for this article.
References
1.
Baghi, H. , Gabrys, R., & Ferrara, S. (1991, April). Applications of computer-adaptive testing in Maryland. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
2.
Benjamin, L. , Cavell, T. A., & Shallenberger, W. R., III. (1984). Staying with initial answers on objective tests: Is it a myth?Teaching of Psychology, 11, 133-141.
3.
Bock, R. D. , & Mislevy, R. J. (1982). Adaptive EAP estimation of ability in a microcomputer environment. Applied Psychological Measurement, 37, 431-444.
4.
Crawford, C. (1928). The technique of study. Boston: Houghton Mifflin.
5.
Crocker, L. , & Benson. J. (1980). Does answer changing affect test quality?Measurement and Evaluation in Guidance, 12, 223-239.
6.
Cronbach, L. J. , & Snow, R. E. (1977). Aptitudes and instructional methods. New York: Irvington.
7.
Cummings, O. W. (1981). Impact of response changes on objective test characteristics and outcomes for junior high school students. Measurement and Evaluation in Guidance, 14, 32-37.
8.
Feldt, L. S. , Forsyth, R. A., & Alnot, S. D. (1986). Iowa Tests of Educational Development: Form X-8. Iowa City: The University of Iowa.
9.
Ferrara, S. , Frances, A., Gilmartin, D., Knott, T., Michaels, H., Pollack, J., Schuder, T., Vaeth, R., & Wise, S. (1996, April). A qualitative study of the information examinees consider during item review on a computer-adaptive test. Paper presented at the annual meeting of the National Council on Measurement in Education, New York.
10.
Gershon, R. C. , & Bergstrom, B. (1991, April). Individual differences in computer-adaptive testing: Anxiety, computer literacy and satisfaction. Paper presented at the annual meeting of the American Educational Research Association, Chicago.
11.
Legg, S. M. , & Buhr, D. C. (1992). Computerized adaptive testing with different groups. Educational Measurement: Issues and Practice, 11, 23-27.
12.
Lunz, M. E. , Bergstrom, B. A., & Wright, B. D. (1992). The effect of review on student ability and test efficiency for computerized adaptive tests. Applied Psychological Measurement, 16, 33-40.
13.
Matthews, C. O. (1929). Erroneous first impressions on objective tests. Journal of Educational Psychology, 20, 280-286.
14.
Mazzeo, J. , & Harvey, A. L. (1988). The equivalence of scores from automated and conventional educational and psychological tests: A review of the literature (College Board Rep. No. 88-8, ETS RR No. 88-21). Princeton, NJ: Educational Testing Service.
15.
McMorris, R. F. , DeMers, L. P., & Schwartz, S. P. (1987). Attitudes, behaviors, and reasons for changing responses following answer-changing instruction. Journal of Educational Measurement, 24, 131-143.
16.
McMorris, R. F. , & Weiderman, A. H. (1986). Answer changing after instruction on answer changing. Measurement and Evaluation in Counseling and Development, 18, 93-101.
17.
Moe, K. C. , & Johnson, M. F. (1986). Participants’reactions to computerized testing. East Lansing, MI: National Center for Research on Teacher Learning. (ERIC Document Reproduction Service No. ED 282 935).
18.
Mueller, D. J. , & Wasser, V. (1977). Implications of changing answers on objective test items. Journal of Educational Measurement, 14, 9-13.
19.
Schmitt, F. L. , Urry, V. W., & Gugel, J. J. (1978). Computer assisted tailored testing: Examinee reactions and evaluations. Educational and Psychological Measurement, 38, 265-273.
20.
Schwarz, S. P. , McMorris, R. F., & DeMers, L. P. (1991). Reasons for changing answers: An evaluation using personal interviews. Journal of Educational Measurement, 28, 163-171.
21.
Shatz, M. A. , & Best, J. B. (1987). Students’ reasons for changing answers on objective tests. Teaching of Psychology, 14, 241-242.
22.
Smith, M. , White, K., & Coop, R. (1979). The effect of item type on the consequences of changing answers on multiple choice tests. Journal of Educational Measurement, 16, 203-208.
23.
Spielberger, C. D. (1980). Preliminary professional manual for the Test Anxiety Inventory. Palo Alto, CA: Consulting Psychologists Press.
24.
Spray, J. A. , Ackerman, T. A., Reckase, M. D., & Carl, J. E. (1989). Effect of medium of item presentation on examinee performance and item characteristics. Journal of Educational Measurement, 26, 261-271.
25.
Stocking, M. L. (1997). Revising item responses in computerized adaptive tests: A comparison of three models. Applied Psychological Measurement, 21, 129-142.
26.
Stone, G. E. , & Lunz, M. E. (1994). The effect of review on the psychometric characteristics of computerized adaptive tests. Applied Measurement in Education, 7, 211-222.
27.
Vispoel, W. P. (1993). Computerized adaptive and fixed-item versions of the ITED Vocabulary subtest. Educational and Psychological Measurement, 53, 779-788.
28.
Vispoel, W. P. (1998). Reviewing and changing answers on computer-adaptive and self-adaptive vocabulary tests. Journal of Educational Measurement, 35, 329-346.
29.
Vispoel, W. P. , Rocklin, T. R., & Wang, T. (1994). Individual differences and test administration procedures: A comparison of fixed-item, adaptive, and self-adapted testing. Applied Measurement in Education, 53, 53-79.
30.
Vispoel, W. P. , Rocklin, T. R., Wang, T., & Bleiler, T. (1999). Can examinees use a review option to obtain positively-biased ability estimates on a computerized adaptive test?Journal of Educational Measurement, 36, 141-157.
31.
Vispoel, W. P. , Wang, T., de la Torre, R., Bleiler, T., & Dings, J. (1992, April). How review options, administration mode, and anxiety influence scores on computerized vocabulary tests. Paper presented at the meeting of the National Council on Measurement in Education, San Francisco. (ERIC Document Reproduction Service No. ED 346 161)
32.
Waddell, D. L. , & Blankenship, J. C. (1995). Answer changing: A meta-analysis of the prevalence and patterns. Journal of Continuing Education in Nursing, 25, 155-158
33.
Wainer, H. (1993). Some practical considerations when converting a linearly administered test to an adaptive format. Educational Measurement: Issues and Practice, 12, 15-20.
34.
Wang, M. , & Wingersky, M. (1992, April). Incorporating post-administration item response revision into a CAT. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.
35.
Wise, S. L. (1996, April). A critical analysis of the arguments for and against item review in computerized adaptive testing. Paper presented at the annual meeting of the National Council on Measurement in Education, New York.
