Test item review is a specialized type of content analysis conducted to identify and correct test item flaws early in the test development process. Test item reviewers not only examine the targeted content of a test but also remove inappropriate content and balance various types of incidental content. An automated content analysis implementation of Hiller’s verbal ambiguity scales and Laffal’s General Concept Dictionary of English was used to examine 576 multiple-choice test items before and after test item review and revision by experienced item editors. Hiller’s scales detected some problems with item clarity. Laffal’s categories detected content imbalance between test forms but not inappropriate item content.
Alexa, M. (1997). Computer-assisted text analysis methodology in the social sciences. Mannheim, Germany: ZUMA.
2.
Alexa, M., & Zuell, C. (1999). A review of software for text analysis. Mannheim, Germany: ZUMA.
3.
Bengston, D. N., & Fan, D. P. (1999). An innovative method for evaluating strategic goals in a public agency. Evaluation Review, 23(1), 77-100.
4.
Biber, D., Conrad, S., & Reppen, R. (1998). Corpus linguistics: Investigating language structure and use. Cambridge, UK: Cambridge University Press.
5.
Bott, M. M., Ford, J. M., Hennen, M. E., & Dovers, G. (1999, June 9-11). The development of an HR consultant certification process. Symposium presented at the 1999 IPMAAC Conference, Orlando, FL.
6.
Chung, G.K.W. K., & O’Neil, H. F. (1997). Methodological approaches to online scoring of essays (Technical Report No. CSE-TR-461). Los Angeles: National Center for Research on Evaluation, Standards, and Student Testing. (ERIC Document Reproduction Service No. ED 418 101)
7.
Cuilenburg, J. J., Kleinnijenhuis, J., & Ridder, J. A. (1988). Artificial intelligence and content analysis: Problems and strategies for computer text analysis. Quality and Quantity, 22, 65-97.
8.
Evans, W. (1996). Computer-supported content analysis: Trends, tools and techniques. Social Science Computer Review, 14(3), 269-279.
9.
Fan, D. P. (1988). Predictions of public opinion from the mass media: Computer content analysis and mathematical modeling. New York: Greenwood.
10.
Ford, J. M. (1999). A general automated content analysis model: Implementation notes for applied research (Working paper). Washington, DC: U.S. Office of Personnel Management.
11.
Frakes, W. B., & Baeza-Yates, R. (1992). Information retrieval: Data structures and algorithms. Upper Saddle River, NJ: Prentice Hall.
12.
Franzosi, R. (1998). Narrative as data: Linguistic and statistical tools for the quantitative study of historical events. International Review of Social History, 43, 81-104.
13.
Gottschalk, L. A. (1995). Content analysis of verbal behavior: New findings and computerized clinical applications. Mahwah, NJ: Lawrence Erlbaum.
14.
Haladyna, T. M. (1999). Developing and validating multiple choice test items (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum.
15.
Hiller, J. H. (1998). Applying computerized text measurement strategies from Project Essay Grade (PEG) to military and civilian organizational needs. Paper presented at the 1998 Annual Meeting of the American Educational Research Association, San Diego, CA.
16.
Holmes, D. I. (1994). Authorship attribution. Computers and the Humanities, 28(2), 87-106.
17.
Krippendorff, K. (1980). Content analysis: An introduction to its methodology. Beverly Hills, CA: Sage.
18.
LaDuca, A., Downing, S. M., & Henzel, T. R. (1995). Systematic item writing and test construction. In J. C. Impara (Ed.), Licensure testing: Purposes, procedures and practices (pp. 117-148). Lincoln, NE: Buros Institute of Mental Measurements.
19.
Laffal, J. (1990). A concept dictionary of English. Essex, CT: Gallery Press.
20.
Landauer, T. K., Foltz, P. W., & Laham, D. (1998). An introduction to Latent Semantic Analysis. Discourse Processes, 25(2/3), 259-284.
21.
Lebart, L., Salem, A., & Berry, L. (1998). Exploring textual data. Boston: Kluwer.
22.
Nacos, B. L., Shapiro, R. Y., Young, J. T., Fan, D. P., Kjellstrand, T., & McCaa, C. (1991). Content analysis of news reports: Comparing human coding and a computer-assisted method. Communication, 12, 111-128.
23.
Page, E. B., & Petersen, N. S. (1995). The computer moves into essay grading: Updating the ancient test. Phi Delta Kappan, 76(7), 561-565.
24.
Page, E. B., Poggio, J. P., & Keith, T. Z. (1997). Computer analysis of student essays: Finding trait differences in the student profile. Paper presented at the 1997 Annual Meeting of the American Educational Research Association, Chicago.
25.
Peladeau, N. (1998). WordStat: Content analysis module for SimStat. Montreal, Quebec, Canada: Provalis Research.
26.
Plumlee, L. B. (1980). A guide to the development of job knowledge tests: A reference kit for measurement specialists (2nd ed.) (Personnel Research Report 80-27). Washington, DC: U.S. Office of Personnel Management, Personnel Resources and Development Center.
27.
Popping, R. (2000). Computer-assisted text analysis. Thousand Oaks, CA: Sage.
28.
Potter, W. J., & Levine-Donnerstein, D. (1999). Rethinking validity and reliability in content analysis. Journal of Communication Research, 27, 258-284.
29.
Riffe, D., Lacy, S., & Fico, F. G. (1998). Analyzing media messages: Using quantitative content analysis in research. Mahwah, NJ: Lawrence Erlbaum.
30.
Roberts, C. W. (1997). Text analysis for the social sciences: Methods for drawing statistical inferences from texts and transcripts. Mahwah, NJ: Lawrence Erlbaum.
31.
Stetz, T. A., Ford, J. M., & O’Leary, B. (2000). Automated content analysis of critical incidents. Paper presented at the 15th Annual Conference of the Society for Industrial/Organizational Psychology, New Orleans, LA.
32.
Weber, R. P. (1990). Basic content analysis (2nd ed.). Thousand Oaks, CA: Sage.
33.
Weitzman, E. A., & Miles, M. B. (1995). Computer programs for qualitative data analysis: A software sourcebook. Thousand Oaks, CA: Sage.
