A new procedure for defining achievement levels on continuous scales was developed using aspects of Guttman scaling and item response theory. This procedure assigns examinees to levels of achievement when the levels are represented by separate pools of multiple-choice items. Items were assigned to levels on the basis of their content and hierarchically defined level descriptions. The resulting level response functions were well-spaced and noncrossing. This result allowed well-spaced levels of achievement to be defined by a common percent-correct standard of mastery on the level pools. Guttman patterns of mastery could be inferred from level scores. The new scoring procedure was found to have higher reliability, higher classification consistency, and lower classification error, when compared to two Guttman scoring procedures.
Get full access to this article
View all access options for this article.
References
1.
ACT Inc. (1994a). Setting achievement levels on the 1994 National Assessment of Educational Progress in Geography and U.S. History and the 1996 National Assessment of Educational Progress in Science.Iowa City IA: Author.
2.
ACT Inc. (1994b). Work Keys targets for instruction: Applied mathematics.Iowa City IA: Author.
3.
ACT Inc. (1995). Characteristics of the Work Keys assessments.Iowa City IA: Author.
4.
American Psychological Association (1985). Standards for Educational and Psychological Testing. Washington DC: Author.
5.
Andrich, D. (1985). An elaboration of Guttman scaling with Rasch models for measurement. In N. Tuma (Ed.), Sociological Methodology(pp. 33–80). San Francisco: Jossey-Bass.
6.
Berk, R. A. (1984). Selecting the index of reliability. In R. A. Berk (Ed.), A guide to criterionreferenced test construction(pp. 267–290). Baltimore: The Johns Hopkins University Press.
7.
Bock, R. D. (1997). A brief history of item response theory. Educational Measurement: Issues and Practice, 16(4), 21–33.
8.
Bock, R. D. , Thissen, D., & Zimowski, M. F. (1997). IRT estimation of domain scores. Journal of Educational Measurement, 34, 197–211.
9.
Boulton-Lewis, G. M. (1987). Recent cognitive theories applied to sequential length measuring knowledge in young children. British Journal of Educational Psychology, 57, 330–342.
10.
Cliff, N. (1983). Evaluating Guttman scales: Some old and new thoughts. In H. Wainer & S. Messick (Eds.), Principals of modern psychological measurement: A festschrift for Frederick M. Lord. Hillside NJ: Erlbaum.
11.
Edwards, A. L. (1957). Techniques of attitude scale construction.Englewood Cliffs NJ: Prentice-Hall.
12.
Forsyth, R. A. (1991). Do NAEP scales yield valid criterion-referenced interpretations?Educational and Psychological Measurement: Issues and Practice, 10(3), 3–9.
13.
Fox, J. E. , & Tipps, R. S. (1995). Young children’s development of swinging behaviors. Early Childhood Research Quarterly, 10, 491–504.
14.
Guttman, L. (1950). The basis for scalogram analysis. In S. A. Stouffer, L. Guttman, E. A.Suchman, P. A. Lazarsfeld, S. A. Star, & J. A. Clausen (Eds.), Measurement and prediction(pp. 283–301). Princeton NJ: Princeton University Press.
15.
Huynh, H. (1990). Error rates in competency testing when test retaking is permitted. Journal of Educational Statistics, 15, 39–52.
16.
Impara, J. C. & Plake, B. S. (1998). Teachers’ ability to estimate item difficulty: A test of the assumptions in the Angoff standard setting method. Journal of Educational Measurement, 35, 69–81.
17.
Katz, S. , & Akpom, C. A. (1976). A measure of primary sociobiological functions. International Journal of Health Services, 6(3), 493–507.
18.
Kolen, M. J. , & Brennan, R. L. (1995). Test equating: Methods and practices.NewYork: Springer-Verlag.
19.
Kolen, M. J. , Zeng, L., & Hanson, B. A. (1996). Conditional standard errors of measurement for scale scores using IRT. Journal of Educational Measurement, 33, 129–140.
20.
Lord, F. M. , & Wingersky, M. S. (1984). Comparison of IRT true-score and equipercentile observedscore “equatings.”Applied Psychological Measurement, 8, 453–461.
21.
Lund, M. , Foy, D., Sipprelle, C., & Strachan, A. (1984). The combat exposure scale: A systematic assessment of trauma in theVietnamWar. Journal of Clinical Psychology, 40, 1323–1328.
22.
Masters, G. N. , Adams, R., & Lokan, J. (1994). Mapping student achievement. International Journal of Educational Research, 21, 595–609.
23.
May, K. ,& Nicewander, W. A. (1994). Reliability and information functions for percentile ranks. Journal of Educational Measurement, 31, 313–325.
24.
Meijer, R. R. (1994). The number of Guttman errors as a simple and powerful person-fit statistic. Applied Psychological Measurement, 18, 311–314.
25.
Mislevy, R. J. (1998). Implications of market-basket reporting for achievement-level setting. Applied Measurement in Education, 11, 49–53.
26.
Mislevy, R. J. , & Bock, R. D. (1990). BILOG 3: Item analysis and test scoring with binary logistic models(2nd ed.). Mooresville IN: Scientific Software.
27.
National Assessment Governing Board (1990). Setting appropriate achievement levels for the National Assessment of Educational Progress: Policy framework and technical procedures.Washington DC: Author.
28.
National Center for Education Statistics (1994). The NAEP 1992 Technical Report(Report No. 23-TR20). Washington DC: Author.
29.
Pommerich, M. , & Nicewander, W. A. (in press). Estimating average domain scores. Journal of Educational Measurement.
30.
Schulz, E. M. , Kolen, M. J., & Nicewander, W. A. (1997). A study of modified Guttman and IRTbased level scoring procedures for Work Keys assessments(ACT Research Report 97–7). Iowa City IA: ACT Inc.
31.
Schulz, E. M. , Perlman, C., Rice, W. K., Jr.,& Wright, B. D. (1992). Vertically equating reading tests: An example from Chicago Public Schools. In M. Wilson (Ed.), Objective Measurement: Theory into Practice(Vol. 1, pp. 138–154). Norwood NJ: Ablex.
32.
Subkoviak, M. J. (1984). Estimating the reliability of mastery-nonmastery classifications. In R. A. Berk (Ed.), A guide to criterion-referenced test construction(pp. 267–290). Baltimore: The Johns Hopkins University Press.
33.
Wang, T. , Kolen, M. J., & Harris, D. J. (1997). Conditional standard errors, reliability, and decision consistency performance levels using polytomous IRT. In D. J. Harris (Ed.), Reliability issues with performance assessments: A collection of papers (ACT Research Report 97–3). Iowa City IA: ACT Inc.
34.
Wang, T., Kolen, M. J., & Lee, S. (1993). Assessing inter-form consistency and equivalency based on IRT parameters. Unpublished manuscript, Iowa City IA: American College Testing.
35.
Wilson, M. (1989). A comparison of deterministic and probabilistic approaches to measuring learning structures. Australian Journal of Education, 33, 127–140.
36.
Wright, B. D. , & Stone, M. H. (1979). Best test design.Chicago: MESA Press.
