Sage Journals: Discover world-class research

Abstract

This article describes a novel variation of the Body of Work method that uses construct maps to overcome problems of transparency, rater inconsistency, and scores gaps commonly occurring with the Body of Work method. The Body of Work method with construct maps was implemented to set cut-scores for two separate K-12 assessment programs in a large Midwestern state and was compared with a previous standard setting for one of the K-12 assessments programs that used the traditional Body of Work method. Data from the standard settings were used to investigate the procedural, internal, and external validity of the Body of Work method with construct maps. Results suggested that the method had strong procedural, internal, and external validity evidence to support its application.

Keywords

standard setting body of work construct maps validity evidence large-scale assessment

Get full access to this article

View all access options for this article.

References

Angoff

W. H.

(1971). Scales, norms, and equivalent scores. In Thorndike

R. L.

(Ed.), Educational measurement (2nd ed., pp. 508-597). Washington, DC: American Council on Education.

Cizek

G. J.

Bunch

M. B.

(2007). Standard setting: A guide to establishing and evaluating performance standards on tests. Thousand Oaks, CA: Sage.

Hambelton

R. K.

Pitoniak

M. J.

(2006). Setting performance standards. In Brennan

R. L.

(Ed.), Educational measurement (4th ed.; pp. 433-470). Washington, DC: American Council on Education.

Kingston

N. M.

Kahl

S. R.

Sweeney

Bay

(2001). Setting performance standards using the Body of Work method. In Cizek

G. J.

(Ed.), Standard setting: Concepts, methods, and perspectives (pp. 219-248). Mahwah, NJ: Erlbaum.

Kingston

N. M.

Tiemann

G. C.

(2012). Setting performance standards on complex assessments: The Body of Work method. In Cizek

G. J.

(Ed.), Setting performance standards: Foundations, methods, and innovations (pp. 201-224). New York, NY: Routledge.

Kane

M. T.

(1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64, 425-461.

Kane

M. T.

(2001). So much remains the same: Conception and status of validation in setting standards. In Cizek

G. J.

(Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 53-81). Mahwah, NJ: Erlbaum.

Lewis

D. M.

Mitzel

H. C.

Green

D. R.

(1996, June). Standard setting: A bookmark approach. In D. R.

Green

(Chair), IRT-based standard setting procedures utilizing behavioral anchoring. Symposium presented at the Council of Chief State School Officers National Conference on Large-Scale Assessment, Phoenix, AZ.

Luecht

R. M.

(2008, February). Assessment engineering. In symposium Assessment engineering: Moving from theory to practice. Presented at the annual meeting of the Association of Test Publishers, Dallas, TX.

10.

Livingston

S. A.

Zieky

M. J.

(1982). Passing scores: A manual for setting standards of performance on educational and occupational tests. Princeton, NJ: ETS.

11.

Mitzel

H. C.

Lewis

D. M.

Patz

R. J.

Green

D. R.

(2001). The Bookmark procedure: Psychological perspectives. In Cizek

G. J.

(Ed.), Standard setting: Concepts, methods, and perspectives (pp. 249-281). Mahwah, NJ: Erlbaum.

12.

Perie

(2008). A guide to understanding and developing performance-level descriptors. Educational Measurement: Issues and Practice, 37, 15-29.

13.

Radwan

Rogers

W. T.

(2006). A critical analyses of the Body of Work method for setting cut-scores. Alberta Journal of Educational Research, 52, 65-79.

14.

Wilson

(2005). Constructing measures: An item response modeling approach. Mahwah, NJ: Erlbaum.

15.

Wyse

A. E.

(2013). Construct maps as a foundation for standard setting. Manuscript submitted for publication.

16.

Zieky

M. J.

Perie

Livingston

S. A.

(2008). Cutscores: A manual for setting standards of performance on educational and occupational tests. Princeton, NJ: ETS.

A Body of Work Standard-Setting Method With Construct Maps

Abstract

Keywords

Get full access to this article

References