Evaluating the Bookmark Judgments of Standard-Setting Panelists

Abstract

The purpose of this study is to describe a new approach for evaluating the judgments of standard-setting panelists within the context of the bookmark procedure. The bookmark procedure is widely used for setting performance standards on high-stakes assessments. A many-faceted Rasch (MFR) model is proposed for evaluating the bookmark judgments of the panelists, and its use illustrated with standard-setting judgments from the Michigan Educational Assessment Program (MEAP). Panelists set three performance standards to create four performance levels (Apprentice, Basic, Met, and Exceeded). The content area used to illustrate the model is mathematics in Grade 3. The analyses suggest that the MFR model provides a promising framework for examining bookmark judgments.

Keywords

standard setting bookmark method Rasch measurement theory

Get full access to this article

View all access options for this article.

References

Brennan

R. L.

(2001). Generalizability theory. New York, NY: Springer.

Cizek

G. J.

(1996). Standard setting guidelines. Educational Measurement: Issues and Practice, 15, 12-21.

Cizek

G. J.

Bunch

M. B.

(2007). Standard setting: A guide to establishing and evaluating performance standards on tests. Thousand Oaks, CA: Sage.

Engelhard

(2009). Evaluating the judgments of standard-setting panelists using Rasch measurement theory. In Smith

E. V.

Jr. Stone

G. E.

(Eds.), Criterion referenced testing: Practice analysis to score reporting using Rasch measurement models (pp. 312-346). Maple Grove, MN: JAM Press.

Hambleton

R. C.

Pitoniak

M. J.

(2006). Setting performance standards. In Brennan

(Ed.), Educational measurement (4th ed., pp. 433-470). Westport, CT: Praeger.

Heldsinger

(2006). Accounting for unit of scale in standard setting methodologies (Unpublished doctoral dissertation). Murdoch University, Perth, Western Australia.

Jaeger

R. M.

(1988). Use and effect of caution indices in detecting aberrant patterns of standard-setting judgments. Applied Measurement in Education, 1, 17-31.

Kane

(1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64, 425-461.

Karabatsos

(2000). A critique of Rasch residual fit statistics. Journal of Applied Measurement, 1, 152-176.

10.

Linacre

J. M.

(2007). A user’s guide to FACETS: Rasch-model computer programs. Retrieved from www.winsteps.com

11.

Masters

G. N.

Wright

B. D.

(1997). The partial credit model. In van der Linden

W. J.

Hambleton

R. K.

(Eds.), Handbook of modern item response theory. New York, NY: Springer.

12.

Michigan Department of Education. (2006). 2006 MEAP Standard Setting: Mathematics, Reading, Writing, Science, and Social Studies. Lansing, MI: Author.

13.

Mitzel

H. C.

Lewis

D. M.

Patz

R. J.

Green

D. R.

(2001). The bookmark procedure: Psychological perspectives. In Cizek

G. J.

(Ed.), Setting performance standards: Concepts, methods and perspectives (pp. 249-281). Mahwah, NJ: Lawrence Erlbaum.

14.

Plake

B. S.

(2007, April). Standard setters: Stand up and take a stand! 2006 Career Award address presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL.

15.

Saal

F. E.

Downey

R. G.

Lahey

M. A.

(1980). Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88, 413-428.

16.

van der Linden

W. J.

(1982). A latent trait method for determining intrajudge consistency in the Angoff and Nedelsky techniques of standard setting. Journal of Educational Measurement, 19, 295-308.

17.

Yin

Sconing

(2007). Evaluating standard errors of cut scores for item rating and mapmark procedures: A generalizability theory approach. Educational and Psychological Measurement, 68, 25-41.