A New Standard-Setting Method for Performance Assessments: The Dominant Profile Judgment Method and Some Field-Test Results

Abstract

Traditional standard-setting methods are not well suited for applications with polytomously scored performance assessments. The present article presents a standard-setting method-the dominant profile judgment (DPJ) method-designed for use with profiles of polytomous scores on exercises in a performance-based assessment. The method is direct, in that it guides standard-setfing panelists in the articulation of their standard-setting policies. Further, it allows complex policy statements that could incorporate compensatory and/or conjunctive components. A detailed description of the method is provided, Results of an application of this standard-setting method are presented. Recommendations for improvements in the method are discussed.

Get full access to this article

View all access options for this article.

References

Angoff, W. H. (1971). Scales, norms, and equivalent scores. In R. L. Thomdike (Ed.), Educational measurement (2nd ed., pp. 508-600). Washington, DC: American Council on Education.

Hambleton, R. K. , & Plake, B. S. (1995). Using an extended Angoff procedure to set standards on complex performance assessments. Applied Measurement in Education, 8, 41-56.

Hambleton, R. K. , & Slater, S. C. (1995, April). Reliability issues and methods for credentialing examinations. Paper presented at the annual meeting of the American Educational Research Association, San Francisco.

Jaeger, R. M. (1982). An iterative structured judgment process for establishing standards on competency tests: Theory and application. Educational Evaluation and Policy Analysis, 4, 461-475.

Jaeger, R. M. (1995). Setting performance standards through two-stage judgmental policy capturing. Applied Measurement in Education, 8, 15-40.

Jaeger, R. M. , Hambleton, R. K. , & Plake, B. S. (1995, April). Eliciting configural performance standards through a sequenced application of complementary methods. Paper presented at the annual meetings of the American Educational Research Association and the National Council on Measurement in Education, San Francisco.

Nedelsky, L. (1954). Absolute grading standards for objective tests. Educational and Psychological Measurement, 14, 2-19.

Putnam, S. E. , Pence, P. , & Jaeger, R. M. (1995). A multi-stage dominant profile method for setting standards on complex performance assessments. Applied Measurement in Education, 8, 57-84.

Sireci, S. G. , & Biskin, B. H. (1993). Measurement practices in national licensing examination programs: A survey. CLEAR Exam Review, pp. 21-25.

10.

Zieky, M. (1995). Historical perspective on standard setting for large-scale assessments. In Proceedings of the NCES-NAGB Conference on Large-Scale Assessments. Washington, DC: National Center for Education Statistics.