Sage Journals: Discover world-class research

Abstract

Assessment and evaluation at all levels of educational systems have become policy priorities for many countries. Two common reasons for this are student learning expectations and accountability. Although much effort has been put into the creation and refinement of content standards, standardized tests, and methods for using testing results, there has been less attention paid to the development of performance standards (proficiency levels) that greatly affect policy decision making. The present study investigates the Objective Standard Setting Model (OSS) as an improved criterion-referenced method for setting multilevel performance standards. To demonstrate how OSS can be employed for performance standard setting, our study used data from preservice teachers completing an assessment literacy test in a pre- and posttest environment. Using OSS, performance standard levels of proficient and excellent were established with clear content-related descriptions of growth in student content mastery.

Keywords

performance standards objective standard setting model assessing student learning

Get full access to this article

View all access options for this article.

References

American Federation of Teachers (AFT), National Council on Measurement in Education (NCME), & National Education Association (NEA) (1990). Standards for teacher competence in the educational assessment of students. Retrieved from https://eric.ed.gov/?id=ED323186

American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (2014). Standards for educational and psychological testing. Washington, DC: American Educational Research Association.

American Institutes for Research. (2016a). Recommending performance standards for Ohio’s State tests. Retrieved from http://oh.portal.airast.org/ocba/wp-content/uploads/OST_Standard_Setting_Technical_Report_ELA_Math.pdf

American Institutes for Research. (2016b). 2014-15 growth model for educator evaluation: Technical report. Retrieved from https://www.engageny.org/file/147081/download/2014-15-technical-report-for-growth-measures.pdf?token=4Kdm3PMf

Angoff

(1971). Scales, norms and equivalent scores. In Thorndike

R. L.

(Ed.), Educational measurement (pp. 506-600). Washington, DC: American Council on Education.

Beltyukova

Stone

Fox

(2004). Equating student satisfaction measures. Journal of Applied Measurement, 4(3), 62-69.

Bhola

D. S.

Impara

J. C.

Buckendahl

C. W.

(2005). Aligning tests with states’ content standards: Methods and issues. Educational Measurement Issues and Practice, 22(3), 21-29.

Borsboom

(2005). The concepts of validity. In Borsboom

(Ed.), Measuring the mind: Conceptual issues in contemporary psychometrics (pp. 149-172). Cambridge, UK: Cambridge University Press.

Candisky

(2016). Ohio set to lower math standards amid test-score drops. The Columbus Dispatch. Retrieved from http://www.dispatch.com/content/stories/local/2016/06/14/state-school-board-likely-to-lower-proficiency-standards-on-math-tests.html

10.

Carey

Manwaring

(2011). Growth models and accountability: A recipe for remaking ESEA. Washington, DC: Education Sector Reports

11.

Cizek

G. J.

(1993). Reactions to the National Academy of Education report, “setting performance standards for student achievement.” Washington, DC: Assessment Governing Board. Retrieved from https://eric.ed.gov/?id=ED360397

12.

Collins

(2014). Houston, we have a problem: Teachers find no value in the SAS Education Value-Added Assessment System (EVAAS(r)). Education Policy Analysis Archives, 22(98). Retrieved from https://epaa.asu.edu/ojs/article/view/1594/1388

13.

Colosi

(2005). Negatively worded questions cause respondent confusion. Paper presented at the Joint Statistical Meetings, Minneapolis, MN. http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.556.243

14.

Darling-Hammond

(2002). Standards, assessments, and educational policy: In pursuit of genuine accountability. Princeton, NJ: Educational Testing Services. Retrieved from https://www.ets.org/Media/Research/pdf/PICANG8.pdf

15.

Dauphinee

W. D.

Blackmore

D. E.

Smee

Rothman

A. I.

Reznick

(1997). Using the judgments of physician examiners in setting the standards for a national multi-center high stakes OSCE. Advances in Health Science Education, 2, 201-211.

16.

Ebel

R. L.

(1979). Essentials of educational measurement. Englewood Cliffs, NJ: Prentice-Hall.

17.

FairTest: The National Center for Fair and Open Testing. (2007). Norm-referenced achievement tests. Author. Retrieved from https://www.cga.ct.gov/ed/tfs/20150622_Minority%20Teacher%20Recruitment/20160125/MTR%20Taskforce%20Norm%20Referenced%20vs%20Criterion%20Referenced%20Tests.pdf

18.

FairTest: The National Center for Fair and Open Testing. (2012). What’s wrong with standardized tests? Author. Retrieved from https://www.fairtest.org/facts/whatwron.html

19.

Glass

G. V.

(1978). Standards and criteria. Journal of Educational Measurement, 15, 237-262.

20.

Holloway

J. H.

(2003). Using data to improve student achievement. Educational Leadership, 60(5), 74-76.

21.

Jaeger

R. M.

(1982). An iterative judgement process for establishing standards on competency tests: Theory and application. Educational Evaluation and Policy Analysis, 4, 461-475. Retrieved from https://www.jstor.org/stable/1163656?seq=1#page_scan_tab_contents

22.

Kane

(1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64, 425-461.

23.

Kenyon

D. M.

MacGregor

Ryu

J. R.

Cho

Louguit

(2006). Annual technical report for ACCESS for ELLs English language proficiency test, series 100, 2004-2005 administration. Retrieved from https://www.wida.us/get.aspx?id=142

24.

Lewis

D. M.

Mitzel

H. C.

Green

D. R.

(1996). Standard setting: A bookmark approach. In Green

D. R.

(Chair), IRT-based standard setting procedures utilizing behavioral anchoring. Symposium conducted at the Council of Chief State School Officers National Conference on Large-Scale Assessment, Phoenix, AZ.

25.

Linacre

M. J.

(2012). WINSTEPS Rasch measurement computer program. Chicago, IL: MESA Press.

26.

MacDougal

Stone

G. E.

(2015). Fortune-tellers or content specialists: Challenging the standard in medical education programs. Journal of Contemporary Medical Education, 3, 134-142.

27.

Martone

Sireci

S. G.

(2009). Evaluating alignment between curriculum, assessment, and instruction. Review of Educational Research, 79, 1332-1362.

28.

McCaffrey

D. F.

Koretz

Lockwood

J. R.

Hamilton

L. J.

(2003). Evaluating value-added models for teacher accountability. Santa Monica, CA: RAND.

29.

Mertler

C. A.

Campbell

C. S.

(2005, April). Measuring teacher’s knowledge & application of classroom assessment concepts: Development of the Assessment Literacy Inventory. Paper presented at the annual meeting of the American Educational Research Association, Monteal, Quebec, Canada. Retrieved from https://eric.ed.gov/?id=ED490355

30.

Moore

Toth

Marazano

R. J.

(2017). The essentials for standards-driven classrooms: A practical instructional model for every student to achieve rigor. West Palm Beach, FL: Learning Sciences International. Retrieved from https://www.learningsciences.com/media/catalog/product//e/s/esdc_lookinside.pdf

31.

National Academy of Education. (1993). Setting performance standards for student achievement. Washington, DC: Author.

32.

National Center for Education Statistics. (2016). Digest of education statistics: 2016. Retrieved from https://nces.ed.gov/programs/digest/d16/

33.

Nedelsky

(1954). Absolute grading standards for objective tests. Educational and Psychological Measurement, 14, 3-19.

34.

New York State Department of Education. (2014). New York state 2014-2015 growth reporting system users guide. Retrieved from https://www.engageny.org/file/12711/download/grs-users-guide.pdf

35.

Oescher

Kirby

P. C.

Paradise

L. V.

(2014). Validating state-mandated criterion-referenced achievement tests with norm-referenced test results for elementary and secondary students. The Journal of Experimental Education, 60, 141-150.

36.

Organization for Economic Cooperation and Development. (2013). Synergies for better learning: An international perspective on evaluation and assessment. Paris, France: Author. Retrieved from http://www.oecd.org/education/school/synergies-for-better-learning.htm

37.

Pibal

Cesnik

H. S.

(2011). Evaluating the quality-quantity trade-off in the selection of anchor items: A vertical scaling approach. Practical Assessment, Research and Evaluation, 16(6), 1-12. Retrieved from pareonline.net/pdf/v16n6.pdf

38.

Plake

B. S.

Melican

G. J.

Mills

C. N.

(1991). Factors influencing intrajudge consistency during standard setting. Educational Measurement: Issues and Practice, 10(2), 15-16.

39.

Prince

C. D.

Schuermann

P. J.

Guthrie

J. W.

Witham

P. J.

Milanowski

A. T.

Thorn

C. A.

(2009). The other 69 percent: Fairly rewarding the performance of teachers of non tested subjects and grades—Guide to implementation: Resources for applied practice. Washington, DC: Center for Educator Compensation Reform.

40.

Rasch

(1980). Probalistic models for some intelligence and attainment tests (With foreward and after word by B. D. Wright). Chicago, IL: University of Chicago Press. (Original work published 1960)

41.

Schmoker

Marzano

R. J.

(1999). Using standards and assessment. Educational Leadership, 56(6), 17-21.

42.

Silber

K. H.

Foshay

W. R.

(2010). Handbook for improving performance in the workplace. San Francisco, CA: Wiley.

43.

Smith

E. V.

Conrad

K. M.

Chang

Piazza

(2002). An introduction to Rasch measurement for scale development and person assessment. Journal of Nursing Measurement, 10, 189-206.

44.

Stewart

(2013). Teacher evaluation: An international perspective (2013 International Summit on the Teaching Profession Report). Retrieved from Asia Society: https://asiasociety.org/education/teacher-evaluation-international-perspective

45.

Stone

G. E.

(1995). Objective standard setting. Presented at the National meeting of the American Educational Research Association, San Francisco, CA.

46.

Stone

G. E.

(1996). The construction of meaning: Replicating objectively derived standards. Presented at the National Meeting of the American Educational Research Association, New York, NY.

47.

Stone

G. E.

(2001). Objective standard setting (or truth in advertising). Journal of Applied Measurement, 1, 187-201.

48.

Stone

G. E.

(2004). Objective standard setting. In Smith

(Ed.), Understanding Rasch measurement (Chapter 11). Mahwah, NJ: Lawrence Erlbaum Associates.

49.

Stone

G. E.

(2009). Establishing criterion measures on graded essays using objective standard-setting for judge-mediated examinations. Presented at the National meeting of the American Educational Research Association, San Diego, CA.

50.

Stone

G. E.

Koskey

K. A.

Sondergeld

T. A.

(2011). Construct definition in Angoff and objective standard-setting: Playing in a house of cards without a full deck. Educational and Psychological Measurement, 71, 942-962.

51.

Wilkerson

J. R.

Lang

W. S.

(2007). Assessing teacher competency. Thousand Oaks, CA: Corwin Press.

52.

Wright

B. D.

Stone

M. H.

(1979). Best test design. Chicago, IL: Mesa Press.

53.

Zieky

Perie

(2006). A primer on setting cut scores on tests of educational achievement. Princeton, NJ: Educational Testing Services.

Objective Standard Setting in Educational Assessment and Decision Making

Abstract

Keywords

Get full access to this article

References