Proposed is a new method of standard setting referred to as response vector for mastery (RVM) method. Under the RVM method, the task of panelists that participate in the standard setting process does not involve conceptualization of a borderline examinee and probability judgments as it is the case with the Angoff and bookmark methods. Also, the RVM-based computation of a cut-score is not based on a single item (e.g., marked in an ordered item booklet) but, instead, on a response vector (1/0 scores) on items and their parameters calibrated in item response theory or under the recently developed D-scoring method. Illustrations with hypothetical and real-data scenarios of standard setting are provided and methodological aspects of the RVM method are discussed.
AngoffW. H. (1971). Scales, norms, and equivalent scores. In ThorndikeR. L. (Ed.), Educational measurement (2nd ed., pp. 508-600). American Council on Education.
2.
AtanasovD. V.DimitrovD. M. (2019). DELTA: A computer program for D-scoring and equating of test data (v.2.0). National Center for Assessment of the Education & Training Evaluation Commission.
3.
BaldwinP. (2018). Some problems with the analytical argument in support of RP67 in the context of the bookmark standard setting method. Applied Psychological Measurement, 43(6), 481-492. https://doi.org/10.1177/0146621618800272
4.
BaldwinP.MargolisM.ClauserB. E.MeeJ.WinwardM. (2019). The choice of response probability in bookmark standard setting: An experimental study. Educational Measurement, 39(1), 37-44. https://doi.org/10.1111/emip.12230
5.
BeretvasS. N. (2004). Comparison of bookmark difficulty locations under different item response models. Applied Psychological Measurement, 28(1), 25-47. https://doi.org/10.1177/0146621603259903
6.
BerkR. A. (1986). A consumer’s guide to setting performance standards on criterion-referenced tests. Review of Educational Research, 56(1), 137-172. https://doi.org/10.3102/00346543056001137
7.
ChangL. (1999). Judgmental item analysis of the Nedelsky and Angoff standard-setting methods. Applied Measurement in Education, 12(2), 151-165. https://doi.org/10.1207/s15324818ame1202_3
8.
CizekG. J.BunchM. B.(2007). Standard stetting: A guide to establishing and evaluating performance standards on test. Sage. https://doi.org/10.4135/9781412985918
ClauserB. E.MargolisM. J.CaseS. M. (2006). Testing for licensure and certification in the professions. In: BrennanR. L. (Ed.) Educational Measurement (4th ed., pp. 701-731). Praeger.
11.
Davis-BeckerS. L.BuckendahlC.GerrowJ. (2011). Evaluating the bookmark standard setting method: The impact of random item ordering. International Journal of Testing, 11(1), 24-37. https://doi.org/10.1080/15305058.2010.501536
12.
DimitrovD. M. (2016). An approach to scoring and equating tests with binary items: Piloting with large-scale assessments. Educational and Psychological Measurement, 76(7), 954-975. https://doi.org/10.1177/0013164416631100
13.
DimitrovD. M. (2018). The delta scoring method of tests with binary items: A note on true score estimation and equating. Educational and Psychological Measurement, 78(5), 805-825. https://doi.org/10.1177/0013164417724187
14.
DimitrovD. M. (2020). Modeling of item response functions under the D-scoring method. Educational and Psychological Measurement, 80(1), 126-144. https://doi.org/10.1177/0013164419854176
15.
DimitrovD. M.AlsadaawiA. (2019, September). A standard setting method using the D- scoring method: Procedures and application to assessment for teacher certification [Paper presentation]. 2019 Conference of the International Association for Educational Assessment. Baku, Azerbaijan.
16.
DimitrovD. M.Al-ShamraniA. (2019, March). A new approach to setting cutting scores for mastery in language testing [Paper presentation]. Second Applied Linguistics & Language Teaching International Conference and Exhibition, Dubai, UAE.
17.
DimitrovD. M.AtanasovD. V. (2021). Latent D-scoring modeling: Estimation of item and person parameters. Educational and Psychological Measurement, 81(2), 388-404. https://doi.org/10.1177/0013164420941147
18.
DimitrovD. M.GhamdiH.AlahmadiM. (2020, December). Setting cutting scores for mastery on Mawhibah’s multiple cognitive ability tests (Technical Report [TR 12-2020]). Education & Training Evaluation Commission, Riyadh, Saudi Arabia.
EganK. L.FerraraS.SchneiderM. C.BartonK. E. (2009). Writing performance level descriptors and setting performance standards for assessments of modified achievement standards: The role of innovation and importance of following conventional practice. Peabody Journal of Education, 84(4), 552-577. https://doi.org/10.1080/01619560903241028
21.
FerraraS.LewisD. M. (2012). The item-descriptor (ID) matching method. In CizekG. J. (Ed.), Setting performance standards: Foundations, methods, and innovations (2nd ed., pp. 255-282). Routledge.
22.
FerraraS.PerieM.JohnsonE. (2008). Matching the judgmental task with standard setting panelist expertise: The item-descriptor (ID) matching procedure. Journal of Applied Testing Technology, 9(1), 1-22.
23.
FerraraS.SwaffieldS.MuellerL. (2009). Conceptualizing and setting performance standards for alternate assessments. In SchaferW. D.LissitzR. W. (Eds.), Alternate assessments based on alternate achievement standards: Policy, practice, and potential (pp. 93-111). Brookes.
24.
GrabovskyI.WainerH. (2017). The cut-score operating function: A new tool to aid in standard setting. Journal of Educational and Behavioral Statistics, 42(3), 251-263. https://doi.org/10.3102/1076998617696495
25.
HambletonR. K. (2001). Setting performance standards on educational assessments and criteria for evaluating the process. In CizekG. J. (Ed.), Setting performance standards (pp. 89-115). Lawrence Erlbaum.
26.
HambletonR. K.PitoniakM. J. (2006). Setting performance standards. In BrennanR. L. (Ed.), Educational measurement (4th ed.). American Council on Education/Praeger.
27.
HambletonR. K.PlakeB. S. (1995). Using an extended Angoff procedure to set standards on complex performance assessments. Applied Measurement in Education, 8(1), 41-55. https://doi.org/10.1207/s15324818ame0801_4
28.
HambletonR. K.SwaminathanH.RogersH. J., (1991). Fundamentals of item response theory. Sage.
29.
HauserR. M.EdleyC. F.Jr.KoenigJ. A.ElliottS. W. (2005). Measuring literacy: Performance levels for adults. National Academies Press
30.
HuynhH. (1998). On score locations of binary and partial credit items and their applications to item mapping and criterion-referenced interpretation. Journal of Educational and Behavioral Statistics, 23(1), 35-56. https://doi.org/10.3102/10769986023001035
31.
HuynhH. (2006). A clarification on the response probability criterion RP67 for standard settings based on bookmark and item mapping. Educational Measurement: Issues and Practice, 25(2), 19-20. https://doi.org/10.1111/j.1745-3992.2006.00053.x
32.
KaneM. T. (1994). Validating the performance standards associated with passing scores. Review of Educational Research, 64(3), 425-461. https://doi.org/10.3102/00346543064003425
33.
KaneM. T. (2001). So much remains the same: Conception and status of validation in setting standards. In CizekG. J. (Ed.), Setting performance standards (pp. 53-88). Lawrence Erlbaum.
34.
KarantonisA.SireciS. G. (2006). The bookmark standard-setting method: A literature review. Educational Measurement: Issues and Practice, 25(1), 4-12. https://doi.org/10.1111/j.1745-3992.2006.00047.x
35.
LewisD. M.GreenR. (1997, June). The validity of PLDs[Paper presentation]. National Conference on Large Scale Assessment, Colorado Springs, CO, United States.
36.
LewisD. M.MitzelH. C.GreenD. R.PatzR. J. (1999). The bookmark standard setting procedure. McGraw-Hill.
37.
LewisD. M.MitzelH. C.MercadoR. L.SchulzE. M. (2012). The bookmark standard setting procedure. In CizekG. J. (Ed.), Setting performance standards: Foundations, methods, and innovations (2nd ed., pp. 225-254). Routledge.
38.
LinJ. (2006). The bookmark procedure for setting cut-scores and finalizing performance standards: Strengths and weaknesses. Alberta Journal of Educational Research, 52(1), 36-52.
39.
MillsC. N.JaegerR. M. (1998). Creating descriptions of desired student achievement when setting performance standards. In HanscheL. (Ed.), Handbook for the development of performance standards (pp. 73-85). U.S. Department of Education and Council of Chief State School Officers.
40.
MitzelH. C.LewisD. M.PatzR. J.GreenD. R. (2001). The bookmark procedure: Psychological perspectives. In CizekG. J. (Ed.), Setting performance standards: Concepts, methods and perspectives (pp. 249-281). Lawrence Erlbaum.
41.
MourguesC. V.TanM.HeinS.Al-HarbiK.AljughaimanA.GrigorenkoE. L. (2016). The relationship between analytical and creative cognitive skills from middle childhood to adolescence: Testing for the threshold theory in the Kingdom of Saudi Arabia. Learning and Individual Differences, 52, 137-147. https://doi.org/10.1016/j.lindif.2015.05.005
PhillipsG. W. (2012). The benchmark method of standard setting. In CizekG. J. (Ed.), Setting performance standards: Foundations, methods, and innovations (2nd ed., pp. 323-346). Routledge.
44.
PlakeB. S.CizekG. J. (2012). Variations on a theme: The modified Angoff, extended Angoff, and Yes/No standard setting methods. In CizekG. J. (Ed.), Setting performance standards: Foundations, methods, and innovations (2nd ed., pp. 181-200). Routledge.
45.
RickerK. L. (2006). Setting cut-scores: A critical review of the Angoff and modified Angoff methods. Alberta Journal of Educational Research, 52(1), 53-64.
SchulzE. M.MitzelH. (2005, April). The mapmark standard setting method [Paper presentation]. Annual meeting of the National Council on Measurement and Education, Montreal, Canada.
48.
SchultzE. M.MitzelH. C. (2011). A mapmark method of standard setting as implemented for the National Assessment Governing Board. Journal of Applied Measurement, 12(2), 165-193.
49.
ShengY. (2008). Markov chain Monte Carlo estimation of normal ogive IRT models in MATLAB. Journal of Statistical Software, 25(8), 1-15. https://doi.org/10.18637/jss.v025.i08
50.
SkaggsG.TessemaA. (2001, April). Item disordinality with the bookmark standard setting procedure [Paper presentation]. Annual meeting of the National Council for Measurement in Education annual meeting, Seattle.
51.
SkorupskiW. P.HambletonR. K. (2005). What are panelists thinking when they participate in standard-setting studies?Applied Measurement in Education, 18(3), 233-256. https://doi.org/10.1207/s15324818ame1803_3
52.
van der LindenW. J. (1982). A latent trait method for determining intrajudge inconsistency in the Angoff and Nedelsky techniques of standard-setting. Journal of Educational Measurement, 19, 295-308. https://doi.org/10.1111/j.1745-3984.1982.tb00135.x
WilliamsN. J.SchulzE. M. (2005, April). An investigation of response probability (RP) values used in standard setting[Paper presentation]. Annual meeting of the National Council on Measurement in Education, Montreal, Quebec, Canada.
55.
WyseA. E. (2011). The similarity of bookmark cut scores with different response probability values. Educational and Psychological Measurement, 71(6), 963-985. https://doi.org/10.1177/0013164410395577
56.
WyseA. E.BunchM.DevilleC.VigerS. G. (2014). Body of work standard-setting method with construct maps. Educational and Psychological Measurement, 74(2), 236-262. https://doi.org/10.1177/0013164413502037
57.
ZiekyM. J. (2001). So much has changed: How the setting of cutscores has evolved since the 1980s. In CizekG. J. (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 19-51). Lawrence Erlbaum.
58.
ZwickR.SenturkD.WangJ.LoomisS. C. (2001). An investigation of alternative methods for item mapping in the National Assessment of Educational Progress. Educational Measurement: Issues and Practice, 20(2), 15-25. https://doi.org/10.1111/j.1745-3992.2001.tb00059.x
