Generalizability theory (G theory) allows researchers to assess the many sources of variance inherent in complex standard setting procedures involving the determination of cut scores. The flexibility of G and D studies provides a way to conceptualize and quantify the results of different standard settings once the universe of admissible observations and the universe of generalization are defined. The current article applies a multivariate single-facet design for estimating standard errors of cut scores. For practical purposes, several multivariate D study designs are used to investigate what effect various panel sizes and test lengths have on the precision of the standard setting process. The current study demonstrates the advantages and usefulness of multivariate G theory in determining the accuracy of cut scores in practical applications of standard setting procedures.
AllenN. L.JenkinsF.KulickE.ZelenakC. A. (1997). Technical report of the NAEP 1996 state assessment program in mathematics. Washington, DC: National Center for Education Statistics.
2.
AngoffW. H. (1971). Scales, norms, and equivalent scores. In ThorndikeR. L. (Ed.), Educational measurement (2nd ed., pp. 508-600). Washington, DC: American Council on Education.
3.
Arce-FerrerA.YinP. (2007, April). Standard errors of cut scores for vertically scaled assessments: A generalizability theory study of Angoff-based standard setting. Paper presented at the annual meeting of the National Council on Measurement in Education, Chicago, IL.
4.
BrandonP. R. (2004). Conclusions about frequently studied modified Angoff standard-setting topics. Applied Measurement in Education, 17(1), 59–88.
5.
BrennanR. L. (1995). Standard setting from the perspective of generalizability theory. In BourqueM. L. (Ed.) Proceedings of the joint conference on standard setting for large-scale assessments (Vol. II, pp. 269–287). Washington, DC: National Center for Education Statistics and National Assessment Governing Board.
6.
BrennanR. L. (2001a). Generalizability theory. New York, NY: Springer-Verlag.
7.
BrennanR. L. (2001b). Manual for mGENOVA (Iowa Testing Programs Occasional Paper No. 47). Iowa City: University of Iowa.
8.
BrennanR. L.LockwoodR. E. (1980). A comparison of the Nedelsky and Angoff cutting score procedures using generalizability theory. Applied Psychological Measurement, 4, 219-240.
9.
ChangL. (1999). Judgmental item analysis of the Nedelsky and Angoff standard-setting methods. Applied Measurement in Education, 12, 151-165.
10.
ChangL.HocevarD. (2000). Models of generalizability theory in analyzing existing faculty evaluation data. Applied Measurement in Education, 13, 255-275.
11.
CizekG. J. (2012a). The forms and functions of evaluations of the standard setting process. In CizekG. J. (Ed.), Setting performance standards: Foundations, methods, and innovations (pp. 165-178). New York, NY: Routledge.
12.
CizekG. J. (2012b). An introduction to contemporary standard setting: Concepts, characteristics, and contexts. In CizekG. J. (Ed.), Setting performance standards: Foundations, methods, and innovations (pp. 3-14). New York, NY: Routledge.
13.
CizekG. J.BunchM. B. (2007). Standard setting—A guide to establishing and evaluating performance standards on tests. Thousand Oaks, CA: Sage.
14.
ClauserB. E.HarikP.MargolisM. J.McManusI. C.MollonJ.ChisL.WilliamsS. (2009). An empirical examination of the impact of group discussion and examinee performance information on judgments made in the Angoff standard-setting procedure. Applied Measurement in Education, 22, 1-21.
15.
HambletonR. K. (2001). Setting performance standards on educational assessments and criteria for evaluating the process 1, 2. In CizekG. J. (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 89-116). Mahwah, NJ: Lawrence Erlbaum.
16.
HambletonR. K.BrennanR. L.BrownW.DoddB.ForsythR. A.MehrensW. A.. . . ZwickR. (2000). A response to “setting reasonable and useful performance standards” in the National Academy of Sciences’ grading the nations report card. Educational Measurement: Issues and Practice, 19(2), 5-14.
17.
HambletonR. K.PitoniakM. J. (2006). Setting performance standards. In BrennanR. L. (Ed.), Educational measurement (4th ed., pp. 433-470). Washington, DC: American Council on Education.
18.
KaneM.WilsonJ. (1984). Errors of measurement and standard setting in mastery testing. Applied Psychological Measurement, 8, 107-115.
19.
KramerA.MuijtjensA.JansenK.DüsmanH.TanL.van der VleutenC. (2003). Comparison of a rational and an empirical standard setting procedure for an OSCE. Medical Education, 37, 132-139.
20.
LeeG.LewisD. M. (2001, April). A generalizability theory approach toward estimating standard errors of cutscores set using the bookmark standard setting procedure. Paper presented at the annual meeting of the National Council on Measurement in Education, Seattle, WA.
21.
LeeG.LewisD. M. (2008). A generalizability theory approach to standard error estimates for bookmark standard settings. Educational and Psychological Measurement, 68, 603-620.
PlakeB. S.CizekG. J. (2012). Variations on a theme: The modified Angoff, extended Angoff, and yes/no standard setting methods. In CizekG. J. (Ed.), Setting performance standards: Foundations, methods, and innovations (pp. 181-200). New York, NY: Routledge.
24.
ReckaseM. D. (1998). Setting standards to be consistent with an IRT item calibration. Iowa City, IA: ACT.
25.
ReckaseM. D. (2000, April). The ACT/NAGB standard setting process: How “modified” does it have to be before it is no longer a modified-Angoff process?Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
26.
TannenbaumR. J.KatzI. R. (2008). Setting standards on the core and advanced iSkills™ assessments (ETS RM-08-04). Princeton, NJ: Educational Testing Service.
27.
TzouH.WuY.-F.LinC. J. (2008, March). Validating the performance standards in 2006 TASA-MAT standard setting. Paper presented at the 2008 annual meeting of the National Council on Measurement in Education, New York City, NY.
28.
VerhoevenB. H.van der SteegA. F. W.ScherpbierA. J. J. A.MuijtjensA. M. M.VerwijnenG. M.van der VleutenC. P. M. (1999). Reliability and credibility of an Angoff standard setting procedure in progress testing using recent graduates as judges. Medical Education, 33, 832-837.
29.
VerhoevenB. H.VerwijnenG. M.MuijtjensA. M. M.ScherpbierA. J. J. A.van der VleutenC. P. M. (2002). Panel expertise for an Angoff standard setting procedure in progress testing: Item writers compared to recently graduated students. Medical Education, 36, 860-867.
30.
WangM. W.StanleyJ. C. (1970). Differential weighting: A review of methods and empirical studies. Review of Educational Research, 4, 663-704.
31.
WuY.-F.TzouH. (2010, April). Evaluating the utility and validity of IRT-based approaches in the modified Angoff standard-setting method. Paper presented at the annual meeting of the National Council on Measurement in Education, Denver, CO.
32.
YinP.SconingJ. (2006, April). Estimating standard errors of cut scores for Angoff-based and bookmark-based procedures: A generalizability theory approach. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco, CA.
33.
YinP.SconingJ. (2008). Estimating standard errors of cut scores for item rating and mapmark procedures—A generalizability theory Approach. Educational and Psychological Measurement, 68, 25-41.
