The choice of response probability in the bookmark method has been shown to affect outcomes in important ways. These findings have implications for the validity of the bookmark method because panelists’ inability to internally adjust when given different response probabilities suggests that they are not performing the intended judgment task. In response to the concerns these findings raise, proponents of the bookmark method argue that such concerns can be addressed by using a response probability of .67. A crucial part of their argument includes the often-repeated claim that the .67 value corresponds with the maximum information for a correct response, which is believed to be beneficial in some way. In this article, it is shown that this claim is mistaken; that the formula upon which the .67 result is based is incorrect; that (for the relevant measurement model) there is no difference between the information for a correct response, for an incorrect response, or for the item overall; and, more generally, that the “maximize information” approach is based on the wrong likelihood function altogether.
AndrichD. (1978). Application of a psychometric rating model to ordered categories which are scored with successive integers. Applied Psychological Measurement, 2, 581-594.
2.
BakerF. B.KimS. H. (2004). Item response theory: Parameter estimation techniques (2nd ed.). New York, NY: Dekker.
3.
BeretvasS. N. (2004). Comparison of bookmark difficulty locations under different item response models. Applied Psychological Measurement, 28, 25-47.
4.
BirnbaumA. (1968). Some latent trait models and their use in inferring an examinee’s ability. In LordF. M.NovickM. R. (Eds.), Statistical theories of mental test scores (pp. 397-479). Reading, MA: Addison-Wesley.
5.
BockR. D. (1972). Estimating item parameters and latent ability when responses are scored in two or more nominal categories. Psychometrika, 37, 29-51.
6.
BradlowE. T. (1996). Teacher’s corner: Negative information and the three-parameter logistic model. Journal of Educational and Behavioral Statistics, 21, 179-185.
DeMarsC. (2010). Item response theory. Oxford, UK: Oxford University Press.
9.
EfronB.HinkleyD. V. (1978). Assessing the accuracy of the maximum likelihood estimator: Observed versus expected Fisher information. Biometrika, 65, 457-483.
10.
HambletonR. K.PitoniakM. J. (2006). Setting performance standards. In BrennanR. L. (Ed.), Educational measurement (4th ed., pp. 433-470). Washington, DC: American Council on Education.
HauserR. M.EdleyC. F.Jr.KoenigJ. A.ElliottS. W. (2005). Measuring literacy: Performance levels for adults. Washington, DC: National Academies Press.
13.
HuynhH. (1994, October). Some technical aspects of standard setting. In CrockerL.ZiekyM. (Eds), Proceedings of the joint conference on standard setting for large scale assessment programs, volume II (pp. 75-91). Washington, DC: National Assessment Governing Board and National Center for Education Statistics.
14.
HuynhH. (1998). On score locations of binary and partial credit items and their applications to item mapping and criterion-referenced interpretation. Journal of Educational and Behavioral Statistics, 23, 35-56.
15.
HuynhH. (2000a, April). On Bayesian rules for selecting 3PL binary items for criterion-referenced interpretation and creating ordered item booklets for bookmark standard setting. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
16.
HuynhH. (2000b, April). On item mapping and statistical rules for selecting binary items for criterion-referenced interpretation and bookmark standard setting. Paper presented at the annual meeting of National Council on Measurement in Education, New Orleans, LA.
17.
HuynhH. (2006). A clarification on the response probability criterion RP67 for standard settings based on bookmark and item mapping. Educational Measurement: Issues and Practice, 25(2), 19-20.
18.
KarantonisA.SireciS. G. (2006). The bookmark standard-setting method: A literature review. Educational Measurement: Issues and Practice, 25(1), 4-12.
19.
KolstadA.CohenJ.BaldiS.ChanT.DeFurE.AngelesJ. (1998). The response probability convention used in reporting data from IRT assessment scales: Should NCES adopt a standard. Washington, DC: American Institutes for Research.
20.
LewisD. M.GreenD. R.MitzelH. C.BaumK.PatzR. J. (1998, April). The bookmark standard setting procedure: Methodology and recent implementations. Paper presented at the annual meeting of the National Council for Measurement in Education, San Diego, CA.
21.
LewisD. M.MitzelH. C.GreenD. R. (1996, June). Standard setting: A bookmark approach. In GreenD. R., IRT-based standard-setting procedures utilizing behavioral anchoring. Symposium conducted at the Council of Chief State School Officers National Conference on Large-Scale Assessment, Phoenix, AZ.
22.
LewisD. M.MitzelH. C.MercadoR. L.SchulzE. M. (2012). The bookmark standard setting procedure. In CizekG. J. (Ed.), Setting performance standards: Foundations, methods, and innovations (2nd ed., pp. 225-253). New York, NY: Routledge.
23.
LinJ. (2006). The bookmark procedure for setting cut-scores and finalizing performance standards: Strengths and weaknesses. Alberta Journal of Educational Research, 52(1), 36-52.
24.
LindsayB. G.LiB. (1997). On second-order optimality of the observed Fisher information. The Annals of Statistics, 25, 2172-2199.
25.
LoriéW. A. (2002). Setting defensible cut scores: Canonical pseudo-responses, item types, and performance standards (Unpublished doctoral dissertation). Stanford University, Stanford, CA.
26.
MagisD. (2015). A note on the equivalence between observed and expected information functions with polytomous IRT models. Journal of Educational and Behavioral Statistics, 40, 96-105.
27.
MastersG. N. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149-174.
28.
MitzelH. C.LewisD. M.PatzR. J.GreenD. R. (2001). The bookmark procedure: Psychological perspectives. In CizekG. (Ed.), Setting performance standards: Concepts, methods and perspectives (pp. 249-281). Mahwah, NJ: Lawrence Erlbaum.
29.
MurakiE. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159-176.
30.
SamejimaF. (1969). Estimation of latent ability using a response pattern of graded scores (Psychometrika Monograph No. 17). Richmond, VA: Psychometric Society.
31.
SamejimaF. (1973). A comment on Birnbaum’s three-parameter logistic model in the latent trait theory. Psychometrika, 38, 221-233.
32.
SkaggsG.HeinS. F.AwuorR. (2007). Setting passing scores on passage-based tests: A comparison of traditional and single-passage bookmark methods. Applied Measurement in Education, 20, 405-426.
33.
SkaggsG.TessemaA. (2001, April). Item disordinality with the bookmark standard setting procedure. Paper presented at the annual meeting of the National Council on Measurement in Education, Seattle, WA.
34.
SuhY.BoltD. M. (2010). Nested logit models for multiple-choice item response data. Psychometrika, 75, 454-473.
35.
van der LindenW. J. (1998). Bayesian item selection criteria for adaptive testing. Psychometrika, 63, 201-216.
36.
van der LindenW. J. (2010). Item response theory. In McGawB.BakerE.PetersonP. (Eds.), International encyclopedia of education (3rd ed., pp. 81-88). Oxford, UK: Elsevier.
37.
VeerkampW. J. J. (1996). Statistical methods for computerized adaptive testing (Unpublished doctoral thesis). University of Twente, Enschede, The Netherlands.
38.
WilliamsN. J.SchulzE. M. (2005, April). An investigation of response probability (RP) values used in standard setting. Annual meeting of the National Council on Measurement in Education, Montreal, Québec, Canada.
