A critical methodological challenge in standard setting arises in small-sample, high-dimensional contexts where the number of items substantially exceeds the number of examinees. Under such conditions, conventional data-driven methods that rely on parametric models (e.g., item response theory) often become unstable or fail due to unreliable parameter estimation. This study investigates two families of data-driven methods: information-theoretic and unsupervised clustering, offering a potential solution to this challenge. Using a Monte Carlo simulation, we systematically evaluate 15 such methods to establish an evidence-based framework for practice. The simulation manipulated five factors, including sample size, the item-to-examinee ratio, mixture proportions, item quality, and ability separation. Method performance was evaluated using multiple criteria, including Relative Error, Classification Accuracy, Sensitivity, Specificity, and Youden’s Index. Results indicated that no single method is universally superior; the optimal choice depends on the examinee mixture proportion. Specifically, the information-theoretic method QIR (quantile information ratio) excelled in scenarios with a dominant non-competent group, where high specificity was critical. Conversely, in highly selective contexts with balanced proficiency groups, the clustering methods CHI (Calinski-Harabasz index) and sum of squared error (SSE) demonstrated the highest classification effectiveness. Bayesian kernel density estimation (BKDE), however, consistently performed as a robust, balanced method across conditions. These findings provide practitioners with a clear decision framework for selecting a defensible, data-driven standard-setting method when traditional approaches are infeasible.
AngoffW. H. (1971). Scales, norms, and equivalent scores. In ThorndikeR. L. (Ed.), Educational measurement (2nd ed., pp. 508–600). American Council on Education.
2.
ArbelaitzO.GurrutxagaI.MuguerzaJ.PérezJ. M.PeronaI. (2013). An extensive comparative study of cluster validity indices. Pattern Recognition, 46(1), 243–256. https://doi.org/10.1016/j.patcog.2012.07.021
3.
BanerjeeA.PujariA. K.PanigrahiC. R.PatiB. (2021). Entropy-based cluster selection. In PanigrahiC. R.PatiB.PattanayakB. K.AmicS.LiK. C. (Eds.), Advances in intelligent systems and computing (pp. 313–321). Springer. https://doi.org/10.1007/978-981-33-4299-6_26
4.
BennettR. E. (2015). The changing nature of educational assessment. Review of Research in Education, 39(1), 370–407.
5.
BergerM. (2013). Criterion-referenced testing. In VolkmarF. R. (Ed.), Encyclopedia of autism spectrum disorders (pp. 823–823). Springer. https://doi.org/10.1007/978-1-4419-1698-3_146
6.
BirnbaumA. (1968). Some latent trait models and their use in inferring an examinee’s ability. In LordF. M.NovickM. R. (Eds.), Statistical theories of mental test scores (pp. 397–479). Addison-Wesley.
7.
BoliB. I. A.-D.WeiC. H. (2024). Bayesian classifier based on Robust Kernel density estimation and Harris Hawks optimisation. International Journal of Internet and Distributed Systems, 6(1), 1–23. https://doi.org/10.4236/ijids.2024.61001
8.
CalinskiT.HarabaszJ. (1974). A dendrite method for cluster analysis. Communications in Statistics—Theory and Methods, 3(1), 1–27. https://doi.org/10.1080/03610927408827101
ChengY.LiuC.BehrensJ. (2015). Standard error of ability estimates and the classification accuracy and consistency of binary decisions. Psychometrika, 80(3), 645–664. https://doi.org/10.1007/s11336-014-9407-z
CizekG. J.BunchM. B. (2007). Standard setting: A guide to establishing and evaluating performance standards on tests. Sage.
13.
CohenJ. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum Associates.
14.
DaviesD. L.BouldinD. W. (1979). A cluster separation measure. IEEE Transactions on Pattern Analysis and Machine Intelligence, PAMI-1(2), 224–227. https://doi.org/10.1109/TPAMI.1979.4766909
15.
de AyalaR. J. (2009). The theory and practice of item response theory (pp. xv, 448). The Guilford Press.
16.
DeborahL. J.BaskaranR.KannanA. (2010). A survey on internal validity measure for cluster validation. International Journal of Computer Science & Engineering Survey, 1(2), 85–102. https://doi.org/10.5121/ijcses.2010.1207
EbelR. L. (1972). Essentials of educational measurement. Prentice-Hall.
19.
EscobarM. D.WestM. (1995). Bayesian density estimation and inference using mixtures. Journal of the American Statistical Association, 90(430), 577–588. https://doi.org/10.1080/01621459.1995.10476550
GlaserR. (1963). Instructional technology and the measurement of learning outcomes: Some questions. American Psychologist, 18(8), 519–521. https://doi.org/10.1037/h0049294
23.
GreenD. M.SwetsJ. A. (1966). Signal detection theory and psychophysics. Wiley.
24.
HalkidiM.BatistakisY.VazirgiannisM. (2002). Clustering validity checking methods: Part II. SIGMOD Record, 31(3), 19–27. https://doi.org/10.1145/601858.601862
25.
HalkidiM.VazirgiannisM.BatistakisY. (2000). Quality scheme assessment in the clustering process. In ZighedD. A.KomorowskiJ.ŻytkowJ. (Eds.), Principles of data mining and knowledge discovery (PKDD 2000) (Vol. 1910, pp. 265–276). Springer. https://doi.org/10.1007/3-540-45372-5_26
26.
HambletonR. K.EignorD. R. (1979). A practitioner’s guide to criterion-referenced test development, validation, and test score usage [2nd ed., Laboratory of Psychometric and Evaluative Research Report No. 70]. School of Education, University of Massachusetts.
27.
HastieT.TibshiraniR.FriedmanJ. (2009). The elements of statistical learning: Data mining, inference, and prediction (2nd ed.). Springer.
28.
HofsteeW. K. B. (1983). The case for compromise in educational selection and grading. In AndersonS. B.HelmickJ. S. (Eds.), On educational testing (pp. 109–127). Jossey-Bass.
29.
InmanH. F.BradleyE. L. (1989). The overlapping coefficient as a measure of agreement between probability distributions and point estimation of the overlap of two normal densities. Communications in Statistics—Theory and Methods, 18(10), 3851–3874. https://doi.org/10.1080/03610928908830127
30.
JacobsonN. S.TruaxP. (1991). Clinical significance: A statistical approach to defining meaningful change in psychotherapy research. Journal of Consulting and Clinical Psychology, 59(1), 12–19.
31.
JaegerR. M. (1989). Certification of student competence. In LinnR. L. (Ed.), Educational measurement (3rd ed., pp. 485–514). Macmillan.
32.
JiangZ.OuyangJ.ShiD.PanJ.XuL.CaiF. (2024). Forming intervals of predicted total scores for cut-off scores evaluation: A generalizability theory application with bootstrapping. Current Psychology, 43, 27778–27792. https://doi.org/10.1007/s12144-024-06306-9
33.
JiaoH.LissitzR. W.MacreadyG.WangS.LiangS. (2011). Exploring levels of performance using the mixture Rasch model for standard setting. Psychological Test and Assessment Modeling, 53(4), 499–522.
KaneM. T. (2001). So much remains the same: Conception and status of validation in setting standards. In CizekG. J. (Ed.), Setting performance standards: Concepts, methods, and perspectives (pp. 53–88). Lawrence Erlbaum Associates Publishers.
36.
KimB.KimJ.YiG. (2017). Analysis of clustering evaluation considering features of item response data using data mining technique for setting cut-off scores. Symmetry, 9(5), Article 5. https://doi.org/10.3390/sym9050062
37.
KraemerH. C. (1992). Evaluating medical tests: Objective and quantitative guidelines. Sage.
38.
LavineM. (1995). On an approximate likelihood for quantiles. Biometrika, 82(1), 220–222.
39.
LewisD. M.MitzelH. C.GreenD. R. (1996). Standard setting: A bookmark approach [Paper presentation]. National Conference on Large-scale Assessment, Council of Chief State School Officers, Boulder, CO, United States.
40.
LewisG.CollishawS.ThaparA.HaroldG. T. (2014). Parent–child hostility and child and adolescent depression symptoms: The direction of effects, role of genetic factors and gender. European Child & Adolescent Psychiatry, 23(5), 317–327. https://doi.org/10.1007/s00787-013-0460-4
41.
LiF.CohenA.KimS.-H.ChoS.-J. (2009). Model selection methods for mixture dichotomous IRT models. Applied Psychological Measurement, 33(5), 353–373.
42.
LivingstonS.ZiekyM. (1982). Passing scores: A manual for setting standards of performance on educational and occupational tests. Educational Testing Service.
43.
LordF. M. (1980). Applications of item response theory to practical testing problems. Erlbaum.
44.
MacQueenJ. B. (1967). Some methods for classification and analysis of multivariate observations. In Le CamL. M.NeymanJ. (Eds.), Proceedings of the fifth Berkeley symposium on mathematical statistics and probability (Vol. 1, pp. 281–297). University of California Press.
NovoselovaN.TomI. (2012). Entropy-based cluster validation and estimation of the number of clusters in gene expression data. Journal of Bioinformatics and Computational Biology, 10(6), 1250011. https://doi.org/10.1142/S0219720012500114
47.
PastoreM.CalcagnìA. (2019). Measuring distribution similarities between samples: A distribution-free overlapping index. Frontiers in Psychology, 10, Article 1089. https://doi.org/10.3389/fpsyg.2019.01089
48.
PitoniakM. J. (2003). Standard setting methods for complex licensure examinations [Unpublished doctoral dissertation]. University of Massachusetts Amherst.
RousseeuwP. J. (1987). Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20, 53–65. https://doi.org/10.1016/0377-0427(87)90125-7
51.
SadlerB. P.StokesS. L. (2022). Item response theory and Fisher information for small tests. In NgH. K. T.HeitjanD. F. (Eds.), Recent advances on sampling methods and educational statistics (pp. 233–250). Springer. https://doi.org/10.1007/978-3-030-92007-6_13
52.
SenS.CohenA. S. (2023). The impact of sample size and various other factors on estimation of dichotomous mixture IRT models. Educational and Psychological Measurement, 83(3), 520–555. https://doi.org/10.1177/00131644221094325
SilvermanB. W. (1986). Density estimation for statistics and data analysis. Chapman and Hall.
55.
SireciS. G. (2001). Standard setting using cluster analysis. In CizekC.J. (Ed.), Standard setting: Concepts, methods, and perspectives (pp. 339–354). Lawrence Erlbaum.
56.
SireciS. G.RobinF.PatelisT. (1999). Using cluster analysis to facilitate standard setting. Applied Measurement in Education, 12, 301–325.
57.
SwanwickT.ForrestK.O’BrienB. C., & Association for the Study of Medical Education. (Eds.). (2018). Understanding medical education: Evidence, theory and practice (3rd ed.). Wiley-Blackwell.
58.
VermaR. K.TiwariR.ThakurP. S. (2023). Partition coefficient and partition entropy in fuzzy C-means clustering. Journal of Scientific Research and Reports, 29(12), 1–6. https://doi.org/10.9734/jsrr/2023/v29i121812
59.
YoudenW. J. (1950). Index for rating diagnostic tests. Cancer, 3(1), 32–35. https://doi.org/10.1002/1097-0142(1950)3:1<32::aid-cncr2820030106>3.0.co;2-3
60.
ZhengY.CheonH.KatzC. M. (2020). Using machine learning methods to develop a short tree-based adaptive classification test: Case study with a high-dimensional item pool and imbalanced data. Applied Psychological Measurement, 44(7–8), 499–514.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
