ESCO skill classifiers, which scan job ads to identify skills at the finest level of the taxonomy, are widely used in international statistical projects and by national employment agencies. However, to our knowledge, no systematic evaluation of these classifiers has been conducted across large sets (i.e., thousands) of ESCO skills. We introduce a method for evaluating ESCO skill classifiers, addressing two key challenges: the large number of skills (up to around 14 000) and severe class imbalance. Our approach relies on matrix sampling of skills and job ads, with clustering and stratification, and uses bootstrapping to estimate standard errors for classifier comparisons. We apply the method to three classifiers using a sample of Luxembourgish IT and finance job ads. Our results indicate that the classifiers achieve acceptable accuracy despite low recall. Notably, the text matching classifier performs competitively, even against methods based on large language models.
Cedefop. Online job vacancies and skills analysis: a Cedefop pan-European approach. Saloniki, Greece: Publication Office of the European Union, 2019. https://doi.org/10.2801/567701
2.
KhaoujaIKassouIGhoghoM. A survey on skill identification from online job ads. IEEE Access2021; 9: 118134–118153.
3.
European Commission. ESCO implementation manual. Brussels, Belgium: Publication Office of the European Union, 2018.
NayaVBiedGCaillouP, et al.Designing labor market recommender systems: the importance of job seeker preferences and competition. HAL Open Sci2022. https://inria.hal.science/hal-03540319
7.
AscheriAKiss-NagyAMarconiG, et al.Competition in urban hiring markets: evidence from online job advertisements. Eurostat Statistical Working Papers, 2021. https://doi.org/10.2785/667004
8.
PruskiCda Costa PereiraCDa SilveiraM, et al.Bridging Skill Gaps: Combining generative and symbolic AI for personalized lifelong learning pathways. In: International Conference on Artificial Intelligence in Education, Palermo, Italy, 2025.
9.
BholaAHalderKPrasadA, et al. Retrieving skills from job descriptions: A language model based extreme multi-label classification. In: Proceedings of the 28th International conference on computational linguistics, 2020, https://doi.org/10.18653/v1/2020.coling-main.513
10.
NguyenKZhangMMontariolS, et al. Rethinking skill extraction in the job market domain using large language models. 2024, arXiv, https://doi.org/10.48550/arXiv.2402.03832
11.
ZhangMJensenKSonniksS, et al. Skillspan: hard and soft skill ex-traction from English job postings. In: Proceedings of the 2022 conference of the North American chapter of the association for computational linguistics: human language technologies, 2022, pp.4962–4984. Seattle, United States: Association for Computational Linguistics.
12.
AlexopoulosP. Semantic modeling for data. California, United States: O’Reilly, 2020.
13.
LeonFGavrilescuMFloriaS, et al.Hierarchical classification of transversal skills in job advertisements based on sentence embeddings. Information2024; 15: 51.
TamburriDVan Den HeuvelWGarrigaM. Dataops for societal intelligence: a data pipeline for labor market skills extraction and matching. arXiv. 2021. https://doi.org/10.1109/IRI49571.2020.00063
ClaviéBSouliéG. Large language models as batteries-included zero-shot ESCO skills matchers. arXiv2023, preprint arXiv:2307.03539. https://doi.org/10.48550/arXiv.2307.03539
MinatelDdos SantosNda SilvaA, et al.Influence of data stratification criteria on fairer classifications. J Inf Data Manage2025; 16. 10.5753/jidm.2025.4677
28.
RaschkaS.Model evaluation, model selection, and algorithm selection in machine learning. ArXiv2020, https://arxiv.org/abs/1811.12808
DumelleMHighamMVer HoefJ, et al. A comparison of design-based and model-based approaches for finite population spatial sampling and inference. Methods Ecol Evol2022; 13: 2018–2029.
31.
SärndalCSwenssonBWretmanJ. Model assisted survey sampling. New York: Springer Science, 2003.
32.
SolonGHaiderSWooldridgeJ. What are we weighting for?J Hum Resour2015; 50: 301–316.
33.
HaliburtonLLeusmannJWelschR, et al.Uncovering labeler bias in machine learning annotation tasks. AI and Ethics2025; 5: 2515–2528.
34.
CosgroveJSosteroMBertoniE. Mapping DigComp digital competences to the ESCO skills framework for analysis of digital skills in EU online job advertisements. Publications Office of the European Union, 2024. https://doi.org/10.2760/197497
Colin CameronAMillerD. A practitioner’s guide to cluster-robust inference. J Hum Resour2015; 50: 317–372.
37.
HassijaVChamolaVMahapatraA, et al.Interpreting black-box models: a review on explainable artificial intelligence. Cognit Comput. 2024, 16, 45–74.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
