Sage Journals: Discover world-class research

Abstract

AI applications in finance including those for the probability of default modeling largely involve using ML classification tools. Oversampling the very minor (very underrepresented) class of defaulted borrowers seems to be a must-be-done step always. However, by crunching more than a thousand of confidence intervals for the classification accuracy metrics, we demonstrate when such oversampling is worth engaging in. Moreover, we argue to what portion of total initial sample size such oversampling should be carried out. Our findings are valuable primarily for the credit risk modeling and Internal Ratings Based (IRB) banks, but are not limited to those and have general applications for the binary classifications in ML domain.

Keywords

credit risk precision recall F1 classification clustering segmentation IRB

Get full access to this article

View all access options for this article.

References

Agresti

Coull

B. A.

(1998). Approximate is better than “exact” for interval estimation of binomial proportions. The American Statistician, 52, 119–126. https://doi.org/10.1080/00031305.1998.10480550 , restricted access. https://math.unm.edu/james/Agresti1998.pdf, open access, accessed on December 30, 2023.

Altman

E. I.

(1968). Financial ratios, discriminant analysis and the prediction of corporate bankruptcy. The Journal of Finance, 23(4), 589–609.

Altman

E. I.

(2018). A fifty-year retrospective on credit risk models, the Altman Z-score family of models and their applications to financial markets and managerial strategies. Journal of Credit Risk, 14(4), 1–34. http://doi.org/10.21314/JCR.2018.243 , restricted access. https://mebfaber.com/wp-content/uploads/2020/11/Altman_Z_score_models_final.pdf, open access, accessed on December 26, 2023.

Altman

E. I.

Sabato

Wilson

(2008). The value of qualitative information in sme risk management. https://pages.stern.nyu.edu/ealtman/SME_EA_GS_NW.pdf

Audigier

Niang

Resche-Rigon

(2021). Clustering with missing data: Which imputation model for which cluster analysis method? https://arxiv.org/pdf/2106.04424.pdf. Online; accessed 15 January 2022.

BCBS (2000). Credit ratings and complementary sources of credit quality information. Working Paper No. 3. https://www.bis.org/publ/bcbs_wp3.pdf, open access, accessed on June 11, 2024.

BCBS (2006). Basel II: International convergence of capital measurement and capital standards: A revised framework - comprehensive version. https://www.bis.org/publ/bcbs128.pdf, open access, accessed on June 11, 2024.

BCBS (2017). Implications of fintech developments for banks and bank supervisors. Basel Committee for Banking Supervision Consultative Paper. https://www.bis.org/bcbs/publ/d415.htm, free access, accessed on Aug. 13, 2022.

Beaver

W. H.

(1966). Financial ratios as predictors of failure. Journal of Accounting Research, 4, 71–111.

10.

Bisogno

Restaino

Di Carlo

(2018). Forecasting and preventing bankruptcy: A conceptual review. African Journal of Business Management, 12(9), 231–242.

11.

Bräuning

Malikkidou

Scalone

Scricco

(2020). A new approach to early warning systems for small European banks. In International conference on machine learning, optimization, and data science (pp. 551–562). Springer.

12.

Brown

L. D.

Cai

T. T.

DasGupta

(2001). Interval estimation for a binomial proportion. Statistical Science, 16, 101–133. https://www.jstor.org/stable/2676784, restricted access. http://www-stat.wharton.upenn.edu/lbrown/Papers/2001a%20Interval%20estimation%20for%20a%20binomial%20proportion%20(with%20T.%20T.%20Cai%20and%20A.%20DasGupta).pdf, open access, accessed on December 26, 2023.

13.

Carreras

Miccinesi

Wilcock

Preston

Nieboer

Deliens

Groenvold

Lunder

van der Heide

Baccini

Korfage

I. J.

Rietjens

J. A. C.

Jabbarian

L. J.

Polinder

van Delden

Kars

Zwakman

Deliens

Verkissen

M. N.

Eecloo

Faes

Pollock

Seymour

Caswell

Wilcock

Bramley

Payne

Preston

Dunleavy

Sowerby

Miccinesi

Bulli

Ingravallo

Carreras

Toccafondi

Gorini

Lunder

Červ

Simonič

Mimić

Kodba-Čeh

OzbiČ

Groenvold

Arnfeldt

Thit Johnsen

and ACTION consortium (2021). Missing not at random in end of life care studies: Multiple imputation and sensitivity analysis on data from the ACTION study. BMC Medical Research Methodology, 21(1), 13. https://doi.org/10.1186/s12874-020-01180-y

14.

Chawla

N. V.

Bowyer

K. W.

Hall

L. O.

Kegelmeyer

W. P.

(2002). SMOTE: Synthetic minority over-sampling technique. Journal of Artificial Intelligence Research, 16, 1–26. https://doi.org/10.1613/jair.953 , open access, accessed on August 27, 2024.

15.

Chen

Marshall

B. R.

Zhang

Ganesh

(2006). Financial distress prediction in China. Review of Pacific Basin Financial Markets and Policies, 09(02), 317–336. 10.1142/S0219091506000744 . https://doi.org/10.1142/S0219091506000744, restricted access.

16.

Cheong

C. W.

Ramasamy

(2019). Bank failure: A new approach to prediction and supervision. Asian Journal of Finance & Accounting, 11, 111–140.

17.

Citterio

(2020). Bank failures: Review and comparison of prediction models. https://ssrn.com/abstract=3719997

18.

Cole

R. A.

Taylor

(2020). Predicting bank failures using a simple dynamic hazard model. Available at SSRN 1460526.

19.

Dendramis

Tzavalis

Cheimarioti

(2020). Measuring the default risk of small business loans: Improved credit risk prediction using deep learning. Athens University of Economics and Business, School of Economic Sciences Working Paper No. 12-2020. https://www.dept.aueb.gr/sites/default/files/allWP-12-20-Dendram-Tzaval-Cheimar-12-11-20_0.pdf, open access, accessed on May 31, 2024.

20.

Dunnigan

(2008). Confidence interval calculation for binomial proportions. https://www.mwsug.org/proceedings/2008/pharma/MWSUG-2008-P08.pdf, open access, accessed on May 29, 2024.

21.

Durand

Le Quang

et al (2021). What do bankrupcty prediction models tell us about banking regulation? Evidence from statistical and learning approaches. https://xtra.economix.fr/pdf/dt/2021/WP_EcoX_2021-2.pdf?1.0

22.

The Economist (2024). Can Nvidia be dethroned? Meet the startups vying for its crown. A new generation of AI chips is on the way. https://www.economist.com/business/2024/05/19/can-nvidia-be-dethroned-meet-the-startups-vying-for-its-crown, open access, accessed on May 29, 2024.

23.

Engelmann

Hayden

Tasche

(2003). Testing rating accuracy. Risk (Concord, NH), 16, 82–86. https://www.researchgate.net/publication/215991100_Testing_Rating_Accuracy, open access, accessed on December 26, 2023

24.

Europarliament (2023). EU AI act: First regulation on artificial intelligence. The use of artificial intelligence in the EU will be regulated by the AI act, the world’s first comprehensive AI law. find out how it will protect you. https://www.europarl.europa.eu/news/en/headlines/society/20230601STO93804/eu-ai-act-first-regulation-on-artificial-intelligence, open access, accessed on May 29, 2024.

25.

Fantazzini

Figini

(2009). Random survival forests models for SME credit risk measurement. Methodology and Computing in Applied Probability, 17, 29–45. https://doi.org/10.1007/s11009-008-9078-2 , restricted access.

26.

Faraj

A. A.

Mahmud

D. A.

Rashid

B. N.

(2021). Comparison of different ensemble methods in credit card default prediction. UHD Journal of Science and Technology, 5, 20–25. https://doi.org/10.21928/uhdjst.v5n2y2021.pp20-25 , open access, accessed on December 31, 2023.

27.

Feng

J. Q.

(2023). The fallacy in the paradox of achilles and the tortoise. https://doi.org/10.48550/arXiv.2310.03768. http://arxiv.org/abs/2310.03768. ArXiv:2310.03768 [math].

28.

Ferriani

Cornacchia

Farroni

Ferrara

Guarino

Pisanti

(2019). An early warning system for less significant Italian banks. https://www.bancaditalia.it/pubblicazioni/qef/2019-0480/QEF_480_19.pdf. Bank of Italy Occasional Paper No. 480.

29.

Fuster

Goldsmith-Pinkham

Ramadorai

Walther

(2018). Predictably unequal? The effects of machine learning on credit markets. https://doi.org/10.1111/jofi.13090, open access, accessed on Dec. 30, 2023.

30.

Geng

Bose

Chen

(2015). Prediction of financial distress: An empirical study of listed chinese companies using data mining. European Journal of Operational Research, 241(1), 236–247. https://doi.org/10.1016/j.ejor.2014.08.016 , https://www.sciencedirect.com/science/article/pii/S0377221714006511. restricted access

31.

Hanson

Schuermann

(2006). Confidence intervals for probabilities of default. Journal of Banking & Finance, 30(8), 2281–2301. https://doi.org/10.1016/j.jbankfin.2005.08.002 , https://www.sciencedirect.com/science/article/pii/S0378426605002128. restricted access. https://papers.ssrn.com/abstract=766345, open access (accessed on April 29, 2025).

32.

Hayek

F. A. V.

(1974). The pretence of knowledge. Nobel Lecture. https://www.nobelprize.org/prizes/economic-sciences/1974/hayek/lecture/, free access, accessed on Aug. 05, 2022.

33.

Heymans

M. W.

Twisk

J. W. R.

(2022). Handling missing data in clinical research. Journal of Clinical Epidemiology, 151, 185–188. https://doi.org/10.1016/j.jclinepi.2022.08.016 , https://www.sciencedirect.com/science/article/pii/S0895435622002189. open access, accessed on January 23, 2024

34.

Jabeur

S. B.

Fahmi

(2018). Forecasting financial distress for French firms: A comparative study. Empirical Economics, 54, 1173–1186. https://doi.org/10.1007/s00181-017-1246-1 , restricted access.

35.

Johnson

M. A.

Mamun

(2012). The failure of lehman brothers and its impact on other financial institutions. Applied Financial Economics, 22(5), 375–385. https://doi.org/10.1080/09603107.2011.613762

36.

Kim

Cho

Ryu

(2020). Corporate default predictions using machine learning: Literature review. Sustainability, 12, 1–11. https://doi.org/10.3390/su12166325 , open access, accessed on December 31, 2023.

37.

Kim

Cho

Ryu

(2021). Predicting corporate defaults using machine learning with geometric-lag variables. Investment Analyst Journal, 50, 161–175. https://doi.org/10.1080/10293523.2021.1941554 , open access, accessed on December 31, 2023.

38.

Kocagil

Reyngold

Stein

Ibarra

(2002). Moody’s RiskCalc

^{TM}

Model for Privately-Held U.S. Banks. http://www.rogermstein.com/wp-content/uploads/riskcalc-usbanks.pdf

39.

Kočenda

Iwasaki

(2022). Bank survival around the world: A meta-analytic review. Journal of Economic Surveys, 36, 108–156.

40.

Korol

Korodi

(2010). Predicting bankruptcy with the use of macroeconomic variables. Economic Computation and Economic Cybernetics Studies and Research, 44, 201–219. https://www.researchgate.net/publication/289639976_Predicting_bankruptcy_with_the_use_of_macroeconomic_variables , limited access

41.

Koziarski

(2021). Potential anchoring for imbalanced data classification. Pattern Recognition, 120, 108114. https://doi.org/10.1016/j.patcog.2021.108114

42.

Kristóf

(2021). Bank failure prediction in the COVID-19 environment. Asian Journal of Economics and Finance, 3(1), 157–171.

43.

Kristóf

Virág

(2020). A comprehensive review of corporate bankruptcy prediction in Hungary. Journal of Risk and Financial Management, 13(2), 35.

44.

Kumar

P. R.

Ravi

(2007). Bankruptcy prediction in banks and firms via statistical and intelligent techniques—A review. European Journal of Operational Research, 180(1), 1–28. https://doi.org/10.1016/j.ejor.2006.08.043 , restricted access.

45.

Liu

(2021). A minority oversampling approach for fault detection with heterogeneous imbalanced data. Expert Systems With Applications, 184, 115492. https://doi.org/10.1016/j.eswa.2021.115492 , restricted access.

46.

Liu

Yang

Wang

Xiong

(2022). Applying machine learning algorithms to predict default probability in the online credit market: Evidence from China. International Review of Financial Analysis, 79, 101971. https://doi.org/10.1016/j.irfa.2021.101971 , restricted access.

47.

Merćep

Mrčela

Birov

Kostanjčar

(2021). Deep neural networks for behavioral credit rating. Entropy, 23, 806–816. https://doi.org/10.3390/e23010027 , open access, accessed on May 29, 2024.

48.

Mirkin

(2016). Clustering: A data recovery approach. 2nd edition. Chapman & Hall. https://doi.org/10.1201/9781420034912, open access, accessed on Jan. 10, 2024.

49.

Moody’s Analytics (2016). RiskCalcTM Banks 4.0. https://www.moodysanalytics.com/-/media/products/riskcalc-banks-4.pdf. The publication year is not explicitly disclosed, though we may refer to the copyright year.

50.

Moscatelli

Parlapiano

Narizzano

Viggiano

(2020). Corporate default forecasting with machine learning. Expert Systems with Applications, 161, 113567.

51.

Nunes

A. R.

Morais

Sardinha

(2021). Use of learning mechanisms to improve the condition monitoring of wind turbine generators: A review. Energies, 14, 7129. https://doi.org/10.3390/en14217129 , restricted access.

52.

Obeid

(2021). Bank failure prediction in the arab region using logistic regression model. https://www.amf.org.ae/sites/default/files/publications/2021-12/bank-failure-prediction-arab-region-using-logistic-regression-model.pdf. Arab Monetary Fund (Working Paper No. 7-2021), Available online.

53.

Ohlson

J. A.

(1980). Financial ratios and the probabilistic prediction of bankruptcy. Journal of Accounting Research, 18, 109–131.

54.

Orawo

L. A.

(2021). Confidence intervals for the binomial proportion: A comparison of four methods. Open Journal of Statistics, 11, 806–816. https://doi.org/10.4236/ojs.2021.115047 , open access, accessed on May 29, 2024.

55.

Pang

Hou

Xia

(2021). Borrowers’ credit quality scoring model and applications, with default discriminant analysis based on the extreme learning machine. Technological Forecasting and Social Change, 165, 120462. https://doi.org/10.1016/j.techfore.2020.120462 , restricted access.

56.

Pereira

R. C.

Santos

Rodrigues

Henriques Abreu

(2019). MNAR imputation with distributed healthcare data. In Progress in artificial intelligence, volume 11805. ISBN 978-3-030-30243-6 (pp. 184–195). https://doi.org/10.1007/978-3-030-30244-3_16

57.

Quan

Lei

Shi

(2019). Review of bankruptcy prediction using machine learning and deep learning techniques. Procedia Computer Science, 162, 895–899. https://doi.org/10.1016/j.procs.2019.12.065 , open access, accessed on December 30, 2023.

58.

Raschka

Mirjalili

(2019). Python machine learning. Machine learning and deep learning with Python, scikit-learn, and TensorFlow 2. 3rd edition. Packt. https://www.amazon.com/Python-Machine-Learning-scikit-learn-TensorFlow/dp/1789955750, restricted access; codes: https://github.com/rasbt/python-machine-learning-book-3rd-edition, open access, accessed on Dec. 30, 2023.

59.

Rubin

D. B.

(1976). Inference and missing data. Biometrika, 63(3), 581–592. https://doi.org/10.1093/biomet/63.3.581

60.

Shibitov

Mamedli

(2019). The finer points of model comparison in machine learning: Forecasting based on Russian banks’ data. http://www.cbr.ru/content/document/file/87572/wp43_e.pdf. Online; accessed on September 08, 2020.

61.

Shrivastava

Jeyanthi

P. M.

Singh

(2020). Failure prediction of indian banks using smote, lasso regression, bagging and boosting. Cogent Economics & Finance, 8(1), 1729569.

62.

Song

Zhu

Deng

H. P.

(2021). Research on an adaptive upsampling algorithm for photovoltaic panel segmentation. Journal of Chinese Computer Science, 42, 1485–1491. http://xwxt.sict.ac.cn/EN/Y2021/V42/I7/1485 , open access, accessed on December 26, 2023.

63.

Szepannek

Luebke

(2021). Facing the challenges of developing fair risk scoring models. Frontiers in Artificial Intelligence. https://doi.org/10.3389/frai.2021.681915, open access, accessed on Dec. 30, 2023.

64.

Tian

(2017). Financial ratios and bankruptcy predictions: An international evidence. International Review of Economics & Finance, 51, 510–526.

65.

Tinoco

Wilson

(2014). Financial distress and bankruptcy prediction among listed companies using accounting, market and macroeconomic variables. International Review of Financial Analysis, 30, 394–419. https://doi.org/10.1016/j.irfa.2013.02.013 , restricted access. https://core.ac.uk/download/pdf/20482286.pdf, open access, accessed on December 30, 2023.

66.

Wilson

E. B.

(1927). Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association, 22(158), 209–212. http://www.jstor.org/stable/2276774 , open access, accessed on December 30, 2023.

67.

Yuksel

Dincer

Hacioglu

(2015). Camels-based determinants for the credit rating of Turkish deposit banks. International Journal of Finance & Banking Studies, 4(4), 1–17.

Confidence Intervals for the Model Performance Metrics Under the Imbalanced Classification: Evaluating SMOTE’s Impact on Metrics’ Reliability

Abstract

Keywords

Get full access to this article

References