Ball-strike judgments in Major League Baseball (MLB) remain susceptible to human error, potentially affecting game outcomes and competitive fairness. This study develops an AI umpire using deep learning to assess the consistency of human umpire decisions. A deep neural network trained on 362,734 pitches from the 2019 MLB season achieved an accuracy of 93%. The AI umpire’s high-confidence predictions (probability >0.9 or <0.1) were compared against actual umpire calls to identify systematic discrepancies. Results indicate that these discrepancies were not randomly distributed but concentrated in specific game contexts—particularly early innings and first-pitch situations—suggesting situational influences on judgment. Moreover, some umpires exhibited spatial biases by consistently expanding or contracting the strike zone relative to the AI model’s zone. These findings suggest that umpire decisions are shaped by game dynamics rather than being solely rule-bound. This study proposes a novel AI-based evaluation framework for officiating consistency, illustrating the potential of AI to support more objective and reliable assessments than human umpires.
MillsBM.Technological innovations in monitoring and evaluation: evidence of performance impacts among Major League Baseball umpires. Labour Econ2017; 46: 189–199.
3.
GreenEDanielsDP.Impact aversion in arbitrator decisions. SSRN 2391558. 2015. SSRN.
4.
HuangJHsuHJ.Approximating strike zone size and shape for baseball umpires under different conditions. Int J Perform Anal Sports2020; 20(2): 133–149.
5.
FlannaganKSMillsBMGoldstoneRL.The psychophysics of home plate umpire calls. Sci Rep2024; 14(1): 2735.
6.
GuéretteJBlaisCFisetD.Verbal aggressions against Major League Baseball umpires affect their decision making. Psychol Sci2024; 35(3): 288–303.
7.
HsuM.Umpire home bias in major league baseball. J Sports Econ2024; 25(4): 423–442.
8.
MacMahonCStarkesJL.Contextual influences on baseball ball-strike decisions in umpires, players, and controls. J Sports Sci2008; 26(7): 751–760.
9.
ChenDLMoskowitzTJShueK.Decision making under the gambler’s fallacy: evidence from asylum judges, loan officers, and baseball umpires. Q J Econ2016; 131(3): 1181–1242.
10.
LeeKHanKKoJ.Analyzing the impact of the automatic ball-strike system in professional baseball: a case study on KBO league data. arXiv preprint arXiv: 2407.15779. Epub ahead of print 22July2024. DOI: 10.48550/arXiv.2407.15779
11.
SchmidhuberJ.Deep learning in neural networks: an overview. Neural Netw2015; 61: 85–117.
12.
ShresthaAMahmoodA.Review of deep learning algorithms and architectures. IEEE Access2019; 7: 53040–53065.
13.
CholletF.Deep learning with Python. Simon and Schuster, 2021.
14.
WilsonACRoelofsRSternM, et al. The marginal value of adaptive gradient methods in machine learning. Adv Neural Inf Process Syst2017; 30: 4151–4161.
15.
JohnsonJMKhoshgoftaarTM.Survey on deep learning with class imbalance. J Big Data2019; 6(1): 1–54.
16.
BaiYMeiSWangH, et al. Don’t just blame over-parametrization for over-confidence: theoretical analysis of calibration in binary classification. In: International conference on machine learning, Vienna, Austria (virtual conference): Cambridge, MA, 1 July 2021, pp.566–576. PMLR.
17.
KullMSilva FilhoTFlachP. Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. In: Artificial intelligence and statistics, Fort Lauderdale, FL,10 April 2017, pp.623–631. Cambridge, MA: PMLR.
18.
LiuCWangWLiuH, et al. Application of Hawk-eye technology to sports events. In: 2022 2nd international conference on information technology and contemporary sports (TCS), Guangzhou, China, 24–26 June 2022, pp.1–5. New York, NY: IEEE.
19.
CantOKovalchikSCrossR, et al. Validation of ball spin estimates in tennis from multi-camera tracking data. J Sports Sci2020; 38(3): 296–303.
20.
SpitzJMoorsPWagemansJ, et al. The impact of video speed on the decision-making process of sports officials. Cogn Res Princ Implic2018; 3(1): 16.
21.
TeixeiradaSilvaJANazarovetsSCarbochJ, et al. The video assistant referee in football. Sports Eng2024; 27(1): 14.
22.
SpitzJWagemansJMemmertD, et al. Video assistant referees (VAR): the impact of technology on decision making in association football referees. J Sports Sci2021; 39(2): 147–153.
23.
CarlosLPEzequielRAntonK.How does Video Assistant Referee (VAR) modify the game in elite soccer?Int J Perform Anal Sports2019; 19(4): 646–653.
24.
BloßNSchorerJLoffingF, et al. Physical load and referees’ decision-making in sports games: a scoping review. J Sports Sci Med2020; 19(1): 149–157.
25.
ParsonsCASulaemanJYatesMC, et al. Strike three: discrimination, incentives, and evaluation. Am Econ Rev2011; 101(4): 1410–1435.
26.
SchweizerGPlessnerHMacMahonC.Judgment and decision-making. In: SchülerJWegnerMPlessnerH, et al. (eds) Sport and exercise psychology: theory and application. Cham: Springer International Publishing, 2023, pp.93–115.
27.
HouisonRJLamont-MillsAKotiwM, et al. Strike 3… out! investigating pre-game moods, performance, and mental health of softball umpires. Sports2024; 12(2): 50.
28.
SimonHA.Invariants of human behavior. Ann Rev Psychol1990; 41(1): 1–20.
29.
GoelA.Is technology a complement or substitute to refereeing in sports? A review. Int J Sports Technol Sci2024; 2(1): 13–23.
30.
OyeniranCOAdewusiAOAdelekeAG, et al. Ethical AI: addressing bias in machine learning models and software applications. Comput Sci IT Res J2022; 3(3): 115–126.
31.
GaonkarBCookKMacyszynL.Ethical issues arising due to bias in training AI algorithms in healthcare and data sharing as a potential solution. AI Ethics J2020; 1(1): 1–9.
32.
RudinC.Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell2019; 1(5): 206–215.
