Restricted accessResearch articleFirst published online 2026
Evaluating analysis methods for coast guard reports freeform text: a case study on resource-constrained natural language processing with search and rescue reports
After a U.S. Coast Guard (USCG) search and rescue (SAR) case, USCG personnel create an after-action report containing a textual narrative of the situation and Coast Guard response efforts. Data analysts explored how to identify reports involving cases with a verified person in the water. With restricted access to compute resources and limiting policy, large language models (LLMs) could not be utilized, so statistical (‘classical’ and non-neural) methods were considered for training a classification model to identify SAR case outcomes from report texts. The dataset was severely imbalanced toward the negative class, and the texts were extremely messy, with many typos and abbreviations. Therefore, an extensive text cleaning pipeline was developed and tested for improving classification performance. The Iterative Token Elimination Algorithm (iTEA) was developed to increase differences in vocabulary between classes. Model improvement was further explored through augmentation of the feature space using non-text data. The best model was an XGBoost model, achieving 0.762 recall and precision (and 0.959 accuracy). Errors from the test set are analyzed to guide future improvements until LLMs can be used, which are expected to improve performance and reduce text cleaning requirements.
ZhouJLiXWangS, et al. NER-based military simulation scenario development process. J Def Model Simul2023; 20: 563–575.
2.
ArslanYAllixKVeiberL, et al. A comparison of pre-trained language models for multi-class text classification in the financial domain. In: Companion proceedings of the web conference 2021. WWW ’21, Ljubljana, 19–23 April 2021, pp. 260–268. New York: Association for Computing Machinery.
3.
MinaeeSKalchbrennerNCambriaE, et al. Deep learning–based text classification: a comprehensive review. ACM Comput Surv2021; 54(3): 1–40.
4.
WangZPangYLinY, et al. Adaptable and reliable text classification using large language models. In: 2024 IEEE international conference on data mining workshops (ICDMW), Abu Dhabi, 9 December 2024, pp. 67–74. New York: IEEE.
XuXAminMSanteeWR.Probability of survival decision aid (PSDA) [Technical report]. Natick, MA: U.S. Army Research Institute of Environmental Medicine, 2008.
10.
XuXAllenARiouxT, et al. Refinement of probability of survival decision aid (PSDA) [Technical report]. Natick, MA: U.S. Army Research Institute of Environmental Medicine, 2014.
WahbaYMadhavjiNSteinbacherJ.Attention is not always what you need: towards efficient classification of domain-specific text. In: AraiK (ed.) Intelligent computing. Cham: Springer Nature Switzerland, 2023, pp. 1159–1166.
13.
WahbaYMadhavjiNSteinbacherJ.A comparison of SVM against pre-trained language models (PLMs) for text classification tasks. In: NicosiaGOjhaVLa MalfaE, et al. (eds.) Machine learning, optimization, and data science. Cham: Springer Nature Switzerland, 2023, pp. 304–313.
14.
RudinC.Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell2019; 1: 206–215.
15.
RostamZRKKertészG.Advances in pre-trained language models for domain-specific text classification: a systematic review. ACM Trans Intell Syst Technol2025; 16: 1–41.
16.
BeltagyILoKCohanA.SciBERT: a pretrained language model for scientific text. In: InuiKJiangJNgV, et al. (eds.) Proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP). Hong Kong, China: Association for Computational Linguistics, 2019, pp. 3615–3620. https://aclanthology.org/D19-1371/
ChalkidisIFergadiotisMMalakasiotisP, et al. LEGAL-BERT: the muppets straight out of law school. In: CohnTHeYLiuY (eds.) Findings of the association for computational linguistics: EMNLP 2020. Kerrville, TX: Association for Computational Linguistics, 2020, pp. 2898–2904. https://aclanthology.org/2020.findings-emnlp.261/
19.
LiuZHuangDHuangK, et al. Finbert: A pre-trained financial language representation model for financial text mining. In: BessiereC (ed.) Proceedings of the twenty-ninth international joint conference on artificial intelligence, IJCAI-20. Bremen: International Joint Conferences on Artificial Intelligence Organization, 2021, pp. 4513–4519.
20.
BurkeLPazdernikKFortinD, et al. Nukelm: pre-trained and fine-tuned language models for the nuclear and energy. ESARDA Bull2021; 63: 30–40.
21.
VasankariLKoskiA.GenAI in the military: trends and opportunities. Scand J Mil Stud2025; 8: 416–434.
22.
SaricaSLuoJ.Stopwords in technical language processing. PLoS ONE2021; 16: e0254937.
23.
KimYZhangO.Credibility adjusted term frequency: a supervised term weighting scheme for sentiment analysis and text classification. In: Balahur A, Van Der Goot E, SteinbergerR, et al. (eds.) Proceedings of the 5th workshop on computational approaches to subjectivity, sentiment and social media analysis. Baltimore, MD: Association for Computational Linguistics, 2014, pp. 79–83. https://aclanthology.org/W14-2614/
24.
ZubiagaA (ed). Exploiting class labels to boost performance on embedding-based text classification. In: Proceedings of the 29th ACM international conference on information & knowledge management. CIKM ’20. New York: Association for Computing Machinery, 2020, pp. 3357–3360.
25.
LiangZZhaoYXuH, et al. A hybrid model integrating RoBERTa, TF-IDF, and attention mechanism for medical query intent classification. Sci Rep2025; 15: 42847.
26.
KaibassovaDMukhametzhanovaBTokseitD, et al. Document analysis via combined vectorization and machine learning approaches. Int J Innov Res Sci Stud2025; 8: 2195–2204. https://ijirss.com/index.php/ijirss/article/view/8356
RehurekRSojkaP.Gensim–python framework for vector space modelling. Brno: NLP Centre, Faculty of Informatics, Masaryk University, 2011.
29.
MikolovTChenKCorradoG, et al. Efficient estimation of word representations in vector space. In: 1st international conference on learning representations, ICLR 2013, Scottsdale, AZ, 2–4 May 2013, pp. 1–12. Kerrville, TX: Workshop Track Proceedings.
30.
LeQMikolovT.Distributed representations of sentences and documents. In: XingEPJebaraT (eds.) Proceedings of the 31st international conference on machine learning, proceedings of machine learning research, volume 32. Bejing, China: PMLR, 2014, pp. 1188–1196.
31.
ChenTGuestrinC.Xgboost: a scalable tree boosting system. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. KDD ’16, San Francisco, CA, 13–17 August 2016, pp. 785–794. New York: Association for Computing Machinery.
32.
BestgenY.Please, don’t forget the difference and the confidence interval when seeking for the state-of-the-art status. In: CalzolariNBéchetFBlacheP, et al. (eds.) Proceedings of the thirteenth language resources and evaluation conference. Marseille: European Language Resources Association, 2022, pp. 5956–5962. https://aclanthology.org/2022.lrec-1.640/
33.
BerziABerényiEKépesZ, et al. NLP-based removal of personally identifiable information from hungarian electronic health records. Front Artif Intell2025; 8: 1585260.
34.
MaJ.Military reinforcement learning with large language model–based agents: a case of weapon selection. J Def Model Simul. Epub ahead of print 15March2025. DOI: 10.1177/15485129251323291.
35.
SchaddMSternheimAMBlankendaalR, et al. How a machine can understand the command intent. J Def Model Simul2025; 22: 41–58.
Supplementary Material
Please find the following supplemental material available below.
For Open Access articles published under a Creative Commons License, all supplemental material carries the same license as the article it is associated with.
For non-Open Access articles published, all supplemental material carries a non-exclusive license, and permission requests for re-use of supplemental material or any part of supplemental material shall be sent directly to the copyright owner as specified in the copyright notice associated with the article.
