Predicting time to clearance of sport-related concussions using machine learning

Abstract

Objective

To evaluate whether integrating longitudinal clinical data improves machine learning (ML)-based prediction of time to medical clearance following sport-related concussion (SRC) and to identify clinical features most strongly associated with classification of either ‘prolonged’ recovery (≥ 30 days) or ‘normal’ recovery (< 30 days).

Methods

A retrospective cohort of 217 athletes (mean age 26.94 years) from the USF Concussion Center (2021–2025) was analyzed. Six ML classifiers were trained on Visit 1 features (n = 48) and combined Visit 1 + Visit 2 features (n = 95). Internal validation was performed using Leave-One-Out Cross-Validation (LOOCV).

Results

Prolonged recovery occurred in 81.1% of the cohort. Adding Visit 2 features improved accuracy in 66% of models, with XGBoost achieving the highest accuracy (0.84, +5% gain over Visit 1). Specificity remained low (0.00–0.34) due to class imbalance. VOR Vertical Headache and its change score were the most frequent predictors of prolonged recovery, present in 81% and 100% of models, respectively. Treatment presence between visits emerged as the strongest predictor of normal recovery.

Conclusions

Longitudinal clinical data modestly improves ML-based SRC recovery predictions. Vestibulo-oculomotor symptoms - particularly headache provoked during vertical VOR testing - are robust prognostic indicators. These findings support the utility of granular VOMS subscores for early risk stratification and targeted rehabilitation. External validation is required before clinical deployment. Code: https://github.com/MeganTran6023/Sport-Related-Concussions_Machine-Learning. IRB: USF STUDY003514.

Introduction

Recent trends show an annual increase in sports participation in the United States, and subsequently an increased potential for concussions like traumatic brain injury (TBI). ¹ Specifically, given the increased exposure from sports participation, current estimates suggest that over 3.8 million concussions occur in the US annually, a steady increase over previous decades (i.e., 1.7 concussions per 10,000 athlete exposures in 1988–1989 to 3.4 in 2003–2004 ² and to 4.47 in the period between 2009 and 2014. ³ This problem is further exacerbated by the fact that estimations also believe that up to 50% of concussions go unreported. ¹ As such, it is essential that licensed medical practitioners (e.g., physicians, neurologists, neuropsychologists, and emergency medicine specialists) accurately diagnose TBIs following suspected incidence. To do so they utilize a combinatfion of tests, such as Balance Error Scoring System (BESS) ⁴ and Vestibular Ocular Motor Screening (VOMS) ⁵ to assess cognitive, observational, and visual outcomes related to TBI symptoms. ⁶ However, these preliminary assessments are typically only sufficient to diagnose TBIs without being able to adequately predict a player’s time to clearance. ⁷ Fortunately, the use of machine learning (ML) in the healthcare field allows for improvement to diagnostic capabilities, while also allowing for improved prediction modalities for longitudinal prognosis (e.g., time to clearance). ⁸ These ML methods can be implemented using data collected at the time of the original diagnosis or can combine longitudinal data to improve prognosis capabilities. ⁹

The objective of this study was to utilize machine learning models classifying recovery duration for medical clearance (i.e., return to sport) following sport-related TBI. This is done via the integration of data from gold-standard clinical assessments (e.g, BESS, ⁴ VOMS, ⁵ ImPACT, ¹⁰ etc.) collected across multiple time points from licensed medical practitioners. Specifically, the study aimed to evaluate whether the predictive accuracy of time to clearance improves given longitudinal data, while also identifying and quantifying the specific assessment features that most strongly drive predicting time to clearance. By identifying assessment features that demonstrate a predictive signal, this work provides exploratory insights that could inform the future development of evidence-based protocols for clinical and applied contexts. Ultimately, this study will support a framework for longitudinal monitoring of TBI related to both individualized return-to-play decisions and broader clinical guidelines. This study is primarily comparative in scope: rather than proposing a single deployable clinical tool, it benchmarks six ML classifiers across longitudinal data to identify which model architectures and clinical features best support future development of a validated prediction tool. External validation and prospective testing are required before any model described here could be considered for clinical deployment. The intended target population for a future validated version of this model consists of athletes diagnosed with sport-related concussion (SRC) presenting to a sports medicine or concussion specialty clinic within one year of injury. The intended users are licensed clinicians experienced in concussion management (e.g., sports medicine physicians, neurologists, athletic trainers), who would use model output to supplement—not replace—clinical judgment regarding return-to-play timelines (TRIPOD+AI Item 3b).

Related works

Gold standard concussion assessments

In sports-related activities, athletes are at risk of the acute effects of concussion. ¹¹ This includes decreased verbal/visual memory and processing speed during the acute time period defined as 1 – 14 days post-concussion. ¹² Consequently, these impairments translate to declines in both cognitive outcomes and athletic performance. ¹¹ Because TBI affects several aspects of an athlete’s mental and physical abilities, multiple cognitive assessments have been used for screening/diagnosing TBIs. These tools are used to evaluate cognitive, observational, and visual outcomes related to TBIs (Appendix Table 10).

While it is recommended that a variation of the Sport Concussion Assessment Tool (SCAT) is used in any TBI assessment, given its multi-modal approach to concussion screening, ¹³ it is also common for other tests (e.g., King Devick for oculomotor assessment) to be administered in conjunction with the SCAT, so as to increase the overall feature set. ¹³ Similarly, while there are several versions of these assessments, many of them have modifications (e.g., Balance Error Scoring System (BESS) and modified BESS (mBESS)) which may provide changes in sensitivity for the assessment of different populations/conditions. ¹⁴ Interestingly, the ImPACT Test battery also uses Post-Concussion Symptom Scale (PCSS), which explains why some administration cases do not use both tests together for concussion diagnosis and evaluation.^15,16 Neurocognitive testing is incorporated to better identify concussion in athletes after an injury as solely depending on symptoms is not sufficient enough for proper diagnosis. ¹⁷ PCSS is a computerized neurocognitive test many health care professionals use to determine the number and severity of symptoms an athlete experiences following a concussion. ¹⁸ Langevin et al. found that the assessment only indicates a low to moderate degree of correlation between the frequency of symptoms reported by a concussed athlete using the PCSS test and the Dizziness Handicap Inventory, Headache Disability Inventory, and Neck Disability Index. ¹⁹ Symptom assessments are crucial in concussion testing, as understanding an individual’s specific profile enables clinicians to implement a personalized recovery plan. ²⁰ Common physical symptoms, such as headaches, dizziness, and light sensitivity, often appear immediately following the injury. It is imperative to record a patient’s symptoms for multiple reasons including establishing a baseline and tracking its progression if recorded longitudinally.^21–23

Machine learning for health prediction

Given the increased application of ML in healthcare, supervised learning has been extensively deployed as a means for processing neurological disease-based datasets to output explainable results. ²⁴ This is possible due to the large amount of digitally-available data (e.g., from gold standard assessments, digital devices, patient reported outcomes, etc.) which relate to different neuro-cognitive functions (e.g., motor, memory, and executive function) of interest. ²⁵ The presence of these types of data are extensively impactful for the standardization of diagnosing TBIs. ²⁶ However, many of these tools, focused on diagnosing TBIs, only account for a single time point of data which makes it difficult to predict future outcomes. ²⁷ Further, as there are various types of features (e.g., binary responses, continuous values, etc.) across multiple neuro-cognitive domains, ML is necessary to employ for analyzing all underlying patterns. ²⁴ Consolidating ML with existing evaluation tools allows for clinicians to capture complex relationships between various assessment features that are not easily identifiable. ²⁸ This integration can improve clinical and patient outcomes by enabling earlier, more accurate diagnosis, personalized prognosis, and data-driven treatment planning that supports timely interventions and informed clinical decision-making. ²⁹ Recent methodological literature in the TBI and concussion domain has increasingly focused on enhancing model robustness through advanced deep learning frameworks and validation strategies. For instance, Ref. 30 demonstrated how convolutional neural network (CNN) architectures and robust preprocessing—such as addressing class imbalance through oversampling—can improve classification accuracy across temporal stages of pathology, offering a valuable perspective on the validation required for clinical translation.

Bergeron et al. found that models Naive Bayes and Random Forest were top performers in predicting concussion resolution in high school athletes within 7, 14, or 28 days. ³¹ Top features that drove this high model performance includes difficulty concentrating, sensitivity to light/noise, and balance issues. ³¹ Chu et al. reported that the CatBoost model outperformed traditional statistical methods in both predictive and discriminative aspects when predicting concussion recovery time and protracted recovery after using clinical data involving the Vestibular Ocular Motor Screening (VOMS), King-Devick Test, and the C3 Logix Trails Test. ³² Thomas and Arnett found that their Random Forrest model performed the best in classifying concussed college athletes recovery timeframes as typical (≤ 28 days) or prolonged (>28 days). ³³ These recent, impactful studies add to existing literature with the use ML models to capture nonlinear relationships inside complex datasets to properly determine recovery timeframes for concussed patients. ³¹ Thomas and Arnett’s study highlight an issue from human-driven statistical method that ML models overcome which is to use past data for prospective concussion recovery predictions. ³³ Furthermore, Chu et al. models required less features than traditional methods to accurately predict concussion recovery time—specifically they used 11 features for prediction while traditional models used 25–27 features. ³²

Methodology

This study was reported in accordance with the TRIPOD+AI checklist. ³⁴ The completed checklist is provided in the Appendix, with locations indicated by page number.

Cohort

The dataset utilized in this study was provided from the University of South Florida (USF) Concussion Center via the USF Research Electronic Data Capture (REDCap) server. The study population consisted of 3,038 patients diagnosed with a concussion between 2017 and 2026 at USF facilities. This dataset includes patient data with multiple visits collected from 2021 to 2025. Within this dataset, rows represent each patient with respective visit information while columns represent a combination of patient intake and examination data. Within the full USF Concussion Center database, patients were categorized based on one of four mechanisms of injury causing concussion (i.e., Sports Related Concussions (this study), Motor Vehicle Accidents, Falls, and Other – represented by assaults, collision with random impediments, etc. Initially, the full dataset included 3038 unique patient records. Following data preprocessing (as further depicted in the subsequent section), 217 unique patients with concussion remained for ML analysis. Of the 217, 80 (36.9%) are males and 137 (63.1%) are females, with an average age of 26.94 years. Treatment Present is a binary variable (0/1) indicating whether a patient received any form of treatment following their initial visit. Treatment types span both pharmacological and non-pharmacological approaches: Selective Serotonin Reuptake Inhibitors (SSRI), amantadine, stimulants, preventative headache medication, vestibular therapy, physical therapy, chiropractic care, psychological treatment, neuropsychological treatment, neurology treatment, and cognitive therapy. Of the 217 patients, 168 received at least one form of treatment Treatment Present = 1) and 49 did not (Treatment Present = 0). Anxiety and depression are the most common mood disorders in the dataset, where the prevalence of depression ranges from 6% to 34% ³⁵ and the prevalence of anxiety is 46.72% in youth athletes. ³⁶ GAD-7 Score (General Anxiety Disorder-7): This was used to quantify anxiety status at the time of the visit. PHQ-9 Score (Patient Health Questionnaire-9): This was used to quantify depression/mood status at the time of the visit.

To account for the ‘treatment_present’ variable, we included features for Selective Serotonin Reuptake Inhibitors (ssri_tx), psychological treatment (psych_tx), neuropsychological evaluation or treatment (neuropsych_tx), and cognitive therapy (cognitive_tx), each of which was coded as a binary value (0/1) to indicate whether the patient received the treatment during their single intake visit.

Finally, clearance is determined by being either asymptomatic or back to baseline levels of symptoms that were present pre-injury in addition to normal functional measures based on VOMS, BESS, CNS vital signs, return to school. Physicians experienced in the diagnosis and management of concussions make that determination.

Data preprocessing

To clean the dataset with 3038 unique patients, a multi-step iterative process was employed, beginning with the establishment of specific cleaning standards: a column missingness threshold of 0.9, a row missingness threshold of 0.8, and a threshold step of 0.1 for subsequent rounds. During the step-by-step cleaning phase, the percentage of missing values was calculated for every row and column, leading to the removal of any records or features that exceeded the established thresholds. Following each removal cycle, the standards were adjusted by the threshold step and the process was repeated until the data stabilized, resulting in an intermediate dataset of 2,338 rows and 49 columns. Finally, outliers were addressed by filtering for clinically relevant timelines, specifically retaining records where the duration to the first visit was between 0 and 365 days and the duration to clearance was at least 1 day. This rigorous refinement process produced a final dataset of 1,865 rows and 49 essential columns such that the full data frame includes no null values. This process was chosen to avoid the use of data imputation, as imputation in the medical data tends to lead to bias and it is noted that there are still no optimal imputation solutions in the medical domain.^37,38 After all preprocessing steps filtering for patients with two hospital visits, this resulting dataset included 1201 unique patients with 217 of the total with sports-related concussions.

Subsequently, both feature inclusion and engineering was performed to acknowledge clinically relevant variables related to either time instances and/or longitudinal changes experienced by patients. For feature inclusion ‘prior head injury’ and ‘history of mood disorders’ were selected for inclusion for Visit 1, whereas ‘prior head injury’, ‘history of mood disorders’, and ‘treatment presence’ were selected for inclusion for Visit 2 (i.e., as treatment was not formally administered until after the collection of data in the first visit). In addition, difference features were engineered for Visit 2 by subtracting Visit 1 values from corresponding Visit 2 values for all shared base features, excluding ‘prior head injury’, ‘history of mood disorders’, and ‘treatment presence’, to provide the models with information related to longitudinal changes between visits. This process resulted in 49 features for Visit 1 and 95 features for Visit 2. Data for patients with 2+ visits was included–highlighting only the baseline and second visit–as there were not enough patients with 3 or more visits within the normal one month time to clearance classifier.

Machine learning

Each model were trained separately for each visit using the same hyperparameter tuning, training, and testing pipeline. Visit 1 models used the clinical dataset that included new features prior head injury and history of mood disorders in addition to the original features remaining from preprocessing. Visit 2 models used the clinical dataset that included new features prior head injury, history of mood disorders, and treatment presence, the original features remaining from preprocessing, as well as the difference variants of the original features. The model’s prediction calculations are expressed in this section via Equations 1-9 (TRIPOD+AI Item 12g,22).

Light Gradient Boosting Machine (LGBM) is a Gradient Boosting Decision Tree that incorporates techniques Gradient-based One-Side Sampling (GOSS) and Exclusive Feature Bundling (EFB). This ML model optimizes the number of features to focus on to quickly and accurately make predictions using a small dataset while reducing memory usage. ³⁹ Similar to other gradient boosting frameworks, LGBM constructs an additive model by minimizing a regularized objective function of the form

$L={\sum }_{i=1}^{n}l\left(y_i,{\widehat{y}}_i\right)+{\sum }_{k=1}^K\mathrm{\Omega }\left(f_k\right)$

(1)

In the objective function, n denotes the number of training samples, y _i represents the true label for the ith sample, and ${\widehat{y}}_i$ is the corresponding model prediction. The function l(·) denotes the loss function measuring prediction error. The term f _k represents the kth decision tree in the ensemble, where K is the total number of trees. The regularization term Ω(f _k ) penalizes model complexity by constraining tree depth, number of leaves, and leaf weights, thereby reducing overfitting. By leveraging GOSS to retain instances with large gradients and EFB to bundle mutually exclusive features, LGBM efficiently optimizes this objective, making it well suited for achieving the study objectives when working with limited data and computational resources.

Decision Tree Classifier applies a divide and conquer algorithm for each feature to determine the most optimal order of splits that best capture nonlinear patterns in a dataset. ⁴⁰ Partitioning the problems into binary sub-outputs makes it easy to track what order of features leads to the prediction of a patient’s time to clearance that results in the highest predictive accuracy. ⁴¹ Three splitting criteria choices used for this model, along with their respective mathematical formulations, are as follows:

a) Gini Impurity

$G\left(S\right)=1-{\sum }_{k=1}^K{p}_k^2$

(2)

• b) Entropy

$H\left(S\right)=-{\sum }_{k=1}^K{p}_k\mathit{log}\left(p_k\right)$

(3)

• c) Log Loss

$Loss=-\frac{1}{N}{\sum }_{i=1}^N\left[y_i\hspace{0.17em}\mathit{log}\left(p_i\right)+\left(1-y_i\right)\mathit{log}\left(1-p_i\right)\right]$

(4)

In the splitting criteria equations, S denotes the set of samples at a given node, and K is the total number of classes. The term p _k represents the proportion of samples in S that belong to class k. For log loss, N denotes the total number of samples, y _i is the true binary class label for sample i, and p _i is the predicted probability that the sample belongs to the positive class. These criteria quantify node purity and guide the selection of optimal feature splits. A clarification statement to be made is that the output of the models are the binary class labels 0 (‘normal’ recovery) or 1 (‘prolonged’ recovery).

Random Forest Random Forest combines multiple decision trees to improve the generalizability of the model’s outputs. ⁴¹ Given that our dataset has 48 features and 97 features for initial and second visits, respectively, combining multiple decision trees provides a holistic understanding of feature influence on the model’s high accuracy rather than relying on a single order of splits.

Formally, Random Forest predictions can be analyzed using the margin function, defined as

$m\left(X,Y\right)=P_{\mathrm{\Theta }}\left(h\left(X,\mathrm{\Theta }\right)=Y\right)-\max \hspace{0.17em}{P}_{\mathrm{\Theta }}\left(h\left(X,\mathrm{\Theta }\right)=j\right),$

(5)

which measures how much more strongly the ensemble supports the correct class compared to the most competitive incorrect class, reflecting prediction confidence.

The corresponding generalization error is given by

$PE=P_{X,Y}\left(m\left(X,Y\hspace{0.17em}\right)\hspace{0.17em}<\hspace{0.17em}0\right),$

(6)

which represents the probability that the ensemble misclassifies an input, linking model accuracy to the strength and diversity of the individual trees. In the margin function, X represents the input feature vector, Y denotes the true class label, and h(X, Θ) is the prediction of an individual tree parameterized by random variables Θ, which control feature selection and bootstrapped sampling. The probability P_Θ(·) is taken over the ensemble of trees. The generalization error PE measures the likelihood that the ensemble predicts an incorrect class for a randomly drawn input-output pair (X, Y).

Support Vector Classifiers (SVCs) SVCs perform well with small datasets since its time and space complexity is proportional to the input dataset size. ⁴² SVCs are a specific type of Support Vector Machines (SVMs). For classification problems, SVMs attempt to linearly separate points from a dataset into distinct groups on a feature space of higher dimension. ⁴³ This separation is achieved by solving the following optimization problem:

${\mathit{min}}_{w,b,\xi }\hspace{0.5em}\frac{1}{2}\parallel w{\parallel }^2+C{\sum }_{i=1}^n{\xi }_i$

(7)

In the optimization problem, w represents the weight vector defining the separating hyperplane, and b is the bias term. The slack variables ξ _i quantify the degree of misclassification for the ith sample. The regularization parameter C controls the trade-off between maximizing the margin and minimizing classification error. A larger C places greater emphasis on correctly classifying training samples, while a smaller C allows for a wider margin with increased tolerance for misclassification. Given its practicability with small datasets, this model was utilized as part of this study.

XGBoost This model is a widely used implementation of gradient tree boosting that is designed for scalability, efficiency, and strong performance, even when working with sparse datasets. Its effectiveness stems from greedy optimization in which each new tree is trained to model the residual errors left by the previous ensemble of trees. By iteratively reducing residuals, the model progressively improves its predictions, leading to a robust and highly accurate boosting framework. ⁴⁴ Formally, XGBoost minimizes the following regularized objective function where n denotes the number of training samples, y _i is the true label, and ${\widehat{y}}_i$ is the predicted output for sample i.

$L={\sum }_{i=1}^{n}l\left(y_i,{\widehat{y}}_i\right)+{\sum }_{k=1}^K\Omega \left(f_k\right)$

(8)

The loss function l(·) measures prediction error, while f _k denotes the kth tree in the ensemble, with K being the total number of trees. The regularization term Ω(f _k ) penalizes model complexity by incorporating constraints on tree structure and leaf weights, improving performance.

Ridge Regression This is useful for addressing the collinearity problem that occurs in general linear regression analyses without dropping any of the features from the original set of independent variables. ⁴⁵ The equation used introduces an ℓ₂ regularization term into the standard least-squares objective and is defined as

$\widehat{\beta }=\mathit{arg}{\mathit{min}}_{\beta }\left(\parallel y-X\beta {\parallel }^2+\lambda \parallel \beta {\parallel }^2\right)$

(9)

In the optimization expression, X denotes the design matrix of input features, y is the vector of observed class labels, and β represents the regression coefficients. The regularization parameter λ controls the strength of the ℓ₂ penalty, with larger values enforcing greater shrinkage of coefficients. This constraint reduces variance and stabilizes coefficient estimates in the presence of multicollinearity. This way, each feature has some impact on the prediction of the binary classes for our dataset which enables us to analyze their individual contribution to the prediction’s accuracy.

Study design

As ML models are intended to predict whether each patient’s time to clearance is within a month or over a month we used a binary encoding method to classify these states. Specifically, this study treats recovery prognosis as a binary classification task of recovery duration, where the models differentiate between ‘normal’ (< 30 days) and ‘prolonged’ (≥ 30 days) recovery timelines. Class 0 denotes recovery within a month and Class 1 denotes recovery over a month. One month was selected as the threshold since this is the general clinical recovery timeframe for sports-related concussions. ⁴⁶ This was completed to not only evaluate if the models can predict the time to clearance, but also demonstrate increases in prediction accuracy comparing between first and second visit datasets (TRIPOD+AI Item 8a).

Leave One Out Cross Validation (LOOCV) was utilized to determine the model’s accuracy for predicting the patients’ time to medical clearance for both initial and second visits. This method was selected due to the dataset’s small size. ⁴⁷ For LOOCV with a dataset of size N, one data point is used in the training set while the remaining N - 1 data points is the testing set. This procedure is repeated until each data point has been used once as the training instance. ⁴⁸

No patients or members of the public were involved in the design, conduct, reporting, or dissemination plans of this research.

ML Model Hyperparameter Tuning LightGBM, Decision Tree, Random Forest, SVC, and XGBoost were hyperparameter tuned using an 80:20 train–test split. For these models, hyperparameters were optimized using randomized search via ParameterSampler, ⁴⁹ where 50 distinct parameter combinations were randomly sampled from predefined distributions; the configuration achieving the highest validation accuracy was selected. This number of combinations was selected given a performance plateau. In contrast, the Ridge classifier was tuned using k-fold cross-validation, where the data were split into k folds, the model was trained on k − 1 folds and validated on the remaining fold. This process was repeated k times, the validation scores were averaged across folds for each candidate alpha (α). Subsequently, the α with the highest average score was selected and the final Ridge model was re-fit on the full dataset using this optimal α. Since the Ridge model only had one hyperparameter to tune (i.e. α), its hyperparameter tuning process differed from those of the other models implemented in the paper. ⁵⁰ Details for hyperparameter tuning for each model are explained in the following paragraphs.

LightGBM - Hyperparameter tuning used a randomized search with n_iter = 50 over a predefined parameter distribution. The search space included num_leaves in {31, 50, 70}, max_depth in {-1, 10, 20, 30}, learning_rate in {0.01, 0.05, 0.1, 0.2}, n_estimators in {100, 200, 500, 1000}, min_child_samples in {10, 20, 30, 50}, subsample in {0.6, 0.8, 1.0}, and colsample_bytree in {0.6, 0.8, 1.0}. Parameter combinations from ParameterSampler were evaluated in a manual loop to select the best-performing configuration.

Decision Tree - Hyperparameter tuning was performed via randomized sampling over a defined grid with n_iter = 50. The search space included criterion in {gini, entropy, log_loss}, max_depth in {None, 5, 10, 20, 30}, min_samples_split in {2, 5, 10}, and min_samples_leaf in {1, 2, 5}. Sampled parameter sets were evaluated iteratively to identify the best validation accuracy.

Random Forest - Randomized hyperparameter search with n_iter = 50 was applied using sampled combinations from a parameter distribution that included n_estimators in {100, 200, 500}, criterion in {gini, entropy}, max_depth in {None, 10, 20, 30}, min_samples_split in {2, 5, 10}, min_samples_leaf in {1, 2, 5}, and max_features in {sqrt, log2}. Each sampled configuration was trained and evaluated to determine the best-performing model.

XGBoost - Hyperparameter tuning used a randomized search with n_iter = 50. The sampled search space included max_depth in {3, 5, 7}, learning_rate in {0.05, 0.1, 0.2}, n_estimators in {50, 100, 150}, subsample in {0.7, 0.85, 1.0}, colsample_bytree in {0.7, 0.85, 1.0}, gamma in {0, 0.1, 0.3}, min_child_weight in {1, 3, 5}, reg_alpha in {0, 0.1, 0.5}, and reg_lambda in {0.5, 1.0, 1.5}. Parameter sets were sampled and evaluated in a manual random search loop.

Support Vector Classifier (SVC) - Hyperparameter tuning was conducted using randomized sampling over a discrete parameter set. The search space consisted of C in {0.1, 10, 100} and gamma in {1, 0.1, 0.01}. The experiment was restricted to a linear kernel, as this was the only configuration allowing feature importance extraction. Due to limited computational resources, 27 sampled combinations were evaluated iteratively, and the configuration yielding the highest accuracy was selected.

Ridge Regression/Ridge Classifier - Hyperparameter tuning targeted the regularization strength alpha. A candidate set alpha in {0.001, 0.01, 0.1, 1, 5, 10, 15, 20} was used. For some experiments, a random subset of these values (up to n_iter = 50) was sampled and evaluated using k-fold cross-validation, with the best-performing alpha selected based on validation performance.

Class Imbalance The dataset exhibits substantial class imbalance: 176 patients (81.1%) belong to the ‘prolonged’ recovery class (Class 1) versus 41 patients (18.9%) in the normal recovery class (Class 0). No resampling techniques (e.g., SMOTE, random oversampling) were applied in the primary analysis, as the authors judged that synthetic data generation could misrepresent the true clinical distribution. ⁵¹ Instead, class weighting was incorporated into the pipeline to penalize misclassification of the minority class (Class 0) proportionally to class frequency. The effect of this imbalance is reflected in near-zero specificity values in the unweighted pipeline and is discussed as a primary limitation of the reported accuracy and recall metrics (TRIPOD+AI Item 13).

Feature Importance (FI) To identify and quantify which assessment features most strongly influence predictions of time to clearance, FI analyses were conducted using methods appropriate to each model type on a final refitted model. For tree-based models (i.e., Decision Tree, Random Forest), importance was derived from the reduction in Gini impurity attributable to each feature. Specifically, these FI values in addition to their respective Gini impurity values were extracted from each trained tree-based model.^52,53 Conversely, for gradient-boosted models (i.e., LightGBM, XGBoost) importance was based on gain, defined as the total reduction in loss from splits using a given feature. ⁵⁴ For linear SVC, FI was quantified using the absolute magnitude of the learned coefficients, reflecting each feature’s influence on the decision function. ⁵⁵ For Ridge, permutation importance was derived by measuring the decrease in predictive accuracy resulting from random permutation of individual features. ⁵⁶ Appendix Table 11 denotes formal feature names related to specific features of importance.

Results

All six classifiers produce binary class labels as their primary output: Class 0 denotes predicted ‘normal’ recovery (<30 days to medical clearance) and Class 1 denotes prediction of ‘prolonged’ recovery (≥30 days). Predicted class probabilities are available for probabilistic models (LightGBM, XGBoost, Random Forest, Decision Tree, Ridge) and can be inspected via the public repository. Classification thresholds were set at 0.5 for all models; threshold optimization (e.g., via ROC analysis) was not performed at this stage given the exploratory scope of the study (TRIPOD+AI Item 15).

ML Model Metrics The classification of a TBI patient’s time to clearance was completed to gain additional insights on assessment features that drive high predictive accuracy. Table 1 presents the accuracy (i.e., proportion of all correct predictions), precision (i.e., proportion of predicted positives that are true positives), recall (i.e., proportion of actual positives (Class 1) correctly identified), F1 (harmonic mean of precision and recall, reflecting their balance), and specificity (i.e., proportion of actual negatives (Class 0) correctly identified) for each model’s performance on both visits. Model performance was also evaluated using balanced accuracy (accounts for class imbalance by averaging sensitivity and Specificity), MCC (provides a single-value summary robust to class imbalance), and the Brier score (measures probabilistic calibration). No clinical utility analysis (e.g., decision curve analysis) was performed at this stage, as the study is comparative and exploratory rather than proposing a deployment-ready tool. Bootstrap 95% confidence intervals (1,000 resamples) are reported for all metrics to quantify estimation uncertainty (TRIPOD+AI Item 12e). Given only Visit 1 data, SVC and Random Forest performed the best with accuracies of 0.81, recall of 1.00, F1-score of 0.9 for both models. Precision was highest from both LightGBM and XGBoost for Visit 1 data at 0.84. With the introduction of Visit 2 data, XGBoost provided the highest accuracy (0.84), recall (0.99), and F1-score (0.91) with an adequate precision metric of 0.84. However, Decision Tree had the best precision of 0.86. Overall, specificity values were low with LightGBM performing the best using only Visit 1 data (0.29) whereas Decision Tree performed the best when expanding the dataset to include Visit 2 features (0.34). This could be attributed to the small dataset used for both visits as there were more samples for class 1 than in class 0. As a result, ML models are prone to over predict for the dominant class to optimize a single summary metric (e.g. overall error rate). ⁵⁷ The event-per-variable (EPV) ratio for models in the first visit is 0.84 and 0.43 for second visit.

Table 1.

Visit 1 – model performance with 95% bootstrap confidence intervals.

Model	Accuracy	Balanced Accuracy	Precision	Recall	F1	Specificity	MCC	Brier
LightGBM	0.77	0.57	0.84	0.89	0.87	0.29	0.23	0.16
	[0.77, 0.87]	[0.51, 0.63]	[0.78, 0.88]	[0.94, 0.99]	[0.86, 0.93]	[0.05, 0.29]	[0.04, 0.39]	[0.12, 0.21]
Dec. Tree	0.77	0.61	0.82	0.85	0.83	0.17	0.23	0.17
	[0.71, 0.83]	[0.53, 0.68]	[0.79, 0.90]	[0.82, 0.92]	[0.82, 0.90]	[0.19, 0.49]	[0.07, 0.38]	[0.14, 0.21]
Rand. For.	0.81	0.50	0.81	1.00	0.90	0.00	0.00	0.15
	[0.76, 0.86]	[0.50, 0.50]	[0.76, 0.86]	[1.00, 1.00]	[0.86, 0.93]	[0.00, 0.00]	[0.00, 0.00]	[0.12, 0.18]
XGBoost	0.79	0.58	0.84	0.95	0.89	0.22	0.26	0.15
	[0.77, 0.87]	[0.52, 0.65]	[0.78, 0.89]	[0.94, 0.99]	[0.86, 0.93]	[0.08, 0.32]	[0.08, 0.42]	[0.12, 0.19]
SVC	0.81	0.50	0.81	1.00	0.90	0.00	-0.03	0.16
	[0.75, 0.86]	[0.49, 0.50]	[0.76, 0.86]	[0.98, 1.00]	[0.86, 0.92]	[0.00, 0.00]	[-0.06, 0.00]	[0.13, 0.19]
Ridge	0.80	0.51	0.81	0.97	0.88	0.02	0.03	0.18
	[0.74, 0.84]	[0.48, 0.55]	[0.76, 0.87]	[0.94, 0.99]	[0.85, 0.91]	[0.00, 0.12]	[-0.09, 0.18]	[0.16, 0.19]

Note: Values within brackets represent the 95% confidence intervals.

Notably, the integration of class weighting yielded superior specificity across all models compared to the initial machine learning implementation. By assigning greater weights to minority class observations, this strategy mitigated the inherent bias toward the majority class, ensuring that class imbalance did not impede the predictive integrity of the models.

FI Scores and Average Effect on Predictive Outcome Listing out the top 20 features that influence each model’s prediction is important for knowing which assessments to use for future TBI identification and treatment. To further understand the specific effect each of the top 20 features has on the model’s predictive output, plotting their average effects for each target class as well as their FI scores was executed. Results for the top 20 features by model and visit number are presented in Tables 1 and 2. Imp. represents Importance Scores in these tables. Additionally, to aid in identifying high-yield assessments in longitudinal modeling, we quantified FI by examining both rank order and selection frequency across models. This is displayed in Tables 3 and 4. From these tables several key features are highlighted. From Visit 1, four key features (i.e., NPC Headache, VMST Dizziness, VMST Headache, and VOMS Headache) show up across all 6 models (Table 3) with high feature importance scores as presented in Table 2. Similarly, expanding to the Visit 2 feature set, three key features (i.e., VOR Vertical Headache, Treatment Present, and VOR Vertical Headache Difference) show up across all 6 models (Table 4) with high feature importance scores presented in Table 1.

Table 2.

Counts of specific treatment received.

Treatment column	Count
amantadine_tx	17
stimulant_tx	15
preventative_headache_tx	46
vestibular_tx	86
pt_tx	68
chiro_tx	66
psych_tx	31
neuropsych_tx	21
neurology_tx	5
cognitive_tx	8

Table 3.

Visit 2 – model performance with 95% bootstrap confidence intervals.

Model	Accuracy	Balanced Accuracy	Precision	Recall	F1	Specificity	MCC	Brier
LightGBM	0.82	0.58	0.84	0.91	0.88	0.24	0.31	0.14
	[0.78, 0.88]	[0.53, 0.64]	[0.78, 0.88]	[0.97, 1.00]	[0.87, 0.93]	[0.06, 0.29]	[0.13, 0.47]	[0.11, 0.17]
Dec. Tree	0.80	0.56	0.86	0.91	0.88	0.34	0.13	0.17
	[0.69, 0.80]	[0.49, 0.64]	[0.78, 0.88]	[0.81, 0.91]	[0.80, 0.88]	[0.13, 0.42]	[-0.02, 0.29]	[0.14, 0.21]
Rand. For.	0.82	0.57	0.83	0.98	0.90	0.15	0.25	0.14
	[0.77, 0.87]	[0.51, 0.63]	[0.78, 0.88]	[0.96, 1.00]	[0.87, 0.93]	[0.05, 0.26]	[0.07, 0.41]	[0.11, 0.17]
XGBoost	0.84	0.60	0.84	0.99	0.91	0.17	0.27	0.15
	[0.76, 0.86]	[0.53, 0.68]	[0.79, 0.89]	[0.90, 0.97]	[0.86, 0.92]	[0.14, 0.42]	[0.09, 0.42]	[0.12, 0.18]
SVC	0.81	0.50	0.81	0.99	0.89	0.00	0.00	0.15
	[0.76, 0.86]	[0.50, 0.50]	[0.76, 0.86]	[1.00, 1.00]	[0.86, 0.93]	[0.00, 0.00]	[0.00, 0.00]	[0.12, 0.19]
Ridge	0.80	0.50	0.81	0.97	0.89	0.05	0.01	0.18
	[0.71, 0.83]	[0.46, 0.55]	[0.76, 0.86]	[0.89, 0.97]	[0.83, 0.90]	[0.00, 0.17]	[-0.12, 0.17]	[0.16, 0.20]

Note: Values within brackets represent the 95% confidence intervals.

Table 4.

Statistical significance of accuracy performance gains across visits per model.

Model	Visit 1 accuracy	Visit 2 accuracy	Gain [95% confidence interval]	Significant
XGBoost	0.79	0.84	0.051	True
			[0.023, 0.083]
LightGBM	0.77	0.82	0.045	True
			[0.018, 0.074]
Dec. Tree	0.77	0.80	0.028	True
			[0.009, 0.051]
Rand. For.	0.81	0.82	0.009	False
			[0.000, 0.023]
SVC	0.81	0.81	0.000	False
			[0.000, 0.000]
Ridge	0.80	0.80	0.000	False
			[0.000, 0.000]

Further, Figures 1–3 present the average predictive effect for each of the top 20 features per model and visit. In these figures, the blue bars predict towards Class 0 and red bars predict towards Class 1. For the identified features from Visit 1 (i.e., NPC Headache, VMST Dizziness, VMST Headache, and VOMS Headache), Figures 1–3 highlight that these features tend to predict towards Class 1 with 19 of 24 instances (79.17%) of these features being highlighted in red. Similarly, for Visit 2, the VOR Vertical Headache and VOR Vertical Headache Difference features predicted towards Class 1 (i.e., red) across 11 of 12 instances (91.67%). Conversely, for Visit 2, the treatment present feature predicted towards Class 0 (i.e., black) in all 6 models.

Figure 1.

Data preprocessing flowchart.

Figure 2.

Average effects – Visit 1 vs Visit 2. Blue bars predict towards Class 0 and red bars predict towards Class 1.

Figure 3.

Average effects – Visit 1 vs Visit 2. Blue bars predict towards Class 0 and red bars predict towards Class 1.

Discussion

The primary objective of this study was to evaluate the utility of machine learning (ML) models in predicting time to medical clearance via binary classification following sport-related traumatic brain injury (TBI), with a secondary aim of assessing whether the integration of longitudinal clinical data across two visits improves predictive accuracy compared to a single-visit model. To accomplish these goals, six ML models—LightGBM, Decision Tree, Random Forest, XGBoost, Support Vector Classifier, and Ridge Regression—were trained and evaluated using gold-standard clinical assessment data from the USF Concussion Center. Traditional statistical approaches commonly applied in concussion research, such as linear or logistic regression, are typically constrained by assumptions of linearity, independence, and normally distributed errors, which may not adequately capture the complex, nonlinear interactions inherent in multidimensional clinical datasets^32,33; In contrast, ML models are well-suited to handle high-dimensional feature spaces, mixed data types, and nonlinear relationships, while also offering the advantage of feature importance quantification to support interpretable, evidence-based clinical decision-making. ³¹ Furthermore, the feasibility of implementing ML in SRC research has been demonstrated across a growing body of work, including studies predicting concussion resolution within 7, 14, and 28 days using clinical symptom profiles, ³¹ recovery time using vestibular and oculomotor screening data, ³² and recovery trajectories in collegiate athletes. ³³ Building upon this foundation, the present study extends prior work by incorporating longitudinal data across two clinical visits and applying feature engineering—specifically, difference features capturing symptom change between visits—to further enhance the predictive framework and identify clinically actionable assessment targets for personalized return-to-play decision-making.

Model Performance Focusing on accuracy is important because it informs clinicians on properly diagnosing and treating patients. ⁵⁸ As such, highlighting improved accuracy when including secondary visit features is of interest. Following this inclusion 66% of the models demonstrated an increase in accuracy when including Visit 2 features with XGBoost yielding both the highest change in accuracy (i.e., from Visit 1 to Visit 2 prediction) as well as second visit accuracy alone (84%).

A reason behind the upwards trend in accuracy between initial and second visits in 66% of the ML models is attributed to applying domain specific feature engineering. Deriving new features from accounting for the interactions between multiple original features to enhance the models’ predictive performance as they provide additional context to said field. ⁵⁹ This is evident in the usage of difference features for the second visits dataset, which represents the difference between the second visit and initial visit values of the original features list and allow the model to make more correct predictions for both classes

(Figures 4–6). It is interesting to note that for Visit 2, models show an increase of correctly predictions for Class 0 compare to initial visits which may be due to the additional difference features and treatment presence in the dataset. Thus, this hints about these added features’ potential to be clinically significant. Also, the dataset consisted of 176 patients belonging in Class 1 and 41 patients in Class 0. From this, the model will have a high chance of correct predictions since it will overpredict for Class 1 (Figures 4–6). Table 5 supports this as the recall values are close to 1 while that of specificity is close to 0. While there is support for why specificity is close to 0, from a clinical standpoint, it is important to highlight the need for accurately identifying true negatives (e.g. Class 0) rather than only correctly classifying true positives (e.g. Class 1). By keeping TBI patients under medical treatment longer than needed, aspects such as cognitive and physical abilities may decline. ⁶⁰

Figure 4.

Average effects – Visit 1 vs Visit 2. Blue bars predict towards Class 0 and red bars predict towards Class 1.

Figure 5.

Confusion matrix – Visit 1 vs Visit 2.

Figure 6.

Confusion matrix – Visit 1 vs Visit 2.

Table 5.

Description of dataset between outcome groups.

Metric	‘Normal’ recovery (Class 0)	‘Prolonged’ recovery (Class 1)
Total Patients (N)	41	176
No Treatment	21	28
Treatment Present	20	148
Male Patients	17	63
Female Patients	24	113
Days to Clearance (Mean ± SD)	20.59 ± 5.33	98.18 ± 90.54
Days from Injury to First Visit (Mean ± SD)	6.98 ± 3.24	29.53 ± 50.52
Days from 1st to 2nd visit	12.56 ± 4.58	28.39 ± 19.36

XGBoost has the highest accuracy change between its Visit 1 and Visit 2 accuracy scores as well as has the highest accuracy for Visit 2 alone. This is attributed to the model’s majority voting combination method of base models and boosting ensemble learning method. ⁶¹ The accuracy score for XGBoost’s second visits aligns with work completed in Thomas et al. of 82.5% ⁶² regarding the differentiation of concussed versus healthy cohorts. Also, as overfitting is minimized for this model, this means it can generalize well to unseen data. These model characteristics allow XGBoost to successfully output accurate classifications. ⁶¹

The lack of improvement in accuracy across both visits for SVC and Ridge can be attributed to their shared limitation as linear models. While the linear kernel in SVC was applied due to its simple implementation on limited dataset sizes, ⁶³ and Ridge regression is inherently a linear method, ⁴⁵ neither is capable of capturing the complex, non-linear relationships between the input assessment features and the target classes.^64,65 The SVC accuracy scores observed in this study align with the approximately 81% accuracy reported by ⁶⁶ for detecting prior concussions in retired athletes, and the Ridge classifier’s predictive accuracy-measured by Mean Squared Error (MSE) and Mean Absolute Error (MAE)-is comparable to LASSO, identified as the top-performing model for predicting sports-related TBI in NCAA athletes. ⁶⁷ Despite these encouraging benchmarks, the additional difference-based features derived from the second visit provided little new information beyond what was already encoded in the original variables, yielding no meaningful gain in predictive accuracy across visits for either model.

Feature Importance and Average Effect The evident cross-model agreement (e.g., showing convergence of similar features in top 20 importance lists as well as their average effect on prediction for days to clearance) is present given the nature in which these rely on similar supervised learning methods. ⁶⁸ Not only is the feature similarity due to the model’s learning method, but there is a possible true relationship present between those features and the prediction of a patient’s time to clearance. ⁶⁹

Features that appear across all models denote that these must be assessed and incorporated into a TBI patient’s personalized treatment plan as these were found to be most important to the time to clearance prediction per the models’ results. Further, if adding a base feature and its difference variant together from the second visit dataset result in a total occurrence count of at least 9, then it is deemed to have high clinical significance. Specific features that reflect this are explained in detail in the following paragraphs (Tables 6–9).

Table 6.

Top 20 feature scores for machine learning models – Visit 1.

LightGBM		Decision Tree		Random Forest		XGBoost		SVC		Ridge
Feature	Imp.	Feature	Imp.	Feature	Imp.	Feature	Imp.	Feature	Imp.	Feature	Imp.
Cervical Flexion	169.712095	Saccades Horizontal Headache	0.101640	Cervical Flexion	0.104531	VOMS Headache	3.696006	VOMS Fogginess	0.132768	VOMS Fogginess	0.101229
Left Cervical Rotation	137.412696	VOR Horizontal Dizziness	0.075271	Left Cervical Rotation	0.104429	Saccades Horizontal Headache	2.603712	Saccades Horizontal Fogginess	0.066317	VMST Headache	0.075422
PHQ-9 Score	117.477276	Right Cervical Rotation	0.074313	Right Cervical Rotation	0.085136	VOR Horizontal Headache	1.998317	Smooth Pursuits Fogginess	0.066310	Saccades Horizontal Fogginess	0.028571
Left Lateral Flexion	113.081305	PHQ-9 Score	0.069096	PHQ-9 Score	0.081346	Left Cervical Rotation	1.726872	VMST Headache	0.002160	VOR Vertical Headache	0.023502
GAD-7 Score	89.292642	Previous Head Injury	0.067866	Left Lateral Flexion	0.064458	Cervical Flexion	1.720641	VOR Vertical Headache	0.002008	VOMS Headache	0.016436
Right Cervical Rotation	81.066041	Left Lateral Flexion	0.067476	Cervical Extension	0.057082	VMST Dizziness	1.582768	Smooth Pursuits Dizziness	0.001783	Saccades Vertical Dizziness	0.015515
Cervical Extension	61.320671	VOR Vertical Dizziness	0.066531	GAD-7 Score	0.051770	Right Cervical Rotation	1.547318	VOMS Dizziness	0.001707	VMST Dizziness	0.013057
NPC Headache	34.372974	Saccades Vertical Fogginess	0.061698	VOR Horizontal Headache	0.029481	NPC Headache	1.426695	NPC Headache	0.001482	Smooth Pursuits Dizziness	0.009370
VOR Horizontal Dizziness	29.291269	Cervical Flexion	0.057859	NPC Headache	0.025966	Smooth Pursuits Headache	1.311398	Saccades Vertical Dizziness	0.001470	VOR Vertical Dizziness	0.008295
VOR Vertical Dizziness	28.412666	VOMS Headache	0.056673	VOR Vertical Headache	0.025339	PHQ-9 Score	1.248514	NPC Dizziness	0.001383	Saccades Horizontal Dizziness	0.003687
VMST Headache	26.949200	Left Cervical Rotation	0.048231	VOR Vertical Dizziness	0.024689	VOR Horizontal Dizziness	1.074843	Smooth Pursuits Headache	0.001161	NPC Dizziness	0.003226
Smooth Pursuits Fogginess	24.984704	VMST Headache	0.040304	VMST Dizziness	0.024522	Left Lateral Flexion	0.953129	NPC Fogginess	0.001006	Cervical Flexion	0.002304
VOR Horizontal Headache	19.395496	NPC Headache	0.039941	VOR Horizontal Dizziness	0.023149	Previous Head Injury	0.920716	History of Mood Disorder	0.000703	Left Lateral Flexion	0.000614
VMST Dizziness	18.911416	NPC Dizziness	0.038961	VMST Headache	0.022981	VMST Headache	0.869875	VOMS Headache	0.000573	Previous Head Injury	0.000000
Saccades Horizontal Headache	16.961958	VOR Vertical Fogginess	0.032568	Smooth Pursuits Headache	0.021290	VOR Vertical Headache	0.789330	VOR Horizontal Headache	0.000502	Smooth Pursuits Headache	-0.000461
VOMS Headache	16.128770	Cervical Extension	0.026422	Saccades Vertical Headache	0.020215	GAD-7 Score	0.781334	VOR Horizontal Fogginess	0.000365	NPC Headache	-0.000768
VOR Vertical Headache	15.519637	VMST Dizziness	0.026287	Saccades Horizontal Headache	0.018980	Smooth Pursuits Fogginess	0.644567	VOR Vertical Fogginess	0.000329	VOR Horizontal Fogginess	-0.001536
History of Mood Disorder	13.483883	GAD-7 Score	0.011150	VOMS Headache	0.018487	VOMS Dizziness	0.629748	VMST Fogginess	0.000318	Saccades Vertical Headache	-0.001843
Previous Head Injury	12.862027	VOMS Dizziness	0.007611	VMST Fogginess	0.017119	VOR Vertical Dizziness	0.570209	Saccades Vertical Headache	0.000246	VMST Fogginess	-0.001843
Saccades Horizontal Dizziness	12.215913	Saccades Horizontal Dizziness	0.004396	Saccades Horizontal Dizziness	0.016843	Saccades Vertical Headache	0.542436	VMST Dizziness	0.000237	PHQ-9 Score	-0.002151

Bold text and colored cells denote features that show up across all ML models.

Table 7.

Top 20 feature scores for machine learning models – Visit 2.

LightGBM		Decision Tree		Random Forest		XGBoost		SVC		Ridge
Feature	Imp.	Feature	Imp.	Feature	Imp.	Feature	Imp.	Feature	Imp.	Feature	Imp.
Cervical Extension	132.997165	Treatment Present	0.209153	PHQ-9 Score	0.158378	PHQ-9 Score	7.520260	Treatment Present	0.277474	VOR Vertical Headache Difference	0.112289
PHQ-9 Score	132.629271	NPC Headache Difference	0.130095	Treatment Present	0.107146	Treatment Present	6.863630	VMST Headache Difference	0.229089	VMST Headache Difference	0.086329
Treatment Present	102.996456	GAD-7 Score	0.093987	Right Cervical Rotation	0.091488	VOR Horizontal Headache	4.832900	VOR Vertical Headache Difference	0.225829	VOR Vertical Headache	0.035484
GAD-7 Score	99.256491	Right Cervical Rotation	0.080408	VOR Vertical Headache	0.072016	VOR Vertical Headache Difference	4.686990	NPC Headache	0.183861	Saccades Vertical Headache Difference	0.035023
Left Cervical Rotation	94.990231	Smooth Pursuits Fogginess	0.070633	Cervical Extension	0.067849	VMST Headache Difference	4.647623	Smooth Pursuits Headache Difference	0.124145	NPC Headache	0.027957
Right Cervical Rotation	85.222731	VOR Horizontal Dizziness Difference	0.056453	Left Cervical Rotation	0.060300	VMST Dizziness Difference	3.845895	Saccades Vertical Headache Difference	0.114409	VOMS Headache Difference	0.020891
Cervical Flexion	65.853179	VOR Horizontal Dizziness	0.055287	Cervical Flexion	0.054013	Right Cervical Rotation	3.588670	VMST Headache	0.107742	VMST Dizziness Difference	0.018894
VOR Horizontal Dizziness Difference	50.793655	Cervical Extension	0.052546	VOR Horizontal Dizziness	0.047586	VOR Vertical Headache	3.425193	VOR Vertical Headache	0.091372	VOR Horizontal Fogginess	0.014439
VOR Vertical Headache	41.973389	Left Cervical Rotation	0.049496	Left Lateral Flexion	0.040433	VOR Horizontal Dizziness	3.338928	Saccades Vertical Headache	0.090034	Left Cervical Rotation	0.010753
VMST Dizziness Difference	40.289953	VOR Vertical Headache	0.047843	VMST Dizziness Difference	0.040085	Left Cervical Rotation	2.505095	VOR Horizontal Headache Difference	0.088041	PHQ-9 Score	0.009831
Left Lateral Flexion	39.628385	PHQ-9 Score	0.042504	VOR Horizontal Dizziness Difference	0.038589	VOR Horizontal Dizziness Difference	2.120906	VMST Dizziness	0.064442	Treatment Present	0.009677
VOR Vertical Headache Difference	29.793191	Saccades Horizontal Headache Difference	0.032939	GAD-7 Score	0.038330	GAD-7 Score	2.113929	History of Mood Disorder	0.063288	VMST Headache	0.009677
VOR Horizontal Headache Difference	29.516739	Left Lateral Flexion	0.009242	VOR Vertical Headache Difference	0.021061	VOR Horizontal Headache Difference	1.921161	NPC Headache Difference	0.061687	VOR Horizontal Headache	0.008141
VOR Horizontal Dizziness	28.416646	Smooth Pursuits Headache Difference	0.008370	VMST Dizziness	0.018404	Cervical Extension	1.817560	Saccades Horizontal Headache	0.060007	VOR Horizontal Dizziness	0.007527
Saccades Horizontal Headache Difference	25.220808	VOR Horizontal Headache Difference	0.007884	VOR Horizontal Headache Difference	0.015961	VMST Dizziness	1.443005	VOR Horizontal Fogginess	0.058896	Cervical Flexion	0.007066
VMST Headache Difference	21.213691	VMST Dizziness Difference	0.007234	VOMS Headache Difference	0.015678	NPC Headache Difference	1.431498	VOR Vertical Fogginess	0.053299	Saccades Vertical Headache	0.005837
VOR Horizontal Fogginess	13.139492	VOR Vertical Dizziness	0.006691	VOR Horizontal Headache	0.013487	Left Lateral Flexion	1.201151	VMST Fogginess	0.053299	Saccades Horizontal Headache	0.005684
NPC Headache Difference	9.903239	Saccades Horizontal Fogginess	0.006682	Saccades Vertical Headache Difference	0.012717	Cervical Flexion	1.153562	Saccades Horizontal Dizziness	0.052364	VOMS Headache	0.003994
Previous Head Injury	9.243800	VOR Vertical Headache Difference	0.006217	VMST Headache Difference	0.012322	Saccades Horizontal Headache Difference	1.048383	Smooth Pursuits Headache	0.051990	VMST Fogginess	0.003994
VMST Dizziness	9.105162	Saccades Vertical Headache Difference	0.003349	Saccades Horizontal Headache Difference	0.008727	Saccades Horizontal Headache	0.707218	VOMS Headache	0.051990	Smooth Pursuits Dizziness	0.003840

Bold text and colored cells denote features that show up across all ML models. Olive and forest green colored cells denote base feature and difference variant that sum up to 9 or more for occurence total.

Table 8.

Feature frequency across models — Visit 1.

Feature	Occurrence in no. of models
NPC Headache	6
VMST Dizziness	6
VMST Headache	6
VOMS Headache	6
Cervical Flexion	5
Left Cervical Rotation	5
Left Lateral Flexion	5
VOR Vertical Dizziness	5
VOR Vertical Headache	5
GAD-7 Score	4
PHQ-9 Score	4
Previous Head Injury	4
Right Cervical Rotation	4
Saccades Horizontal Dizziness	4
Saccades Horizontal Headache	4
Smooth Pursuits Headache	4
VOR Horizontal Dizziness	4
VOR Horizontal Headache	4
Cervical Extension	3
NPC Dizziness	3
Saccades Vertical Headache	3
Smooth Pursuits Fogginess	3
VMST Fogginess	3
VOMS Dizziness	3
VOR Vertical Fogginess	3
History of Mood Disorder	2
Saccades Horizontal Fogginess	2
Saccades Vertical Dizziness	2
Smooth Pursuits Dizziness	2
VOMS Fogginess	2
VOR Horizontal Fogginess	2
NPC Fogginess	1
Saccades Vertical Fogginess	1

Green colored cells denote features that show up across all models.

Table 9.

Feature frequency across models — Visit 2.

Feature	Occurrence in No. of Models
VOR Vertical Headache	6
Treatment Present	6
VOR Vertical Headache Difference	6
VOR Horizontal Dizziness	5
VMST Headache Difference	5
VMST Dizziness Difference	5
VOR Horizontal Headache Difference	5
Left Cervical Rotation	5
PHQ-9 Score	5
Cervical Extension	4
Right Cervical Rotation	4
GAD-7 Score	4
Left Lateral Flexion	4
NPC Headache Difference	4
VOR Horizontal Dizziness Difference	4
Saccades Horizontal Headache Difference	4
Saccades Vertical Headache Difference	4
Cervical Flexion	4
VMST Dizziness	3
VOR Horizontal Fogginess	3
VOR Horizontal Headache	3
Saccades Horizontal Headache	3
Smooth Pursuits Headache Difference	3
VMST Headache	2
NPC Headache	2
VOMS Headache Difference	2
VOMS Headache	2
Saccades Vertical Headache	2
VMST Fogginess	2
History of Mood Disorder	1
VOR Vertical Dizziness	1
Saccades Horizontal Dizziness	1
Smooth Pursuits Fogginess	1
Previous Head Injury	1
VOR Vertical Fogginess	1
Smooth Pursuits Headache	1
Smooth Pursuits Dizziness	1
Saccades Horizontal Fogginess	1

Green colored cells denote features that show up across all models. Cells in orange and bold identify variable pairs (base feature and difference variant) whose combined occurrences show up at least 75% of the time.

The high frequency of Vestibular Ocular Motor Screening (VOMS)-related features across top-performing models warrants a nuanced discussion regarding potential redundancy. Measures such as Near Point of Convergence (NPC) Headache, Visual Motor Stability (VMST) Headache, and Vestibulo-Ocular Reflex (VOR) Vertical Headache appear as primary predictors in nearly all model architectures. While these features originate from the same clinical battery and share variance related to global symptom provocation, their independent selection suggests they capture distinct physiological stressors. Unlike an aggregate VOMS composite score, which serves as a general diagnostic marker, the individual sub-assessments isolate specific vestibular and oculomotor pathways; for instance, the identification of VOR Vertical Headache as more prognostic than horizontal variants highlights the importance of keeping granular data even when features appear collinear.

For the first visit, the patient records for Near Point of Convergence headache (i.e., npc_headache), Visual Motor Stability dizziness (i.e. vmst_dizziness), Visual Motor Stability headache (i.e. vmst_headache), and Vestibular Ocular Motor Screening headache (i.e. voms_headache) appear on every ML models’ top 20 features list. This predicted favorably for a time to clearance of over a month for 81% of models for npc_headache, vmst_headache, and voms_headache and for 66% of the models for vmst_dizziness. The model where the features did not have an average effect for over a month time to clearance prediction is for SVC. This discrepancy is attributed to the type of kernel used as a linear kernel does not properly capture complex relationships between the input and target output. ⁶⁴ Vestibular Ocular Motor Screening (VOMS) is a well-established physical examination tool for concussion with evidence supporting both its diagnostic sensitivity and prognostic value.^70,71 The composite score produced by the VOMS assessment has been identified as the most accurate data point in the diagnosis of concussion with vertical saccades and horizontal vestibular/ocular reflex testing demonstrating the greatest diagnostic impact. ⁷⁰ In addition, 40% of the total unique patients in the data received vestibular therapy (vestibular_tx). This includes various exercises that improve overall life, vertigo, gaze, and posture. ⁷² However, the presence of dizziness during visual motor stability testing and headache symptoms overall - particularly during near point convergence (NPC) and visual motor stability (VMST) testing at the initial assessment - has not been previously identified as having independent clinical significance beyond the overall VOMS score. Similarly, headache produced during vertical vestibular ocular reflex testing and its difference from between first and second visit has not been previously identified as unique in its significance. It is important to mention that variants of the general symptom follow the same trend of increasing or decreasing across VOMS sub-assessments. This may appear redundant, as the individual components — Near Point of Convergence (NPC), Visual Motor Sensitivity Test (VMST), Vestibulo-Ocular Reflex (VOR) horizontal and vertical, Smooth Pursuits, and Saccades — all contribute to the same overarching symptom domains of headache, dizziness, and fogginess. Because the VOMS is administered as a sequential battery, a patient presenting with an elevated baseline headache prior to testing will likely carry that symptom load across every sub-component, making it difficult to disentangle component-specific provocation from general symptom burden without intra-assessment change-from-baseline scoring.

An attempt to represent inter-visit change is made using a _diff variant of each feature, where the difference value reflects the recorded score at the second visit minus the value at the initial visit. However, these difference features still capture changes in specific VOMS components between clinical visits rather than isolating symptom provocation within a single assessment. Within the VOMS battery specifically, this distinction is clinically meaningful: VOR Vertical Headache and its difference variant (VOR Vertical Headache Difference) both appeared across all six Visit 2 models, suggesting that headache provoked during vertical vestibulo-ocular reflex testing and its change over time carries independent prognostic signal beyond what is captured by the composite VOMS score or by horizontal-plane VOR testing alone. The directional asymmetry between vertical and horizontal VOR headache frequency across models further supports retaining granular sub-assessment data rather than collapsing to an aggregate score.

To better identify which VOMS components carry genuine clinical significance beyond global symptom burden, a conservative criterion is proposed: a VOMS feature should be considered clinically significant only when both its original variant and its difference variant appear in the Top 20 Features list across models. For example, VMST Headache and VMST Headache Difference both appearing would indicate that the specific physiological stressor isolated by visual motor stability testing distinct from the NPC or VOR components contributes independently to recovery prediction at both the initial and follow-up visit. This paired-feature criterion helps separate true component-level signal from the shared variance attributable to overall symptom severity, and provides a more principled basis for prioritizing which VOMS sub-assessments to emphasize in longitudinal clinical monitoring protocols.

Treatment presence (i.e. treatment_present) appeared in all six ML models’ top 20 features list. It also ranks in the top 3 features for 81% of the models and has the highest average effect for Class 0 (Figures 1–3). Provided this, observing patient TBI recovery longitudinally using this feature is imperative since prescribing some kind of treatment to a patient after the first visit also holds clinical importance in the prediction of time to medical clearance. Treatment presence consists of having one of the 11 specific types which can be generalized as a mix of both pharmacological and non-pharmacological treatments: Selective Serotonin Reuptake Inhibitors (SSRI), amantadine, stimulant, preventative headache, vestibular, physical therapy (pt), chiropractic, psychological, neuropsychological, neurology, and cognitive. Various approaches to TBI rehabilitation is imperative following a patient’s diagnosis for restoring one’s capabilities to eventually return to play. ⁷³ From this interpretation, it can be unclear whether the presence of treatment is directly related to recovery time or if there is another confounding variable that results in this finding. Providing treatment after initial visit reduces a concussed athlete’s perception of pain and improves performance on both cognitive and physical tests measured by gold standard assessments. The improvement in these individual components would then link with the time to medical clearance, and not solely on the treatment. Presence of treatment appears with VOR Horizontal Dizziness Difference, VOR Vertical Headache Difference, Saccades Horizontal Headache Difference, and VMST Dizziness Difference for the four models that exhibited improvement in accuracy (Table 1). This could mean that the difference values for the listed assessed features play a role in recovery time along with treatment presence. ⁷⁴

Additionally there is a large indication of importance for vestibulo-ocular reflex (VOR) - vertical headache (e.g. vor_vert_headache) as it appeared the most across all features when accounting for both initial and secondary visits. Specifically, in 81% of initial visit models and 100% of second visit models. Based on the output this specific feature may be considered the most important, especially as its difference variant (e.g., vor_vert_headache_diff) also appears in all six models for the second visits. Also, this is important to identify as this pair of features commonly predicts towards Class 1 (i.e., time to clearance greater than a month). The addition of difference features provided all models additional information to learn from and establish a true relationship between the two inputs vor_vert_headache and vor_vert_headache_diff with the predicted time to clearance. ⁵⁹ Given this relationship, it is imperative to longitudinally assess this feature as it serves to be of clinical importance in the prediction of time to medical clearance. Disruptions to the visual and vestibular signals occur in a patient experiencing TBI from sports-related activities. ⁷⁵ It has been found that headaches correlate with visual-vestibular mismatch, which prolongs recovery if left untreated. ⁷⁶ Thus, medical personnel must create treatment plans that directly address headaches stemming from injuring the vestibulo-ocular reflex to ensure for effective TBI recovery.

Assessment features vestibulo-ocular reflex (VOR) - horizontal dizziness (i.e. vor_horiz_dizziness) and vestibulo-ocular reflex (VOR) - horizontal dizziness difference (i.e. vor_horiz_dizziness_diff) sum to 9 total occurrences based off of their respective values in Table 4. For second visits, 81% of models and 100% of models display that the base feature and its difference variant (i.e. vestibulo-ocular reflex (VOR) - horizontal dizziness and vestibulo-ocular reflex (VOR) - horizontal dizziness difference) drive for Class 1 predictions respectively. Stemming from these findings, it can be deduced that the ML models have extensive information about vestibulo-ocular reflex (VOR) - horizontal dizziness after accounting for both that and its difference variants which allows for a strong relationship between these inputs with the target prediction outcome time to clearance. ⁶⁹ Moreover, these need to be longitudinally assessed in TBI patients as they are important to the prediction of time to medical clearance. Dizziness is a core symptom of vestibular migraine ⁷⁶ which can result from activities such as concussions. Patients with TBI report more instances of headaches compared to dizziness, ⁷⁷ which explains the difference between its frequency of occurring across the models as a clinically important assessment feature with that of vor_vert_headache along with its difference variant.

Features that are not as frequently occurring across ML models as the ones mentioned earlier still hold some degree of significant clinical importance (Tables 1 and 2). Since they appeared in the top 20 features list, the ML models highlight that feature to be essential for predicting time to medical clearance following concussion.

Thus, personalized treatment plans can involve any combination of the features in the top 20 features scores list (Table 2) as combining multi modal approaches translates to improved sports-related TBI recovery. ⁷⁸ It is highly encouraged to first account for VOR Vertical Headache, VOR Vertical Headache Difference, Treatment Present, VOR Horizontal Dizziness, VOR Horizontal Dizziness Difference, VMST Dizziness, and VMST Dizziness Difference before including additional assessment features as these were found to be of highest clinical significance due to their frequency across ML models. The choice of combination will depend on the patient’s attributes (past records, condition, etc.).

While XGBoost achieved the highest overall accuracy (0.84) with Visit 2 features and statistically significant improvement across visits in the initial method implementation, we do not designate a single final deployable model at this stage. The six models serve as a comparative framework for identifying robust clinical predictors rather than as ready-for-deployment tools. Model specification details and code are available at https://github.com/ MeganTran6023/Sport-Related-Concussions_Machine-Learning to support future replication and external validation (TRIPOD+AI Item 22).

Comparison with Previous Studies Previous research studied concussion diagnosis at a single time point with gold standard assessments using ML models such as XGBoost and Random Forest.^62,79,80 A different study on predicting sports injuries in football uses multiple time points and extracts important features that drive models’ predictions; however, it is not specific to concussions from sports related activities. ⁸¹ This current study utilizes not only multiple time points for analysis, but also determines essential assessment features that drive models to demonstrate an improvement in accuracy between initial and second visits.

To extend the detection of clinically important features determined by the models, this study incorporates feature engineering to generate new features, mainly the difference variants of the base original features. These new features provide models more information about the existing features and their interactions to produce informed results. ⁸²

Through these novel implementations, this study paves the way to optimize assessments for clinicians to provide high quality personalized rehabilitation protocols for athletes with sports-related TBI injuries.

Clinical Relevance The intended users of a future validated version of this model are licensed clinicians experienced in concussion management (e.g., sports medicine physicians, neurologists). Input features are derived from standardized clinical assessments (VOMS, BESS, ImPACT) already routinely administered in concussion care. Clinicians would need to input structured assessment data at one or two time points; no specialized computing expertise is required beyond use of a provided interface. However, poor-quality or missing input data — particularly VOMS subscores — should prompt clinical judgment rather than model reliance, as the current model was not trained on imputed data and may underperform when key features are absent (TRIPOD+AI Item 27b).

The trend of an increase in concussion can be primarily attributed to increased awareness in the diagnosis of concussions by a trained medical professional^83,84 This may have led to more consistent reports of concussion cases that may have gone unnoticed in previous studies/findings. ⁸⁵ Clinicians who treat concussion patients may benefit from predictive models in several important ways. ⁸⁶ The results explore the feasibility of identifying individuals at risk for prolonged recovery during initial assessment, which could eventually serve as a preliminary signal for investigating targeted interventions. For example, identifying aspects of the VOMS assessment as discussed above early may inform clinicians and result in earlier and/or more targeted physical therapy to address these higher risk features. Early initiation of physical therapy has already been identified as beneficial in recovery ⁸⁷ and ML models may serve as a tool to identify patients most likely to benefit. Furthermore, enhancing recovery efficiency not only improves quality of life, but also facilitates a quicker return to normal function or athletic participation. ⁸⁸ In sports settings, recognizing athletes at risk for extended recovery can inform individualized treatment strategies and help establish realistic expectations for both the athlete and the team.

Limitations Although the results indicate a positive direction for concussion screening and treatment, there are limitations to this study. First, the relatively small dataset size represents an important limitation of this work as the limited number of samples could affect the model’s ability to generalize to TBI patients with diverse demographic backgrounds and clinical profiles. As a result, the findings may not fully reflect the variability observed in broader patient populations. ⁸⁹ In addition, having a greater number of data points belonging to Class 0 would be beneficial, as this would allow for a more robust evaluation of the ML model’s performance, particularly in assessing its ability to classify and predict time to medical clearance accurately. This is a common issue when it comes to handling ML problems with clinical data where classifiers learn best on the class with most data points because they attempt to optimize a single aggregate metric while overlooking the distribution of the data across the target classes. ⁵⁷

From the reported results, the small and uneven balance between outcome classes 0 and 1 (176 patients and 41 patients respectively) resulted in low specificity scores. Additionally, the inflated both accuracy and recall scores could potentially be attributed to overpredicting for the dominant class. Balancing techniques such as class weighting, resampling, or a combination of other methods should be implemented to ensure the reported results are not skewed by this limitation. Another aspect to note is that overprediction for a class could result in false positives and/or negatives. Clinically speaking, incorrectly predicting a patient’s time to recover earlier than the actual time to clearance leads to undesired effects such as worsening existing concussive symptoms ⁹⁰ and reduced responsiveness to brain-derived neurotrophic factor (BDNF). ⁹¹ Conversely, mistakenly predicting a patient to recover longer than normal would hinder the concussed patient’s physical and psychological state, reducing their quality of life. ⁹²

Formal statistical filtering, initially piloted using independent t-tests and Mann-Whitney tests, was excluded from the final preprocessing workflow as it failed to enhance longitudinal predictive performance. Instead, feature selection prioritized iterative cleaning based on missingness thresholds and clinical relevance to maintain a comprehensive feature space for the machine learning models. While the omission of formal statistical testing during preprocessing is acknowledged as a limitation to analytical rigor, future research should investigate alternative statistical frameworks and structured dimensionality reduction to optimize feature utility and address the event-per-variable ratio.

A notable limitation of this analysis is the absence of formal statistical testing to compare performance differences between visits and the lack of reported confidence intervals for key metrics. While 66% of the models showed numerical improvements in accuracy with the addition of Visit 2 data, including a 5% increase for the XGBoost model, the statistical significance of these changes was not formally evaluated. Additionally, without confidence intervals, the precision of performance values-such as the peak accuracy of 0.84 -remains unquantified. ⁹³ This lack of statistical rigor is underscored by the small cohort of 217 patients and the potential for distributional bias inherent in the Leave-One-Out Cross-Validation (LOOCV) method.

A supplementary factor to consider is the gender imbalance in the dataset used. Of the 176 total number of patients in Class 1, 113 (64.2%) are female and 63 (35.8%) are male. Of the 41 total number of patients in Class 0, 24 are female and 17 are male. From this, these findings may be biased to a female population. More specifically, that means that there is a possibility that for some or all models, the correctly classified Class 0 and 1 predictions are mainly females. This is important to note as previous works show that females take longer to recover from sports-related TBI than males do across different sports related activities (i.e., basketball, rugby, soccer, and squash) ⁹⁴ (TRIPOD+AI Item 3c). As our dataset does not specify the sports for each sports-related concussions, the reason why more females are in Class 1 compared to Class 0 may be due to the nature of the sports each participated in. Another study that also used a small dataset size found that female collegiate soccer athletes who sustained a sports-related TBI experienced a longer recovery time than that of male collegiate soccer athletes, ⁹⁵ but this was limited by its small dataset size as well as having an gender imbalance.

Another limitation relates to the features utilized during the ML stage. Although a feature selection method was applied to identify the most essential predictors, some of the retained features were based on self-reported data. Self-reported measures are inherently subject to bias, which can introduce uncertainty into the model.^96,97 This bias may reduce the reliability of the model’s predicted class outcome and perceived clinical importance of certain features, potentially affecting their true relationship with days to medical clearance.

The lack of improvement in accuracy across visits can largely be attributed to the reliance on linear models. In the case of the SVC, the use of a linear kernel-while appropriate given the limited dataset size and its straightforward implementation-likely restricted the model’s ability to capture complex, non-linear relationships between assessment features and class labels. ⁶⁴ As a result, the model may have failed to leverage additional information introduced at the second visit, leading to unchanged performance. Similarly, the Ridge regression model’s inherently linear nature limits its capacity to model non-linear interactions in the data. ⁴⁵ Consequently, the difference-based features derived from the second visit may not have contributed meaningful new information beyond what was already represented in the first-visit features, resulting in comparable predictive accuracy across visits.

The current prediction method of binary classification oversimplifies the prediction of patients. The assignment of classes ‘under a month’ and ‘equal to or greater than 1 month’ sacrifices granular clinical utility. A reformulation of this method would be to utilize granular, multiclass buckets. To ensure nuanced predictions, the buckets could be divided into acute, typical, and prolonged recovery for both improved methodological accuracy and clinical significance. ⁹⁸

Also, the current dataset lacks neurocognitive data outside of symptom reporting. This type of data is useful regarding concussion assessments. For instance, tests for this specific aspect accounts for specific trends in a patient’s symptoms during recovery as recording whether a patient is taking some neurocognitive treatment is not sufficient enough to conclude that the outcome predictions are of high accuracy. ¹⁷

Treatment presence does not account for the varying time for each treatment’s therapeutic effect (e.g. Some Selective Serotonin Reuptake Inhibitor (SSRI) treatment takes 8 weeks to have a full therapeutic effect ⁹⁹ If recording patient’s data before the time frame for certain treatments a patient received, this would then skew the interpretation of what features are most significant to their predicted concussion recovery. To resolve this, it would be beneficial to collect data on features associated with the patient that account for each treatment’s time-frame to have a more accurate depiction of the results.

Although the filtering process did eliminate the majority of features not significant for concussion days to clearance classification, this still resulted in redundant features kept (ie. headache, dizziness, etc.). An attempt to differentiate the specific components of the exams was done when calculating the difference between the second and initial visits. Furthermore, the acquired sample is reduced given the number of features the dataset has. This is prone to overfitting and instability in feature importance listings. ¹⁰⁰ A possible future implementation is recursive feature elimination (RFE) for feature selection. This method should be used in relation to the inclusion of more patients and datatypes collected over time, resulting in a dataset that will include a more robust, objective set of measures. In this setting RFE would be able to systematically removes the least important features/types from a dataset to improve model performance, reduce overfitting, and enhance interpretability.

RFE-MF (MissForest) was found to outperform four of the classic data imputation methods (mean/mode imputation, kNN, MICE, and MF) in addressing the critical need for data accuracy in medical research, where it helps mitigate challenges that can impair clinical decision-making and ultimately affect the quality of patient care. ¹⁰¹

While LOOCV is recommended for small datasets such as the one utilized in the study, using this method leads to distributional bias. In turn, this leaks information pertaining to the removed item as the test set to the model as well as reduces performance of commonly-used ML models. ¹⁰² Therefore, stratified repeated holdout and a modified version of k-fold cross-validation is recommended to avoid this bias ¹⁰³ While Leave-One-Out Cross-Validation (LOOCV) is a standard validation strategy for small datasets, its results remain highly contingent upon the specific characteristics of the cohort. Although LOOCV maximizes data utility to minimize estimation bias, these unbiased outputs do not inherently ensure clinical generalizability. This limitation is often a consequence of the model overfitting to the unique noise within the sample. Therefore, while the methodology may yield stable internal performance metrics, it does not guarantee that the model will demonstrate comparable accuracy across external, heterogeneous datasets.

Despite its limitations, both the point estimate (accuracy) and variance are constant when using LOOCV. The lack of fluctuation in both of these measures allows for reproducible and deterministic findings for analysis.

The small sample size used is prone to overfitting. This was primarily addressed through strategic model selection and the implementation of regularization techniques. Specifically, the Ridge classifier employs L2 regularization to shrink the coefficients of features with weaker associations to the outcome groups, which reduces variance and stabilizes the model’s estimates. ¹⁰⁴

Future Work The focus of this study is to ensure that the ML models accurately predict concussed patients’ time to clearance from sport-related TBI with the integration of longitudinal data and gold-standard assessments per our objectives. While we are optimistic about the acquired results, there are ways to expand upon this current study.

Future work should prioritize increasing the overall dataset size. A larger dataset would allow the ML models to learn more stable and representative patterns, ⁸⁹ which in turn would improve their ability to identify the most important features for accurately predicting time to medical clearance. With more data, the influence of noise and bias would be reduced, leading to clearer insights into which variables truly contribute to prediction performance and improving confidence in the model’s results. Moreover, having a balanced dataset between both genders and target Class classifications (Class 0 and Class 1) would allow the results to best describe a generalized set of athletes which will be applicable to patients who experienced other methods of TBI.

Expanding the analysis beyond athletic-based injuries is another important direction. While the current methodology has shown success in TBI cases resulting from athletic concussions, applying this approach to a broader range of TBI patients would improve its clinical relevance. ¹⁰⁵ Extending the framework to non-athletic injuries would support the development of more general clinical guidelines and help determine whether the same assessment features and prediction strategies remain effective across different injury mechanisms. This broader application would also motivate further research into TBI individualized treatment and recovery within the wider medical field.

Including patients with more than two clinical visits could further strengthen analysis. Additional visits provide more longitudinal clinical data, allowing models to better capture changes in patients’ symptoms over time. ¹⁰⁶ With richer temporal information, ML models would be better positioned to accurately predict a TBI patient’s time to clearance and reflect the progression of recovery more reliably.

Additional features that incorporates objective methods of evaluating concussed patients should also be included. These will not only allow clinicians to more accurately diagnose an injured athlete, they also can diagnose and provide personalized treatment in a timely manner. This is imperative if the injured athlete sustains the concussion before adolescence since this is the period of major brain development. ¹⁰⁷

Currently, the LOOCV method done on the current dataset lacks external validation, temporal validation, optimism correction and calibration assessment, which is necessary for determining gernalizability. ¹⁰⁸ This is a given especially since the small dataset and the distribution for the two outcome classes are prone to overfitting. Future works will run the listed elements to evaluate its clinical applicability in the concussion domain.

Running the feature selection method prior to the modeling instead of within the modeling’s cross-validation folds increases the risk for data leakage. ¹⁰⁹ Future works will repeat the analysis where the variable selections are performed nested within the validation process to account for internal validity.

Future work could also focus on incorporating algorithmic modifications to the existing SVC and Ridge models utilized in this study ¹¹⁰ or using more nonlinear ML models. This will enable a more informed listing of clinically important assessment features given the model’s demonstration of an increase in accuracy provided longitudinal patient sports-related concussion data. From this, proper personalized protocols can be put together for athletes.

Finally, applications in digital health should be explored. Integrating ML models into technologies such as wearable devices would help translate the study’s findings into real-world clinical use. These tools could allow patients to monitor their condition outside of clinical settings, while simultaneously providing clinicians with real-time data to support decision-making. ¹¹¹ Such integration would enhance continuous assessment, improve personalized treatment planning, and extend the practical impact of the proposed methodology.

Conclusion

This study is primarily comparative in scope: rather than proposing a single deployable clinical tool, it benchmarks six ML classifiers across longitudinal data to identify which model architectures and clinical features best support future development of a validated prediction tool. External validation and prospective testing are required before any model described here could be considered for clinical deployment.

Utilizing longitudinal clinical assessments to predict time to medical clearance represents a preliminary approach that offers exploratory insights into diagnosis and return-to-play decision-making. By applying ML models to longitudinal data, patterns not readily detectable by licensed medical practitioners were identified. These models achieved high predictive accuracy and highlighted specific assessment features that significantly influence clearance timelines. Collectively, these exploratory findings highlight assessments that may warrant investigation in future validation studies and suggest the potential for a longitudinal monitoring framework to inform return-to-play decisions and guidelines. The findings can be applied to future return-to-play decisions by enabling clinicians to better anticipate recovery trajectories and focus on the most informative assessments to best formulate personalized sports- related TBI rehabilitation protocols.