Abstract
Background:
Eye-tracking technology for detecting eye movements has been gaining increasing attention as a possible assessment and monitoring tool for sport-related concussion (SRC).
Purpose:
To determine the diagnostic accuracy of a rapid number-naming task with eye tracking, the King-Devick Eye Tracking (K-D ET) assessment, in identifying SRC.
Study Design:
Cohort study.
Methods:
One female and 1 male team of United States collegiate rugby-15 players competing during the 2018 season were recruited. Variables assessed were total saccades, saccade velocity, total fixations, fixation duration, fixation polyarea, and test duration. A generalized estimating equation was used to examine group (concussion vs nonconcussion), time (baseline vs postinjury/postseason), and sex-based differences for each outcome measure. In addition, the different components of diagnostic accuracy of the K-D ET were calculated.
Results:
Baseline K-D ET assessment for 49 participants (25 male, 24 female) were assessed at the beginning of the season, with 28 participants who did not sustain a head injury during the season completing the postseason assessments and 6 participants completing a postinjury (suspected concussion) assessment. Significant differences were observed between concussed and nonconcussed groups for total saccades (P = .024), fixation duration (P = .007), and fixation polyarea (P = .030), with differences from baseline to follow-up observed for saccade velocity (P = .018) in both groups. Sex-based differences were noted for total fixations (P = .041), fixation polyarea (P = .036), and completion time (P = .035). No significant Group × Time interactions were noted. The K-D ET test duration indicated high specificity (0.86) but not high sensitivity (0.40). No other variables reported high sensitivity or specificity.
Conclusion:
Other than completion time of the K-D ET test, no K-D ET oculomotor parameter was highly sensitive or specific in the diagnosis of concussion in this study.
Sport-related concussion (SRC) is a relatively common injury in sport, occurring because of the transfer of injurious biomechanical forces to the head that result in a heterogeneous range of signs and symptoms. 25 It is vital for the clinician and patient to understand that symptoms may be present immediately or occur hours after injury, and the quantity, severity, and duration of concussion symptoms vary between people and consecutive concussions. 25 The evaluation of SRC by medical physicians, athletic trainers, sports physical therapists, and other clinicians involves assessing various components such as balance, cognition with memory recall, and self-reported symptoms. 22,34,35
Recent innovations in eye-tracking technology using camera-based video-oculography equipment have enabled clinicians to utilize eye movement measurements as well. However, its use has been reported to be limited, with many clinicians relying on their own clinical assessments for concussion. 31 Nonetheless, it may be a reliable tool and more research is needed to establish its appropriateness and the interpretation of findings. 32
The King-Devick (K-D) assessment provides the option of an infrared video oculography–based 120-Hz sampling rate eye tracker (K-D Eye Tracking [ET] assessment). 16,18 The unit provides measurement on duration (length of time on the number-naming task), number of saccades, peak saccade velocity, number of fixations, fixation duration (intersaccadic interval [ISI]), number of blinks, average fixation polyarea, and average saccade velocity, with the latter 2 being company-based metrics. The standard K-D test, a widely used vision-based assessment tool for concussion, 8 consists of a rapid number-naming task and has been validated as a sensitive sideline performance measure for concussion detection. 2,11,15,17,19 –21 As the K-D test is a reading task; it is sensitive to impairments of eye movements and thus indirectly measures saccades and other eye movement functions. 9,5,12 However, with the recently available eye-tracking component, it is valuable to provide more evidence on its diagnostic accuracy for SRC now that the system’s reliability has been established. 18
Recent research 17,18 has demonstrated excellent reliability when using the K-D ET unit, and a previous study by Hecimovich et al 16 utilized the K-D ET unit on youth Australian footballers for SRC. The eye-tracking data in that study were limited to saccades (eye movements that quickly shift the eye’s focus between 2 fixed points), number of fixations (maintaining a visual gaze but with small involuntary eye movements), and blinks.
The primary aim of the current study was to investigate the diagnostic accuracy of the 120-Hz K-D ET system for clinical use in the recognition of SRC. Specifically, the objectives were to determine the diagnostic accuracy of the current K-D ET test in identifying SRC that occurred from game participation and to perform a comparative analysis on saccadic and fixational activity between baseline and postinjury in concussed athletes. We hypothesized that, based on previous research, 9 –11,15 –21 the eye-tracking variables assessed (total saccades, average saccade velocity, average fixation duration, total fixations, and average fixation polyarea [amplitudes during fixational eye movements]) would not provide useful diagnostic measurements for SRC and that K-D ET test duration would be a useful measure.
Methods
Study Participants
A prospective observational cohort study was conducted on 1 female and 1 male team of United States collegiate rugby-15 players competing during the 2018 season. All members of the 2 teams were invited to participate in the study. Exclusion criteria included a history of neurological or ophthalmological disease (other than refractive error), learning disability, or visual dysfunction and taking central nervous system–active medications. We initially enrolled 49 participants consisting of 25 healthy male (20 aged 18-21 years, 5 aged ≥22 years) and 24 healthy female (18 aged 18-21 years, 6 aged ≥22 years) rugby-15 players. The university’s institutional review board approved all procedures, and consent was obtained from the players and team coach before enrollment.
K-D ET Testing Procedure
All baseline K-D ET testing was performed at the university in a dedicated room. Before the testing, each participant had the procedure explained, had it demonstrated in detail, and was given the opportunity to ask any questions.
Each participant sat in front of a laptop computer that contained the K-D ET system, including an eye-tracking unit, an infrared-based video-oculographic rig (120-Hz VT3-Mini; EyeTech Digital Systems). Participants were seated with a head-to-camera distance of 60 cm to ensure proper eye contact with the infrared camera, eyes aligned to the center of the computer screen, ensuring the screen was in focus and clear. The eye-tracking unit was positioned close to the bottom of the laptop screen while ensuring that it did not cover the display. Green lights on the eye-tracking unit indicated correct participant positioning. For optimum conditions, the overhead lighting in the room was dimmed and outside window glare was minimized by darkened window shades. Participants were instructed to complete a pretest validity screen by looking straight ahead and then at yellow targets in each corner of the screen. If the pupil, corneal reflections, and/or crosshairs were not visible, out of focus, or unsteady in any of the viewing locations, the participant was cycled through the spatial location again to optimize the settings. At this point calibration commenced. With this, the participants followed a red target across the screen with their eyes.
After calibration, a validation was conducted by verifying the eye position at 5 spatial checkpoints on the screen. Accurate viewing with a proper calibration changed the spatial checkpoints to blue. If successfully accomplished, the K-D ET test began. If the calibration was not accurate, calibration was repeated for best results. Corrective lenses were worn only if required for reading. Participants were instructed to read aloud the single digit displayed from left to right, top to bottom, as quickly as possible without making any errors. Participants were instructed not to use their hand or finger to help follow the numerical pattern. If the participant made an error and quickly corrected it, no error was recorded. An error was recorded for each omission. The K-D ET system provides post hoc eye movement and test analysis for the following measures: total fixations, total saccades, average saccade velocity, blinks, ISI (saccade latency and fixation duration), fixation polyarea, and individual section and overall completion time.
All players completed a baseline K-D ET test during the preseason. Players with a suspected concussion during training or match play, determined by the team’s athletic trainer, were removed from further participation and assessed with the Sport Concussion Assessment Tool–5. These players underwent testing with the K-D ET the following day. All players (including those who sustained in-season concussion) were requested to complete an end-of-season K-D ET assessment. However, in the concussion group, the end of seasons results were not included within our analysis.
The K-D ET assessment was used strictly for research purposes and not for clinical and/or return-to-play decisions. The participating teams were unaware of the research findings.
Statistical Analysis
All measures (total saccades, saccade velocity, total fixations, fixation duration, fixation polyarea, and test completion time) in the concussed and nonconcussed groups are reported as mean ± standard deviation, with between-group differences determined using independent t tests and within-group differences determined using paired t tests (data met normality assumptions). A generalized estimating equation (GEE) model with normal distribution, identity link (player ID treated as subject with repeated measures), and fixed effects for concussion group, time, and sex was used to examine group (concussed vs nonconcussed), time (baseline vs postinjury/postseason), and sex differences for each outcome measure.
Parameter estimates (estimated marginal mean, standard error, and 95% confidence interval [CI]) were reported for concussion groups, and Wald χ2 and P values were reported for tests of the model’s fixed effects. Group × Time interaction models were also examined, with Group × Time interaction test results and interaction parameter estimates reported. If a follow-up K-D ET test was more than 3 seconds slower than baseline, suggesting worsened performance, it was classified as positive. This cutoff is frequently used.
15
For all other measures (total saccades, saccade velocity, total fixations, fixation duration, and fixation polyarea) the cutoff values were calculated in 2 separate ways. A cutoff was assigned as the value corresponding to the mean difference between test-retest in healthy controls in a previous reliability study.
18
The diagnostic accuracy (sensitivity, specificity, positive likelihood ratio, negative likelihood ratio, positive predictive value, and negative predictive value) with 95% CI for all variables was calculated using MedCalc. A receiver operator characteristic (ROC) analysis providing the area under the curve (AUC) with sensitivity (true-positive rate) and 100 – specificity (false-positive rate) was conducted using NCSS (version 21.0.1, NCSS, LLC) where a concussion was present (1) versus not present (0). The ROC analysis direction indicative of concussion for each variable was set as the direction opposite that of the mean group change of the control group. For example, if the control group had a decrease in the total saccades at follow-up, then an increase in the number of saccades was the direction set within the ROC analysis as indicative of concussion. Area under the ROC curve (AUC) values of 0.5 are considered of no clinical value (null hypothesis AUC, 0.5). Subsequent diagnostic accuracy using the cutoff values determined by the ROC analysis was not reported when the AUC was not significant.
For all comparisons, significance was set at P < .05.
The components of diagnostic accuracy (sensitivity, specificity, positive likelihood ratio, negative likelihood ratio, positive predictive value, and negative predictive value) were calculated using MedCalc (www.medcalc.org/calc/diagnostic_test.php). The sensitivity and specificity were quantified as being excellent (0.9-1.00), very good (0.8-0.9), good (0.7-0.8), sufficient (0.6-0.7), or bad (<0.6). 29
Results
In total, 49 participants (25 male, 24 female) were assessed on the K-D ET at the beginning of the season with an established baseline. Fifteen participants who did not sustain a concussion did not complete postseason testing. Therefore, 34 participants completed the postseason assessments, with 6 of these participants suspected to have sustained a concussion. In the suspected concussion group (5 male, 1 female), 8 concussions were recorded from the 6 participants, with one of the players (male) being requested to test on the K-D ET on 3 occasions (in this instance, only the first concussion was recorded within diagnostic accuracy testing).
Mean outcome values and between-group differences, as well as differences from baseline to postinjury (concussed group) or postseason (nonconcussed group), are shown in Table 1. The base model results examining group, time, and sex-based differences are presented in Table 2 and account for the differences between groups at baseline while controlling for sex. The interaction model with concussion Group × Time interaction is also reported; no significant interactions were detected, and wide CIs were noted.
Differences in the Means From Baseline to Postinjury (Concussed Group; n = 6) or Postseason (Nonconcussed Group; n = 28) a
a Data are reported as mean ± SD. Boldface P values indicate statistically significant differences between groups compared (P < .05).
General Linear Model Results: Estimated Marginal Means for Concussed Versus Nonconcussed and Model Effects for Basic Model and Interaction Model a
a Boldface P values indicate statistically significant differences (P < .05).
b Compared with all other group combinations, which are set to zero (ie, concussed at baseline, concussed at postinjury/postseason, and nonconcussed at postinjury/postseason).
Total Saccades
A between-group difference was detected at baseline (P = .037) (Table 1) and also in the GEE model (P = .024) (Table 2). The diagnostic accuracy of the K-D ET test for total saccades indicated insufficient bad sensitivity and specificity (Table 3). Furthermore, this test was not significant when assessed using the ROC analysis (Table 4).
Diagnostic Accuracy of the King-Devick Eye Tracking Test a
a Boldface values indicate sensitivity or specificity as being >0.8 (classified as very good or excellent).
Receiver Operator Characteristic Analysis a
a AUC, area under the receiver operating characteristic curve.
Saccade Velocity
The nonconcussed group reported significantly higher postseason saccade velocity (P = .034) (Table 1). The GEE model detected an overall change over time (P = .018) (Table 2). No other significant between-group differences were determined for saccade velocity. The diagnostic accuracy of the K-D ET test for saccade velocity indicated insufficient sensitivity and specificity (Table 3). Furthermore, this test was not significant when assessed using the ROC analysis (Table 4).
Total Fixations
No significant between-group differences were determined for total fixations (Tables 1 and 2). The diagnostic accuracy of the K-D ET test for total fixations indicated insufficient sensitivity and specificity (Table 3). Furthermore, this test was not significant when assessed using the ROC analysis (Table 4).
Fixation Duration
The nonconcussed group reported significantly lower postseason fixation duration (P = .034) (Table 1). The GEE model also detected a between-group difference (P = .007) (Table 2). No other significant between-group differences were determined for fixation duration. The diagnostic accuracy of the K-D ET test for fixation duration indicated insufficient sensitivity and specificity (Table 3). Furthermore, this test was not significant when assessed using the ROC analysis (Table 4).
Fixation Polyarea
The nonconcussed group reported significantly higher baseline fixation polyarea (P = .016) (Table 1). The GEE model also detected a between-group difference (P = .030) (Table 2). No other significant between-group differences were determined for fixation polyarea. The diagnostic accuracy of the K-D ET test for fixation polyarea indicated insufficient sensitivity and specificity (Table 3). Furthermore, this test was not significant when assessed using the ROC analysis (Table 4).
K-D ET Completion Time
No significant between-group differences were determined for completion time (Tables 1 and 2). The diagnostic accuracy of the K-D ET test duration is presented within Table 3. The K-D ET completion time had high specificity (0.86; 95% CI, 0.68-0.96) but insufficient sensitivity.
Discussion
In this study, we evaluated the K-D ET system, a small-scale device that can be used in a variety of clinical settings, in order to examine its potential utility as a diagnostic aid for SRC, with a focus on the eye-tracking measurements. With the eye-tracking additions to the K-D test as a bundled single package, it is necessary to assess its clinical utility. We determined that the K-D ET test duration was the most useful feature for diagnosing concussion with very good specificity (0.86). No other included variables reported very good or excellent sensitivity or specificity: total saccades (sensitivity, 0.50; specificity, 0.43); average saccade velocity (sensitivity, 0.67; specificity, 0.25); average fixation duration (sensitivity, 0.50; specificity, 0.46); and fixation polyarea (sensitivity, 0.67; specificity, 0.57).
The ocular motor variables measured within this study were total saccades, average saccade velocity, total fixations, ISI (designated on the K-D ET as average fixation duration), average fixation polyarea, and overall completion time. Two company-based metrics are available with the K-D ET: average saccade velocity and average fixation polyarea. Thus, these 2 metrics cannot be used when assessing eye movements between the K-D ET and other similar research or clinically with other eye-tracking units. Onge et al 26 and Hecimovich et al 18 reported on both of these metrics, and Marchant et al 24 reported on average saccade velocity. All 3 of those studies used the K-D ET unit, and as the metrics are components on the unit, it was determined that the current study report on them for use in future research. The analysis in the current study employed the ROC curve, with results providing an important indicator of accuracy to inform clinical practice. 3,14
The results from this study did not indicate that total saccades, average saccade velocity, total fixations, average fixation duration, and average fixation polyarea were useful diagnostic tools after SRC. The large reported CIs suggest that larger studies would provide a more precise estimate of diagnostic accuracy. Furthermore, the K-D ET unit recorded eye measurements with a 120-Hz sampling rate and an angular error of approximately 0.5° (drift, <0.3°). Andersson et al 1 suggested that sampling errors decrease with eye-tracker units that are at or above 200 Hz, and a 500-Hz unit can better discriminate saccades at difference velocities. 27 Future research will need to confirm the appropriate sampling frequency and eye movements to be measured with SRC.
There have been several studies that have used frequencies at 60 Hz to 500 Hz in SRC research, as reported from a systematic review and meta-analysis by Snegireva et al. 32 For example, DiCesare et al 6 utilized a 60-Hz Tobii X2-60 Eye Tracker and reported significant differences in fixation and initial fixation error for prosaccade and self-paced tasks with concussed participants (mean age, 16.8 ± 1.2 years) and similarly aged nonconcussed participants. Conversely, Rizzo et al 28 utilized an Eyelink 1000 unit at 500 Hz and the standard K-D test on recently concussed participants and controls and noted that the average ISIs for task-specific saccades were significantly longer among concussed patients compared with controls and the concussed patients had a greater number of saccades during number reading in comparsion to control, but no differences in saccade peak velocity, duration, or amplitude between both groups.
Although there have been published reports on eye tracking and SRC, comparing the results from those studies to those of the current study is problematic. First, the K-D ET utilizes a horizontal reading task that requires the participant to read as fast as they can without error, 10 whereas other studies, such as DiCesare et al, 6 required participants to follow a target moving in a sinusoidal pattern, and Taghdiri et al 33 had participants gaze back and forth between 2 static targets; these are only 2 examples of several various protocols used in eye-tracking studies. 31 Second, the K-D ET uses a 120-Hz unit that, ideally, should be compared against similar sampling rate units. Third, the variables measured in eye-tracking SRC studies reach across smooth pursuit, fixations, antisaccades, memory-guided saccades, and self-paced saccades. 32 Lastly, how long (time) after injury the test was conducted would also need to match when comparing across other eye-tracking units.
Overall, the use of camera-based video-oculography for concussion recognition or a criterion for return to play will more than likely be based on clinician preference. Research reporting oculomotor deficits after concussion 15 does exist; however, using a diagnostic tool solely focused on oculomotor dysfunction may not be appropriate and may fail to recognize concussion signs and symptoms related to other systematic dysfunctions. For example, a systematic review on eye-tracking technology 32 reported that most saccadic and pursuit deficits may be missed with a clinical examination, and thus the use of eye-tracking devices is a viable option and may help guide the clinician on recovery and thus return to competition.
The study findings demonstrated a specificity for the duration of the K-D ET test similar to that recently reported in a meta-analysis by Harris et al. 15 However, the sensitivity of the K-D ET test duration was not comparable to that reported within the meta-analysis. This may be due to the relatively small sample size of this study, as the 95% CI for the sensitivity of this study does include the pooled sensitivity estimate from the meta-analysis. Another explanation may be our 1-day delay in assessing the K-D test postinjury, as studies within the Harris et al 15 meta-analysis completed the K-D test immediately after injury. The value of using or measuring oculomotor function with SRC may be growing, and it has recently been reported in a large cohort study that impaired oculomotor function (impaired/symptomatic vestibular ocular reflex) is a factor associated with prolonged recovery from concussion in adolescents. 13 While this is important, it must be noted that the K-D ET was not included in that particular study, and there is a paucity of empirical evidence to support tools such as the K-D ET in identifying those with SRC or at high risk of a poor prognosis post-SRC. Nonetheless, the findings from the present study may guide future studies that would in turn be used to examine the utility of the K-D ET in the assessment of oculomotor function after SRC.
Limitations
There are some limitations to our study. This study was limited to a small sample size, and the results may not reflect the subtle changes in ocular motor dynamics that occur with concussion. For future studies based on this study’s results, a priori power calculation indicates that large samples would be required to detect any differences if they truly existed (saccade n = 168 through to fixation duration n = 22,282 (repeated-measures analysis of variance between factors, 2 groups, 2 measurements, α = .05, power = 0.8 [G*Power 3.1.9.4]). 7 Hence, large multiteam investigations would be required to further examine the tool’s diagnostic suitability for SRC.
As noted previously, the sampling frequency of 120 Hz utilized with the K-D ET may not be as sensitive in comparison with higher-frequency video-oculographic units, for example, the differences between saccadic and fixation counts, and may introduce aliasing artifacts. 27 However, we have demonstrated good test-retest reliability of the K-D ET in a previous study. 18 Furthermore, ocular motor measurements may have been influenced by the time frame between head injury and assessment with the K-D ET unit. Participants were assessed within a 24-hour time period; however, this may have affected the results, depending on severity of head injury. Ideally, it would have been useful to use the unit immediately postinjury with 24-, 48-, and 72-hour follow-ups, but the conditions at the training and match facilities did not allow this to occur.
The recommended K-D rapid number-naming task component is for participants to establish a baseline measurement by using the best (fastest) time of 2 trials 10 minutes apart without errors. 9 In the current study, 1 baseline measurement was conducted because of limited time constraints. This limitation is highlighted by Galetta et al, 10 who reported that the K-D test has been shown to have learning effects associated with repeat testing improvement of 3.4 seconds in median times. The learning effect reported by Galetta et al 10 has also not been investigated over longer periods of time and may not be applicable given that the K-D ET was repeated many weeks after baseline testing. However, all participants in the current study had been previously exposed to K-D testing, and intraclass correlation coefficients of 0.85 to 0.95 have been reported for the test-retest reliability of the rapid number-naming task. 9,23,30 Lastly, this study did not include blinks or peak saccadic velocity, as they have not been shown to be affected in SRC. 31
Conclusion
In this study, the primary aim was to extend the evidence on the diagnostic accuracy of the K-D ET for clinical use in the recognition of SRC. Other than completion time of the K-D ET test, no K-D ET oculomotor parameters were found to be highly sensitive or specific in the diagnosis of concussion in this study. This is vital, as concussion can result in altered visual dynamics, 4 and therefore a clinician needs to have confidence in the diagnostic accuracy of the eye-tracking unit.
Footnotes
Acknowledgment
The authors thank Eric Pimental, certified athletic trainer, for providing the athletic training coverage for the rugby matches and dutifully informing the researchers on injuries sustained during competition. We thank Dongyub Back and the University of Northern Iowa athletic training students for their involvement with participant assessment and logistics. Finally, we thank the University of Northern Iowa’s men’s and women’s rugby teams for their willingness to partake in this study.
Final revision submitted August 27, 2022; accepted September 15, 2022.
The authors declared that they have no conflicts of interest in the authorship and publication of this contribution. AOSSM checks author disclosures against the Open Payments Database (OPD). AOSSM has not conducted an independent investigation on the OPD and disclaims any liability or responsibility relating thereto.
Ethical approval for this study was obtained from the University of Northern Iowa (IRB 19-0029)