Examining the Reliability and Validity of Coding Perceived Force Severity and Bracing in the NHL Concussion Spotter Program

Abstract

Background:

Data obtained from the National Hockey League (NHL) have shown that a risk prediction model, including both visible signs and mechanisms of injury, improves the identification of possible concussion. However, only about half of concussions diagnosed by club medical staff in the NHL exhibit visible signs. At present, the NHL concussion spotter protocol does not include central league spotters’ subjective judgments of the severity of forces associated with a direct hit to the head (perceived force severity [PFS]) or whether players brace before a hit (bracing).

Purpose:

To examine the interrater reliability, preliminary validity, and association with concussion diagnosis of central league spotter determinations of PFS and bracing.

Study Design:

Cross-sectional study.

Methods:

Video footage of 1071 events after a direct or indirect blow to the head were observed from the 2020-2021 and 2021-2022 NHL seasons. These events were classified into 4 groups: concussion with visible signs; concussion without visible signs; no concussion with visible signs; and no concussion without visible signs. A total of 50 events were randomly selected from the total events in each group. Then, 2 raters (NHL central league spotters) coded PFS for each of the 200 video events as low, medium, or high. Bracing was coded as no bracing, insufficient bracing, or full bracing.

Results:

Interrater reliability was fair to moderate for the categorical and continuous ratings of both PFS (κ = 0.36 and 0.45, respectively) and bracing (κ = 0.40 and 0.49, respectively). There was no significant association between concussion diagnosis and either PFS (Z = 0.00, P = .99) or bracing (Z = 0.77, P = .44). Exploratory, post hoc analyses suggested a possible relationship between bracing and reduced concussion risk among a select subsample of events with no visible signs (r = −0.29, P < .01).

Conclusion:

The interrater reliability for PFS and for bracing was fair to moderate. Neither PFS nor bracing were significantly related to concussion diagnosis, but they were significantly associated with other visible signs and mechanisms of injury.

Keywords

video analysis early diagnosis concussion ice hockey visible signs

Concussion is a clinical diagnosis based on observable signs and reported symptoms.²² The ability to detect potential concussions quickly during athletic events is critical for player health. As such, concussion spotter programs have been implemented in several sports to identify visual signs of concussion.^3,9 The optimal combination of signs, symptoms, mechanisms of injury, and other behaviors are still being explored and refined.⁶ Data obtained from the National Hockey League (NHL) show that a risk prediction model that includes both visible signs and mechanisms of injury improves the identification of possible concussion³; however, there is still room for improvement. For example, only about half of players with concussions diagnosed by club medical staff in the NHL exhibit visible signs.⁷ The development of more sensitive and specific models that quickly identify a larger proportion of concussions may improve player health and reduce gametime disruptions (eg, players being sent off the ice for unnecessary evaluations).

The current NHL concussion protocol includes real-time video monitoring during games to detect possible concussions. A staff of observers known as “central league spotters” monitors all games from the NHL Player Safety Room in New York City. All central league spotters are certified athletic trainers and physical therapists who have clinical experience working in elite hockey and have received extensive training to detect visible signs after a direct or indirect blow to the head. In addition, in-arena spotters monitor play in real time and refer all “big hits” for further review by the central league spotters. In-arena spotters are off-ice officials.

Visible signs of concussion include being slow to get up, clutching the head (due to lack of predictive value, clutching of the head was removed as a visible sign after the 2021-2022 season), balance problems/motor incoordination, lying motionlessness, and a blank/vacant look. The central league spotters also code the mechanisms of injury, including head to ice, another player's shoulder to head or upper torso, and being punched with an ungloved fist. Using an objective risk-based algorithm derived in part from previous work,³ central league spotters notify team medical staff whether an evaluation is discretionary or mandatory. Discretionary events include a rink-side check-in with a certified athletic trainer from the player's team to determine whether an off-ice evaluation is required. Mandatory events require that a player leave the playing environment immediately and receive a full Sport Concussion Assessment Tool–5 (Concussion in Sport Group) evaluation and clinical examination in a nondistracting environment. The diagnosis of concussion is made by NHL team physicians, who follow the definition of concussion set forth by the Concussion in Sport Group.¹⁸

There are costs associated with delayed concussion diagnosis, including increased symptom expression and more complicated recovery.^2,17 Identifying and refining heuristics that can be used by medical personnel to identify potential concussion in real time and make rapid decisions is of critical importance. Although often discussed anecdotally in describing player contact, perceived force severity (PFS) and bracing are 2 factors that have not been systematically evaluated in current predictive models. The current NHL protocol does not include spotters’ subjective judgments of the severity of acceleration forces associated with a direct hit to the head (ie, PFS). A growing body of research in recent years has focused on exploring whether measurable linear or rotational forces are associated with the likelihood of sustaining a concussion. This work is rooted in the assumption that a harder hit may be more likely to lead to diagnosis of a concussion.²¹ Further, “big hits” garner significant public and media attention as it is assumed that these hits are more dangerous and therefore more likely to lead to brain injuries. Empirical or quantitative evidence to support that these perceived “big hits” are associated with increased likelihood of concussion diagnosis is still evolving.

Similarly, the current NHL protocol does not include spotters’ subjective judgments of whether players anticipated and braced for the ensuing hit (bracing). It has been argued that if athletes can anticipate events, they can prepare for the impact or avoid it altogether.¹⁴ Bracing the head or the neck ahead of an impact may help to improve dynamic stabilization, ultimately decreasing acceleration/deceleration forces and reducing risk of sustaining a concussion.¹⁴ In a study of rugby players, anticipating an oncoming collision (ie, bracing) was one of several factors independently associated with concussion diagnosis.⁹ Despite this, no studies have examined whether bracing is associated with concussion diagnosis in hockey.

With the above considerations in mind, we examined the interrater reliability and preliminary validity of PFS and bracing as part of a spotter program in professional hockey. The overall goals of the study were to examine the reliability and preliminary validity of spotter PFS and bracing judgments to determine if PFS and bracing might improve concussion detection and triage in the NHL. We hypothesized that PFS and bracing could be reliably coded and that higher PFS and less bracing would be associated with an increased likelihood of subsequent concussion diagnosis.

Methods

Procedures and Measures

All data used in this study were de-identified and analyzed after receiving approval from the University of Missouri—Kansas City Office of Research Compliance. Video footage of 1071 events after a direct or indirect blow to the head from the 2020-2021 and 2021-2022 NHL seasons were reviewed. These events were classified into 1 of 4 groups: concussion with visible signs; concussion without visible signs; no concussion with visible signs; and no concussion without visible signs. A total of 50 events were randomly selected from each of the 4 categories. Two independent raters (certified athletic trainers who work as central league spotters) retrospectively reviewed and systematically coded PFS and bracing for these 200 events; of these events, 100 were concussive events (out of 198 total concussive events during the study period), 50 of which had visible signs (out of a possible of 89 events) and 50 of which did not have visible signs (out of a possible 109 events). Another 100 of these events were nonconcussive events, 50 with visible signs identified by spotters (out of 773 nonconcussive events with visible signs during the study period) and 50 with hits to the head but no visible signs (50 randomly selected events from 100 identified plays that included contact to the head/upper torso but no visible signs). The raters were blinded to the categories of these events and whether or not an event resulted in a diagnosed concussion. See Figure 1 for a CONSORT (Consolidated Standards of Reporting Trials) flow diagram of these events, Table 1 for a summary of the 4 groups, and Table 2 for a summary of the visible signs observed in each of the 4 groups.

Figure 1.

CONSORT (Consolidated Standards of Reporting Trials) diagram of coded events included in the study. NHL, National Hockey League.

Table 1

Summary of the 4 Groups of Events

	Yes Visible Signs	No Visible Signs
Yes concussion	50	50
No concussion	50	50

Table 2

Frequency of Visible Signs Observed in Each of the 4 Event Groups^a

	Visible Signs
	Motionless	Balance Problems	Blank/Vacant	Slow to Get Up	Clutching the Head^b
No Cx, no VS	0	0	0	0	0
Yes Cx, no VS	0	0	0	0	0
No Cx, yes VS	0	2	0	47	19
Yes Cx, yes VS	8	18	2	45	9

Cx, concussion; Motionless, lying motionless, blank/vacant, blank or vacant look; VS, visible signs.

Due to the lack of predictive value, clutching of the head was removed as a visible sign after the 2021-2022 season.

The raters retrospectively reviewed the events for coding and could review video footage of the events as many times as needed, with the option to rewind and play the footage in slow motion using VideoCast, an internal video editor program. In instances where there was discrepant coding between raters, an independent rater's score was used to determine the final ratings for PFS and bracing. Coders were provided with the following definitions to guide their coding. PFS was rated from 1 to 3, corresponding to low, medium, and high presumed force, respectively. Bracing was rated from 0 to 23, corresponding to no bracing, insufficient bracing, or full bracing, with higher scores indicating more effective bracing. The definitions used by the raters for the bracing ratings were as follows:

No bracing: events involving a collision or blindside hit where the player seems to be unaware of the pending contact and appears to take no action to prepare to absorb the hit.

Insufficient bracing: events where the player recognizes contact is pending but is not able to effectively prepare to absorb the hit.

Full bracing: events where the player is either delivering a hit, or the player recognizes pending contact will occur and appears able to absorb the hit by effectively preparing himself.

Approach to Data Analysis

All analyses were conducted using SPSS Version 29 (IBM Corp). The kappa statistic was employed to examine the interrater reliability of the newly devised rating systems for PFS and bracing. Both categorical and continuous reliability were determined. The Cohen kappa was used to measure the categorical agreement, which does not account for “relative” agreement, between the 2 raters. When examining categorical agreement, the reliability is determined only by whether the raters were in absolute agreement; the degree of difference does not factor into the reliability. For example, if rater 1 indicated that an event had low PFS, this rating would be similarly discrepant from a rater 2 indicating that an event had medium PFS or high PFS. Linear weighted kappa was used to determine continuous reliability, which does account for “relative” agreement between the 2 raters.¹⁶ When examining continuous reliability, the degree of the difference between the ratings is also factored into the reliability rating. Using continuous reliability, high PFS and medium PFS show more agreement than high PFS and low PFS. The Landis and Koch¹⁵ criteria for classifying the strength of obtained kappa values were used: 0.0 to 0.2, slight agreement; 0.2 to 0.4, fair agreement; 0.4 to 0.6, moderate agreement; 0.6 to 0.8, substantial agreement; and 0.8 to 1.0, perfect agreement.

The Mann-Whitney U test was used to determine associations between PFS and bracing and visible signs and concussion diagnosis. The Kruskal-Wallis 1-way analysis of variance was used to compare differences in PFS and bracing among the 4 groups (no concussion with no visible signs, concussion with no visible signs, no concussion with visible signs, and concussion with visible signs). Follow-up pairwise comparisons were conducted to examine specific group differences. The associations across PFS and bracing with visible signs, mechanisms of injury, and concussion diagnosis were determined using point biserial correlations. The alpha level was set at p < .05.

Results

Reliability

The final PFS and bracing ratings (for each event) were obtained when 2 raters independently agreed. An independent tiebreaker determined final ratings for events with discrepant coding, using the same definitions provided to the 2 raters. The 2 raters agreed on 125 ratings for bracing, leaving the tiebreaker to determine 75 final consensus ratings. The 2 raters agreed on 120 PFS ratings, and the tiebreaker determined the consensus rating for 80 events.

Categorical Reliability

There was fair agreement between raters’ categorical judgments of PFS (κ = 0.36; 95% CI, 0.26-0.47; P < .001). Overall, for categorical judgments, there was 60% agreement between raters, with 37% of agreement due to chance alone⁸; removing the agreement due to chance, there was 23% agreement between raters. There was also fair agreement between raters’ categorical judgments of bracing (κ = 0.40; 95% CI, 0.29-0.51; P < .001), with 26% agreement between raters once removing agreement due to chance.⁸ PFS and bracing were not significantly correlated (r = 0.03, P = .661).

Continuous Reliability

There was moderate agreement for continuous ratings of PFS (κ = 0.45; 95% CI, 0.35-0.55; P < .001). There was moderate agreement for continuous ratings of bracing (κ = 0.49; 95% CI, 0.39-0.58; P < .001).

Results for PFS

Mann-Whitney U testing revealed that events with visible signs had significantly higher PFS (Z = 3.25, P = .001). No significant association was found between concussion diagnosis and PFS (Z = 0.00, P = .99). A Kruskal-Wallis test showed significant differences among the 4 groups in PFS (H = 16.56, P < .001). A post hoc examination of pairwise comparisons found that events with concussion and visible signs had increased PFS when compared to events with concussion and no visible signs (Z = −3.76, P < .001) and events with no concussion and no visible signs (Z = −2.45, P = .01). In addition, events with no concussion and visible signs had increased PFS when compared to events with concussion and no visible signs (Z = −2.38, P = .02). Results of these pairwise comparisons can be found in Table 3.

Table 3

Kruskal-Wallis Tests of Perceived Force Severity and Bracing with Multiple Pairwise Comparisons by Group^a

Group Comparisons	Mean Rank^b	Median	Z ^c	P
Perceived Force Severity
Yes Cx, no VS/no Cx, no VS			−1.87	.06
Yes Cx, no VS	45.47	2
No Cx, no VS	55.53	2
Yes Cx, no VS/no Cx, yes VS			−2.38	.02
Yes Cx, no VS	44.08	2
No Cx, yes VS	56.92	2
Yes Cx, no VS/yes Cx, yes VS			−3.76	<.001
Yes Cx, no VS	40.37	2
Yes, Cx, yes VS	60.63	3
No Cx, no VS/no Cx, yes VS			−0.61	.54
No Cx, no VS	48.89	2
No Cx, yes VS	52.11	2
No Cx, no VS/yes Cx, yes VS			−2.45	.01
No Cx, no VS	44.05	2
Yes Cx, yes VS	56.95	3
No Cx, yes VS/yes Cx, yes VS			−1.92	.06
No Cx, yes VS	45.51	2
Yes Cx, yes VS	55.49	3
Bracing
Yes Cx, no VS/no Cx, no VS			−1.75	.08
Yes Cx, no VS	45.91	1
No Cx, no VS	55.09	1
Yes Cx, no VS/no Cx, yes VS			−1.71	.09
Yes Cx, no VS	54.96	1
No Cx, yes VS	46.04	0
Yes Cx, no VS/yes Cx, yes VS			−0.99	.33
Yes Cx, no VS	53.09	1
Yes, Cx, yes VS	47.91	0.5
No Cx, no VS/No Cx, yes VS			−3.64	<.001
No Cx, no VS	59.98	1
No Cx, yes VS	41.02	0
No Cx, no VS/yes Cx, yes VS			−2.85	.004
No Cx, no VS	57.94	1
Yes Cx, yes VS	43.06	0.5
No Cx, yes VS/Yes Cx, yes VS			−0.73	.46
No Cx, yes VS	48.62	0
Yes Cx, yes VS	52.38	0.5

Cx, concussion; PFS, perceived force severity; VS, visible signs.

Mean rank = average of the ranks for all observations within each grouping (higher values indicate higher PFS/bracing for that group compared to the group with lower values).

Z value indicates how the average rank for each group compares to the average rank of all observations (negative values indicate that the group's average rank is less than the overall average rank, positive values indicate that a group's average rank is greater than the overall average rank). Example interpretation from the bracing section for the group comparison between “no Cx, no VS/no Cx, yes VS”: both the median bracing rating (1) and the mean rank (59.98) for the “no Cx, no VS” group were higher than the median bracing rating (0) and mean rank (41.02) for the “no Cx, yes VS” group. Overall, the average rank for the “no Cx, no VS” group is less than the overall average rank across all groups (negative Z value).

Higher PFS was associated with increased likelihood of having visible signs (r = 0.22, P < .01), specifically lying motionless (r = 0.14, P < .05), exhibiting balance problems/motor incoordination (r = 0.18, P < .05), and being slow to get up (r = 0.24, P < .01). Higher PFS was also associated with increased likelihood of being hit in the head or upper torso with a shoulder (r = 0.16, P < .05) and hitting the head on the ice (r = 0.22, P < .01). PFS was not significantly associated with a diagnosis of concussion (r = 0.00, P > .05). Correlations among PFS and concussion, visible signs, and mechanisms of injury can be found in Table 4.

Table 4

Point-biserial Correlations Between PFS and Bracing and Concussion Diagnosis, Visible Signs, and Mechanisms of Injury^a

	Dx	Visible Signs						Mechanism of Injury
	Dx	All VS	Motionless	Balance	Blank/Vacant	STGU	Clutching	Shoulder	Ice	Ungloved Fist
PFS	0.00	0.22^b	0.14^c	0.18^c	0.10	0.24^b	-0.04	0.16^c	0.22^b	0.09
Bracing	−0.05	−0.23^b	−0.02	0.06	0.05	−0.25^b	−0.24^b	−0.05	−0.10	−0.40^b

Balance, balance problems; Clutching, clutching the head; Dx, diagnosis of concussion; PFS, perceived force severity; Shoulder, another player's shoulder to head or upper torso; STGU, slow to get up; VS, visible signs.

Statistically significant correlation: P < .01.

Statistically significant correlation: P < .05.

Results for Bracing

Mann-Whitney U testing revealed that players with visible signs were less likely to have braced before a hit (Z = 3.24, P = .001). No significant association was found between bracing and concussion diagnosis (Z = 0.77, P = .44). A Kruskal-Wallis test showed significant between-group differences in bracing (H = 14.326, P = .022). Post hoc pairwise comparisons found that in events with no concussion and no visible signs, players were more likely to have braced when compared to events with concussion and visible signs (P = .005) and events with no concussion and visible signs (P < .001). Results of the pairwise comparisons can be found in Table 3.

Bracing before a hit was associated with a reduced likelihood of visible signs (r = −0.23, P < .01); specifically, players were less likely to be slow to get up (r = −0.25, P < .01) and clutch their heads (r = −0.24, P < .01). Players were more likely to brace before being punched with an ungloved fist (r = −0.40, P < .01). There was no association between bracing and concussion diagnosis (r = −0.05, P > .05). Point-biserial correlations among bracing and concussion, visible signs, and mechanisms of injury can be found in Table 4.

Exploratory Analysis

Given the meaningful and significant correlation (r = −0.40, P < .001), we further explored the relationship between bracing and being punched with an ungloved fist. Post hoc exploration of this finding revealed perfect separation for events that included a punch with an ungloved fist (9 events, all concussed, 4 with accompanying visible signs). All but one of these events were also coded as having full bracing, with the other having partial bracing (8 of 9 events rated the same by both raters). Given the unique nature of this mechanism of injury, we decided to examine whether these events may have had undue influence on the obtained results. After removing these events, bracing, but not PFS, was significantly associated with a reduced likelihood of concussion diagnosis (n = 191, r = −0.15, P < .05 and r = −0.04, P > .05, respectively), albeit with small effects. This effect was magnified when examining events without visible signs in isolation. In this subsample (n = 95), bracing was also significantly associated with reduced risk of concussion (r = −0.29, P < .01). Contrary to expectations, however, increased PFS was associated with a reduced risk of concussion (r = −0.22, P < .05). No significant associations were found between bracing and PFS in the subsample with visible signs (n = 96, r = −0.00, P > .05 and r = 0.14, P > .05, respectively).

Discussion

The NHL concussion program identifies (in part) possible concussions during game play based on visible signs and mechanisms of injury.²⁰ A prediction model then incorporates visible signs and injury mechanisms to stratify risk. About half of concussions diagnosed by club medical staff in the NHL exhibit visible signs.⁷ Therefore, identification of additional observed indicators of possible concussion may improve diagnosis. Presently, the NHL protocol does not include spotters’ subjective judgments of PFS or whether the athlete braced for the hit. The overall goals of the current study were to examine the reliability and preliminary validity of spotters’ PFS and bracing judgments to determine if PFS and bracing might improve concussion detection and triage in the NHL. Contrary to our hypotheses, we did not find a consistent, clinically significant association between PFS/bracing and concussion diagnosis.

Although a relationship between force and concussion diagnosis (eg, some force is required for concussion diagnosis) has been the subject of much discussion and empirical research, our results do not support PFS for use as part of a concussion prediction tool in professional ice hockey. Our mostly null findings related to PFS and concussion diagnosis are similar to the literature showing a lack of consistent association between measurable linear and rotation acceleration of hits and concussion diagnosis.^10,21 In both cases, a “hard hit,” either subjectively observed in our case or objectively measured by accelerometers, does not reliably predict concussion diagnosis. As objective measures may be more reliable, future models should explore the integrated predictive value of accelerometry, visible signs, and mechanisms of injury. Our results demonstrated that higher PFS was associated with reduced likelihood of concussion in a select subsample of events without visible signs. This unexpected finding may be due to type I error (ie, a false-positive result) given the exploratory nature of the post hoc analyses. That is, in conducting additional comparisons as part of post hoc analyses, the chances of a false-positive result increased. Alternatively, randomly selected “big hits” may have higher PFS than standard hits that cause concussion and have no visible signs.

There have been equivocal findings in the literature regarding bracing. Work in the National Rugby League found that bracing before a hit was associated with increased likelihood of concussion.⁹ In contrast, a laboratory study designed to simulate head-to-head impacts in American football examined different muscle activation strategies and found that preactivation responses (ie, anticipating a hit and bracing for it) effectively reduced the rate of concussion.¹² In the current study, we did not find a consistent association between subjectively coded bracing and concussion diagnosis. As part of the post hoc analyses, we did find a relationship between bracing and reduced risk of concussion (r = −0.29, P < .01). However, this finding was isolated to a subset of players who did not exhibit visible signs during a limited number of nonpunch events. The importance of this finding must be tempered by the exploratory nature of the analyses and the potential risk of type I error. More research in larger samples is needed to determine whether bracing is associated with concussion diagnosis in select player circumstances.

Results revealed fair to moderate interrater reliability for PFS and bracing ratings. Several researchers point out that ratings in this range are below the expected norms for medical use.^4,13 An alternative rating system put forth by McHugh¹⁹ suggests that the level of agreement between raters in this study falls in the minimal to weak range, with <15% of the data being considered reliable. Overall, our results indicate that consistently coding PFS and bracing across raters was difficult in the context of professional hockey games. This may be because high head accelerations take place in the order of milliseconds, far below the threshold of what one can perceive with the naked eye, making them challenging to code, even with the ability to review video footage of events. Suboptimal interrater reliability may contribute to the lack of consistent findings showing an association between concussion and PFS and bracing.

Another explanation for the mostly null findings is that causes of concussion are more nuanced than simply observing a “hard hit” or bracing for impact. Individual characteristics, not considered by simply observing a hit, likely increase the risk of concussion. Two people receiving a hit with the same PFS and bracing in the same way may have different outcomes based on factors, such as previous concussion history^1,23 or preexisting conditions, such as poor sleep,²⁴ psychological symptoms,¹¹ or developmental disorders (attention-deficit/hyperactivity disorder or learning disability)⁵ The location of the hit, number and frequency of hits, and the involvement of protective equipment may also play a role in subsequent concussion diagnosis.²⁵ Future models would benefit from incorporating a larger number of events along with additional individual factors that may influence concussion diagnosis.

Bracing and PFS were related to select visible signs of concussion and mechanisms of injury. It is possible that the consequences of the hit influence subjective judgments of PFS. For example, observing a hit that leaves a person on the ice surface for a prolonged period may, in hindsight, increase perceptions of force. Bracing may also be influenced by surrounding contextual circumstances; for example, if the fans in an arena are particularly loud, that may distract a player's attention away from an ensuing event, making it less likely that they will fully brace. Almost by definition, people brace to be hit when involved in a fight. For the purposes of this study, “being hit with an ungloved fist” occurred exclusively during fights. This was confirmed in the present study (with 8 of 9 events showing full bracing). The influence of bracing on subsequent concussion may depend on the mechanism of injury. The present study found no association between bracing/PFS and concussion in the full sample. In contrast, removing events with visible signs and punches, a statistically significant finding emerged between increased bracing and reduced likelihood of concussion. Data related to the development of risk prediction models continue to evolve and it will be important to continue to consider how additional factors may interact to predict concussion risk.

Limitations

This study has several limitations. There is currently no gold standard objective measure that can be used in the diagnosis of concussion. Rather, at present, the diagnosis of concussion is clinical, based on mechanism of injury, a variety of observable signs, examination findings, and self-reported symptoms. The visible signs and mechanisms of injury that were used in the current study may be specific to the NHL and therefore not generalize well to other levels of hockey or other sports. Different playing characteristics, medical histories, environments, protective equipment, and mechanisms of injury may be important to consider when examining the impact of PFS and bracing on subsequent concussion diagnosis. Rather than randomly identifying select events, future studies should prospectively rate a larger number of consecutive events. This approach may allow for the development of a more comprehensive model that can examine mediators and moderators of concussion risk.

To improve ecological validity, we chose not to blind raters to ensuing visible signs. Future studies examining raters’ subjective perceptions may wish to blind the raters to reduce the bias that visible signs may have on PFS and bracing ratings. Future studies may also examine if raters are consistent over time; that is, if they rate the same events the same way at different time points. Lastly, our null findings of coding PFS and bracing may reflect that the consequences of hard hits most likely to be associated with concussion are already captured by empirically supported visible signs and mechanisms of injury. Therefore, hard hits that occur without leading to these specific signs may reflect observed whole body acceleration/deceleration rather than the short-duration, high-acceleration head strikes that may lead to concussion.

Conclusion

In the present study, we found little evidence that PFS and bracing are associated with concussion diagnosis in professional hockey, with modest interrater reliability. Therefore, PFS and bracing do not appear to add clinically significant value to current concussion prediction models that emphasize visible signs and mechanisms of injury.

Footnotes

Acknowledgements

The authors thank the NHL team physicians and athletic trainers/physiotherapists for their continued support and cooperation. They also thank the coders in this project, who work tirelessly in a critical role.

Final revision submitted April 2, 2024; accepted April 12, 2024.

One or more of the authors has declared the following potential conflict of interest or source of funding: K.E.R. is a part-time employee of the NHL. R.E. is a paid consultant for the NHL, co-chair of the NHL/NHLPA concussion subcommittee, occasionally provides expert testimony in matters related to mild traumatic brain injury and sports concussion, and has received funding from the NFL (NFL-Long) through Boston Children's Hospital. W.M. is the chief medical officer for the NHL and an employee of the NHL. P.C. is a member of the NHL/NHLPA Concussion Subcommittee and a paid consultant for the NHLPA. M.G.H. is a member of the NHL/NHLPA Concussion Subcommittee and a paid consultant for the NHLPA. J.S.D. serves as the chief medical consultant for the NHLPA and is the chief medical officer for CF Montreal of Major League Soccer, roles for which he is remunerated. J.M.B. is a part-time employee of the NHL, is a paid consultant for Med-IQ and Sporting KC, and has received grant funding from Genzyme. AOSSM checks author disclosures against the Open Payments Database (OPD). AOSSM has not conducted an independent investigation on the OPD and disclaims any liability or responsibility relating thereto.

Ethical approval for this study was obtained from the University of Missouri–Kansas City (reference No. 259565).

References

Abrahams

Fie

Patricios

Posthumus

September

. Risk factors for sports concussion: an evidence-based systematic review. Br J Sports Med. 2014;48(2):91-97.

Asken

McCrea

Clugston

Snyder

Houcks

Bauer

RM.

“Playing through it”: delayed reporting and removal from athletic activity after concussion predicts prolonged recovery. J Athl Train. 2016;51(5):329-335.

Bruce

Echemendia

Meeuwisse

Hutchison

Aubry

Comper

Development of a risk prediction model among professional hockey players with visible signs of concussion. Br J Sports Med. 2018;52(17):1143-1148.

Cerullo

Quinn

McCleery

Vounzoulaki

Cooper

Sutton

AJ.

Interrater agreement in dementia diagnosis: a systematic review and meta-analysis. Int J Geriatr Psychiatry. 2021;36(8):1127-1147.

Cook

Karr

Iverson

GL.

Children with ADHD have a greater lifetime history of concussion: results from the ABCD study. J Neurotrauma. 2022;39(1-2):86-92.

Davis

Makdissi

Bloomfield

, et al. International study of video review of concussion in professional sports. Br J Sports Med. 2019;53(20):1299-1304.

Echemendia

Bruce

Meeuwisse

Hutchison

Comper

Aubry

Can visible signs predict concussion diagnosis in the National Hockey League?

Br J Sports Med. 2018;52(17):1149-1154.

Fleiss

Cohen

Everitt

BS.

Large sample standard errors of kappa and weighted kappa. Psychological Bulletin. 1969;72(5):323-327.

Gardner

Howell

Iverson

GL.

A video review of multiple concussion signs in National Rugby League match play. Sports Med Open. 2018;4(1):5.

10.

Guskiewicz

Mihalik

Shankar

, et al. Measurement of head impacts in collegiate football players: relationship between head impact biomechanics and acute clinical outcome after concussion. Neurosurgery. 2007;61(6):1244-1252.

11.

Iverson

Gardner

Terry

, et al. Predictors of clinical recovery from concussion: a systematic review. Br J Sports Med. 2017;51(12):941-948.

12.

Jin

Feng

Mika

Viano

Yang

KH.

The role of neck muscle activities on the risk of mild traumatic brain injury in American football. J Biomech Eng. 2017;139(10). doi:10.1115/1.4037399

13.

Kottner

Audige

Brorson

, et al. Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed. Int J Nurs Stud. 2011;48(6):661-71.

14.

Kung

Suksreephaisan

Perry

Palmer

Page

RA.

The effects of anticipation and visual and sensory performance on concussion risk in sport: a review. Sports Med Open. 2020;6(1):54.

15.

Landis

Koch

GG.

The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-174.

16.

Marasini

Quatto

Ripamonti

Assessing the inter-rater agreement for ordinal data through weighted indexes. Stat Methods Med Res. 2016;25(6):2611-2633.

17.

McCrea

Hammeke

Olsen

Leo

Guskiewicz

Unreported concussion in high school football players. Clin J Sport Med. 2004;14(1):13-17.

18.

McCrory

Meeuwisse

Dvorak

, et al. Consensus statement on concussion in sport—the 5th international conference on concussion in sport held in Berlin, October 2016. Br J Sports Med. 2017;51:838-847.

19.

McHugh

ML.

Interrater reliability: the kappa statistic. Biochem Med (Zagreb). 2012;22(3):276-282.

20.

NHL Public Relations. NHL concussion evaluation and management protocol for 2022-2023 season. NHL. July 15, 2022. Accessed February 18, 2024. https://www.nhl.com/news/nhl-concussion-evaluation-and-management-protocol-for-2022-23-season-335002568

21.

O’Connor

Rowson

Duma

Broglio

SP.

Head-impact-measurement devices: a systematic review. J Athl Train. 2017;52(3):206-227.

22.

Patricios

Schneider

Dvorak

, et al. Consensus statement on concussion in sport: the 6th International Conference on Concussion in Sport-Amsterdam, October 2022. Br J Sports Med. 2023;57(11):695-711.

23.

Putukian

Riegler

Amalfe

Bruce

Echemendia

Preinjury and postinjury factors that predict sports-related concussion and clinical recovery time. Clin J Sport Med. 2021;31(1):15-22.

24.

Riegler

Guty

Thomas

Bradson

Arnett

PA.

Functional outcomes, injury variables, and athlete characteristics associated with post-concussion sleep disturbance. Arch Clin Neuropsychol. 2023;38(2):182-195.

25.

Sokol-Randell

Rotundo

Tierney

Deasy

Cusimano

MD.

Characteristics of potential concussive events in elite male Gaelic football players: a descriptive video-analysis. J Sports Sci. 2021;39(15):1700-1708.