Lessons on Data Collection and Curation From the NFL Injury Surveillance Program

Abstract

Background:

“Research-ready” evidence platforms that link sports data with anonymized electronic health records (EHRs) or other data are important tools for evaluating injury occurrence in response to changes in games, training, rules, and other factors. While there is agreement that high-quality data are essential, there is little evidence to guide data curation.

Hypothesis:

We hypothesized that an EHR used in the course of clinical care and curated for research readiness can provide a robust evidence platform. Our purpose was to describe the data curation used for active injury surveillance by the National Football League (NFL).

Study Design:

Dynamic cohort study.

Level of Evidence:

Level 2.

Methods:

Players provide informed consent for research activities through the collective bargaining process. A league-wide EHR is used to record injuries that come to the attention of the teams’ athletic trainers and physicians, NFL medical spotters, or unaffiliated neurotrauma consultants. Information about football activities and injuries are linkable by player, setting, and event to other sports-related data, including game statistics and game-day stadium quality measures, using a unique player identification designed to protect player privacy. Ongoing data curation is used to review data completeness and accuracy and is adjusted over time in response to findings.

Results:

The core data curation activities include monthly injury summaries to team staff, queries to resolve incomplete reporting, and periodic external checks. Experiences derived from producing more than 100 reports per year on diverse topics are used to update coding training and related guidance documents in response to missing data or inconsistent coding that is observed. Roughly 20% more injuries were recorded for the same “reportable” injuries after switching from targeted reporting to an EHR.

Conclusion:

Research-ready databases need systematic curation for quality and completeness, along with related action plans. More injuries were reported through EHR than through targeted reporting.

Clinical Relevance:

Evidence-driven decision-making thrives on reliable data fine-tuned through systematic use, review, and ongoing adjustments to the curation process.

Keywords

football data curation epidemiology surveillance electronic health records

Get full access to this article

View all access options for this article.

References

Califf

RM.

Benefit-risk assessments at the US Food and Drug Administration: finding the balance. JAMA. 2017;317:693-694.

Califf

Robb

Bindman

, et al. Transforming evidence generation to support health and health care decisions. N Engl J Med. 2016;375:2395-2400.

Center for Devices and Radiological Health, Center for Biologics Evaluation and Research, U.S. Food and Drug Administration. Use of real-world evidence to support regulatory decision-making for medical devices. Guidance for industry and Food and Drug Administration staff. US Food and Drug Administration; August, 2017. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/use-real-world-evidence-support-regulatory-decision-making-medical-devices. Accessed June 16, 2019.

Daniel

Silcox

Bryan

, et al. White paper: Characterizing RWD Quality and Relevancy for Regulatory Purposes. Published October 1, 2018. https://healthpolicy.duke.edu/sites/default/files/atoms/files/characterizing_rwd.pdf. Accessed June 16, 2019.

Dreyer

NA.

Advancing a framework for regulatory use of real-world evidence: when real is reliable. Ther Innov Regul Sci. 2018;52:362-368.

Dreyer

Rodriguez

AM.

The fast route to evidence development for value in healthcare. Curr Med Res Opin. 2016;32:1697-1700.

Gliklich

Dreyer

Leavy

MB.

Registries for Evaluating Patient Outcomes: A User’s Guide. 3rd. ed. Rockville, MD: Agency for Healthcare Research and Quality; 2014.

Gliklich

Dreyer

Leavy

Christian

JB.

21st Century Patient Registries: Registries for Evaluating Patient Outcomes: A User’s Guide: Addendum. 3rd ed. Rockville, MD: Agency for Healthcare Research and Quality; 2018.

Hershman

Anderson

Bergfeld

, et al. An analysis of specific lower extremity injury rates on grass and FieldTurf playing surfaces in National Football League Games: 2000-2009 seasons. Am J Sports Med. 2012;40:2200-2205.

10.

Jacobs

King

TRW

Klippel

, et al. Beyond the decade: strategic priorities to reduce the burden of musculoskeletal disease. J Bone Joint Surg Am. 2013;95:e1251-e1256.

11.

Lessley

Kent

Funk

, et al. Video analysis of reported concussion events in the National Football League During the 2015-2016 and 2016-2017 seasons. Am J Sports Med. 2018;46:3502-3510.

12.

Mack

Hershman

Anderson

, et al. Higher rates of lower extremity injury on synthetic turf compared with natural turf among National Football League athletes: epidemiologic confirmation of a biomechanical hypothesis. Am J Sports Med. 2019;47:189-196.

13.

Matava

Gortz

The University of the National Football League: how technology, injury surveillance, and health care have improved the safety of America’s game. J Knee Surg. 2016;29:370-378.

14.

Miksad

Abernethy

AP.

Harnessing the power of real-world evidence (RWE): a checklist to ensure regulatory-grade data quality. Clin Pharmacol Ther. 2017;103:202-205.

15.

Office of Medical Policy and the Office of Translational Sciences, Center for Drug Evaluation and Research, US Food and Drug Administration. Use of Electronic Health Record Data in Clinical Investigations: Guidance for Industry. Silver Spring, MD: US Food and Drug Administration; July, 2018. https://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UCM501068.pdf. Accessed June 16, 2019.

16.

Powell

Schootman

A multivariate risk analysis of selected playing surfaces in the National Football League: 1980 to 1989. An epidemiologic study of knee injuries. Am J Sports Med. 1992;20:686-694.

17.

Sherman

Anderson

Dal Pan

, et al. Real-world evidence—what is it and what can it tell us? N Engl J Med. 2016;375:2293-2297.