Translating Data Analytics Into Improved Spine Surgery Outcomes: A Roadmap for Biomedical Informatics Research in 2021

Abstract

Study Design:

Narrative review.

Objectives:

There is growing interest in the use of biomedical informatics and data analytics tools in spine surgery. Yet despite the rapid growth in research on these topics, few analytic tools have been implemented in routine spine practice. The purpose of this review is to provide a health information technology (HIT) roadmap to help translate data assets and analytics tools into measurable advances in spine surgical care.

Methods:

We conducted a narrative review of PubMed and Google Scholar to identify publications discussing data assets, analytical approaches, and implementation strategies relevant to spine surgery practice.

Results:

A variety of data assets are available for spine research, ranging from commonly used datasets, such as administrative billing data, to emerging resources, such as mobile health and biobanks. Both regression and machine learning techniques are valuable for analyzing these assets, and researchers should recognize the particular strengths and weaknesses of each approach. Few studies have focused on the implementation of HIT, and a variety of methods exist to help translate analytic tools into clinically useful interventions. Finally, a number of HIT-related challenges must be recognized and addressed, including stakeholder acceptance, regulatory oversight, and ethical considerations.

Conclusions:

Biomedical informatics has the potential to support the development of new HIT that can improve spine surgery quality and outcomes. By understanding the development life-cycle that includes identifying an appropriate data asset, selecting an analytic approach, and leveraging an effective implementation strategy, spine researchers can translate this potential into measurable advances in patient care.

Keywords

biomedical informatics health information technology data analytics machine learning implementation science big data;spine surgery

Introduction

Spine surgery has a proud history of applying rigorous research and technological innovations to advance the care of patients with complex spine disease. Historically, many technological advances came from biomedical engineering, which has contributed to improved imaging modalities, implant technology, and fusion biologics. However, there is increasing acknowledgment that spine surgeons still face substantial uncertainty related to basic treatment questions, such as the likelihood of surgical success and chance of postoperative complications.¹ This recognition has increased the focus on using data science to transform spine surgery practice. Supporting this mission, there has been explosive growth in healthcare data – an estimated 16,000 exabytes in 2018.² The increased availability of data assets has expanded opportunities to use biomedical informatics tools to improve virtually all aspects of spine care, including: diagnosis and imaging classification; treatment selection and risk prediction; perioperative management; and administrative tasks.^3,4

Nonetheless, data science has not yet transformed spine surgery in the way it has some areas of medicine and society. Despite the increasing availability of large data assets and advanced computing power, we remain far from the goal set by the Institute of Medicine in 2007 to have 90% of clinical decisions supported by accurate, timely clinical information by the year 2020.⁵ To narrow this divide, spine surgeons must understand how key decisions related to dataset selection, analytic techniques, and implementation strategy influence the clinical impact of health information technology (HIT).⁶ Recognizing these important considerations (Figure 1), this review will provide a HIT roadmap to help realize the potential of data assets and biomedical informatics tools to improve spine surgery practice.

Figure 1.

A diagram depicting the process of developing and implementing new health information technology in spine surgery. EHR indicates electronic health record.

Types of Data Assets

An overview of data assets available for spine surgery informatics research is shown in Table 1.

Table 1.

A summary of the Strengths, Limitations, and Ideal Uses for Data Assets used in Spine Surgery Biomedical Informatics Research.

Data asset	Strengths	Limitations	Uses
Administrative (claims) data	▪ Large sample size ▪ Inexpensive ▪ Clinical and cost data ▪ Population coverage ▪ Structured data	▪ Unreliable data accuracy ▪ Limited breadth of data (e.g. no lab or imaging data) ▪ Delayed availability (due to coding and processing)	▪ Population-level outcome trends⁷ ▪ Health policy and cost analysis⁸ ▪ Linkage with clinical registries⁹
Spine surgery registries	▪ Relatively large sample size ▪ High quality, “real world” data ▪ Condition-specific data collection (e.g. patient-reported outcomes, imaging) ▪ Structured data	▪ Expensive to establish and maintain ▪ Narrow clinical focus and data collection ▪ Delayed data availability	▪ Quality improvement programs¹⁰ ▪ Comparative effectiveness studies¹¹ ▪ Hypothesis generation for clinical trials¹²
Electronic Health Records	▪ Real-time data acquisition ▪ Wide breadth of data (e.g. clinical, imaging, free-text) ▪ Inexpensive to access ▪ Large sample size	▪ Inconsistent data quality ▪ Often lack patient-reported outcomes ▪ May lack generalizability ▪ Unstructured data	▪ Real-time safety alerts¹³ ▪ Integration with clinical registries¹⁴ ▪ Quality and outcomes research¹⁵
Mobile data	▪ Real-time, real-world data collection ▪ Detailed physical function data ▪ Rapid data acquisition	▪ Limited availability ▪ Socioeconomic barriers to use ▪ Uncertain patient acceptance ▪ Interpreting clinical importance can be challenging ▪ Interoperability and data storage	▪ Physical function assessments¹⁶ ▪ Real-world data acquisition¹⁷ ▪ Physiologic data collection ▪ Low patient burden
Biobanks	▪ Individualized ▪ Biological detail	▪ Expensive ▪ Limited availability ▪ Patient privacy concerns	▪ Precision medicine (e.g. risk prediction, drug targeting)¹⁸ ▪ Linkage with EHR/registry data¹⁹

Administrative Datasets

Administrative datasets based on billing claims have been used frequently in spine surgery research, likely due to their widespread availability, relatively low cost, structured data, and population-level coverage.^20-22 These datasets have provided important insights into the effectiveness of policy interventions,²³ surgical costs,⁸ and population-level trends.^8,7 However, diagnoses from billing codes are often imprecise and lack imaging data,²⁴ limiting the ability to evaluate clinical outcomes. For example, billing data have limited ability to distinguish key surgical variables, such as the number of levels treated or the use of minimally invasive techniques, confounding comparative effectiveness research efforts. Although technically complex, linking administrative and clinical registry data can help overcome some of these limitations and broaden potential applications.⁹

Spine Surgical Registries

Spine surgery registries are experiencing increasing growth and attention. A 2015 systematic review identified 25 registries representing 14 countries.²⁵ Among the most recent, the American Spine Registry has emerged as a successor to the Quality Outcomes Database with the goal of unifying neurosurgery and orthopedic registries efforts.^25,26 Other registries, such as the International Spine Study Group and European Spine Study group, have focused on particular spine populations, such as deformity.^27-29 These registries offer advantages over claims data, including data quality control, detailed patient characteristics, and inclusion of patient-reported outcomes.^25,10,30 These attributes have generated substantial enthusiasm among both surgeons and hospital administrators.³¹ Nonetheless, few registries capture imaging data, and standards for processing and storing these data are lacking. Additionally, considering maintenance fees and the need for a full-time employee for data review, establishing a multicenter registry can cost millions of dollars.^30,32,33 Finally, most registries are not designed to collect real-time patient data. Linking registries with electronic health records (EHR) and mobile health data offers an opportunity to decrease their cost and expand potential uses.^34,35

Electronic Health Records

EHRs represent an expansive and underutilized source of spine surgery data. Currently, at least 98% of hospitals have adopted an EHR system, creating vast quantities of patient data, updated in real time.³⁶ The EHR offers spine surgeons valuable opportunities to both develop and implement informatics tools. While many surgeons are familiar with using the EHR for simple research tasks (e.g. identifying patients by procedure code), its full potential has largely been untapped. For example, automated workflows are capable of populating quality improvement registries,^35,37 though such pipelines are not routine. Additionally, multidimensional EHR data can be used in real-time to support evidence-based decision-making. For example, a model predicting surgical complications evaluated 285 clinical, demographic, administrative, and laboratory variables to develop a prediction tool that processes EHR data in real-time to provide risk predictions at the point-of-care.³⁸ In spine surgery specifically, the use of real-time EHR analytics to support decision-making has been less common, though there have been notable successes, such as clinical decision-support for guiding appropriate spine imaging.^39,40 Challenges to leveraging insights from the EHR include the frequent use of unstructured data (e.g. clinic notes), non-random missing data, and inconsistent data quality.^41,42 Additionally, generating multicenter datasets is often challenging because many EHRs, even from major vendors, store data in unique, institution-specific ways. Nonetheless, with continued efforts in areas such as natural language processing,⁴³ opportunities to replace manual chart abstraction with sophisticated EHR queries continue to expand and are likely to assume a growing role in spine surgery research and quality improvement. Likewise, broad adherence to interoperability standards will facilitate the implementation of analytic pathways and clinical decision support across health systems.⁴⁴

Mobile Data

Mobile health (mHealth) is at the vanguard of biomedical informatics, with both researchers and “Big Tech” companies vying to capitalize on the increasing use of smartphones and wearable technology.⁴⁵ By its nature, mHealth removes many barriers of having patients complete outcomes questionnaires, and in this way, might pave the way for seamlessly collecting population-based physical outcomes data. Indeed, there is expanding evidence for the role of mHealth in postoperative monitoring after spine surgery, ^17,46,47 and there are increasingly available commercial applications intended to aid post-discharge patient surveillance.^48-50 Particularly notable, one study used a mobile application to aid postoperative monitoring for over 1,600 enhanced recovery after surgery patients. More generally, mHealth used in spine surgery has shown success in collecting patient-reported outcome measures,⁵¹ decreasing surgical cancellations,⁵² monitoring postoperative recovery,^17,46 and guiding postoperative rehabilitation.⁵³ In other fields, mHealth applications have also been used to support behavioral modification related to factors that may impact spine surgery outcomes (e.g. cardiovascular disease, medication compliance).^54,55 Despite such promises, there remain important obstacles to more widespread use of mHealth. Several studies have shown that only a minority of patients use such applications regularly,^17,51 and despite promising reports, rigorous evidence demonstrating improved outcomes or decreased cost is lacking.⁵⁶ For example, despite increasing use of mobile sensor data to study activity measures, such as step count,^57,58 there is sparse evidence demonstrating the extent to which such real-time measures reliably capture physical function or quality of life.⁵⁸ Additional barriers to expanding mHealth include patient reservations related to privacy protection and technology familiarity, socioeconomic disparities in access,^59,60 and uncertainties related to data and evidence quality.⁶¹ Finally, there remains an ongoing need to integrate mHealth technology with existing EHR systems, which is often a complicated and costly endeavor.⁶² As these barriers are overcome, spine surgery practice will benefit from new efficiencies and care pathways, while researchers will derive new insights from high-frequency, real-world data collection.

Biobanks

Genomic, proteomic, and metabolomic (i.e., ‘omic’) data assets serve an essential role in tailoring treatment selection and outcome prediction to individual patient characteristics. Biobanks have been slow to take hold in spine surgery. Current spine-related biobanks focus on tumor samples, such as the Chordoma foundation biobank, and spinal cord injury.^63-65 However, novel insights regarding osteoarthritis from the UK Biobank demonstrate that other areas of spine surgery, particularly degenerative disease, could benefit from these pooled resources.⁶⁶ To maximize their impact, ‘omic’ data should be integrated with more complete clinical information. Given the substantial resources required, more widespread adoption of spine surgery biobanks will require support from funding bodies and innovative solutions from data scientists, such as linking biochemical data with clinical EHR platforms.⁶⁷

Analytical Techniques

As important as selecting an appropriate dataset is the analytical approach used to investigate those data. While some authors describe a continuum between fully human-guided and machine-guided statistical techniques,⁶⁸ we will distinguish traditional regression techniques from newer machine-guided approaches.⁴ Each of these analytical techniques contains multiple nuances and variations, including approaches to handling clustered and longitudinal data. Detailed reviews are available on such topics.^4,69-71 Our goal is to provide an overview of the key advantages and weaknesses of each approach, along with the applications each is best suited to address (Table 2).

Table 2.

Strengths, Weaknesses, and Ideal Use of Regression Versus Machine Learning Techniques.

	Regression models	Machine learning
Strengths	Familiar to researchers and clinical spine surgeons High model transparency Established techniques to test statistical significance of observed differences	Able to model complex patterns and unstructured data Not bound by pre-existing assumptions Superior predictive power (in some circumstances)
Weaknesses	Assumptions of linearity and additivity Difficulty modeling unstructured data Decreased predictive power (in some circumstances)	Decreased model transparency Higher sample size requirements Less familiar to spine surgeon researchers
Ideal Use	Risk models using structured data Conducting inference related to treatment outcome and cost Evaluating policy interventions	Modeling high volumes of unstructured data (e.g. real-time EHR output, mobile health data) Interpreting imaging data, mobile activity sensors

Regression Models

Regression models – including linear, logistic, and proportional hazards regression – are the traditional workhorse of statistical modeling. Regression models are generally designed to evaluate categorical and linear predictors, but techniques also exist for modeling non-linearity, including restricted splines and fractional polynomials.⁷² While several approaches exist to help automate variable selection and prevent overfitting,^73,74 variable selection and other modeling choices – such as interaction testing – remain heavily influenced by expert knowledge.⁷² While regression models are effective at risk prediction, they are particularly valuable for testing the statistical significance of observed variations, including surgical costs, clinical outcome, and health policy interventions.^11,75,76 Finally, regression results are generally easy to interpret, facilitating the identification of clinically relevant relationships and possibly enhancing surgeon acceptance of risk predictions.⁷²

Machine Learning

Machine learning refers to the intersection of statistics and computer science dedicated to using computing power to make predictions by recognizing patterns within data.⁷¹ Most applications of machine learning familiar to spine surgeons would be categorized as supervised learning, which involves training a model to predict a known outcome (e.g., postoperative complications, a fracture on CT).^71,77 By comparison, unsupervised learning involves using computers to detect new patterns in data, such as defining disease categories without preexisting constraints. Due to their advantages detecting novel classifications within high dimensional data, unsupervised approaches are likely to assume a dominant role in the future, though at present remain relatively uncommon in spine surgery and clinical medicine.

While variable interactions and spline transformations can extend regression techniques, they are largely bound by assumptions related to linearity and additivity (i.e., predictor variables have an additive effect on the outcome). By comparison, machine learning can accommodate much more complex patterns and unstructured data that may more accurately reflect spine surgery practice.⁷¹ A variety of machine learning techniques, including random forests, support vector machines, and convolutional neural networks have been developed for this purpose.⁷⁸ Yet machine learning approaches have important shortcomings, including a lack of interpretability (i.e., the “black box” problem) or clinical applicability, and higher sample size requirements.^79,80 Advances in “interpretable machine learning” have helped address some of these shortcomings but still do not fully replicate an inherently interpretable modeling structure.⁸¹

Selecting an Analytical Approach

Overall, regression techniques are better suited to making inferences (e.g. are outcomes from fusion better than decompression), given their greater transparency and well-defined approaches for determining statistical significance. Machine learning may offer advantages when engaging in prediction, though such gains are far from certain.⁸² Benefits of machine learning are likely to be most pronounced when dealing with complex datasets, and large sample sizes (e.g. thousands of cases) are often needed to yield stable predictions.^68,80 These limitations, combined with the relatively simple nature of many clinical datasets, likely explain the fact that machine learning approaches have often shown modest if any advantages compared to regression in many spine clinical prediction studies.^83-86 Consequently, investigations using machine learning for clinical predictions should demonstrate sufficient improvements in predictive performance to justify the loss of interpretability.

By comparison, machine learning has shown greater success when dealing with complex data assets, such as high volume EHR data, mobile sensors, and imaging analysis.^4,87 For example, machine learning approaches have been used to aid preoperative planning in deformity surgery,^88,89 and also to classify gait abnormalities based on mobile sensor data.^90,91 Likewise, machine learning approaches have proven effective at analyzing high-volumes of EHR data in real-time to aid postoperative risk predictions at the point-of-care.⁹² Other innovative efforts, such as integrating high-volume clinical and imaging data with expert opinion to improve patient classification in spondylolisthesis, are ongoing.⁹³

Future Perspectives

While regression techniques remain a mainstay in spine surgery research, there are a variety of approaches that have received scant attention and may open new analytic opportunities in the future. For example, multilevel models are well-suited to modeling hierarchical data (e.g. distinguishing patient vs. surgeon effects) as well as longitudinal trends (e.g. postoperative recovery trajectory).^94,95 Likewise, spine surgeons should consider making use of emerging techniques like generalized additive models, which allow substantial flexibility in modeling complex relationships while preserving interpretability.⁹⁶ Finally, as large data assets continue to expand, so too will the role for machine learning techniques, particularly unsupervised approaches that may identify novel phenotypes of complex disease.⁹⁷ Therefore, the emerging challenge for spine researchers is learning how best to deploy these powerful resources.

Implementation and Evaluation

Rigorous analytics applied to appropriate data assets serve as the foundation for effective HIT, such as clinical decision support predicting postoperative complications or tools to help select osteotomy sites for planning deformity correction. However, to effectively impact spine surgical practice, new HIT must be adopted by diverse stakeholders within complex healthcare systems. These challenges may be particularly prominent in spine surgery, where surgeon preference and institutional traditions remain important influences on management practices. Many of the concepts relevant to implementing HIT may be unfamiliar to spine surgeons, but identifying how and when such approaches can be used is key to moving biomedical informatics from the research setting into clinical practice.

Human-centered Design

Human-centered design (HCD) and evaluation refers to an iterative process that involves users throughout the design lifecycle to ensure that new HIT meets the needs and preferences of end-users.^98,99 After an initial HIT prototype is developed based on user-specified requirements,⁹⁸ formal usability and usefulness testing should be completed in a simulated environment prior to clinical implementation.¹⁰⁰ A number of mixed methods approaches can be employed to assess usability, such as the think-aloud technique, which elicits users’ thoughts and feelings as they use the new technology.¹⁰¹ This think-aloud approach has been used to evaluate a virtual reality vertebroplasty simulator and a novel outcome assessment tool for spine trauma, identifying potential problems and suggestions for improvement.^102,103 Another approach, cognitive walkthroughs, involves a trained evaluator analyzing the cognitive processes required to use new HIT, thereby identifying potential discrepancies between designers’ and users’ understanding of a task.¹⁰⁴ This technique was used to evaluate a dashboard for presenting predicted patient-reported outcomes to spine surgery patients.⁹⁸ Alternatively, heuristic evaluation uses human-computer interaction experts to identify usability problems based on established heuristic principles that may be missed with user testing.^104,105 This approach was used in combination with cognitive walkthrough to optimize the patient-reported outcome dashboard noted above.¹⁰⁶ After completing these types of evaluations, field testing in clinical settings can reveal real-world problems not identified in a laboratory environment.^107,108 An exhaustive discussion of the HCD process is beyond the scope of this review, and many of the approaches involved, particularly the mixed methods techniques, may be unfamiliar to most spine surgeons. Consequently, surgeons seeking to implement new HIT should seek out methods experts to assist in this process.

Sociotechnical Analysis

Sociotechnical analysis provides a conceptual framework to evaluate the interconnected organizational, human, and technical elements impacting the adoption of HIT.¹⁰⁹ Sociotechnical analysis focuses on the following aspects of implementation: the hardware and computing infrastructure; clinical content; human-computer interface; people; clinical workflow and communication; organizational policies, procedures, and culture; and system measurement and monitoring after implementation.¹¹⁰ In doing so, this approach provides a foundation for studying key implementation measures, such as barriers and context.¹¹¹ Sociotechnical analysis is typically pursued through qualitative interviews with stakeholders, though surveys and EHR interrogation can also be used.¹¹² This approach has rarely been used in spine research, though one study conducted a sociotechnical analysis to evaluate clinical video telehealth for spinal cord injury patients.¹¹³ There have also been limited successes using this approach to inform the implementation of clinical decision support in other surgical populations, such as patients with traumatic brain injury and patients requiring orthopedic imaging.^114,115 Spine surgeons developing new HIT should consider conducting a sociotechnical analysis to improve the likelihood that their intervention will be successfully integrated into clinical practice.

EHR Log Analysis

Traditional approaches to understanding how clinicians interact with HIT include interviews, surveys, and direct observation.¹¹⁶ While informative, such methods are labor and resource intensive and may not capture the full variability in care processes. Addressing these short-comings, EHR log analysis evaluates the time users spend performing different EHR-related tasks.¹¹⁶ This technique can be used to assess usage behaviors and clinical workflow, describe HIT demands, and evaluate the impact of HIT on care processes.¹¹⁶ This technique has been used to study time demands by surgical residents and currently represents an untapped opportunity for spine surgeons to collaborate with informatics experts to evaluate clinical practices and new HIT interventions.^117,118

Implementation Trials

The most rigorous approach for evaluating new HIT is an implementation trial, which typically assumes a cluster-randomized design.¹¹⁹ These studies are often designed to evaluate effectiveness outcomes, such as a trial for a machine learning-derived early warning system for intraoperative hypotension.¹³ However, focusing only on effectiveness creates a missed opportunity to study key implementation outcomes, such as context, barriers, and facilitators.¹²⁰ Implementation trials for spine disease have evaluated the role of mobile phone-based postoperative rehabilitation and an online application for managing low back pain, providing high-level evidence of the effectiveness of these interventions.^53,121 While labor and resource intensive, for high-stakes HIT interventions—including those that may warrant reimbursement from payers—implementation trials remain the gold standard for demonstrating an impact on health outcomes and care delivery.

Challenges and the Path Forward

Realizing the potential of biomedical informatics to transform spine surgery will involve navigating a variety of challenges and considerations, which are summarized in Figure 2. Among the most important challenges spine surgeons should consider are:

Figure 2.

A summary of the key challenges and considerations in developing, implementing, and maintaining health information technology.

Click Fatigue

With the increasing adoption of EHRs, spine surgeons, like most physicians, are inundated with alerts, more than half of which are overridden.¹²² To reduce click fatigue, researchers should focus on identifying when data analytics tools are most likely to impact clinical outcomes.^123,124 Likewise, adoption of HIT interventions will be enhanced by focusing on design strategies that reduce the cognitive workload demanded of busy spine surgeons.¹²⁵

Model Maintenance

Like any medical device, successful predictive models must be maintained over time and across different healthcare settings, adding to their long-term costs.¹²⁶ Counterintuitively, the more effectively a model impacts practice and improves outcomes, the more its performance may diminish over time with changing conditions.¹²⁷ Similarly, changing practice patterns and patient characteristics often lead to a decay in model performance over time.¹²⁸ Furthermore, many predictive models suffer from poor portability across institutions,⁴¹ as was found in a model predicting infections after spine surgery.¹²⁹ More efficient systems for sharing, testing, and updating prediction models across institutions are needed in spine surgery and medicine more broadly, particularly to make these tools accessible to smaller institutions with limited information technology resources.¹³⁰

Regulation and Oversight

As HIT interventions assume increasingly prominent roles in spine surgery practice, the role of government regulation must be defined. A recent review found that nearly half of healthcare applications did not describe their content source,¹³¹ and several popular healthcare applications have been removed for poor clinical accuracy.⁶² Given the high-risk nature of spine surgery, surgeons seeking to broadly implement new HIT (e.g. to guide patient selection) should proactively consider engaging with regulatory bodies to preserve innovation while ensuring the rigor of HIT interventions.

Stakeholder Acceptance

To increase acceptance of HIT among spine surgeons, researchers must address doubts related to the quality of their underlying evidence and how these interventions interact with existing clinical practices.⁶² Soliciting diverse surgeon feedback early in HIT development is therefore key to decreasing conflict between established practices and new interventions. Finally, data analytics tools should augment rather than replace clinical experience, and explicitly incorporating surgeon judgment into predictive models may enhance stakeholder acceptance.¹³²

Ethical Challenges

Relying on purely data-driven, particularly machine-based predictions to guide spine surgery decision-making has the potential to accentuate disparities based on race and socioeconomic status. Specifically, models built to mimic human decision-making may reinforce known disparities in treatment access and outcomes.¹³³ Furthermore, data assets may not contain adequate representations of minority groups, leading to decreased predictive performance in those populations.^133,134 Recognizing these potential challenges will allow spine surgeons to maximize the ethical use of HIT.

Conclusions

The growth in HIT has provided access to data and computing resources previously unattainable in spine surgery, which has contributed to a rapid rise in informatics research. Like nearly all technology, biomedical informatics in spine surgery is subject to the “hype cycle model” described by Gartner Inc., summarizing the path toward sustained use of new innovations.¹³⁵ At present, we are likely experiencing the peak of inflated expectations. To truncate the trough of disillusionment associated with unmet expectations, spine surgery researchers should recognize the strengths and limitations of diverse data assets and analytic tools, while also leveraging effective HIT implementation strategies. Through navigating these complex considerations, spine research may move toward a plateau of productivity, where new HIT innovations produce meaningful advances in spine surgery quality and outcomes.

Footnotes

Acknowledgments

We thank Ms. Kelley Foyil for her assistance with manuscript proof reading and editing.

Declaration of Conflicting Interests

The author(s) declared no potential conflicts of interest with respect to the research, authorship, and/or publication of this article.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Dr. Ray reports: stock/equity in Acera surgical; consulting support from Depuy/Synthes, Globus, and Nuvasive; royalties from Depuy/Synthes, Nuvasive, Acera surgical. Dr. Kelly received personal fees from The Journal of Bone and Joint Surgery. Dr. Molina reported equity in Augmedics and consulting fees from Depuy/Synthes and Kuros. Dr. Greenberg was supported by an Early Career Award from the Thrasher Research Fund (Award #15024) and a National Research Service Award from the Agency for Healthcare Research and Quality (Award #1F32HS027075-01A1). Dr. Ray has no funding related to this work. Dr. Ray received research support from the Defense Advanced Research Projects Agency, Department of Defense, Missouri Spinal Cord Injury Foundation, National Institute of Health/NINDs, Hope Center, and Johnson & Johnson. Dr. Foraker received no funding specifically related to this study. Dr. Foraker reports research support from the Washington University Institute for Public Health, National Institutes of Health, Global Autoimmune Institute, Agency for Healthcare Research and Quality, Siteman Investment Program, Alzheimer’s Drug Discovery Foundation, and Children’s Discovery Institute. Dr. Ghogawala received no funding specific to this study. Dr. Ghogawala received research support from the Patient-Centered Outcomes Research Institute and the National Institutes of Health. Dr. Yen reported no funding related to this submission. Dr. Limbrick reported no funding related to this submission. Dr. Limbrick received research support from the National Institutes of Health, the Patient-Centered Outcomes Research Institute, the Hydrocephalus Association, Medtronic Inc., Karl Storz Inc., and Microbot Medical, Inc. Dr. Limbrick also received philanthropic equipment contributions for humanitarian relief work from Karl Storz, Inc. and Aesculap, Inc. Dr. Kelly reported no funding related to this submission. Dr. Kelly received research support from the Setting Scoliosis Straight Foundation and the International Spine Study Group Foundation. The funding sources for this study had no role in the study design, collection of the data, writing of the manuscript, or decision to submit the manuscript for publication.

ORCID iDs

Jacob K. Greenberg, MD, MSCI

Zoher Ghogawala, MD

Michael P. Kelly, MD, MSc

References

National Quality Forum. NQF-Endorced Measures for Surgical Procedures, 2015-2017. Washington, DC; 2017.

Dash

Shakyawar

Sharma

Kaushik

. Big data in healthcare: management, analysis and future prospects. Journal of Big Data. 2019;6(1):54.

Lee

Grabowski

Habboub

Mroz

. The impact of artificial intelligence on quality and safety. Global Spine J. 2020;10(1):99s–103s.

Loftus

Tighe

Filiberto

, et al. Artificial Intelligence and surgical decision-making. JAMA Surg. 2020;155(2):148–158.

McGinnis

Aisner

Olsen

. The Learning Healthcare System: Workshop Summary. National Academies Press; 2007.

Rumsfeld

Joynt

Maddox

. Big data analytics to improve cardiovascular care: promise and challenges. Nat Rev Cardiol. 2016;13(6):350–359.

Cram

Landon

Matelski

, et al. Utilization and outcomes for spine surgery in the United States and Canada. Spine (Phila Pa 1976). 2019;44(19):1371–1380.

Kahn

Ellimoottil

Dupree

Park

Ryan

. Variation in payments for spine surgery episodes of care: implications for episode-based bundled payment. J Neurosurg Spine. 2018;29(2):214–219.

Columbo

Martinez-Camblor

MacKenzie

, et al. Comparing long-term mortality after carotid endarterectomy vs carotid stenting using a novel instrumental variable method for risk adjustment in observational time-to-event data. JAMA Netw Open. 2018;1(5):e181676.

10.

Asher

McCormick

Selden

Ghogawala

McGirt

. The National Neurosurgery Quality and Outcomes Database and NeuroPoint Alliance: rationale, development, and implementation. Neurosurg Focus. 2013;34(1):E2.

11.

Karsy

Chan

Mummaneni

, et al. Outcomes and complications with age in spondylolisthesis: An evaluation of the elderly from the Quality Outcomes Database. Spine (Phila Pa 1976). 2020;45(14):1000–1008.

12.

Park-Reeves Syringomyelia Research Consortium. Published 2020. Updated March 30, 2021. Accessed April 4, 2020. https://park-reeves.wustl.edu/Default.aspx. Park-Reeves Syringomyelia Research Consortium

13.

Wijnberge

Geerts

Hol

, et al. Effect of a machine learning-derived early warning system for intraoperative hypotension vs standard care on depth and duration of intraoperative hypotension during elective noncardiac surgery: the hype randomized clinical trial. JAMA. 2020;323(11):1052–1060.

14.

Ghogawala

Dunbar

Essa

. Lumbar spondylolisthesis: modern registries and the development of artificial intelligence. J Neurosurg Spine. 2019;30(6):729–735.

15.

Stopa

Yan

Dasenbrock

Kim

Gormley

. Variance reduction in neurosurgical practice: the case for analytics-driven decision support in the era of big data. World Neurosurg. 2019;126:e190–e195.

16.

Rao

Phan

Maharaj

Pelletier

Walsh

Mobbs

. Accelerometers for objective evaluation of physical activity following spine surgery. J Clin Neurosci. 2016;26:14–18.

17.

Glauser

Ali

Gardiner

, et al. Assessing the utility of an IoS application in the perioperative care of spine surgery patients: the neuropath pilot study. Mhealth. 2019;5:40.

18.

Brabetz

Leary

SES

Grobner

, et al. A biobank of patient-derived pediatric brain tumor models. Nat Med. 2018;24(11):1752–1761.

19.

Suki

Wildrick

Sawaya

. A time-tested information system in neurosurgical oncology. Front Oncol. 2018;8:593.

20.

Karhade

Larsen

AMG

Cote

Dubois

Smith

. National databases for neurosurgical outcomes research: options, strengths, and limitations. Neurosurgery. 2018;83(3):333–344.

21.

Oravec

Motiwala

Reed

, et al. Big data research in neurosurgery: a critical look at this popular new study design. Neurosurgery. 2018;82(5):728–746.

22.

Patel

Singh

Nunley

Minhas

. administrative databases in orthopaedic research: pearls and pitfalls of big data. JAAOS. 2016;24(3):172–179.

23.

Villelli

Das

Yan

Huff

Zou

Barbaro

. Impact of the 2006 Massachusetts health care insurance reform on neurosurgical procedures and patient insurance status. J Neurosurg. 2017;126(1):167–174.

24.

Lawson

Louie

Zingmond

, et al. A comparison of clinical registry versus administrative claims data for reporting of 30-day surgical comatioplicns. Ann Surg. 2012;256(6):973–981.

25.

Asher

Speroff

Dittus

Parker

Davies

Selden

. The National Neurosurgery Quality and Outcomes Database (N2QOD): a collaborative North American outcomes registry to advance value-based spine care. Spine (Phila Pa 1976). 2014;39(22 Suppl 1):S106–S116.

26.

American Spine Registry. American spine registry, the national quality improvement registry for Spine care. Published 2020. Updated March 30, 2021. Accessed November 22, 2020. https://www.americanspineregistry.org. American Spine Registry

27.

European Spine Study Group. Published 2020. Updated March 30, 2021. Accessed December 24, 2020. http://www.spine-essg.com. European Spine Study Group;

28.

International Spine Study Group. Published 2020. Updated March 30, 2021. Accessed December 24, 2020. https://issgf.org. International Spine Study Group

29.

HSG Research. Published 2020. Updated March 30, 2021. Accessed December 24, 2020. https://www.settingscoliosisstraight.org/hsg-research/. Setting Scoliosis Straight

30.

Pugely

Martin

Harwood

Ong

Bozic

Callaghan

. Database and Registry Research in Orthopaedic Surgery: Part 2: Clinical Registry Data. JBJS. 2015;97(21):1799–1808.

31.

Ismael

Villafañe

Cabitza

Banfi

Berjano

. Spine surgery registries: hope for evidence-based spinal care? J Spine Surg. 2018;4(2):456–458.

32.

American College of Surgeons. ACS NSQIP Hospital Participation Requirements. Published 2020. Updated March 30, 2021. Accessed March 18, 2020. https://www.facs.org/quality-programs/acs-nsqip/joinnow/participation. American College of Surgeons

33.

American College of Surgeons. ACS NSQIP Frequently Asked Questions. Published 2020. Updated March 30, 2021. Accessed April 9, 2020. https://www.facs.org/quality-programs/acs-nsqip/faq. American College of Surgeons

34.

Pasquali

Jacobs

Shook

, et al. Linking clinical registry data with administrative data using indirect identifiers: implementation and validation in the congenital heart surgery population. Am Heart J. 2010;160(6):1099–1104.

35.

Pittman

Miranpuri

. Neurosurgery clinical registry data collection utilizing Informatics for Integrating Biology and the Bedside and electronic health records at the University of Rochester. Neurosurg Focus. 2015;39(6):E16.

36.

The Office of the National Coordinator for Health Information Technology. Hospitals Participating in the CMS EHR Incentive Programs,’ Health IT Quick-Stat #45. Published 2017. Updated March 30, 2021. Accessed March 18, 2020. https://dashboard.healthit.gov/quickstats/pages/FIG-Hospitals-EHR-Incentive-Programs.php

37.

Azad

Kalani

Wolf

, et al. Building an electronic health record integrated quality of life outcomes registry for spine surgery. J Neurosurg Spine. 2016;24(1):176–185.

38.

Bihorac

Ozrazgat-Baslanti

Ebadi

, et al. MySurgeryRisk: Development and Validation of a Machine-learning Risk Algorithm for Major Complications and Death After Surgery. Ann Surg. 2019;269(4):652–662.

39.

Gershanik

Schneider

, et al. Impact of IT-enabled Intervention on MRI Use for Back Pain. Am J Med. 2014;127(6):512–518.e511.

40.

Chen

Bhambhvani

Hom

, et al. Effect of electronic clinical decision support on imaging for the evaluation of acute low back pain in the ambulatory care setting. World Neurosurg. 2020;134:e874–e877.

41.

Goldstein

Navar

Pencina

Ioannidis

. Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review. J Am Med Inform Assoc. 2017;24(1):198–208.

42.

Goldstein

Pomann

Winkelmayer

Pencina

. A comparison of risk prediction methods using repeated observations: an application to electronic health records for hemodialysis. Stat Med. 2017;36(17):2750–2763.

43.

Shah

Bini

Vail

. Data for registry and quality review can be retrospectively collected using natural language processing from unstructured charts of arthroplasty patients. Bone Joint J. 2020;102(7 Supple B):99–104.

44.

Karhade

Schwab

Del Fiol

Kawamoto

. SMART on FHIR in spine: integrating clinical prediction models into electronic health records for precision medicine at the point of care. Spine J. 2020.

45.

Banks

. Tech giants, armed with wearables data, are entrenching in health research. Nat Med. 2020;26(1):4–5.

46.

Debono

Bousquet

Sabatier

Plas

J-Y

Lescure

J-P

Hamel

. Postoperative monitoring with a mobile application after ambulatory lumbar discectomy: an effective tool for spine surgeons. Eur Spine J. 2016;25(11):3536–3542.

47.

Debono

Corniola

Pietton

Sabatier

Hamel

Tessitore

. Benefits of Enhanced Recovery after Surgery for fusion in degenerative spine surgery: impact on outcome, length of stay, and patient satisfaction. Neurosurg Focus. 2019;46(4):E6.

48.

Bahadori

Collard

Williams

Swain

. A review of current use of commercial wearable technology and smartphone apps with application in monitoring individuals following total hip replacement surgery. J Med Eng Technol. 2020;44(6):324–333.

49.

von Glinski

Ishak

Elia

, et al. Emerging insight in the use of an active post discharge surveillance program in spine surgery: a retrospective pilot study. World Neurosurg. 2020;139:e237–e244.

50.

Bai

Mobbs

Walsh

Betteridge

CMW

. mHealth Apps for enhanced management of spinal surgery patients: a review. Front Surg. 2020;7:81.

51.

Ponder

Ansah-Yeboah

Charalambous

, et al. A Smartphone app with a digital care pathway for patients undergoing spine surgery: development and feasibility study. JMIR Perioper Med. 2020;3(2):e21138–e21138.

52.

Stewart

Fayed

Henault

Kalantar

Voyadzis

J-M

. Use of a smartphone application for spine surgery improves patient adherence with preoperative instructions and decreases last-minute surgery cancellations. Cureus. 2019;11(3):e4192–e4192.

53.

Hou

Yang

, et al. The effectiveness and safety of utilizing mobile phone–based programs for rehabilitation after lumbar spinal surgery: multicenter, prospective randomized controlled trial. JMIR Mhealth Uhealth. 2019;7(2):e10201.

54.

Park

Beatty

Stafford

Whooley

. Mobile phone interventions for the secondary prevention of cardiovascular disease. Prog Cardiovasc Dis. 2016;58(6):639–650.

55.

Morawski

Ghazinouri

Krumme

, et al. Association of a smartphone application with medication adherence and blood pressure control: the MedISAFE-BP randomized clinical trial. JAMA Intern Med. 2018;178(6):802–809.

56.

Goz

Spiker

Brodke

. Mobile messaging and smartphone apps for patient communication and engagement in spine surgery. Ann Transl Med. 2019;7(Suppl 5):S163–S163.

57.

Basil

Sprau

Eliahu

Borowsky

Wang

Yoon

. Using smartphone-based accelerometer data to objectively assess outcomes in spine surgery. Neurosurgery. 2021;88(4):763–772.

58.

Stienen

Rezaii

, et al. Objective activity tracking in spine surgery: a prospective feasibility study with a low-cost consumer grade wearable accelerometer. Sci Rep. 2020;10(1):4939.

59.

Abelson

Symer

Peters

Charlson

Yeo

Mobile health apps and recovery after surgery: What are patients willing to do?

Am J Surg. 2017;214(4):616–622.

60.

Abelson

Kaufman

Symer

Peters

Charlson

Yeo

Barriers and benefits to using mobile health technology after operation: a qualitative study. Surgery. 2017;162(3):605–611.

61.

Sim

. Mobile Devices and Health. N Engl J Med. 2019;381(10):956–968.

62.

Eapen

Turakhia

McConnell

, et al. Defining a mobile health roadmap for cardiovascular health and disease. J Am Heart Assoc. 2016;5(7):e003119.

63.

International Spinal Cord Injury Biobank (ISCIB), Canada. Published 2020. Updated March 30, 2021. Accessed December 22, 2020. https://biobanking.org/biobanks/view/500#:∼:text=Objective%3A%20The%20purpose%20of%20the,spinal%20cord%20injury%20(SCI). Biobank Resource Centre

64.

Biobank. Published 2020. Updated March 30, 2021. Accessed December 22, 2020. https://www.chordomafoundation.org/research/biobank/. Chordoma Foundation

65.

Zhou

Wang

Huang

Xiao

. Epidemiological characteristics of primary spinal osseous tumors in Eastern China. World J Surg Oncol. 2017;15(1):73.

66.

Tachmazidou

Hatzikotoulas

Southam

, et al. Identification of new therapeutic targets for osteoarthritis through genome-wide analyses of UK Biobank data. Nat Genet. 2019;51(2):230–236.

67.

Clavreul

Soulard

Lemée

J-M

, et al. The French glioblastoma biobank (FGB): a national clinicobiological database. J Transl Med. 2019;17(1):133.

68.

Beam

Kohane

. Big data and machine learning in health care. JAMA. 2018;319(13):1317–1318.

69.

Chang

Canseco

Nicholson

Patel

Vaccaro

. The Role of machine learning in spine surgery: the future is now. Front Surg. 2020;7:54.

70.

Walls

. Intensive longitudinal data. In: Little

, ed. The Oxford Handbook of Quantitative Methods: Statistical Analysis, Vol. 2. Oxford University Press; 2013:432–440.

71.

Deo

. Machine learning in medicine. Circulation. 2015;132(20):1920–1930.

72.

Steyerberg

van der Ploeg

Van Calster

. Risk prediction with machine learning and regression methods. Biom J. 2014;56(4):601–606.

73.

Steyerberg

. Selection of main effects. In: Steyerberg

, ed. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. Springer International Publishing; 2019:207–225.

74.

Steyerberg

. Modern estimation methods. In: Steyerberg

, ed. Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. Springer International Publishing; 2019:247–260.

75.

Garriga

Leal

Sánchez-Santos

, et al. Geographical Variation in Outcomes of Primary Hip and Knee Replacement. JAMA Network Open. 2019;2(10):e1914325–e1914325.

76.

Dimick

Ryan

. Methods for evaluating changes in health care policy: the difference-in-differences approach. JAMA. 2014;312(22):2401–2402.

77.

Callahan

Shah

. Machine Learning in Healthcare. In: Sheikh

Wright

Cresswell

Bates

, eds. Key Advances in Clinical Informatics: 2018:279–291.

78.

Richter

Khoshgoftaar

. A review of statistical and machine learning methods for modeling cancer risk using structured clinical data. Artif Intell Med. 2018;90:1–14.

79.

Rudin

. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nat Mach Intell. 2019;1(5):206–215.

80.

van der Ploeg

Austin

Steyerberg

. Modern modelling techniques are data hungry: a simulation study for predicting dichotomous endpoints. BMC Med Res Methodol. 2014;14:137.

81.

Lundberg

Erion

Chen

, et al. Explainable AI for trees: From local explanations to global understanding. arXiv preprint arXiv:190504610. 2019.

82.

Christodoulou

Collins

Steyerberg

Verbakel

Van Calster

. A systematic review shows no performance benefit of machine learning over logistic regression for clinical prediction models. J Clin Epidemiol. 2019;110:12–22.

83.

Senders

Staples

Karhade

, et al. machine learning and neurosurgical outcome prediction: a systematic review. World Neurosurg. 2018;109:476–486.e471.

84.

Goyal

Ngufor

Kerezoudis

McCutcheon

Storlie

Bydon

. Can machine learning algorithms accurately predict discharge to nonhome facility and early unplanned readmissions following spinal fusion? Analysis of a national surgical registry. J Neurosurg Spine. 2019:1–11.

85.

Pedersen

Andersen

MØ

Carreon

Eiskjær

. Applied machine learning for spine surgeons: predicting outcome for patients undergoing treatment for lumbar disc herniation using PRO Data. Global Spine J. 2020:2192568220967643.

86.

Jain

Durand

Burch

Daniels

Berven

Machine Learning for Predictive Modeling of 90-day Readmission, Major Medical Complication, and Discharge to a Facility in Patients Undergoing Long Segment Posterior Lumbar Spine Fusion. Spine (Phila Pa 1976). 2020;45(16): 1151–1160.

87.

Buchlak

Esmaili

Leveque

, et al. Machine learning applications to clinical decision support in neurosurgery: an artificial intelligence augmented systematic review. Neurosurg Rev. 2020;43(5):1235–1253. Epub 2019 August 17.

88.

Galbusera

Niemeyer

Wilke

H-J

, et al. Fully automated radiological analysis of spinal disorders and deformities: a deep learning approach. Eur Spine J. 2019;28(5):951–960.

89.

Lafage

Pesenti

Lafage

Schwab

. Self-learning computers for surgical planning and prediction of postoperative alignment. Eur Spine J. 2018;27(1):123–128.

90.

Sharif Bidabadi

Tan

Murray

Lee

. Tracking foot drop recovery following lumbar-spine surgery, applying multiclass gait classification using machine learning techniques. Sensors. 2019;19(11):2542.

91.

Bidabadi

Murray

Lee

GYF

Morris

Tan

. Classification of foot drop gait characteristic due to lumbar radiculopathy using machine learning algorithms. Gait Posture. 2019;71:234–240.

92.

Meyer

Zverinski

Pfahringer

, et al. Machine learning for real-time prediction of complications in critical care: a retrospective study. Lancet Respir Med. 2018;6(12):905–914.

93.

Ghogawala

Dunbar

Essa

. Artificial intelligence for the treatment of lumbar spondylolisthesis. Neurosurg Clin N Am. 2019;30(3):383–389.

94.

Peugh

. A practical guide to multilevel modeling. J Sch Psychol. 2010;48(1):85–112.

95.

Hoffman

Longitudinal analysis: Modeling within-person fluctuation and change. Routledge; 2015.

96.

Stasinopoulos

Rigby

Heller

Voudouris

De Bastiani

. Flexible regression and smoothing: using GAMLSS in R. CRC Press; 2017.

97.

Shameer

Johnson

Glicksberg

Dudley

Sengupta

. Machine learning in cardiovascular medicine: are we there yet? Heart. 2018;104(14):1156–1164.

98.

Hartzler

Chaudhuri

Fey

Flum

Lavallee

. Integrating patient-reported outcomes into spine surgical care through visual dashboards: lessons learned from human-centered design. EGEMS (Wash DC). 2015;3(2):1133.

99.

Johnson

Zhang

. A user-centered framework for redesigning health care interfaces. J Biomed Inform. 2005;38(1):75–87.

100.

Yen

Bakken

. Review of health information technology usability study methodologies. J Am Med Inform Assoc. 2012;19(3):413–422.

101.

Jaspers

Steen

van den Bos

Geenen

. The think aloud method: a guide to user interface design. Int J Med Inform. 2004;73(11-12):781–795.

102.

Koch

Pfandler

Stefan

, et al. Say, what is on your mind? surgeons’ evaluations of realism and usability of a virtual reality Vertebroplasty simulator. Surg Innov. 2019;26(2):234–243.

103.

Sadiqi

Lehr

Post

, et al. Development of the AOSpine Patient Reported Outcome Spine Trauma (AOSpine PROST): a universal disease-specific outcome instrument for individuals with traumatic spinal column injury. Eur Spine J. 2017;26(5):1550–1557.

104.

Jaspers

. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int J Med Inform. 2009;78(5):340–353.

105.

Yen

Bakken

. A comparison of usability evaluation methods: heuristic evaluation versus end-user think-aloud protocol - an example from a web-based communication tool for nurse scheduling. AMIA Annu Symp Proc. 2009;2009:714–718.

106.

LeRouge

Hasselquist

Kellogg

, et al. Using Heuristic Evaluation to Enhance the Visual Display of a Provider Dashboard for Patient-Reported Outcomes. EGEMS (Wash DC). 2017;5(2):6.

107.

Melnick

Hess

Guo

, et al. Patient-centered decision support: formative usability evaluation of integrated clinical decision support with a patient decision aid for minor head injury in the emergency department. J Med Internet Res. 2017;19(5):e174.

108.

Tham

Swietlik

Deakyne

, et al. Clinical decision support for a multicenter trial of pediatric head trauma: development, implementation, and lessons learned. Appl Clin Inform. 2016;7(2):534–542.

109.

Sittig

Singh

. A new sociotechnical model for studying health information technology in complex adaptive healthcare systems. Qual Saf Health Care. 2010;19(Suppl 3):i68–74.

110.

Yen

McAlearney

Sieck

Hefner

Huerta

. Health information technology (HIT) adaptation: refocusing on the journey to successful HIT implementation. JMIR Med Inform. 2017;5(3):e28.

111.

Kilsdonk

Peute

Jaspers

. Factors influencing implementation success of guideline-based clinical decision support systems: a systematic review and gaps analysis. Int J Med Inform. 2017;98:56–64.

112.

Singh

Spitzmueller

Petersen

, et al. Primary care practitioners’ views on test result management in EHR-enabled health systems: a national survey. J Am Med Inform Assoc. 2013;20(4):727–735.

113.

Martinez

Hogan

Balbale

, et al. Sociotechnical perspective on implementing clinical video telehealth for veterans with spinal cord injuries and disorders. Telemed J E Health. 2017;23(7):567–576.

114.

Brunner

Sheehan

Yanke

, et al. Joint design with providers of clinical decision support for value-based advanced shoulder imaging. Appl Clin Inform. 2020;11(1):142–152.

115.

Greenberg

Otun

Nasraddin

, et al. Electronic clinical decision support for children with minor head trauma and intracranial injuries: a sociotechnical analysis. 2020.

116.

Adler-Milstein

Adelman

Tai-Seale

Patel

Dymek

. EHR audit logs: a new goldmine for health services research? J Biomed Inform. 2020;101:103343.

117.

Maloney

Peterson

Kao

Sherrill

Green

Sachdev

. Surgery resident time consumed by the electronic health record. J Surg Educ. 2020;77(5):1056–1062.

118.

Cox

Farjat

Risoli

, et al.

Documenting or operating: where is time spent in general surgery residency?

J Surg Educ. 2018;75(6):e97–e106.

119.

Stiell

Bennett

. Implementation of clinical decision rules in the emergency department. Acad Emerg Med. 2007;14(11):955–959.

120.

Hamilton

Mittman

. Implementation science in health care. Dissemination and Implementation Research in Health. 2nd ed. Oxford University Press; 2017.

121.

Irvine

Russell

Manocchia

, et al. Mobile-web app to self-manage low back pain: randomized controlled trial. J Med Internet Res. 2015;17(1):e1.

122.

Bedoya

Clement

Phelan

Steorts

O’Brien

Goldstein

. Minimal impact of implemented early warning score and best practice alert for patient deterioration. Crit Care Med. 2019;47(1):49–55.

123.

Singh

Sittig

. Toward electronic medical record alerts that consume less physician time--reply. JAMA Intern Med. 2013;173(18):1756.

124.

Peterson

. Machine learning, predictive analytics, and clinical practice: can the past inform the present? JAMA. 2019.

125.

Mazur

Mosaly

Moore

Marks

. Association of the usability of electronic health records with cognitive workload and performance levels among physicians. JAMA Netw Open. 2019;2(4):e191709.

126.

Amarasingham

Patzer

Huesch

Nguyen

Xie

. Implementing electronic health care predictive analytics: considerations and challenges. Health Aff (Millwood). 2014;33(7):1148–1154.

127.

Lenert

Matheny

Walsh

. Prognostic models will be victims of their own success, unless. J Am Med Inform Assoc. 2019;26(12):1645–1650.

128.

Davis

Greevy

Fonnesbeck

Lasko

Walsh

Matheny

. A nonparametric updating method to correct clinical prediction model drift. J Am Med Inform Assoc. 2019;26(12):1448–1457.

129.

Janssen

DMC

van Kuijk

SMJ

d’Aumerie

Willems

. External validation of a prediction model for surgical site infection after thoracolumbar spine surgery in a Western European cohort. J Orthop Surg Res. 2018;13(1):114.

130.

Adibi

Sadatsafavi

Ioannidis

JPA

. Validation and utility testing of clinical prediction models: time to change the approach. JAMA. 2020;324(3):235–236.

131.

Fougerouse

Yasini

Marchand

Aalami

. A Cross-sectional study of prominent us mobile health applications: evaluating the current landscape. AMIA Annu Symp Proc. 2017;2017:715–723.

132.

Nielsen

Schultz

Langkjaer

, et al. Adjusting early warning score by clinical assessment: a study protocol for a danish cluster-randomised, multicentre study of an individual early warning score (I-EWS). BMJ Open. 2020;10(1):e033676.

133.

Gianfrancesco

Tamang

Yazdany

Schmajuk

. Potential biases in machine learning algorithms using electronic health record data. JAMA Intern Med. 2018;178(11):1544–1547.

134.

Char

Shah

Magnus

. Implementing machine learning in health care - addressing ethical challenges. N Engl J Med. 2018;378(11):981–983.

135.

Gartner Hype Cycle. Published 2020. Updated March 30, 2021. Accessed December 24, 2020. https://www.gartner.com/en/research/methodologies/gartner-hype-cycle. Gartner Inc