Reaching a Tipping Point for Neurorehabilitation Research: Obstacles and Opportunities in Trial Design,Description,and Pooled Analysis

Abstract

The record-breaking pace of COVID-19 vaccine development and implementation depended heavily on collaboration among academic, government, and commercial stakeholders, especially through data-sharing and robust multicenter trials. Collaborative efforts have not been as fruitful in fields such as neurorehabilitation, where non-pharmacological interventions play a much larger role. Barriers to translating scientific advancements into clinical practice in neurorehabilitation include pervasively small study sizes, exacerbated by limited funding for non-pharmacological multicenter clinical trials; difficulty standardizing—and adequately describing—non-pharmacological interventions; and a lack of incentives for individual patient-level data-sharing. These barriers prevent reliable meta-analysis of non-pharmacological clinical studies in neurorehabilitation. This point-of-view will highlight these challenges as well as suggest practical steps that may be taken to improve the neurorehabilitation pipeline between evidence and implementation.

Introduction

The rate of COVID-19 vaccine development and implementation markedly exceeded both scientific expectations ¹ and historical precedent.^2,3 Among many scientific and logistical factors that contributed to this achievement, improved collaborative mechanisms stand out.^4,5 Partnerships between academic, government, and commercial entities have improved vaccine clinical trial capacity and created infrastructure for data-sharing and more robust meta-analyses, crucially expediting the scientific process.^4,6,7

This rewriting of the record books for vaccine development raises an obvious question: How can improved collaboration speed innovation and clinical implementation in other fields? Collaboration between therapists, patients, caregivers, physicians, and engineers is intrinsic to the neurorehabilitation process itself. However, collaboration among neurorehabilitation teams in academia, industry, and government remains underdeveloped due to a number of obstacles. Translating scientific advancements to neurorehabilitation clinical practice remains elusive, especially regarding non-pharmacological interventions, where small study sizes, non-standardized methods, and limited statistical power hamper progress.^{8 -10} This implementation gap, in conjunction with potential pandemic-related funding shortfalls,^{11 -13} makes the need for improved efficiency, collaboration, and reporting in neurorehabilitation particularly urgent. This point-of-view will outline specific challenges facing neurorehabilitation as it aims to improve the multicenter clinical trial (MCT) and data-sharing capabilities of its non-pharmacological investigations. Though we intend for this point-of-view to be broadly applicable across neurorehabilitation, by necessity many of our examples come from the stroke literature.

Obstacles

Industry Sponsorship for Drug MCTs Dwarfs Industry Sponsorship for Device MCTs

The torrid pace of COVID-19 vaccine development benefited from an unparalleled pandemic-inspired collaborative spirit and massive funding. However, collaboration does not require a pandemic: In the 1950s, leading oncologists formed a series of cooperative groups to provide a framework for multicenter cancer studies. Cancer collaborations such as the SWOG Cancer Research Network have evolved into a remarkably successful model for biomedical progress. ¹⁴ Compared to the single-site research that plagues neurorehabilitation, multicenter trials increase sample size, making it easier to reliably detect small effect sizes, reduce the rate of false positive and false negative findings, and note relatively rare adverse events. Multicenter recruitment also improves generalizability to diverse clinical populations. ¹⁵ Not to be underestimated, multicenter collaboration broadens study investigators’ expertise in crafting and executing studies. ¹⁶

With some notable exceptions in stroke^{17 -23}—non-pharmacological neurorehabilitation studies have been hindered by distinct financial and logistical challenges in developing multicenter trials. Whereas cooperative cancer studies are driven by a $97.4 to $141.3 billion global oncology drug market,^24,25 the global market for neurorehabilitation devices is only $1.5 billion. ²⁶ Furthermore, the added costs of site onboarding, fidelity monitoring, and governance processes counterintuitively prevent multisite studies from achieving economies of scale compared to single-site studies. ¹⁵ Contract research organizations have evolved to work with pharmaceutical companies to manage these logistical and administrative burdens. However, the limited financial resources of the non-pharmaceutical industry constrain efforts to properly oversee multicenter trials in neurorehabilitation—especially for researchers working with smaller patient populations (smaller markets) than stroke or traumatic brain injury.

We Don’t Know Which Neurorehabilitation Interventions to Select for MCT Funding

When allocating resources for MCTs, limited funding necessitates choosing wisely. However, the well-established phased pipeline approach for drugs does not fit cleanly for device and other non-pharmacological interventions. There is a lack of consensus among researchers, fueled in large part by a murky literature strewn with underpowered preclinical and pilot studies.^8,27,28 Whereas numerous frameworks for identifying promising preclinical candidates have been put forth^{28 -30} and calls have been made to ground rehabilitation trials on better preclinical and early-stage studies, ³¹ strategies for identifying the best candidates for MCTs in neurorehabilitation remain under-implemented.

For example, even in a niche as specific and promising as using electrical stimulation of the spinal cord to activate spared neural pathways after spinal cord injury, different research teams employ a diverse set of approaches, not limited to the sites of cathode and anode placement, the frequency and shape of electrical pulses, and even the terms used to describe the same technique. ³² Conducting a multicenter trial in an area such as spinal cord stimulation would require researchers to reach consensus on a trial protocol despite limited empirical evidence to suggest that one approach will yield better results than another. This scenario may resemble that faced by oncologists decades ago. Leaders in that field forged onward, developing sophisticated tools such as systematic reviews ³³ and quality-adjusted life years ³⁴ that have been adopted by other fields. Furthermore, stroke investigators have emphasized the need for more rigorously defining and determining optimal dosing of interventions in early-stage research.^35,36 Although developing meaningful mathematical tools to aggregate and appraise early-stage clinical data in neurorehabilitation will be more challenging than in fields like oncology, these efforts will be necessary for neurorehabilitation clinician-scientists to play catch-up. ³⁷

Standardizing Protocols Across Multiple Centers is Harder for Non-Pharmacological Interventions

Even when consensus exists on selecting a rehabilitation intervention for MCT funding, non-pharmacological interventions require much greater effort than drug studies to minimize inter-center variability. First, identifying the “active ingredients” of rehabilitation interventions is not a simple issue—researchers have started to develop conceptual schemes and taxonomies only over the past decade.^38,39 Frameworks for applying non-pharmacological interventions in a standardized fashion do exist for surgical procedures,^40,41 yet are under-utilized in neurorehabilitation. Some practices used to minimize variability include intensive centralized procedural training, frequent on-site and remote external monitoring, minimum enrollment quotas for each site to maintain participation, and others.^{19 -21,40,41} Unfortunately, while the aforementioned stroke rehabilitation studies provide a blueprint for executing non-pharmacological studies across multiple sites, the required funding and dedicated statistical oversight necessary from the planning phases through protocol execution remain rare in neurorehabilitation trials. ⁴² Finally, the slower recruitment in most non-pharmacological neurorehabilitation trials leads to greater challenges posed by shifts in “usual care” during multiyear studies—for example, during the course of the iCARE study, standard-of-care upper extremity rehabilitation evolved to closely resemble the iCARE intervention, thereby confounding study interpretation. ²⁰

It’s Difficult to Analyze Results From Studies Performed and Described in Non-Standardized Fashion

Researchers agree that analyzing and interpreting one’s own data is difficult enough, let alone analyzing and interpreting somebody else’s data without appropriate context. ⁴³ In neurorehabilitation, the inherent variability of the field’s non-pharmacological interventions makes the detailed reporting of experimental methods especially important. Yet, reporting of non-pharmacological interventions remains substandard, ⁴⁴ with fewer than 50% of non-pharmacological trials adequately reporting crucial details of their interventions.^{37,44 -46} Insufficient methodological reporting hinders experimental repeatability, ⁴⁵ secondary analysis,^37,47,48 and clinical implementation. ⁴⁹ Accurate methodological description is just as crucial for control as for investigational interventions. Not only does “usual care” delivered in 1992 or 2002 differ from that delivered in 2022, but usual care often differs across sites participating in an MCT.^20,50 When these differences in control interventions are not clearly described, it becomes impossible to meta-analyze the effects of various interventions against control across different eras and sites.^10,15,37,51 Neurorehabilitation’s particularly poor documentation of its interventions stymies the potential contributions of meta-analysis and contributes to avoidable research waste—which has been estimated to be in excess of $100 billion per year across all fields of medical research combined. ⁵²

Toward Practical Solutions

Better Meta-Analysis Through More Sharing of Individual Patient-Level Data

The obstacles to funding, selecting, and analyzing neurorehabilitation interventions will make it difficult to execute truly pivotal randomized MCTs in many fields of neurorehabilitation for the foreseeable future. We need pragmatic solutions that may be implemented during the interim. Increased sharing of individual patient-level data (IPD) provides a more practical mechanism to harness the cumulative power of the multiple small studies that are so characteristic of this field. Since the 1950s, ⁵³ researchers in specialties such as cardiology,^54,55 oncology, and genetics ⁵⁶ have effectively exploited IPD to generate novel findings using individual patient data meta-analyses (IPDMA). ⁵⁷ IPDMA comprises a specific subset of meta-analysis in which individual participants rather than individual publications serve as the unit of analysis.^58,59 IPDMA may be an important tool with which to extract valuable data, gain insights, and perhaps devise alternate paths to implementing evidence-based neurorehabilitation. However, this requires better definition and standardization of methodology and outcome measurements, especially at earlier phases of clinical research. Clinical research collaborations such as StrokeNet, the Stroke Recovery and Rehabilitation Roundtable, the FOCUS/AFFINITY/EFFECTS trial groups, Northeast ALS Consortium, NIH StrokeNet, NeuroNEXT, and Center TBI demonstrate the potential to develop protocols collaboratively, harmonize research findings, and share data.^{60 -65} Participating in and strengthening these collaborative networks will be key to progress in all fields of neurorehabilitation.

Several other challenges need to be overcome to increase IPD sharing in neurorehabilitation: First, the costs and administrative burdens associated with sharing deidentified data can be particularly high for research teams conducting small single-center studies.^{66 -69} Academic and granting institutions need to increase dedicated funding and core staffing for these efforts. Second, these practices remain disincentivized in terms of earning academic credit.^{70 -72} New metrics already exist to give credit for data-sharing, collaboration, reproducibility, and public engagement^73,74—more institutional academic promotion committees need to consider and incentivize these metrics. Third, despite a willingness to share IPD,^75,76 guidelines for doing so, ⁷⁷ and open repositories, inertia will largely prevent substantial progress in this area until more journals and funders formally require—and enforce—individual patient-level data-sharing. ⁷⁸ Fourth, even the most statistically sound meta-analyses are subject to publication bias: researchers continue to debate how best to account retrospectively for the disproportionate absence of unpublished (mostly negative) results.^{79 -83} Encouraging deposition of published and unpublished data into repositories such as Figshare, GitHub, Vivli, the Virtual International Stroke Trials Archive (VISTA), Open Data Commons for Spinal Cord Injury, and others should help increase the balance of positive and negative data available for secondary analysis.

Accurate Meta-Analysis Through Accurate Description of Study Methods

Sharing patient-level data is not enough. Meta-analysis requires careful consideration of the experimental methods employed across a variety of primary study settings. Efforts to address inadequate published “methods sections” include the CONSORT addendum on reporting non-pharmacological trials, ⁸⁴ the TIDieR statement, ⁸⁵ RCTRACK, ¹⁰ the Rehabilitation Treatment Specification System, ⁸⁶ and increasing use of supplemental digital content—or even an “intervention bank” ⁴⁵ —for detailed methods descriptions unrestricted by the word and media limitations imposed by most journals. ⁴⁶

Most readers of this article will be professionals in neurorehabilitation—but amateurs at statistical design, statistical analysis, and implementation logistics. More professional statisticians and implementation experts must be incorporated into every aspect of neurorehabilitation trial design and execution. Furthermore, more meaningful collaboration between patients, preclinical scientists, and clinical trialists is required to better align research goals with patient priorities, as well as to enhance the rigor of the translational pathway. Preclinical and clinical research groups in all aspects of neurorehabilitation should look to the Stroke Preclinical Assessment Network as a sterling example of collaboration between laboratories to increase statistical power and methodological reliability on the pathway toward selecting treatments for RCTs. ²⁸ By de-emphasizing first- and senior-author publications, academic institutions and funding agencies can better incentivize these essential forms of collaboration, enhancing the likelihood of reliable data aggregation and meaningful trial findings.

Conclusion

Neurorehabilitation faces enormous financial, logistical, and scientific challenges to implementing definitive large-scale multicenter trials. While overcoming these challenges should remain the long-term goal, four practical steps can be taken right now to improve the neurorehabilitation evidence base (Table 1):

• Increase participation in and funding for existing and future neurorehabilitation trial networks;

• Refine and adhere to TIDieR and RCTRACK guidelines to better describe methodology^10,39;

• Increase incentives and enforce obligations to share IPD in public data-sharing platforms;

• Embed professional statisticians, implementation scientists, and patient stakeholders into neurorehabilitation trial design, execution, analysis, and interpretation.

Table 1.

Barriers and proposed solutions

Barriers	Solutions
Money lacking for neurorehabilitation MCTs	Expand academic trial networks
	Increase government and foundation funding for trial networks
Small, unreliable early phase studies	Expand academic trial networks
	More emphasis on dose ranging
Difficult to standardize interventions	Expand academic trial networks
	Refine and adhere to TIDieR and RCTRACK guidelines to describe methodology
Individual level data not fully available	Incentivize and facilitate public sharing of individual patient data
	More statistician involvement from study formulation through study closure

Abbreviation: MCT, multicenter clinical trial.

In a healthcare system that demands evidence-based practice, neurorehabilitation researchers need to strengthen the field’s evidence base. Pooling and meta-analyzing individual patient data from well-described protocols will be key to this effort. These steps, if taken by a larger number of neurorehabilitation investigators, especially outside the realm of stroke, would vastly improve our ability to meta-analyze the results of the small, underpowered studies prevalent in the field. Meta-analysis can facilitate the rational selection of promising early-stage non-pharmacological interventions for large randomized, controlled multicenter trials.

Finally, we must emphasize that this point-of-view is not the first article attempting to highlight obstacles and opportunities to improve the effectiveness and impact of neurorehabilitation research. If the field reaches a “tipping point” of implementing these calls to action, neurorehabilitation investigators could attract more funding, maximize the value of trial subjects’ participation, and shorten the timeline of therapeutic development and implementation for patients with neurological deficits.