Combined Analysis of Cortical (EEG) and Nerve Stump Signals Improves Robotic Hand Control

Abstract

Background. Interfacing an amputee’s upper-extremity stump nerves to control a robotic hand requires training of the individual and algorithms to process interactions between cortical and peripheral signals. Objective. To evaluate for the first time whether EEG-driven analysis of peripheral neural signals as an amputee practices could improve the classification of motor commands. Methods. Four thin-film longitudinal intrafascicular electrodes (tf-LIFEs-4) were implanted in the median and ulnar nerves of the stump in the distal upper arm for 4 weeks. Artificial intelligence classifiers were implemented to analyze LIFE signals recorded while the participant tried to perform 3 different hand and finger movements as pictures representing these tasks were randomly presented on a screen. In the final week, the participant was trained to perform the same movements with a robotic hand prosthesis through modulation of tf-LIFE-4 signals. To improve the classification performance, an event-related desynchronization/synchronization (ERD/ERS) procedure was applied to EEG data to identify the exact timing of each motor command. Results. Real-time control of neural (motor) output was achieved by the participant. By focusing electroneurographic (ENG) signal analysis in an EEG-driven time window, movement classification performance improved. After training, the participant regained normal modulation of background rhythms for movement preparation (α/β band desynchronization) in the sensorimotor area contralateral to the missing limb. Moreover, coherence analysis found a restored α band synchronization of Rolandic area with frontal and parietal ipsilateral regions, similar to that observed in the opposite hemisphere for movement of the intact hand. Of note, phantom limb pain (PLP) resolved for several months. Conclusions Combining information from both cortical (EEG) and stump nerve (ENG) signals improved the classification performance compared with tf-LIFE signals processing alone; training led to cortical reorganization and mitigation of PLP.

Introduction

Longitudinal intrafascicular electrodes (LIFEs) ¹ have been used in recent years as invasive neural interfaces directly connected to the peripheral nervous system for bidirectional control of artificial devices,^2-5 with promising results during short-term trials with amputees. Recently, a new version of 4 thin-film LIFEs (tf-LIFE-4s) was implanted in a right-handed man, who suffered traumatic left-arm transradial amputation, ⁵ to control a robotic hand prosthesis via neural signals.

This study evaluated whether the integrated use of LIFEs and advanced signal processing of EEG and neurographic signals would allow better classification of movement commands. Moreover, cortical reorganization related to training with the robotic hand was monitored through a quantitative EEG analysis.

Methods

Participants and Experimental Protocol

A 26-year-old man was recruited for the study 2 years after amputation of the left arm. Surgical procedures of LIFEs implant and electrode characteristics as well as clinical aspects are detailed elsewhere. ⁵ Briefly, 2 tf-LIFE-4s were inserted in the ulnar and median nerves for 1 month with intense training. The participant did not report any complication during the 12-month follow-up. As previously detailed, ⁵ phantom awareness and presence of phantom limb pain (PLP) were evaluated presurgically using an abbreviated version of the McGill Pain Questionnaire (sfMcGill), the Present Pain Intensity Scale (PPI), the Pain Visual Analoge Scale (VAS), and an open section for description of phantom awareness; participant follow-up was at the end of the training period and 3 months after LIFEs removal. The study was approved by the local Ethics Committee and by the assigned office of Italian Ministry of Health, and an informed consent was signed.

For the first 3 weeks after the implantation, the individual was trained in dispatch motor commands to produce various hand and finger movements without activating stump muscles. In the last week, he was asked to perform a palmar grasp, pinch grasp, and little-finger flexion with the robotic hand as pictures representing these tasks were randomly presented to him on a computer screen. Each visual presentation also served as the theoretical onset of each motor command.

After 4 weeks of training, a clear clinical improvement of PLP with a progressive return to normal perception of the upper limb and of the hand motion was reported (preimplant scores: sfMcGill = 18, PPI = 3, VAS = 38; 1 week post-LIFE removal scores: sfMcGill = 11, PPI = 2, VAS = 23). This improvement disappeared 3 months later (sfMcGill = 17; PPI = 3; VAS = 36).

EEG Recordings and Data Analysis

EEG signals were recorded from the scalp: (1) before surgery (PRE) during voluntary command to perform left hand grip and (2) after LIFEs implant (POST) and at the end of intensive training for motor commands control simultaneously to electroneurographic (ENG) acquisition for the same movement. In the POST session EEG recordings were also performed during right (intact)-hand movement. We used 32 electrodes (scalp sites defined according to the International 10-20 EEG electrode system) mounted on an elastic cap and binaural reference (time constant = 0.1 s; sampling rate = 1024 Hz; presampling analogical filter, 0.48-256 Hz; BrainAmp System, Brain Products GmbH, Gilching, Germany). A semiautomatic procedure based on independent component analysis ⁶ was applied to identify and eliminate artefacts (ie, eye movements, cardiac activity, and scalp muscle contraction). The spatial resolution of the artefact-free EEG data was enhanced by surface Laplacian estimation (regularized 3-D spline function). ⁷

Voluntary movements are ordinarily accompanied by modulation in oscillatory firing of cortical neurons. ⁸ In both the α (8-14 Hz) and β frequency ranges (15-30 Hz), this consists of a power decrease (event-related desynchronization, ERD) over the sensorimotor (central) areas contralateral to the motion, which begins at least 1.5 s before movement onset, rapidly becoming bilateral, followed by an increase (event-related synchronization, ERS) after movement execution. Whereas the presence of an ERD has been linked to the activation of cortical areas related to preparation of movement, the ERS has been associated with inhibited or idling areas.^8,9 Therefore, to quantify the event-related changes of EEG power and to identify ERD as a marker of motor command, we used the ERD/ERS procedure, ⁸ defined as the percentage decrement/increment of the EEG power density at the “event” (imaginary movement onset) compared with a “preevent” baseline. For each EEG channel, time frequency analysis was performed. ¹⁰ A total of 20 single EEG trials were chosen in the time interval of −2000 to 6000 ms, with 0 being the trigger onset. These were convolved by a Morlet wavelet, and the squared absolute values of the convolution over trials were averaged. For each frequency, the time course of the power modulation was obtained as percentage value of the mean of the baseline period (−1100 to −100 ms). Topographical maps of ERD/ERS in α-1 (8-10 Hz), α-2 (11-14 Hz) and β (15-25 Hz) bands were obtained by averaging for each band the time–frequency representation in the period from 500 to 1500 ms. The topographical maps of the different sessions were compared.

To reveal areas interacting with the primary motor cortex, corticocortical connectivity between EEG rhythmic activity from Rolandic and other electrodes was estimated by measuring coherence at a given frequency. To minimize the effect of volume conduction and evidence time-lagged direct interdependence between 2 brain areas, we used the imaginary part of complex coherence. ¹¹ We computed the power spectral density of each EEG channel and the complex cross-spectral density between C3 (or C4) and the other EEG channels by the standard Fast Fourier Transform (FFT) approach, using the Welch technique and Hanning windowing. The spectral and cross-spectral densities were calculated both for 1 s following the trigger (from 500 to 1500 ms) and for the 1-s “baseline.” For each frequency bin f, the complex coherence between signal x and y is defined as follows

$$C_{xy}(f)=\frac{S_{xy}(f)}{\sqrt{S_x(f)S_y(f)}},$$

with C_xy being the cross-spectrum between x and y, and S_x and S_y the spectra of x and y. The absolute value of the imaginary part of the coherence was averaged for the α-1, α-2, and β bands. Change in connectivity as a result of the task (event-related coherence) was then calculated as the difference between the imaginary part of coherence in the event period and in the baseline.

ENG Classification Based on Information Extracted From EEG Signals

ENG processing algorithms and classification for LIFE signals processing are reported elsewhere.^12-15 Despite the trigger video used to label the signals, the exact timing of the motor command was hardly identifiable because of its intertrial jitter. For this reason, the single trial ERD/ERS maps in the POST session were used to obtain a more realistic, EEG-driven time-label of the voluntary commands and to focus the appropriate time window for the ENG signals analysis. In particular, the minimum of single-trial ERD at C4 in the α band in the event period was chosen as the trigger for classification. For 2 grip types—the palmar and pinch grasps—the performances of classifiers, as assessed by the recognition ratio (RR), the ratio between the number of correctly identified grasps and the size of the test set, obtained while using the trigger video, and the information provided by the ERD/ERS analysis were compared. The mutual information index (MI) ¹⁶ was also calculated to assess the improvement achieved with the new EEG-based trigger. In detail, let X and Y be 2 random variables and Z be the joint random variable Z = (X, Y) over the Cartesian product; then, the MI (X, Y) between X and Y is defined as the relative entropy between Z and the product XY. In Bayesian terms, the MI represents the reduction in uncertainty of 1 variable when the other is observed—that is, between the prior and posterior distributions. ¹⁶

Results

EEG Data Analysis

Real-time control of neural (motor) output was achieved by the individual, and 3 different actions of the robotic hand were discriminated. The wavelet denoising allowed the identification of several classes of spikes as fully reported elsewhere. ⁵

What was interesting was that in the POST session, after LIFEs implant and intensive training, the participant regained a physiological modulation of cerebral rhythms in the time immediately preceding voluntary movements—which was completely absent in the PRE session—consisting of an abrupt power decrease (ERD) over the central sensorimotor areas contralateral to the missing hand in α-2 and β bands (Figure 1), as in controls. ¹⁷ The time preceding the real right-hand grip was associated with an ERD maximal to the contralateral central areas in all frequency bands (Figure 1). It is to be noted that in the POST session, the only difference in the cortical rhythms behavior after movement imagery/ execution between right and left cortical motor areas is the substantial absence of the α and β ERS after termination of robotic hand movement (as evidenced by the time–frequency maps Figure 1A).

Figure 1.

A. Top: Time–frequency representation of EEG power modulation (ERD/ERS) at C4, approximately corresponding to the right primary motor cortex location, for voluntary motor command of the missing limb before and after LIFEs implant. Bottom: Time–frequency representation of EEG power modulation at C3, corresponding to the left primary motor cortex location, for voluntary movement of the right hand. B. Scalp topography of ERD (blue)/ERS (red) in α-1, α-2, and β bands in the time period of 500 to 1500 ms for voluntary motor command of the missing limb before and after LIFEs implant (top) and for voluntary movement of the right hand (bottom). C3 and C4 location are evidenced by circles. In time–frequency representations, boxes indicate the frequency bands and time period considered for scalp topography. Abbreviations: ERD/ERS, event-related desynchronization/synchronization; LIFE, longitudinal intrafascicular electrodes.

As shown in Figure 2, comparing the PRE and the POST sessions, the former showed—in the hemisphere contralateral to the movement—reduced levels of coherence in the α and β bands in the “event” period after trigger onset between the right primary sensorimotor area (SM1, C4) and ipsilateral frontal (Fc2, F4) and parietal areas (Cp2, P4). Moreover, a decrease of interhemispheric coherence between bilateral SM1 (C4 vs C3) was also observed. In the POST session, an increased level of synchronization in the α-1 band between the right SM1 area and the frontal (FC2, F4) and parietal areas (CP2, P4, Pz) and between the SM1 of the 2 hemispheres was observed. These patterns of synchronization in the α-1 band were similar to those observed during real movement (C3 vs C4, C3 vs Fz/F3, and C3 vs CP1).

Figure 2.

Scalp topography of event-related imaginary part of coherence in α-1, α-2, and β bands between C4 (left “phantom” arm movement in left and central columns) or C3 (right arm movement, right column) and all EEG channels. Red (blue) color indicates increased (decreased) levels of synchronization with respect to baseline in the time period from 500 to 1500 ms after trigger (0 being the trigger onset). C3 or C4 channels are marked by a full red circle.

Classification of the ENG Signals According to the ERD/ERS

In the session POST, 58% of EEG epochs were selected for the single trial ERD/ERS analysis free of artifact to provide triggers related to EEG activity. The maximum ERD in the α band was found at around 1200 ms (median; 5th-95th percentile; 500-2140 ms). The RRs and MI obtained while using the trigger video and the information provided by the ERD/ERS analysis are shown in Figure 3. The improvement is evident for the recognition of 2 different grip types when using EEG-driven triggers.

Figure 3.

In the session POST (posttreatment), for 2 grip types—palmar (fist) and pinch grasp—the performances (Perf) of classifiers, as assessed by the recognition ratios (RRs) and the mutual information index (MI) achieved by labeling using the trigger video (TrVideo) and the information provided by EEG analysis (TrEEG) are displayed for different trials (1, 2, 3, and 4). The performances and MI achieved by using the trigger video are displayed in blue and azure, respectively; in yellow and red are the same parameters obtained by using the trigger EEG.

Discussion

Among the different approaches to restore the bidirectional link between the external world and the nervous system, the use of implantable intraneural peripheral nervous system interfaces may be valuable.^3-5,18 Results of the present report open a new avenue in this field, with an innovative approach combining simultaneous classification of signals from the sensorimotor areas that dispatch the motor command (via EEG signals) and from the nerve fibers as the final output to the target muscles (via tf-LIFE signals). The results in terms of classification of movements showed that a state control algorithm could be implemented. The information gathered from the EEG signals significantly improved the classification performance to identify the exact timing of the motor command dispatched from brain. In particular, a single trial ERD/ERS analysis to detect EEG modulation linked to the activation of cortical areas related to preparation of movement was carried out. ⁸ By identifying the exact cortical timing of motor command formulation, it was possible to better focus the analysis of LIFE signals in the epochs just following ERD, improving the classification performance significantly.

What was interesting was that EEG data clearly reflected “learning,” based on restoration of the physiological behavior of movement-related cortical oscillations. In the PRE session, the normal modulation of rhythms time locked with movement imagination (α and β ERD) was missing, whereas this was evident after training, in the POST session. ⁸ The only exception to missing α and β rebound (ERS) after termination of robotic hand movement could be a result of the excitability derangement involving the motor cortex contralateral to the missing limb, as previously evidenced through a TMS study ⁵ with an enlargement of the representation area of muscles adjacent to the stump. Moreover, according to previous evidence, ¹⁹ we hypothesize that the missing α and β ERS is partly related to the absence of somatosensory inputs coming from the lost hand to the motor cortex.

Coherence analysis revealed a restored α band synchronization of the Rolandic area with frontal and parietal ipsilateral regions known to participate in movement planning and execution. Consistent with the reduction of the cortical reorganizations within the sensorimotor regions and restoration of physiological firing and connectivity of the areas devoted to motor control, a significant decrease of PLP in our patient was evident. Present results confirm the findings in previous literature.^20,21 We hypothesize that the recovery after 4 weeks of input/output data streaming to and from the periphery achieved through the intensive use of tf-LIFEs led to the reduction of sensorimotor cortical aberrant reorganization, supporting the <3 months mitigation of the phantom limb syndrome.