Abstract
The brain typically uses a rich supply of feedback from multiple sensory modalities to control movement in healthy individuals. In many individuals, these afferent pathways, as well as their efferent counterparts, are compromised by disease or injury resulting in significant impairments and reduced quality of life. Brain–machine interfaces (BMIs) offer the promise of recovered functionality to these individuals by allowing them to control a device using their thoughts. Most current BMI implementations use visual feedback for closed-loop control; however, it has been suggested that the inclusion of additional feedback modalities may lead to improvements in control. We demonstrate for the first time that kinesthetic feedback can be used together with vision to significantly improve control of a cursor driven by neural activity of the primary motor cortex (MI). Using an exoskeletal robot, the monkey's arm was moved to passively follow a cortically controlled visual cursor, thereby providing the monkey with kinesthetic information about the motion of the cursor. When visual and proprioceptive feedback were congruent, both the time to successfully reach a target decreased and the cursor paths became straighter, compared with incongruent feedback conditions. This enhanced performance was accompanied by a significant increase in the amount of movement-related information contained in the spiking activity of neurons in MI. These findings suggest that BMI control can be significantly improved in paralyzed patients with residual kinesthetic sense and provide the groundwork for augmenting cortically controlled BMIs with multiple forms of natural or surrogate sensory feedback.
Introduction
Over the past decade, substantial advances have been made in cortically controlled brain–machine interfaces (BMIs) in both animals and humans (Serruya et al., 2002; Taylor et al., 2002; Carmena et al., 2003; Nicolelis et al., 2003; Kennedy et al., 2004; Musallam et al., 2004; Hochberg et al., 2006). Despite these early successes, first-generation clinical implementations suffered from acute and pervasive problems such as underlying instabilities and poor quality of control (Hochberg et al., 2006). Some of these control issues have been mitigated as current BMI implementations demonstrate robust, stable performance on significantly complex tasks requiring cursor stabilization (Kim et al., 2008; Mulliken et al., 2008), self-feeding using a robotic arm (Velliste et al., 2008), and the voluntary control of paralyzed muscles (Moritz et al., 2008; Pohlmeyer et al., 2009). Additional control improvements may require next-generation BMIs to move beyond the current paradigm in which feedback is only available through the subject's intact visual system to one that incorporates feedback both from vision and other relevant sensory modalities (i.e., proprioceptive, tactile, and force feedback).
Multiple sensory modalities are frequently used to estimate the state of the body with respect to the external environment and are thus critical for normal motor control (Rossetti et al., 1995; van Beers et al., 1999; Sober and Sabes, 2005). Experimental evidence indicates that abilities for on-line control and error correction are highly dependent on the proprioceptive system, especially when visual feedback is unreliable or unavailable (Gordon et al., 1995). Some shortcomings of current BMI control are mirrored in the movements of patients suffering from the loss of proprioceptive feedback as a result of large-fiber sensory neuropathies. These individuals can move by relying on vision, but their movements require great concentration and are slow compared with healthy individuals (Ghez et al., 1995; Sainburg et al., 1995).
The importance of proprioceptive feedback in the control of artificial limbs has been recognized since the early 20th century (Childress, 1980), and its utility for advanced BMI control has been widely acknowledged (Abbott, 2006; Hatsopoulos and Donoghue, 2009). However, its effects on BMI control remain mostly unexplored. We addressed this gap in BMI research by conducting an experiment in which the presence and fidelity of proprioceptive feedback during BMI control were systematically varied. Using a robotic exoskeleton, we evaluated the hypothesis that congruent visual and proprioceptive feedback would improve the movement of a cortically controlled visual cursor compared with a BMI relying solely on vision for feedback. Because of the movement of the arm and its interaction with the exoskeleton, the monkeys received a very rich feedback experience. In addition to visual feedback, these conditions included sensory information about limb position and velocity derived from receptors in the muscles, joints, and skin (Bosco and Poppele, 2001; Johnson et al., 2008) and force and tactile cues from the interaction of their arm with the exoskeleton. Within the scope of this study, we define the term “proprioceptive feedback” to include all movement-related sensory information originating from sensory receptors in the arm.
Materials and Methods
Behavioral task
Two adult male rhesus macaques (Macaca mulatta) were operantly trained to control a cursor in a two-dimensional workspace using a two-link robotic exoskeleton (Scott, 1999). The animals sat in a primate chair with their arm in the exoskeleton. Their shoulder joint was abducted 90° and supported by the manipulandum such that all movements were made within the horizontal plane. Direct vision of the limb was precluded by a horizontal projection screen above the monkey's arm. Visual feedback was available via a visual cursor projected onto the screen. The position of the cursor was controlled by one of two sources: either the position of the end effector of the robot (i.e., the monkey's hand position) or the output of a BMI that decoded the position of the cursor based on recent neural activity. Cartesian coordinates of the visual cursor were determined by digitizing the shoulder and elbow angle along with angular velocity at 500 Hz and transforming these variables into a visual cursor position (in centimeters) using the forward kinematic equations for the exoskeleton.
The random target pursuit (RTP) task required the monkeys to repetitively move a cursor (6 mm diameter circle) to a square target (2.25 cm2). The target appeared at a random location within the workspace (12 × 6 cm), and each time the monkey hit it, a new target appeared immediately in a new random location. To complete a successful trial and receive a juice reward, the monkey was required to sequentially acquire two to seven targets, depending on the experimental condition. Because of their increased difficulty, the number of successful hits per trial was reduced in the BMI conditions to encourage continuous play. Because each trial completion was followed by the immediate presentation of another target, the monkeys typically did not pause between trials resulting in continuous movement trajectories. A trial was aborted if any movement between targets took >2500 ms or if the monkey removed his arm from the exoskeleton. After an aborted trial, a new sequence of randomly presented targets appeared.
Experimental design
Before the experiment, a monkey used its arm to perform a random target pursuit task designed to generate complex movements that thoroughly sampled the position and velocity space of the arm. We refer to this case as the “active movement” (AM) condition (Fig. 1a). Next, the animals observed the replay of target positions and cursor trajectories recorded during the AM phase while they voluntarily maintained a static arm posture in the robotic exoskeleton at a predetermined reference position. Since neurons in the motor cortex demonstrate congruent activity during active movement and observation of action (Tkach et al., 2007; Suminski et al., 2009), we used the neural discharge recorded during the playback to construct a neural decoder that predicted Cartesian position of the cursor based on recorded cell activity. It is important to note that this decoder was trained solely on the observed target positions and cursor trajectories while the monkey's arm remained motionless in an attempt to mimic a training paradigm that could be used in patients with severe motor dysfunction. A new neural decoder was trained during each experimental session.
Experimental apparatus and single-trial kinematic data during the active movement and BMI conditions. Monkeys performed the RTP task by moving a visual cursor (red circle/trace) to randomly placed targets (squares). To receive a reward, the monkey was required to sequentially acquire either two or seven successive targets. a, During AM phase, the position of the visual cursor was controlled by the movements of the monkey's hand (black trace). b–d, In the BMI phase, the position of the visual cursor was decoded from the spiking activity of an ensemble of neurons in motor cortex. b, During the visual feedback-only condition, the animal voluntarily maintained a static posture in the exoskeleton. c, During the visual and proprioceptive feedback condition, the monkey's hand was moved (by the exoskeleton) through the decoded cursor trajectories. d, In contrast, the monkey's hand was moved through a trajectory unrelated to the cursor during the visual and noisy proprioceptive feedback condition.
During the experiment, the monkey used a real-time decoding (BMI) algorithm to move the cursor in the same task based on the activity of an ensemble of recorded motor cortical [primary motor cortex (MI)] neurons under three different conditions. In the first condition, referred to as the visual feedback-only condition, VBMI, the animal moved the cursor only via the neural decoder while maintaining a static arm posture in the robotic exoskeleton at a predetermined reference position (Fig. 1b). If the monkey moved the manipulandum away from this reference position and outside an invisible “hold” region (1 cm radius around the reference position) or removed his arm from the manipulandum, the game was “turned off” (i.e., the visual cursor and target were extinguished) until the monkey resumed proper posture. A gentle elastic force was applied to the manipulandum to keep the animal's arm from drifting away from the reference position, but was not applied within the “hold” region. Therefore, within the hold region, the small inherent friction of the exoskeletal robot served as the only resistance.
The second condition, referred to as the visual and proprioceptive feedback condition, V+PBMI, provided the animal with congruent visual and proprioceptive feedback about the position of the decoded cursor (Fig. 1c). Here, the animal controlled the cursor with the BMI while his arm was moved by the exoskeleton to follow the visual cursor, thereby transforming the monkey's arm from an effector driving the movement into a transducer providing an accurate proprioceptive estimate of the position and velocity of the cursor. Cursor trajectories were reproduced well by movements of the exoskeleton as the median (first quartile, third quartile) correlation coefficient between hand and cursor position was 0.92 (0.81, 0.97) and 0.89 (0.76, 0.95) for the X and Y direction, respectively.
Last, we included a control condition, visual and noisy proprioceptive feedback (V+NBMI), in which the monkey moved the cursor via the BMI while its arm was moved by the exoskeleton through a trajectory that was different from the cursor (Fig. 1d). The movement of the arm was driven by the replay of cursor trajectories recorded during the visual only and the visual and proprioceptive feedback BMI conditions that were completed earlier in the same experimental session. Despite the discrepancy in their feedback signals, hand movements contained some information about BMI cursor movements as the median (first quartile, third quartile) correlation coefficient between the position of the BMI cursor and the hand was 0.62 (0.48, 0.80) and 0.57 (0.46, 0.75) for the X and Y direction, respectively. The purpose of this condition was to confirm that any performance gains observed during the V+PBMI condition were attributable to information contained in the proprioceptive feedback and not task-irrelevant sensory stimulation related to the motion of the limb.
We implemented a proportional-derivative (PD) controller to minimize the deviation between the commanded and realized position of the exoskeleton during the V+PBMI and V+NBMI decoding conditions. Because of the dynamics of the exoskeleton, we found that this controller introduced a time delay between the control signal and the realized position of the monkey's arm, causing a delay in the proprioceptive feedback. To compensate for this delay, we imposed a 100 ms time delay on the visual feedback in all of the decoding conditions, resulting in the temporal alignment of the cursor position and the monkey's hand during the congruent feedback BMI condition.
Trial selection
To assess the accuracy of the proprioceptive feedback given by the PD controller without voluntary control of the arm, we performed a separate control experiment in which we measured the dynamics and average error between the commanded (i.e., the cursor) and actual positions of the robot (and arm) during replayed BMI trajectories with an anesthetized animal. This experiment provided a means to reject trials in the V+PBMI and V+NBMI conditions in which the monkey may have made undesirable arm movements leading to large errors between hand and cursor direction above and beyond the error introduced by the passive dynamics of exoskeleton and monkey's arm.
In this control experiment, the animal was anesthetized (ketamine, 2 mg/kg; dexmedatomidine, 75 μg/kg; atropine, 0.04 mg/kg) and then placed in the primate chair with its arm strapped into the manipulandum. Cursor position was digitized (500 Hz) and recorded independently while the monkey's relaxed arm was moved through the replayed cursor trajectories for ∼5 min. Playback of each trajectory was repeated three times for a total exposure time of 15 min. We computed the cross-correlation between the X and Y cursor and hand positions during passive arm movements to measure the time delay between movement of the cursor and the manipulandum. As expected, a strong correlation (>0.95) was observed between cursor and hand position at average time delays of 98 and 52 ms in the X and Y directions, respectively. To compute the error between cursor and hand positions, we first corrected for the dynamics of the position controller/manipulandum by shifting the hand position data by the appropriate time delay and then used the dot product to compute the angular difference in movement direction (error) between the cursor and hand on a sample-by-sample basis. In this control experiment, when the animal was anesthetized, the error between the cursor and hand direction averaged 41.6 ± 7.5° (μ ± σ) for monkey MK and 42.8 ± 6.7° for monkey B.
Based on the values obtained from the control passive movement experiment, we computed the direction error between the hand and decoded cursor or replayed trajectory on a sample-by-sample basis for the V+PBMI and the V+NBMI conditions, respectively. Trials with an average error exceeding the mean error plus 3 SDs (as obtained from the control experiment described above) were excluded from additional analysis. This threshold was 64.1° for monkey MK and 62.9° for monkey B. We also excluded trials in which the monkey may have been voluntarily contributing to the movement of the exoskeleton. Here, a cross-correlation analysis between the commanded and actual hand position on each trial was used to determine those trials in which the movement of the hand preceded (i.e., led) the movement of the command signal by >0 ms. As a result, across all experiments in both monkeys, 385 of 1210 trials and 335 of 561 trials from the V+PBMI and V+NBMI conditions, respectively, passed both tests and were analyzed further. We varied the correlation delay threshold in the V+PBMI to investigate the effect of the monkeys' voluntary contribution to movement of the exoskeleton on BMI performance. In this analysis, we considered three additional thresholds for the elimination of trials: (1) hand led BMI cursor by >25 ms, (2) hand led cursor by >50 ms, and (3) no lead/lag threshold (i.e., all data were considered). A total of 650 (53%), 860 (71%), and 1210 (100%) of V+PBMI trials were included in the analysis based on these exclusion criteria. A total of 708 and 756 trials were analyzed from the VBMI and AM conditions, respectively.
Electrophysiology
Each animal was chronically implanted with a 100 electrode (400 μm interelectrode separation) microelectrode array (Blackrock Microsystems) in MI contralateral to the arm used for the task (Maynard et al., 1999). The electrodes on each array were 1.5 mm in length. The tips of the electrodes were coated with iridium oxide. During a recording session, signals from up to 96 electrodes were amplified (gain of 5000), bandpass filtered between 0.3 Hz and 7.5 kHz, and recorded digitally (14 bit) at 30 kHz per channel using a Cerebus acquisition system (Blackrock Microsystems). Only waveforms that crossed a user-defined threshold were used for real-time decoding and stored for additional analysis. The neural data used to train and drive the BMI during the real-time decoding conditions were comprised of single and multiunit spiking events that were spike-sorted on-line. On average, 43.67 ± 1.43 and 44.80 ± 2.35 (mean ± 1 SE) neural channels were allocated for use with the decoder per experimental session for monkeys MK and B, respectively. In contrast, the stored spike waveforms that were used in our neural analyses were spike-sorted off-line, using Offline Sorter (Plexon). In total, 609 individual cell samples were identified via Offline Sorter: 337 cell samples across six data sets from monkey MK and 272 cell samples across five data sets from monkey B. Because of the chronic nature of our recordings, we use the term “cell samples” when aggregating over data sets to make clear that these represent independent samples across different data sets but may or may not represent different neurons.
This study was conducted over an extended period of time and the monkeys were trained on multiple variants of the task described above. The 11 data sets (5 from monkey B and 6 from monkey MK) included in our analysis represent all data collected using this particular experimental design. A data set is defined as the simultaneously recorded neural activity during a single recording session. All of the surgical and behavioral procedures were approved by the University of Chicago Institutional Animal Care and Use Committee and conform to the principles outlined in the Guide for the Care and Use of Laboratory Animals.
Robust linear filter decoder
The real-time decoder is implemented using a Wiener filter (Wessberg et al., 2000; Serruya et al., 2002; Fagg et al., 2009). The spikes generated by a single neuron or multiunit activity during 1 s before prediction were binned into a total of 20 time bins, each of which was 50 ms in duration. The predicted cursor position is a linear function of these binned spike counts. The coefficients for the model are selected using a ridge regression approach that trades prediction accuracy on the training set for a smoother prediction surface (Bjèorck, 1996). This approach helps to address the overfitting problem that can occur with small training data sets in high-dimensional feature spaces.
In our approach, a time series of hand position data points is reconstructed from a linear combination of neural responses from many neurons at multiple times as follows:
where S(t) is the signal to be reconstructed, N(i, j) denotes the activity of neuron i at time lag j, ai,j is the corresponding coefficient, C is the number of cells, Tpre is the time before the current time t, and dt is the width of the time bins (50 ms). The set of coefficients (ai,j) that minimizes a cost function that includes the least mean squared difference between the actual and reconstructed signal can be solved for analytically as follows:
where (NTN) is the correlation matrix that contains the pairwise correlations between each pair of neurons at different time bins, α is a regularization constant required for the ridge regression, and I is the identity matrix. Since hand movements are restricted to a plane, independent estimates of the X and Y components of hand position were made to predict the motion of the hand.
Chance decoder performance
We computed the chance percentage level of successful trials for each animal by randomly shuffling the binned firing rate time series for each unit during the VBMI and V+PBMI conditions. Cursor trajectories were then generated (off-line) by applying the actual filters computed during the experiment to the shuffled neural data. Each simulated trajectory was required to hit the same sequence of targets as were presented during the real-time decoding conditions. Just as in the real experiment, cursor trajectories were allowed up to 2.5 s for the cursor to hit a target, and two consecutive target hits constituted a successful trial. We repeated this procedure 1000 times allowing us to compute a mean and SD for the percentage of successful trials. The averaged chance success rate for monkey B was 29.9 ± 0.8 and 36 ± 0.4% (mean + 1 SE) for the VBMI and V+PBMI conditions, respectively. The averaged chance success rate for monkey MK was 38.7 ± 2.2 and 32.6 ± 0.8% for the VBMI and V+PBMI conditions, respectively.
Analysis
Kinematics.
Kinematic parameters (position and direction) of hand and cursor movement in each condition were binned in 50 ms bins and boxcar-smoothed using a 150 ms sliding window for most analyses. To assess performance differences between the active movement and real-time decoding conditions, we used three kinematic measures: (1) normalized time-to-target, (2) normalized path length, and (3) normalized path reversals. The normalized time-to-target metric is defined as the time difference between consecutive target hits divided by the Euclidian distance between the targets. The normalized path length metric is defined as the path length of the cursor between consecutive targets divided by the Euclidian distance between consecutive targets. Normalized path length is a unitless ratio of distance measures and captures the straightness of the cursor path. The normalized path reversal metric is defined as the number of positive-to-negative zero crossings in the movement speed directed along a line connecting consecutive targets. To compute the number of normalized path reversals, we first rotated each trajectory such that it started at the origin and ended at a point on the positive x-axis. We then computed the velocity of the trajectory in the x direction and counted the number of positive-to-negative zero crossings (i.e., the number of times the visual cursor started to move away from the target). Last, we divided the number of zero crossings for each target hit by the Euclidian distance between consecutive targets. Kinematic observations for each metric were pooled by monkey across data sets, and a one-way ANOVA with post hoc t tests was used to determine whether BMI performance differed as a function of experimental condition.
Spiking activity.
In our off-line sorting and analysis, we sought to isolate single units possessing signal-to-noise ratios of 3 or higher. Signal-to-noise ratios were defined for each sorted unit as the difference in mean peak-to-trough voltage divided by twice the mean SD. The mean SD was computed by averaging the SD of the spike waveform over all acquired spikes at each of the 48 sample time points of the waveform. Unlike the preceding kinematic analyses, we considered all observations related to the spiking activity of single cells to be independent and pooled them across subjects for all population analyses.
We performed a principal components analysis (PCA) on the ensemble of neurons recorded in each data set to understand how the different experimental conditions modulate spiking activity. Specifically, we were interested in exploring whether passive movement of the arm in the BMI conditions resulted in an increase in the “flexibility” of the neural activity (i.e., the ability to use more degrees of freedom). The methodology has been previously described (Yu et al., 2009). Briefly, spike trains (1 ms resolution) for each neuron in the ensemble were binned in 20 ms bins and the resulting spike counts were square root transformed to stabilize the spiking noise variance. A smoothed estimate of the firing rate for each neuron was then created by convolving the binned spike trains with a Gaussian kernel (σ = 60 ms). We then used PCA to determine the number of orthogonal bases required to account for 90% of the variance in the ensemble activity. The number of bases in each condition was pooled across both data sets and subjects, and a one-way ANOVA was used to determine whether the number of bases varied across experimental condition.
Mutual information between binned neural data and kinematics (50 ms bins) was calculated at multiple time leads and lags (Paninski et al., 2004). This analysis captures both linear and nonlinear relationships between the two variables by means of signal entropy reduction. The computation yields a measure of the strength of the relationship between the two variables when they are shifted with respect to each other by different time lags. By examining the relative timing of the peak mutual information, we were able to determine at what time lag the modulation of a neuron was most related to the cursor movement. The kinematic probability distributions (one-dimensional distribution of instantaneous movement direction) conditioned on the number of observed spikes were estimated by histograms of the empirical data. To account for biases in this estimation, the information calculated from shuffled kinematic bins (mean of 100 shuffles) was subtracted from the values obtained from the actual data for each cell. Last, the lead/lag mutual information profiles were boxcar-smoothed with a 3 bin window (150 ms).
Electromyograms.
Electromyograms (EMGs) were recorded from the biceps and triceps using differential surface electrodes (Noraxon Myosystem 1200). EMG signals were bandpass filtered between 10 and 450 Hz, amplified (100×), and digitized at 2000 Hz using the Cerebus acquisition system. Residual offsets were subsequently removed from the digitized EMGs, which were then rectified and filtered at 4 Hz with a zero-phase low-pass filter (fourth-order Butterworth). Next, the resulting time series of muscle EMG were normalized by the mean value of the rectified and filtered activity recorded during the active movement phase of the experiment. We then characterized the coordination between muscles at the elbow by estimating the degree of antagonist muscle co-contraction using a measure also known as “wasted contraction” (Thoroughman and Shadmehr, 1999). For the pair of antagonist muscles (biceps and triceps) at each sampling instant, the minimum value of the two normalized EMG signals was selected to yield a time varying co-contraction signal, which represents the magnitude of normalized EMG that is equal and opposite in the antagonist muscle pair. One-way ANOVAs were used to determine differences in muscle activity and co-contraction across conditions.
We used a cross-correlation technique (Suminski et al., 2007) to evaluate the temporal relationship between elbow angular velocity and individual muscle activities during each trial of the active movement and real-time decoding conditions. The resulting correlation time series were averaged across trials within each experimental condition for display purposes. The peak correlation magnitude for successful trials in each condition was Fisher transformed and compared using a one-way ANOVA.
Eye position.
Eye movements were recorded with an infrared oculometer (R-HS-S6; Applied Science Laboratories) located in front of the animal. The eye position was calibrated using the ASL EyeTracker Software Suite while the monkey performed the RTP task with targets placed in a 3 × 3 grid that spanned the rectangular workspace. During the BMI conditions, the Cartesian positions of the monkey's gaze were digitized at 1 kHz, time-stamped with respect to the neural and kinematic data, and saved to disk. These values were regressed against the x and y target positions in the active movement condition to determine the location of gaze in the same coordinate frame as the cursor and target. Using the resulting regression coefficients, x and y gaze position was then correlated with the x and y target positions on a per-trial basis during each of the BMI conditions. Correlation coefficients for each of the BMI conditions were Fisher transformed and the means of the resulting distributions were compared against each other using a one-way ANOVA statistical test.
Results
Both monkeys achieved successful control of the cursor in all three BMI conditions as assessed by the percentage of trials in which two consecutive targets were hit within 2500 ms. Monkey B achieved an average success rate of 75.0 ± 6.2, 75.4 ± 5.9, and 74.4 ± 6.4% (mean + 1 SE; across all recording sessions) for the VBMI, V+PBMI, and V+NBMI conditions, respectively. Monkey MK achieved 81.7 ± 4.5, 82.4 ± 4.1, and 77 ± 6.5% success rates for the VBMI, V+PBMI, and V+NBMI conditions, respectively. The success rates observed during the BMI conditions were less than those observed during active movement (97.2 ± 0.5 and 92 ± 2.5% for monkeys B and MK, respectively) but were substantially greater than the chance rate computed off-line (see Materials and Methods, Chance decoder performance). Because of the daily retraining of the neural decoder, success rates varied from day to day. We found no evidence of day-to-day gains in success rate that have been reported by others using decoders that were fixed across experimental sessions (Ganguly and Carmena, 2009). We were unable to determine the cause of missed trials because the task did not require a specific behavior to initiate a trial. Missed trials were likely attributable to some combination of poor performance and sporadic periods of inattention. Success rates were computed based on all trials and thus do not reflect the trial selection criteria imposed in the subsequent analyses.
Improved performance with congruent feedback
Because our monkeys were trained to relax their arm and no restraints were used during any of our experiments, it was important to select only those trials for our analyses during which the monkeys were receiving intended sensory feedback and to remove any trials during which the monkeys' voluntary movements interfered with the intended sensory information (see Materials and Methods, Trial selection). Examination of the cursor trajectories in each condition revealed that, on average, the movements generated by the BMI incorporating both veridical visual and proprioceptive feedback were straighter and less variable compared with the other BMI conditions. We used three behavioral metrics to quantify the effects of multisensory feedback on the quality of BMI performance: (1) the normalized time-to-target (i.e., the time to reach a target divided by the distance to the target), (2) normalized path length (i.e., the actual path length divided by the target distance), and (3) the normalized number of path reversals (i.e., the number of times the transiently moved away from the target divided by the target distance). One-way ANOVAs with post hoc t tests were used to evaluate the condition-dependent differences in these metrics for each animal separately.
Consistent with our hypothesis, both monkeys moved the visual cursor faster and straighter when using a BMI that provided veridical visual and proprioceptive feedback compared with a BMI with visual feedback alone (Fig. 2a, Table 1). The mean normalized time-to-target was significantly shorter during the V+PBMI condition compared with both the VBMI and the V+NBMI conditions (p < 0.05, for each animal). Monkeys also generated straighter paths under the V+PBMI condition (p < 0.05, for each animal). Interestingly, BMI performance was significantly poorer according to the path length metric during the V+NBMI condition compared with the VBMI condition indicating a detrimental effect of incongruent feedback (p < 0.05, for each animal). Finally, we found that the BMI-generated paths having the least number of trajectory reversals occurred during the V+PBMI condition compared with the VBMI and V+NBMI conditions. In fact, the average percentage difference between V+PBMI and VBMI for these three metrics show a ∼40% improvement when the monkeys moved a visual cursor using a BMI incorporating congruent visual and proprioceptive feedback compared with visual feedback alone (Table 1). According to these metrics, cursor movements generated in the V+PBMI condition reasonably approximated natural reaching movements observed in the AM condition.
Kinematic performance during the active movement and BMI conditions for each monkey. a, Decoding performance is enhanced by the incorporation of congruent naturalistic proprioceptive feedback into the brain–machine interface in the V+PBMI condition (red bar) compared with the visual feedback-only (VBMI, orange bar) and visual and noisy proprioceptive feedback (V+NBMI, gray bar) BMI conditions. Improved performance is evident based on the significantly shorter normalized time-to-target (top) and path length (bottom). The error bars indicate ±1 SE about the mean performance. b, Decoding performance in the V+PBMI condition improved, a reduction in time-to-target and path length, as the movement direction error between the cursor and hand decreased (curves). Larger direction errors caused degradation in BMI control. We varied the time lag threshold between hand and cursor movement to investigate the effect of active movements on BMI performance. Performance decreased proportionally to increases in the time hand movements were allowed to lead cursor movements. The thresholds for elimination of trials were as follows: (1) hand led BMI cursor by >0 ms (red line; metric used other analyses), (2) hand led BMI cursor by >25 ms (dashed red line), (3) hand led cursor by >50 ms (dashed black line), and (4) no lag threshold (black line). Larger direction errors caused degradation in BMI control. The poorest performance was found in the V+NBMI, in which the direction errors were highest (gray circles). Performance in the active movement condition was plotted at an error of 0 because the cursor was controlled by the exoskeleton (blue circles). VBMI performance is represented by the yellow circles.
Normalized behavioral performance per animal
In addition to the presence of congruent multisensory feedback, the degree of congruence also contributed to the improvement in BMI movements we observed. In examining the relationship between task performance and direction error in the V+PBMI condition, we found that, as the discrepancy between cursor and hand movements decreased, both time-to-target and path length decreased (i.e., improved performance) (Fig. 2b, solid red curves). As the elimination criteria became less stringent, performance decreased (Fig. 2b, dashed red, dashed black, and solid black curves). As direction error became large, performance degraded dramatically. The worst performance was observed in the V+NBMI condition, in which the average direction error between the visual cursor and hand movement was greatest (Fig. 2b, gray circle). Finally, in both V+PBMI and V+NBMI, we observed a significant decrease in performance when all of the data were included in the computations (Table 1, denoted as V+PALL,BMI and V+NALL,BMI).
Task performance in the VBMI condition was likely influenced by the 100 ms delay we imposed to compensate for the dynamics of the exoskeleton. To investigate the magnitude of this effect, we compared the performance of the BMI incorporating congruent visual and proprioceptive feedback to the performance of a vision-only BMI having no delay. We collected an additional 10 data sets (5 from monkey MK and 5 from monkey B), in which we imposed no delay on the movement of the visual cursor in the VBMI condition (Table 1, denoted as V0,BMI). For each monkey separately, a two-sample t test was used to evaluate the condition-dependent differences for each performance metric. As expected, we found that performance improved when the 100 ms visual delay was removed (two-sample t test; p < 0.0005 for each metric). However, we found that the BMI incorporating congruent multisensory feedback (even with the imposed 100 ms delay) outperformed the vision only BMI with no delay in the time-to-target (p = 0.019 and p = 0.004, for MK and B) and path length (p = 0.007 and p < 0.0005, for MK and B) metrics by 10 and 15%, respectively (Table 1). There was no difference in the number of trajectory reversals between these two conditions.
We collected surface EMGs from the biceps and triceps in monkey MK to verify that the cursor movements observed during the BMI conditions were not influenced by the monkey's active arm movements. Figure 3a compares the arm kinematics and EMG activity during a representative experiment. Changes in the magnitude of biceps and triceps muscle activity are readily apparent in conditions in which the arm is moved, either by the monkey (AM, blue bar) or the exoskeleton (V+PBMI and V+NBMI, red and gray bars, respectively). To quantify condition-related changes in muscle activity, we computed the trial-averaged EMG magnitude for each muscle and compared them with a one-way ANOVA. The test found a significant effect of experimental condition for both muscles, with EMG magnitude being significantly increased in the V+PBMI and V+NBMI conditions compared with AM and VBMI (Fig. 3b) (p < 0.05). To verify that the exclusion criteria were removing V+PBMI trials when muscle activity may have contributed to improper kinesthetic feedback, we compared EMG magnitude in the population of included and excluded trials for both muscles. We found that muscle activity was significantly reduced in the population of V+PBMI trials that survived the exclusion process (two-sample t test; p = 0.04 and p = 0.03 for biceps and triceps, respectively).
EMG measures for the active movement and BMI conditions. a, Movement of the arm (represented here by the elbow angle) was accompanied by large changes in muscle activity. The monkey was unable to suppress muscle activity in V+PBMI and V+NBMI. The colored bar at the top of the figure shows the transitions among the four experimental conditions. b, The average EMG magnitude for the biceps and triceps in the active movement (blue bar) and three BMI conditions: visual feedback-only (VBMI, yellow bar), visual and proprioceptive feedback condition (V+PBMI, red bar), and visual and noisy proprioceptive feedback condition (V+NBMI, gray bar). Muscle activity significantly increased in the decoding conditions in which the arm was moved. c, This increase in muscle activity was associated with an increase in co-contraction of these antagonist muscles. The error bars indicate ±1 SE about the mean muscle activity. d, We used a cross-correlation analysis to evaluate the relationship between surface electromyograms recorded from the biceps (left) and triceps (right) and the elbow angular velocity. Muscle activity causing movement at the elbow was only observed during the active movement phase and not during the three BMI conditions.
We used two measures to assess whether this increase in muscle activity during the V+PBMI and V+NBMI conditions was related to the monkey's attempt to move the visual cursor by actively moving his arm. First, we computed a measure of the antagonist muscle co-contraction for the biceps and triceps. A one-way ANOVA found a significant increase in the magnitude of co-contraction in the V+PBMI and V+NBMI conditions compared with AM and VBMI (Fig. 3c) (p < 0.05). Next, we used a cross-correlation analysis to examine the relationship between muscle activity (the biceps and the triceps) and elbow angular velocity during the AM and BMI conditions. A one-way ANOVA and post hoc t test found no differences in the peak correlation magnitude between EMG activity and elbow joint velocity across the three BMI conditions (p > 0.05). Furthermore, the correlation magnitudes in the BMI conditions (at the time lag of peak correlation in AM) were not different from zero, suggesting that the monkey was not actively attempting to move its limb during BMI control (Fig. 3d). Peak correlation magnitude between EMG activity and elbow joint velocity significantly increased during the AM condition compared with the BMI conditions (p < 0.05), demonstrating the expected relationship between muscle activity and joint motion during reaching movements (Fig. 3d).
In addition to collection of muscle activity, we recorded the eye movements in monkey MK to confirm that our results are not biased by the position of the monkeys' gaze. A one-way ANOVA found that there was no difference in cross-correlation magnitude between eye position and target location across the three BMI conditions (p = 0.27 and p = 0.61 for X and Y positions, respectively), indicating no effect of gaze position on our behavioral result.
Condition-dependent modulation of spiking activity
We next investigated how the condition-dependent modulation in MI neurons contributed to the performance gains observed when congruent visual and proprioceptive feedback were available to the monkeys during BMI control. We first examined neuronal spike rate modulation during the AM and the three BMI conditions. When viewed over the timescale of the entire experiment, the responses of individual neurons varied substantially across the experimental conditions (Fig. 4a, colored bar). Some neurons seemed to prefer active movement, whereas others preferred individual decoding conditions or some combination of movement and the decoding conditions. This diverse, structured neural activity is illustrated Figure 4a, which shows the normalized binned firing rate (50 ms bins) as a function of time for each of the 61 neurons recorded during a single session. Changes in the experimental condition precisely correlate with substantial changes in the firing rate of individual neurons appearing as vertical striations in Figure 4a.
Diversity in activity of single units during active movement and BMI conditions. a, Time series of binned firing rates for all units recorded during a single session (B0428). Firing rates from each individual neuron were binned (50 ms bin size) and normalized to their maximum firing rate. The resulting time series were then smoothed using a zero-phase, fourth-order, Butterworth, low-pass filter with a cutoff frequency of 0.1 Hz for display purposes. The bins shown in white represent the highest firing rates for each cell, whereas those areas shown in black correspond to very low firing rates. Notice the substantial changes in the firing rates of some cells at the transitions between experimental conditions. The colored bar at the top of the figure shows the transitions between the four experimental conditions. b, A comparison of the firing rates of all neurons in the V+PBMI and V+NBMI with VBMI (left and middle panels, respectively) revealed an increase in the firing rate gain when the arm was moved by the exoskeleton during BMI control. This gain modulation was unrelated to the performance increase, as there was no difference in rate gain between the V+PBMI and V+NBMI conditions (right panel). Each black dot represents the firing rate of a single neuron. The ensemble data was well fit by a linear model in each condition (red line). A firing rate gain of unity is denoted by the black dashed line.
The modulation in the time series of firing rates raised the possibility that the observed changes in BMI performance may be caused by simple changes in the firing properties of neurons and not the introduction of congruent multisensory feedback. We performed two separate analyses to determine whether changes in the rate of spiking could explain the observed differences in BMI performance. First, we computed the average firing rate during included trials for all 609 neuron samples in each BMI condition. Figure 4b contains a set of scatterplots in which each black dot represents the average firing rate of a neuron in the indicated pair of conditions. A regression line was fit to the rate data for all 609 neuron samples (Fig. 4b, red lines) and a one-way analysis of covariance (ANCOVA) with post hoc t test was used to test for differences in the gain (slope) and/or baseline firing rate (intercept) between conditions. We observed a significant firing rate gain (i.e., significantly greater than 1) when comparing the rates measured in the AM condition with every other condition (p < 0.0005; gain = 1.08, 1.11, and 1.17 for VBMI, V+PBMI, and V+NBMI, respectively). Furthermore, there were significant firing rate gains in the V+PBMI and V+NBMI conditions compared with VBMI (p < 0.0005; gain, 1.11 and 1.08, respectively). ANCOVA found no significant gain effect between firing rates in the V+PBMI and V+NBMI conditions (p = 0.63; gain, 1.01). Gain effects were not accompanied by similar changes in baseline firing rate.
Next, we performed principal components analysis on the ensemble of smoothed firing rate time series estimated for each data set. We sought to show that the improvement in BMI performance was not attributable to an increase in the “flexibility” of ensemble neural activity (i.e., the possibility that MI enters a higher dimensional space during some conditions, thereby using more degrees of freedom). We computed the number of orthogonal bases required to account for 90% of the variance in the ensemble activity for each data set. ANOVA found a significant reduction in the number of bases required to explain the data variance in the V+PBMI condition compared with the VBMI condition (p < 0.05; 25 and 31 bases, respectively). This finding demonstrates that the increase in BMI performance is accompanied by increased efficiency in the ensemble spiking activity. The AM and V+NBMI conditions required 30 and 27 bases to account for 90% of the data variance.
Information in spiking neural activity
We computed the mutual information between the instantaneous firing rate of each cell and the instantaneous cursor direction for a range of time lags (Fig. 5). We considered only those neurons whose mutual information peaked at a lag within the range of −600 to 600 ms (499 of 609 neuron samples satisfied this criterion). The magnitude of mutual information varied from neuron to neuron as seen in the three examples shown in Figure 5. Some neurons expressed graded amounts of information about movement direction across the experimental conditions (Fig. 5a), whereas others contained information during a subset of the conditions. We found that the activity of many neurons carried information only during the real-time decoding conditions (Fig. 5b). Mutual information often peaked at positive lags, indicating that cell activity carried information about the future direction of cursor movement (Fig. 5a,b). This is consistent with the idea that these cells causally “drive” the cursor movement (Moran and Schwartz, 1999; Paninski et al., 2004; Suminski et al., 2009). However, during V+PBMI, there were some neurons in which the peak mutual information occurred at near zero or even at negative time lags, indicating that the neural response occurred after the cursor movement, suggestive of a sensory-type response (Fig. 5c).
Examples of the diversity in mutual information temporal profiles (in bits) for single units during the AM and BMI conditions. Each trace represents the mutual information between the firing rate of a single neuron and the cursor movement direction over a range of temporal lags between the two time series. Mutual information profiles are shown for the AM (blue), VBMI (orange), V+PBMI (red), and V+NBMI (gray) conditions.
We were specifically interested in understanding how the strength and temporal relationship of mutual information was modulated by sensory feedback modality in cells whose activity contained significant information about direction in each of the experimental conditions. We included cells in the following analyses if they exhibited significant peak mutual information with the direction of cursor movement in at least one condition. Based on this criterion, 410 (82.2%), 218 (43.7%), 147 (29.5%), and 126 (25.3%) of the 499 neurons were analyzed in the AM, VBMI, V+PBMI, and V+NBMI conditions, respectively.
Summarizing the strength of the mutual information profiles pooled across both monkeys (Fig. 6a), a one-way ANOVA with post hoc t tests found that neural activity carried the greatest amount of information about the direction of cursor movement during the condition that provided the monkeys with congruent visual and proprioceptive feedback about the decoded cursor movement (V+PBMI condition; 0.09 ± 0.005 bits). The addition of congruent proprioceptive feedback yielded a 125% increase in mutual information in neural activity compared with the VBMI condition (0.04 ± 0.002 bits; p < 0.05). This increase was not attributable simply to movement of the arm, as the information magnitude was greater than that observed in the V+NBMI (0.057 ± 0.003 bits; p < 0.05). We found no difference in the amount of information in the neural activity about the direction of cursor movement during the VBMI and V+NBMI conditions. The neural activity recorded in the AM condition carried 0.06 ± 0.003 bits of information about movement direction. It should be noted that, when all 499 neurons (regardless of their significance) or when only those neurons that exhibited a significant information peak in all conditions (97 neurons) were used, the relationship between the magnitudes of mutual information in all conditions remained unchanged to those described above.
Variations in the magnitude and time lag of peak mutual information about movement direction. a, Peak mutual information values (bits) with respect to direction of movement for cell samples used in our analyses during the AM (blue), VBMI (orange), V+PBMI (red), and V+NBMI (gray) conditions. The mean ± 1 SE of the distribution of peak mutual information values for each condition are shown as horizontal black lines with error bars. Values exceeding 0.4 bits in magnitude are reported at that value for display purposes. b–e, Distribution of lags at which mutual information with respect to direction of movement peaks for the cell samples used in our analysis during AM, VBMI, V+PBMI, and V+NBMI. The dotted vertical line intersecting the four histograms represents a time lag of zero. Only neurons having a peak information magnitude significantly greater than zero were included in each condition.
Our results also indicated a shift in the lag at which the mutual information peaked with respect to cursor direction during the BMI conditions compared with the AM condition. The mean lag of peak mutual information during the AM condition was 67 ± 6 ms (Fig. 6b), but shifted to a longer lag time of 189 ± 8 and 209 ± 13 ms during VBMI and V+NBMI conditions, respectively (Fig. 6c,e). This longer delay time during the BMI conditions (∼130 ms) can be explained by the combination of the computational delay imposed by the neural decoder and the visual feedback delay we imposed to compensate for the dynamics of the exoskeleton. Unlike the other conditions, we found a bimodal distribution of peak mutual information lags during the V+PBMI condition (Fig. 6d). We characterized the two modes using a mixture of Gaussians model and an expectation–maximization clustering algorithm (Hastie et al., 2001), which found two modes at lags of −63 ± 14 and 242 ± 10 ms. The actual distribution of lags fit the model with an R value of 0.83. To understand the condition-dependent relationship between the magnitude of information carried by the neural activity and the timing of peak information, we computed the mean mutual information profiles across the two populations of neurons categorized by the sensory and motor modes of the V+PBMI distribution (Fig. 7a,b, respectively). Despite the very different timing of the two modes, we found that the magnitude of peak information in each mode was significantly enhanced in the conditions in which visual and proprioceptive sensory feedback were congruent (AM and V+PBMI) relative to the VBMI and V+NBMI conditions.
Separable neural populations during the congruent visual and proprioceptive feedback condition (V+PBMI). a, Mean (±1 SE) mutual information profiles of cells belonging to the first mode of the bimodal distribution of peak mutual information lags during the V+PBMI condition as seen in top inset. b, Mean (±1 SE) mutual information profiles of cells belonging to the second mode of the bimodal distribution of peak mutual information lags during the V+PBMI condition as seen in the top inset.
Discussion
Our results are the first to demonstrate the utility of feedback modalities other than vision in a cortically controlled brain–machine interface. Congruence in sensory feedback was an important factor driving performance as we observed that both time-to-target and path length decreased (i.e., increased performance) as the error between the visual and proprioceptive estimates of hand position decreased. Improvements in behavioral performance were accompanied by an increase in the magnitude of direction-related mutual information in the spiking activity of MI at time lags indicative of both sensory and motor activity.
A 100 ms visual feedback delay was imposed in all BMI conditions to synchronize the visual and proprioceptive feedback modalities in the V+PBMI, which, by definition, increased the time-to-target values in all conditions. However, even when the delay was removed in the vision-only condition (V0,BMI), the performance in V+PBMI condition (with the delay) was still improved by 10–15% relative to that in the V0,BMI condition. This smaller yet significant performance improvement (comparison of V+PBMI and V0,BMI in Table 1) is likely conservative and could be increased by providing feedback at physiological latencies through improvements in exoskeleton control or via a surrogate methodology (e.g., direct electrical stimulation of the nervous system).
We observed an increase in EMG activity in the conditions in which the arm was moved by the exoskeleton. This muscle activity was not the driver of improved performance as our correlation analysis demonstrated it was unrelated to the monkey's volitional attempt to move the visual cursor with its arm (thereby effecting movement of the BMI cursor). Rather, this muscle activity caused an increase in co-contraction at the elbow. In some V+PBMI trials, the force generated by this undesired muscle activity was sufficient to overcome the force generated by the exoskeleton causing feedback to be incongruent. Therefore, we removed those trials in which the sensory feedback was inappropriate or improperly timed (in the V+PBMI and V+NBMI conditions). This filtering procedure removed a significant number of trials in both conditions but ensured that the sensory feedback was appropriate in each condition.
Comparison with current BMI performance
We chose to use a decoder based on a linear filter because of its computational simplicity and its strong presence in the literature (Serruya et al., 2002; Taylor et al., 2002; Carmena et al., 2003; Musallam et al., 2004; Hochberg et al., 2006; Kim et al., 2008; Mulliken et al., 2008; Ganguly and Carmena, 2009). Other decoding algorithms were not considered because the focus of our paper was on the effect of feedback on BMI control. However, we believe that any gains achieved through the addition of multisensory feedback would transfer, at least partially, to systems using other decoding algorithms.
Our results compare favorably with state-of-the-art BMIs that rely on vision for feedback during closed-loop control. Ganguly and Carmena (2009) trained monkeys to use a linear filter based BMI that decoded the Cartesian position of a visual cursor using MI spiking activity. After performance reached a steady state, the movement time for their monkeys averaged 3.76 s for a 7 cm cursor movement (0.054 s/mm in our normalized time-to-target metric). In contrast, the normalized time-to-target for V+PBMI averaged 0.021 s/mm. This large performance difference may be attributable in part to the numbers of neurons included in the neural decoder. Because Ganguly and Carmena were interested in decoding from a stable population of neurons, they restricted their decoder to at most 15 neurons, whereas our BMI averaged ∼44 neural channels for monkeys MK and B. Performance in V+PBMI also surpassed that reported in a clinical experiment involving two human patients with tetraplegia (Kim et al., 2008). After a 500 ms dwell time was removed from the movement times reported in the manuscript, the time-to-target averaged 2.58 s (0.027 s/mm) and 2.8 s (0.029 s/mm) for the 9.2 cm cursor movements generated by either a position-based linear filter decoder or velocity-based Kalman filter decoder, respectively.
Mechanism for BMI control improvement
The performance increase with congruent sensory feedback was accompanied by three significant changes in the spiking activity of MI neurons during the BMI conditions. First, we found a significant gain modulation in the firing rate of MI neurons during the V+PBMI condition compared with VBMI. This result was not surprising given the well documented effect of passive movement on the activity of neurons in MI (Fetz et al., 1980; Suminski et al., 2009). However, this firing rate gain between VBMI and V+PBMI does not appear to be the source of the performance increase because the firing rate gain between V+PBMI and V+NBMI were not statistically different from one another despite a significant difference in BMI performance between these conditions. We cannot determine whether the decreased performance observed in V+NBMI resulted from a sensory integration strategy in which visual information was weighted more heavily than proprioception or whether the disparity in visual and proprioceptive feedback was simply distracting. We believe the former to be the case, given the similarity in performance in VBMI and V+NBMI, as well as the higher than expected correspondence between the cursor and hand direction in the V+NBMI condition.
Second, principal components analysis on the ensemble neural activity from each data set found a reduction in the average dimensionality of the space spanned by the spiking activity in V+PBMI compared with VBMI. This is inconsistent with the hypothesis that performance improvements may result from greater “flexibility” of the neural activity (i.e., the ability to use more degrees of freedom) when the restrictions on arm movement were relaxed during BMI control (V+PBMI). Instead, the increase in BMI performance in V+PBMI was accompanied by greater efficiency of the neural ensemble possibly resulting from a better estimate of the state of the system because of the integration of visual and proprioceptive feedback (Wolpert and Ghahramani, 2000). In support of this view, we found a strong relationship between BMI performance and the magnitude of the error between cursor and hand movements. As the degree of congruence between the visual and proprioceptive feedback increased, the monkeys' ability to command the BMI cursor was significantly improved. Similarly, the poorest BMI performance was found in VBMI and V+NBMI, when the discrepancy between cursor and hand movements was greatest.
Third, our mutual information analysis provides the strongest evidence supporting the hypothesis that the performance gains observed in V+PBMI were the result of congruent visual and proprioceptive feedback. The bimodal distribution of peak mutual information lag times for the V+PBMI condition suggests two distinct neural populations. The population with positive lag times exhibits a substantial increase in information during V+PBMI relative to VBMI. This enhancement is particularly evident at the positive—“motor”—lag times. These are the only relevant lag times for the neural decoder because they imply a causal relationship between neural modulation and cursor movement. Therefore, we believe that this population is primarily responsible for the enhanced BMI performance observed in the congruent condition.
However, the population with negative lag times exhibits a short-latency “sensory” peak during V+PBMI, which is not evident in the visual or incongruent feedback conditions. This population appears to be sensitive to limb state information via proprioception that is known to reach the cortex in as little as 10 ms (Conrad et al., 1975; Lemon et al., 1976; Evarts and Fromm, 1977; Fetz et al., 1980; Suminski et al., 2009). Although this population may not be directly responsible for the enhanced BMI performance in V+PBMI, it may be mediating the increase in motor information in the first population perhaps via horizontal connectivity between the two populations (Huntley and Jones, 1991).
This “sensory” information may contribute to improved BMI performance in two ways. First, the integration of visual and kinesthetic feedback may provide the monkey a more accurate estimate of the current state of the system. Second, the kinesthetic feedback generated by moving the arm was likely smoothed with respect to the visual feedback because of the dynamics of the arm/exoskeleton. This feedback smoothing may partially contribute to the improved performance that we report because of its effect on the activity of MI.
Application to clinical neuroprostheses
Our findings may have important practical implications for severely motor-disabled patients with residual proprioception. Incomplete spinal cord injury, such as anterior cord and central cord syndromes, results in severe loss of motor function but can leave a certain degree of proprioception intact. Moreover, other conditions such as ALS and locked-in syndrome have devastating motor consequences but may not affect proprioception. With the recent advent of light-weight wearable robotic exoskeletons (Kiguchi et al., 2005; Rocon et al., 2007), these patient populations may be served with a cortically controlled BMI that drives these devices and, in turn, moves the affected but proprioception-intact limbs, thereby augmenting these systems with proprioception.
More importantly, these findings provide a key foundation for additional research involving the integration of different forms of sensory feedback in BMI systems including bidirectional BMIs, in which surrogate feedback is provided via electrical stimulation. Direct stimulation of the primate somatosensory cortex via microelectrodes has been shown to elicit discernable sensory percepts for reach target cuing (Fitzsimmons et al., 2007; London et al., 2008). This paradigm offers the benefit of a bidirectional BMI to a larger group of patients who have lost both motor and sensory function because of complete spinal cord injuries or limb amputation.
Footnotes
-
This work was supported by funding from National Institutes of Health–National Institute of Neurological Disorders and Stroke Grant R01 N545853-01 (N.G.H., A.H.F.) and a postdoctoral fellowship form the Paralyzed Veterans of America Research Foundation (A.J.S.). We thank Josh Coles for his assistance with the experiments. We also thank Lee Miller for his assistance with the design of these experiments and his comments on a previous version of this manuscript.
- Correspondence should be addressed to Nicholas G. Hatsopoulos, Department of Organismal Biology and Anatomy, 1027 East 57th Street, Culver Hall 206, Chicago, IL 60637. nicho{at}uchicago.edu
