Abstract
A variety of studies have shown that motor cortical areas can be activated by observation of familiar actions. Here, we describe single-neuron responses in monkey primary motor (MI) and dorsal premotor (PMd) cortices during passive observation and execution of a familiar task. We show that the spiking modulation, preferred directions, and encoded information of cells in MI and PMd remain consistent during both observation and movement. Furthermore, we find that the presence of a visual target is necessary to elicit this congruent neural activity during observation. These findings along with results from our analysis of the oscillatory power in the beta frequency of the local field potential are consistent with previous imaging and EEG studies that have suggested that congruence between observation and action is a general feature of the motor system, even outside of canonical “mirror” areas. Such congruent activity has proposed relevance to motor learning, mimicry, and communication and has practical applications for the development of motor-cortical neuroprostheses in paralyzed patients.
Introduction
Research over the past two decades has established the existence of “mirror neurons” in monkey ventral premotor cortex (F5) that are activated during both movement and observation of familiar actions performed by other individuals (Gallese et al., 1996; Rizzolatti et al., 1996; Rizzolatti and Craighero, 2004). Importantly, in these cells, the stimuli that evoke neural responses during observation represent the same motor behaviors that the cells themselves are involved in generating. Studies in humans and primates strongly support the notion that this kind of congruence is a feature of observation-evoked motor activity in other motor cortical areas as well (Hari et al., 1998; Rizzolatti and Craighero, 2004; Binkofski and Buccino, 2006; Molnar-Szakacs et al., 2006). Although the phenomenon has been observed in ventral and more recently in dorsal premotor cortices (Cisek and Kalaska, 2004), no published study has examined the phenomenon in primary motor cortex (MI) at the single-cell level.
Research on the effects of action observation in MI has been dominated by human studies, in which gross changes in cortical activity during action observation have been assessed by functional magnetic resonance imaging (Cheng et al., 2007), changes in magnetoencephalogram and EEG power (Hari et al., 1998; Babiloni et al., 2002; Jarvelainen et al., 2004; Muthukumaraswamy and Johnson, 2004; Holmes et al., 2006), or by measuring the evoked responses to transcranial magnetic stimulation or medial nerve stimulation (Fadiga et al., 1995; Maeda et al., 2002; Borroni et al., 2005; Stefan et al., 2005). At the single-cell level, observation-related MI activity has only been used to set the initial parameters for motor decoding schemes (Hochberg et al., 2006; Wahnoun et al., 2006), but the origin, extent, and character of this activity has not been described previously.
Here we describe single-unit responses and local field potentials (LFPs) in MI as well as in the dorsal premotor cortex (PMd) during performance of a familiar task and subsequent observation of replay of the task. The task required the monkeys to guide a cursor using a two-link robotic manipulandum to a series of sequentially appearing targets projected onto a horizontal screen at chest level. In contrast to most previous studies of action observation, the monkeys in this study observed their own, as opposed to a conspecific or human, movements. Our task also had no imposed planning or “hold” period, allowing us to investigate the neural activity related to the generation and observation of self-paced, continuous movements. Our primary goal was to examine the activity elicited by unaltered replay of the task. As a secondary manipulation, to gain insight into the origin of the congruent activity we observed during replay, we also varied whether the target and/or cursor were visible on the screen during playback.
We examined peristimulus spiking around the appearance of the target, directional tuning, and the mutual information between neural firing and the cursor velocity. To reconcile these novel results at the single-cell level with the existing literature on gross neural activation by EEG, we also examined the oscillatory power in the beta frequency band of the LFP signal during performance and observation.
It should be noted that our experiment was not aimed at identifying typical mirror neurons but rather at studying the observation-evoked neural activity in MI on a single-cell level. We identified a population of neurons that exhibit congruence, a major characteristic of mirror neurons, but it is unknown whether the identified neurons would also respond to an interaction between a biological effecter and an object, a criterion necessary for their classification as mirror neurons.
Materials and Methods
Behavioral task
Three rhesus macaques (Macaca mulatta) were operantly trained to control a cursor in two dimensions using a two-link robotic exoskeleton [KINARM (for Kinesiological Instrument for Normal and Altered Reaching Movement); Dr. Stephen Scott, 1999]. The monkey's shoulder joint was abducted 90° such that all movements were made in the horizontal plane, and the cursor was projected onto a horizontal screen in front of the monkey and aligned with the location of the monkey's hand below the display. While his arm rested in the manipulandum, the monkey's view of his own limb was obscured by the projection screen. The shoulder and elbow joint kinematics were sampled at 500 Hz and then transformed to the cursor position using the forward kinematics equations.
The experimental task required the monkeys to repetitively move a cursor (6-mm-diameter circle) to a target (1 cm2). The target appeared at a random location within the workspace, and each time the monkey reached it, a new target appeared immediately in a new random location. To complete a successful trial and receive a juice reward, the monkey was required to sequentially acquire three to seven targets. Because each trial completion was followed by the immediate presentation of another target, the monkeys typically did not pause between trials but rather generated continuous movement trajectories (Fig. 1B). The trial was aborted if any movement between targets took longer than 5000 ms; normally this only happened if the animal was momentarily distracted from the task and removed his arm from the manipulandum.
Experimental apparatus and trajectories during movement and observation phases of the experiment. A, The monkey performs the random target pursuit task in the horizontal plane using a two-link exoskeletal robot. The monkey's arm is hidden from view by the workspace, and an infrared oculometer is positioned ∼36 inches away from the monkey. B, The paths generated by the cursor during replay of the monkey's own movements (blue trace) and by the monkey's hand during the same period when the monkey was observing the replay (red trace). C, The range of motion of the cursor (blue trace) and the monkey's hand (red trace) in the x dimension during observation. The same trend is observed in the y dimension (data not shown).
The experiment consisted of two phases: active movement and observation. In the active movement phase, the animal controlled the cursor via the manipulandum. In the observation phase, the target positions and cursor movements generated during the active movement phase were replayed while the animal maintained a static posture in the robotic exoskeleton. During the observation phase, if the monkey removed his arm from the manipulandum, a pressure sensor mounted in the arm trough was triggered and playback ceased until the monkey returned his arm. Playback was also terminated if the monkey moved the manipulandum outside an invisible hold region (1 cm2 radius) during the observation condition. A gentle elastic force was applied to the manipulandum to keep the animal's arm from drifting away from the required posture but was not applied within the hold region. Therefore, within the hold region, the small inherent friction of the exoskeletal robot served as the only resistance.
The observation phase of the task was divided into three conditions. In the first condition, both the cursor and the target were visible to the animal during playback, just as during performance of the task. This condition was the primary focus of our analysis. In the second condition, the cursor was invisible and only the replayed targets were visible to the monkey. In the third condition, the targets were not displayed, and only the moving cursor was visible. Conditions two and three were included to assess the contribution of different sources of visual information to the observation-related modulation in MI and PMd. The visual target represents the spatial goal of the movement, whereas the cursor is a proxy for the current position of animal's hand. During all observation conditions, the monkey received juice at the completion of every successful trial just as during performance of the task, even when the cursor or target was not visible.
In one monkey (monkey MK), we performed a variation on this experiment in which the monkey observed replay of the monkey's own movement as well as artificially generated cursor movements. In the latter case, the cursor moved at a constant velocity, generating linear paths between targets without pauses. In this variation of the experiment, both the cursor and target were visible at all times. Only data from MI were recorded during this experiment.
The eye movements of the same monkey (MK) were recorded during all experimental conditions with an infrared oculometer (R-HS-S6; Applied Science Laboratories, Bedford, MA) located 36–48 inches in front of the animal. The eye position was calibrated using the ASL EyeTracker Software Suite while the monkey performed a target pursuit task with targets placed at standardized positions that spanned the rectangular workspace. Calibration proceeded iteratively until the tracking of the eye was deemed acceptable.
Neural recordings
Monkeys were chronically implanted with 100-electrode (400 μm interelectrode separation) microelectrode arrays (Cyberkinetics Neurotechnology Systems, Foxborough, MA) in MI and PMd contralateral to the arm used for the task. Monkey RS had an array implanted in MI, monkey RX had an array in PMd, and monkey MK had arrays in both areas. All electrodes were 1 mm in length. During a recording session, signals from up to 96 electrodes were amplified (gain of 5000), bandpass filtered between 0.3 Hz and 7.5 kHz, and recorded digitally (14-bit) at 30 kHz per channel using a Cerebus acquisition system (Cyberkinetics Neurotechnology Systems). Only waveforms (1.6 ms in duration; 48 sample time points per waveform) that crossed a threshold were stored and spike sorted using Offline Sorter (Plexon, Dallas, TX). Signal-to-noise ratios were defined as the difference in mean peak-to-trough voltage divided by twice the mean SD. The mean SD was computed by measuring the SD of the spike waveform over all acquired spikes at each of the 48 sample time points of the waveform and then averaging. All isolated single units used in this study possessed signal-to-noise ratios of three or higher. A total of 18 datasets (12 datasets for animal MK, three datasets for animal RS, and three datasets for animal RX) were analyzed. A dataset is defined as the simultaneously recorded neural activity during one recording session and contained between 300 and 800 trials.
LFP signals were recorded at a 1 kHz sampling rate per channel and amplified using a gain of 5000. The acquired signals were then bandpass filtered from 0.30 to 250 Hz or 0.30 to 500 Hz using the Cerebus data acquisition system.
Analysis
Kinematics.
For most analyses, kinematics were binned in 50 ms bins and boxcar smoothed using a 150 ms sliding window. In calculating the correlation between residual hand movements and the cursor position, raw kinematics at 2 ms resolution were used.
Average peristimulus (target hit) time histograms.
Neural spikes were binned in 5 ms bins, and spike counts were averaged across all target hits within each experimental condition to produce mean peri-target-hit firing profiles from 200 ms preceding each target hit to 500 ms after. We compared the shape and the relative magnitude (ratio of difference between the maximum and minimum mean firing rate after the target hit to the SD of neural activity before target hit) of the profiles generated during the three observation conditions to the profile generated during active movement for each cell.
Mutual information profiles.
Mutual information between binned neural data and kinematics (50 ms bins) was calculated at multiple time leads and lags as in the study by Paninski et al. (2004). The conditional kinematic distributions (one-dimensional distribution of instantaneous directions or two-dimensional distribution of Cartesian hand velocities) conditioned on the number of observed spikes were estimated by histograms of the empirical data. To account for biases in this estimation, the information calculated from shuffled bins (mean of 10 shuffles) was subtracted from the values obtained from the actual data for each cell. Lead/lag profiles were boxcar smoothed with a three bin window.
Directional tuning.
Preferred directions (PDs) were determined for each experimental condition by calculating the mean binned spike count (50 ms bins) as a function of instantaneous movement direction (at π/8 radian resolution). The lag between neural activity and direction for each cell was chosen based on the lag of the peak mutual information (see above, Mutual information profiles). The mean spike counts per direction were fit with a cosine function (Georgopoulos et al., 1982). Cells were considered to be cosine tuned if the correlation between the empirical mean spike counts and the best-fit cosine function was >0.5.
For each cell, the preferred direction was calculated in 50 s consecutive segments within each condition. If the variance in the PD across segments was smaller than 0.5 radians, the cells were deemed to be stably tuned within a condition. Only cells that were both cosine and stably tuned were used in assessing the difference in preferred directions between experimental conditions; the only observation condition in which a significant number of cells met these criteria was the condition when both the target and cursor were visible.
LFP analysis.
A small number of representative channels were selected, and the raw LFPs for each channel were concatenated across blocks of the same condition to produce a channels × conditions × samples matrix; the latter dimension was truncated at the length of the shortest condition. The LFP signal was bandpass filtered (backward and forward to avoid phase distortion) between 1 and 55 Hz using an eighth-order Butterworth filter and was divided into 20 segments within each condition. Next, for each segment, the multi-taper spectrum was calculated in the same frequency band (Chronux neural analysis software; www.chronux.org) and the result was boxcar smoothed using a window length
Eye position.
The analog outputs (two Cartesian dimensions) of the eye-tracking device were digitized at 1 kHz, time-stamped with respect to the neural and kinematic data, and saved to disk. These values were regressed against the x and y target positions in the active movement condition at a lag of 100 ms (which was determined to be optimal) to determine the location of gaze in the same coordinate frame as the cursor and target. At a lag of 100 ms, correlation values between eye position and target position at the time of target acquisition were 0.93 during active movement and 0.86 during observation when the cursor and target was present. The peak correlations between eye position and cursor position during active movement and during the observation condition when the cursor and target was present were 0.81 and 0.72, respectively.
Partial cross-correlation matrices.
In this analysis, we followed the method described by Stark et al. (2006) with the exception that we segmented our continuous data into relatively short epochs of equal lengths rather than organizing it by trials. Like the LFP data, binned time series representing instantaneous cursor and eye directions were concatenated across blocks of each condition and truncated to the length of the shortest condition. These resulting vectors were segmented to produce a number of epochs of equal lengths (5 s). For each segment, partial correlations were computed between binned spike counts (5 ms bins; convolved with a Gaussian kernel of width, 11 bins; and σ = 3 bins) and the cosine of instantaneous cursor and eye directions (instantaneous directions computed as for directional tuning, above). Performing the partial correlation calculation at every possible lead/lag combination (±500 ms, 5ms resolution) produced a matrix of partial cross-correlation values with two dimensions representing different leads and lags for eye and cursor direction. These matrices were Fisher z-transformed and averaged across segments within each unit and then across units (within condition) to produce each of the panels in Figure 6. For each element in the partial cross-correlation matrix (PCCM), a one-tailed t test was performed on the distribution of partial correlation values across cells to determine whether, over all cells, the mean partial correlation value at that combination of lags was significantly greater for the cosine of eye or cursor direction, or neither.
Results
Modulation in neural spiking activity
After each target acquisition in the random target pursuit (RTP) task, a new target appears, the monkey's hand speed dips transiently as he shifts or reverses direction, and the animal initiates a movement to the new target. The latency of this movement initiation in the RTP task is similar to the monkey's natural reaction time measured in other behavioral paradigms, suggesting that the appearance of a new target immediately sets in motion a process of sensorimotor integration leading to the generation of a new movement. During this ∼300 ms period after target appearance, we found that the spike rates of many cells in motor and premotor cortex modulated conspicuously, and so we chose to focus on the period surrounding the appearance of the new target in our comparison of the active movement and observation conditions. For each cell, we assessed the similarity in modulation across conditions by examining both the correlation and relative magnitude of the PSTHs (target hit) for each condition, shown in Figure 2.
Mean peristimulus time histograms of neural activity aligned on target appearance. A, Examples of neural modulation in nine cells during a 700 ms time window aligned on the time of the target appearance (0 ms). Each panel plots the average firing rate in time of one neuron with respect to the appearance of the target during active movement (dark blue trace), target and cursor observation condition (orange trace), target-only observation condition (red trace), and cursor-only observation condition (cyan trace). B, top, shows the distribution of depth of modulation over all recorded neurons per condition measured as the ratio of the difference between maximum and minimum post-target hit firing rate to the SD of neural activity before target hit. MI population data for monkeys MK and RS are shown in dark and light orange colors, respectively. PMd population data for monkeys MK and RX are shown in light blue and dark blue colors, respectively. The median of each distribution is marked by the notch and a horizontal line in each vertical bar. The lower and upper quartiles of each distribution are marked by the bottom and top of each vertical bar, respectively. The vertical dashed lines above and below the vertical bars represent the full range of the data. B, bottom, shows the similarity in neural activity as measured by correlation coefficient of the modulation profiles during active movement with modulation profiles during the three observation conditions.
The PSTHs during active movement for many cells in both MI and PMd were characterized by a transient increase or decrease in mean firing rate after each target hit (Fig. 2A, blue traces for nine different cells). During the first observation condition when both target and cursor were present, cells showed peri-target-hit activity (Fig. 2A, orange traces) that was often very similar to that during the active movement condition (orange vs blue traces). A weaker degree of similarity in the profile and strength of modulation persisted for a subset of cells during the second observation condition, when just the target was present (Fig. 2A, red trace). In the third condition, when only the cursor was present, virtually no target-hit-related modulation occurred, as would be expected because the target itself was not visible (Fig. 2A, cyan trace). Across the population of recorded neurons in each cortical area, we computed the median, upper, and lower quartiles of the depth of modulation after target hits for the active movement phase and for each of the three passive observation conditions (Fig. 2B, top). In addition, the similarity in the peri-target modulation profile evoked during active movement and each of the passive observation conditions for each neuron was assessed by computing the correlation between the profiles. The median correlation coefficient values across the population of recorded neurons indicate that there was the strongest similarity in modulation profiles between the active movement phase and the passive observation phase when both target and cursor were visible (Fig. 2B, bottom).
Information in neural spiking activity
To assess whether the neural activity we observed during the observation phase contained any movement-related information, we calculated the mutual information between the instantaneous binned firing rate of each cell and the binned velocity of the cursor at multiple time leads and lags (Fig. 3A,B). These mutual information profiles can be read like cross-correlations, with each point representing the result of the calculation when the two variables are shifted with respect to each other by a different time lag. However, mutual information estimates have the advantage over methods that assess linear correlations between neural activity and movement parameters because mutual information can also capture nonlinear relationships. By examining the relative timing of peak mutual information, we are able to determine at what time lag the modulation of a neuron is most related to the cursor movement. For example, if the neural activity is most related to the movement of the hand 100 ms in the future, then we interpret this activity as a motor, feedforward signal, “driving” the cursor's movement. Conversely, if the activity of the neuron provides the most information about the cursor movement in the past, we interpret this as either a delayed motor response or a sensory, feedback signal. For example, in one cell (Fig. 3A, top left), the neural activity contained the most information (0.24 bits) about the cursor velocity ∼150 ms in the future in the active movement phase. In other words, the firing rate of the cell measured at time t was most strongly related to the cursor (and hand) velocity occurring t + ∼150 ms later. This temporal relationship is typical of motor cortical neurons that “drive” movement (Moran and Schwartz, 1999; Paninski et al., 2004). However, this cell exhibited very little movement-related information during any of the three passive observation conditions. In contrast, another cell (Fig. 3A, bottom left) showed relatively strong, movement-related information during the observation phase when either both target and cursor were visible or when only target was visible. Interestingly, there were a few cases in which there was more information about the cursor velocity during passive observation (target and cursor visible) than during active movement (Fig. 3A, top right).
Mutual information between cursor velocity and neural activity across experimental conditions. A, Four single-cell examples of mutual information temporal profiles (in bits) as a function of lag time between cursor velocity and neural activity for four cells. Each panel plots the mutual information between the firing rate of a single neuron and the cursor velocity over different relative times between the two. A positive lag time denotes that the neural activity was measured before the cursor velocity, whereas a negative lag time denotes that the neural activity was measured after the cursor velocity. A zero lag time denotes that the neural activity and cursor velocity were measured simultaneously. Mutual information profiles are plotted during active movement (dark blue trace), observation of target and cursor (orange trace), observation of target only (red trace), and observation of cursor only (cyan trace). B, Scatter plot of peak mutual information values (in bits) for all neurons. Monkeys MK, RS, and RX are represented by cross, square, and circle markers, respectively. Different shades of color represent different datasets for each monkey. Data points exceeding a ceiling of 0.1 and 0.05 bits for MI and PMd, respectively, are plotted at those values. C, Mutual information profiles for two cells between cursor velocity and neural activity during active movement, observation of naturalistic movement, and observation of artificial movement. Mutual information profiles are plotted during active movement (dark blue trace), during observation of naturalistic movement (both target and cursor visible; orange trace), and during observation of artificial movement (both target and cursor visible; olive trace). D, Scatter plots of peak mutual information values (in bits) carried by MI neurons during active movement (dark blue), observation of naturalistic movement (orange), and observation of artificial movement (olive) for monkey MK. E, Distribution of peak mutual information time lags during active movement (dark blue), observation of naturalistic movement (orange), and observation of artificial movement (olive). Positive lag implies that neural activity precedes cursor velocity.
Across the population of recorded neurons, we found that the information about cursor velocity followed a similar pattern to the peri-target-hit modulation across conditions. The peak information provided by most cells about cursor velocity was highest during the active movement condition and displayed the same relative order in magnitude for the three observation conditions (Fig. 3B). We observed a similar trend in information magnitudes in both MI and PMd cortical areas.
We also sought to determine whether movement-related modulation was present when the cursor moved to targets in a completely unnatural manner. In particular, the cursor moved in a perfectly straight line to each target at a constant speed that was slower than the peak speeds attained when the cursor was moved by the animal. Surprisingly, the amount of information provided by neurons about cursor velocity during observation of artificial movement was similar to the amount provided during observation of naturalistic movement (Fig. 3D). However, we did observe a difference in the lag at which information carried by neural activity peaked with respect to cursor velocity in the two conditions (Fig. 3C,E). During active movement, cells carried maximum information about the cursor velocities occurring an average ± SE of 64.6 ± 12.8 ms in the future. During observation of naturalistic movement, neural activity provided the most information about cursor velocities taking place an average ± SE of 22.5 ± 24.5 ms in the future. However, during observation of artificial movement, neural activity carried the most information about cursor velocities that happened in the past, at a mean ± SE lag of −166 ± 20.0 ms. This indicated that, on average, neurons exhibited cursor-related modulation that followed or lagged the motion of the cursor. The distributions of mean peak information times during active movement and observation of naturalistic movement were not statistically different from each other (p = 0.13, paired t test). In contrast, the distributions of mean peak information times during active movement and observation of artificial movement were statistically different from each other (p = 4.1 × 10−18, paired t test), and the distributions of mean peak information times during observation of naturalistic movement and observation of artificial movement were also statistically different from each other (p = 1.4 × 10−8, paired t test).
Directional tuning
Mutual information offers a measure of the strength of the dependence of neural activity on kinematic parameters. To gain insight into the degree to which the directional tuning of the cells remained consistent across the active movement and observation conditions, we calculated the preferred direction of each cell in each condition (Fig. 4A) (see Materials and Methods). We observed the largest number of neurons that exhibited stable preferred directions and that were well fit to a cosine function in the active movement phase (Fig. 4B, top). We also assessed the depth of modulation of these neurons as the difference between the preferred and anti-preferred direction activity relative to the baseline (Fig. 4B, bottom). These values decreased monotonically across the three observation conditions. We compared the preferred directions of neurons that exhibited stable and significant cosine tuning in both the active movement phase and the visible target and cursor observation condition (70% of MI neurons, 581 of 829; 60% of PMd neurons, 77 of 128). We found that the distributions of preferred directions during active movement were not statistically different from the distributions of preferred directions during observation in both cortical areas (p = 0.08 for distributions in MI and p = 0.58 for distributions in PMd, Kuiper test). The mean ± SE preferred direction difference was −14.89° ± 2.37° in MI and 0.17° ± 5.49° in PMd (Fig. 4C).
Similarity in preferred directions of cells during active movement and passive observation of movement. A, Two single-cell examples of cosine function fits of the mean firing rates (50 ms bin size) as a function of instantaneous movement direction (binned in 16 possible directions) during observation when both cursor and target are visible. B, Top, Scatter plots of the strength of cosine directional tuning measured by the correlation coefficient between the mean firing rates per direction and the best-fit cosine function over all neurons in MI (left) and PMd (right). Bottom, Scatter plot of the depth of modulation with respect to direction is shown as the ratio of the difference between the maximum and minimum values of the fit function to the mean of the fit function over all neurons in MI (left) and PMd (right). Monkeys MK, RS, and RX are represented by cross, square, and circle markers, respectively. The same symbols but in different shades of color represent different datasets for the same monkey. Data points exceeding 140% limit are plotted at that value. C shows the distributions of differences in preferred directions between active movement and the target and cursor observation condition for MI and PMd cortical areas.
Local field potentials
Based on the observation that power in the beta range of the EEG signal is significantly attenuated during action execution, a number of human EEG studies have used beta-range power as a neurophysiological index of activation within the motor system (Babiloni et al., 2002; Muthukumaraswamy and Johnson, 2004). To replicate this result, we chose a subset of representative channels from each electrode array and computed the power in the 10–25 Hz (beta) band during active movement as well as under the three observation conditions. Our results are consistent with those from the human EEG studies; beta power in both MI and PMd in all three monkeys was smallest during active movement and significantly greater during the three observation conditions (Fig. 5).
Beta frequency range power spectra of the local field potentials recorded from MI and PMd during active movement and passive observation of movement. A, Mean ± SE power in the 10–25 Hz range of the LFP averaged over multiple segments and channels during active movement (dark blue trace), observation of target and cursor (orange trace), observation of target only (red trace), and observation of cursor only (cyan trace). B, Integrated mean power (integrated between 10 and 25 Hz) as a function of experimental condition. Integrated power during each observation condition is statistically different from integrated power during active movement condition for monkeys MK and RX. For monkey RS, integrated power during observation of target only and observation of cursor only is statistically different from integrated power during active movement. MI data shown for monkeys RS and MK, and PMd data shown for monkey RX.
Eye position
To investigate the possibility that motor cortical modulation during the observation phase might be attributable to eye movements, we recorded gaze direction in monkey MK using an infrared eye tracker. We found that, especially in the cursor and target and target-only observation conditions, eye movements were highly correlated with the cursor movements, raising the possibility that the apparent correlation between neural activity and cursor movement we saw during observation was an indirect consequence of the correlation between neural activity and eye movements. To assess the degree to which the observed neural activity was independently related to eye or cursor movement, we followed the method described by Stark et al. (2006). They rely on the method of partial correlations (Fisher, 1970) to assess the residual correlation between two variables when linear correlations with a third variable have been removed, and they expand the method to take into account multiple temporal leads and lags between two variables. Calculating partial correlations between neural activity and kinematic parameters at multiple leads and lags results in a PCCM (Stark et al., 2006). Figure 6, A and B, shows the mean PCCM for eye direction and cursor direction for each of the four experimental conditions averaged over all cells. The z-axis (color bar) represents the mean (across cells) absolute value of the Fisher z-transformed partial correlation between neural activity and the cosine of cursor direction (PCcos(cursor dir)) (Fig. 6A) and the same value with respect to eye direction (PCcos(eye dir)) (Fig. 6B). In each panel, a pixel is the mean value calculated at a particular combination of leads and/or lags (±500 ms) between the neural activity and the two kinematic variables. The lag between neural activity and cursor direction is the same for pixels in the same row, and the lag between neural activity and eye direction is uniform within a column.
Partial cross-correlation matrices of eye and cursor direction for monkey MK. A, Color axis is absolute value of z-transformed partial correlations between neural activity and the cosine of cursor movement direction, with the partial correlation between cursor direction and eye movement direction removed. x-Axis represents the lag time at which the partial correlation between neural activity and eye direction is calculated. y-Axis represents the lag time at which the partial correlation between neural activity and cursor direction is calculated. Positive time values indicate that neural activity precedes movement. Each of four panels represents a different experimental condition. All panels are averages over all cells. B, Same as in A, but color axis now represents the absolute value of z-transformed partial correlations between neural activity and the cosine of eye movement direction. C, Results of performing a single-tailed t test of z-transformed PC values over all cells for each pixel in A and the corresponding pixel in B; yellow, pixel in A is significantly higher than pixel in B; light blue, pixel in B is significantly higher than pixel in A; dark blue, no significant difference. Neural activity is correlated primarily with cursor movement in the first two conditions, but a significant partial correlation with eye movement is revealed in the last two conditions and is especially prominent when only the cursor is present.
The prominent horizontal bar in the active movement panel in Figure 6A indicates that, when linear correlations with the eye direction are removed, neural firing remains strongly correlated with the direction of cursor movement ∼100 ms in the future, as has been described previously (Moran and Schwartz, 1999). In contrast, during active movement, there is virtually no independent relationship between neural activity and eye direction (Fig. 6B, first panel).
In the active movement phase and the cursor-and-target observation condition, PCcos(cursor dir) is significantly higher than PCcos(eye dir) at every combination of leads and lags (one-tailed t test comparing the distributions of the two PC values across cells for each pixel, p > 0.05). In Figure 6C, each pixel is colored according to the outcome of the one-tailed t test (see figure legend). Interestingly, in the target-only condition, neural activity is weakly partially correlated with both cursor direction (horizontal band) and with eye direction (vertical band). Over the population of cells, PCcos(cursor dir) is significantly higher than PCcos(eye dir) at large negative and positive lags in the x-axis, and the opposite is true in the y-axis; near the origin, they are both high and thus there is no significant difference between them. Finally, the fourth panel in Figure 6C shows that, in the cursor-only condition, the partial correlation between the neural activity and the eye dominates.
Residual movements
Although all the monkeys were trained to keep their arms relatively motionless during the observation phase (Fig. 1B,C), it was still possible for the animals to make small, residual movements within the hold region. We examined whether these slight movements might be correlated with cursor movement during the observation phase. We found significant correlations between these residual velocities and the cursor movement for two of the three animals but no significant correlation in the third animal. To rule out the possibility that this residual movement could explain our findings during the observation phase, we reanalyzed one dataset for each monkey by segregating the residual kinematics during observation into bins with high-velocity and low-velocity movements (top and bottom quartiles with respect to speed). Choosing bins with high- or low-speed residual movement (Fig. 7A) generated substantially different correlations between the hand and cursor direction (Fig. 7B) but did not result in a significant change in the distribution of mutual information values between neural activity and cursor direction (Fig. 7C) (Wilcoxon's rank sum test, p = 0.89). Mutual information values in both cases were taken at zero lag to avoid lagging the data over nonconsecutive bins.
Influence of residual hand movement on encoded information during action observation. A, Circular correlation between residual hand movement direction and cursor movement direction during the target and cursor, target-only, and cursor-only observation conditions. Correlations were calculated for bins with high-velocity movements (top quartile; red) and low-velocity movements (bottom quartile; blue). Monkeys MK, RS, and RX are represented by cross, square, and circle markers, respectively. Correlation coefficients that are statistically different from zero are represented by filled-in markers. For monkey MK, all high-velocity cross-correlation values are statistically greater than zero, whereas all low-velocity cross-correlation values are not. B, Mean speed of residual movements (centimeters per second) during the target and cursor, target-only, and cursor-only observations conditions for high- and low-velocity quartiles. C, Peak mutual information values (in bits) between neural firing rate and cursor direction for all MI neurons calculated from high-speed bins (red markers) and low-speed bins (blue markers). There was no significant difference between the two distributions (Wilcoxon's rank sum test, p = 0.89). Monkeys MK, RS, and RX are represented by cross, square, and circle markers, respectively.
Discussion
We have shown that the single-unit activity within both MI and PMd is similar during passive observation of a familiar task and during execution of the same task. As we described, we find that motor cortical activity remains correlated (albeit more weakly) with the cursor movement during observation, although the monkey no longer moves his arm or controls the cursor. This result could be interpreted in a number of ways. One might imagine that this continued correlation between the neural activity and the cursor movement could be attributable to incoming sensory information about the location of the cursor. However, in that case, observation of the cursor itself would be sufficient to elicit the observed neural activity, which we do not observe. Another possibility is that the activity in the motor cortex reflects sensory information about the location of the target and not the cursor. However, in that case, the additional presence of the cursor should not have any effect on the phenomenon (except perhaps as a distraction), which again is inconsistent with our findings. A third possibility, derived from the existing literature comparing the relative motor activation produced by transitive (i.e., goal-directed) versus intransitive behaviors, is that the neural activity during observation is attributable to the covert generation of a motor command and that we observe congruent neural activity during observation because the visual goal, and thus the motor command generated, is the same as during active movement (Fig. 8).
Three possible interpretations of the results. In this highly simplified schematic, during active movement, the monkey integrates sensory information about the position of the target and cursor and generates a motor command, which leads to cursor movement. During observation, we observe congruent neural activity in MI and PMd, although the monkey's intentions to move no longer have a causal relationship with respect to the stimuli. We consider three possible explanations for the congruent neural activity we describe during observation. The first hypothesis (H1) is that the neural activity we observe can be attributed to visual feedback about the location of the cursor. The second hypothesis (H2) is that the neural activity can be directly attributed to visual information about the target. The third hypothesis (H3) is that congruence in neural activity is attributable to the covert generation of a motor plan in response to the visual stimuli.
Our results support this latter interpretation. When both the cursor and target are present, the monkey generates a motor command that is most congruent with the observed cursor movement because the origin and goal of the movement are both known with certainty. When the cursor is invisible, only the goal of the movement is known for certain. Even if the position of the previous target serves as the origin for the internal movement command, this command will most likely be incongruent with one that would be generated from the actual cursor position. This is attributable to the fact that, during execution, the animal often overshoots the target and initiates a new movement from a location other than the previous target. Finally, when only the cursor is present, the monkey is unable to generate a motor command that is congruent with the cursor trajectory because the goal of the movement is not known.
The mutual information analysis supports this interpretation. In the observation phase (with target and cursor present), the peak mutual information between neural modulation and cursor motion occurred at an average lead time of 23 ms (i.e., neural modulation preceded cursor movement), which was not significantly different from the lead time in the active movement phase. This suggests that neural modulation is acting in feedforward driving mode during passive observation.
Our interpretation is further supported by the comparison between replay of the monkey's own cursor movements and artificially generated cursor movements of constant velocity with no pause between targets. We found that the strength of the relationship between the neural activity and the cursor movements was similar in both cases, but the timing of the congruence between action and observation was shifted in the artificial movement condition. We believe this is because the absence of a reaction time after target appearance in the artificial movement condition means that the cursor movement leads the monkey's motor command. In other words, by the time the relevant neural activity occurs, the cursor is already well on its way to the next target, and this latency is reflected in the information profiles.
One difference between our experiments and much of the existing work on action observation is that here the monkeys observed the movement of a cursor in an abstract workspace rather than a more naturalistic stimulus (reaching and grasping by a conspecific or human, for example). However, because of their extensive training, we assume that the monkeys have learned to view the cursor as a reliable surrogate for hand position. Studies in humans have shown that motor cortical responses to observations of tool use, for example, depend on existing knowledge of the mapping between intention and action, and the same sort of familiarity is no doubt required in our study (Jarvelainen et al., 2004; Ferrari et al., 2005).
If our interpretation of this phenomenon is correct and the monkeys are generating covert motor commands during observation that are congruent with the commands generated during the behavior itself, a natural question is how the dissociation between motor cortical modulation and action occurs. One possibility is that the motor cortical activity we observe is being actively gated by other cortical areas. Results from human EEG studies along with our LFP results suggest that an increase in power in the beta range is associated with inhibition of the excitatory state of the motor cortex (Gilbertson et al., 2005). There is some clinical evidence regarding the origin of this inhibition in patients with frontal lobe damage that exhibit “unwilled” automatic movements (Archibald et al., 2001). These clinical studies suggest that the prefrontal, anterior cingulate, and supplementary motor cortices may contribute the necessary inhibition to prevent triggering of movement commands realized in activated motor and premotor cortical areas. Another possibility is that the motor cortex is part of a more distributed network responsible for movement. Therefore, motor cortical activity alone may not be sufficient to elicit action. Without knowing more about the functional roles of the cells from which we are recording, it is difficult to say anything further about the mechanisms intervening between stimulus and response during the observation phase of the experiment.
Our results do, however, suggest that there is a continuum between inhibition and activation in the motor system. Although the monkeys were trained to voluntarily maintain a still posture during the observation phase of the experiment, the manipulandum permitted us to record with high resolution any residual movements within the small hold region. In two of the animals, we found that these minute movements were weakly but significantly correlated with the observed cursor position, especially during the two conditions in which the target was visible. However, in our study, we consistently found the strongest relationship between neural activity and cursor movement in the one monkey that moved the least; in this monkey, there was no significant correlation between residual hand velocities and cursor velocities in any observation condition. Furthermore, in a study examining single-cell activity in dorsal premotor cortex during mental rehearsal, the authors found no relationship between muscle activity (measured via EMG) and neural activity (Cisek and Kalaska, 2004). Finally, as shown in Figure 7C, the strength of the relationship between neural modulation and observed cursor direction measured by mutual information was not significantly different when we compared the bins when the monkey moved the most and when he moved the least, whereas the correlation between residual hand movement and cursor movement was substantially different in the two cases.
In all three monkeys, we observed an increase in beta power during the observation phase of the experiment, with a relative increase in power during the cursor-only condition that mirrored the decrease in single-unit activity during the same period. Although the functional role of beta oscillations remains controversial, beta-range power in human studies has been used as an index of motor cortical activation. Our results are consistent with this literature and suggest that average beta activity is, in fact, a reliable metric for single-cell activation.
We considered the possibility that some of the neural modulation we observed during the observation phase was attributable to the monkey's eye movements. If this were the case, the correlation between the neural activity and the cursor movement during observation could be a secondary consequence of the relatively high correlation between eye and cursor movement. The results of the partial cross-correlation analysis indicate that is not the case; in the first observation condition, once the correlation between the hand and eye was removed, there was virtually no residual relationship between the neural activity and the eye movements. Interestingly, in the cursor-only condition, the neural activity was more directly related to the direction of eye movements than to the direction of cursor movement, suggesting that eye-command-related activity in the other two observation conditions may be masked by the generation of motor commands that are congruent with the cursor movement.
Despite the persistent trends that we have described here, we observed a high level of variation in the strength of the phenomenon across recording days. Because the observation phase of the experiment did not require any overt behavior from the monkey, much of this variation may be attributed to the motivational state of the animal. Motivational variation across days would have a global, cross-conditional effect on our results as opposed to a condition-specific one. Furthermore, on some days, a monkey's motivation to perform the task may dwindle as the experiment progresses and the animal becomes satiated. We avoid this confound using a randomized block design in our experiment.
We have shown that observation of a familiar task elicits covert motor commands in MI and PMd that are congruent with the commands the monkey would generate to actually control the cursor. We conclude that this kind of congruent motor activity can be elicited by observation of a familiar task in both the absence of overt movement and, as anyone who has caught themselves leaning in the opposite direction of a misguided bowling ball already knows, in the absence of an actual link between intention and action. One practical implication of these results is in the development of neural prosthetics for paralyzed individuals, because observed and imagined movements are the only sources of information for generating a mapping between neural activity and behavior. Indeed, our interest in this phenomenon was prompted by our ability to generate a viable mapping between neural activity and cursor position in a monkey performing the RTP task simply from having the monkey observe the task being played by a human experimenter. The neural activity during this observation was sufficient to generate a mapping that allowed the monkey to successfully control the cursor (data not shown). The phenomenon that we described intersects with a number of high-level neural functions such as intention, attention, and sensorimotor integration that are difficult to study or assess in primates. The recent implantation of a motor prosthesis in a human patient (Hochberg et al., 2006) may make it possible to further investigate these aspects of the phenomenon at the single-neuron level.
Footnotes
-
N. G. Hatsopoulos has stock ownership in a company (Cyberkinetics Neurotechnology Systems) that fabricates and sells the multielectrode arrays and acquisition system used in this study.
-
The work is supported by National Institutes of Health/National Institute of Neurological Disorders and Stroke Grant R01 N545853-01. We thank Dawn Paulsen and Aaron Suminski for their assistance with the experiments and data processing.
- Correspondence should be addressed to Nicholas G. Hatsopoulos, University of Chicago, 1027 East 57th Street, Room 202, Chicago, IL 60637. nicho{at}uchicago.edu
