Abstract
Decisions are often made by accumulating evidence for and against the alternatives. The momentary evidence represented by sensory neurons is accumulated by downstream structures to form a decision variable, linking the evolving decision to the formation of a motor plan. When decisions are communicated by eye movements, neurons in the lateral intraparietal area (LIP) represent the accumulation of evidence bearing on the potential targets for saccades. We now show that reach-related neurons from the medial intraparietal area (MIP) exhibit a gradual modulation of their firing rates consistent with the representation of an evolving decision variable. When decisions were communicated by saccades instead of reaches, decision-related activity was attenuated in MIP, whereas LIP neurons were active while monkeys communicated decisions by saccades or reaches. Thus, for decisions communicated by a hand movement, a parallel flow of sensory information is directed to parietal areas MIP and LIP during decision formation.
Introduction
Many decisions benefit from the accumulation of evidence over time and thus display features resembling deliberation and reasoning. This is true even for simple perceptual decisions, which can be studied in animal models. In monkeys trained to discriminate the direction of motion in a dynamic random-dot display, neural correlates of evidence accumulation (Shadlen and Newsome, 1996; Huk and Shadlen, 2005), value (Platt and Glimcher, 1999; Rorie et al., 2010), probability (Yang and Shadlen, 2007), and confidence (Kiani and Shadlen, 2009) have been identified in the firing rates of neurons in the parietal and prefrontal association cortex and in subcortical structures associated with the preparation of eye movements (Horwitz and Newsome, 1999). In these experiments, monkeys are typically trained to communicate their decisions with an eye movement to a choice target. Thus, it is not surprising that such neurons would indicate the outcome of a decision. However, the more remarkable observation is that their graded firing rates represent the gradual formation of a decision—the transformation of momentary evidence from vision to a categorical choice conveyed by the motor system.
To humans, perceptual decisions feel like they are about propositions (e.g., leftward or rightward) and not about the plan of action used to communicate the choice, yet our decisions are governed by an evidence accumulation process similar to the one studied in monkeys (Smith and Ratcliff, 2004; Palmer et al., 2005; Philiastides et al., 2006; Green et al., 2012; O'Connell et al., 2012). Moreover, the accumulating sensory evidence that will ultimately support one or the other choice—an evolving decision variable—has been shown to flow continuously to motor structures in the human brain much as it does in monkeys (McKinstry et al., 2008; Selen et al., 2012; Ding and Gold, 2013). Because decisions can be communicated in a variety of ways, it seems likely that many brain areas could play a role in accumulating evidence toward a choice. This raises the question of whether each area is recruited selectively based on the required action or whether all potential action circuits are recruited in parallel.
To begin to address this question, we trained rhesus monkeys (Macaca mulatta) to indicate perceptual decisions with either a hand or an eye movement while recording from neurons in the medial or lateral intraparietal areas (MIP and LIP, respectively). Just as LIP neurons signal the location of objects that are potential objects of saccadic eye movements (Thier and Andersen, 1998), MIP neurons signal the spatial location of objects that are potential targets of reach movements (Mountcastle et al., 1975; Johnson et al., 1996; Snyder et al., 1997; Batista et al., 1999; Eskandar and Assad, 1999), as well as the outcome of decisions to direct the arm in a particular direction (Snyder et al., 1997; Eskandar and Assad, 1999; Scherberger and Andersen, 2007; Pesaran et al., 2008). Therefore, we hypothesized that neurons in MIP would represent the accumulation of evidence bearing on a decision indicated by a hand movement. We sought to test this and to determine whether such decision-related activity arises selectively in MIP and LIP, depending on the effector system used to communicate the decision. We found that MIP neurons represent the accumulated sensory evidence in support of a decision when the monkey indicated its choice with a hand movement. Such decision-related activity was greatly attenuated in MIP when the monkey used saccades to communicate choices. Interestingly, on hand-movement trials, both LIP and MIP neurons represented an evolving decision variable, consistent with parallel decision mechanisms in the parietal cortex.
Materials and Methods
Two adult male rhesus monkeys (M. mulatta) participated in this study. Both animals underwent extensive training on the random-dot motion task. Monkeys were trained first in the saccade condition, then in the reach condition, and finally in alternating blocks of saccadic and reach movements. Experimental control and stimulus presentation were achieved with either REX (Hays et al., 1982) in combination with the Psychophysics toolbox (Brainard, 1997) or with EXPO (developed by Peter Lennie, currently maintained by Rod Dotson, Center for Neural Science at New York University, New York, NY; https://sites.google.com/a/nyu.edu/expo/). All procedures conformed to the National Institutes of Health Guide for the Care and Use of Laboratory Animals and were approved by the University of Washington Institutional Animal Care and Use Committee.
Behavioral testing.
Monkeys were seated in front of a video screen (LCD) equipped with an infrared touch-sensitive frame (viewing distance, 33 cm; ELO Touch Systems; Tyco Electronics). Eye position was monitored by a video tracking system (Eyelink 1000; SR Research). Monkeys initiated trials by fixating a small point (FP; 0.5° diameter) and touching, with the left hand, a larger circular spot (HOME; 4° diameter) centered 15° below the FP. All intervals between events were randomized using exponential distributions with resampling to conform to a specified range. Thus, after a short delay (200–500 ms; median, 234 ms; exponential parameter, λ = 50 ms), a pair of choice targets appeared to the left and right of the FP in a location such that one was in the response field (RF) of the neuron. After another random delay (500–1000 ms; median, 600 ms; λ = 100 ms), the random dot motion appeared in a 6° aperture centered on the FP. The direction and motion strength were randomized to values −0.512, −0.256, −0.128, −0.64, −032, 0, 0.32, 0.64, 0.128, 0.256, and 0.512, in which the sign indicates the direction toward the RF and the value indicates the probability that a dot plotted at t = t0 will be displaced 0.2° at t = t0 + 40 ms later (i.e., speed = 5° visual angle per second) or else replaced at a random location. Viewing duration ranged from 240 to 1000 ms (median, 450 ms; λ = 400 ms). At the end of the motion stimulus, monkeys were required to maintain eye and hand fixation until either of the fixation spots was turned off (500–1000 ms; median, 656 ms; λ = 300 ms), cueing either an eye or hand movement (Fig. 1). Until this point, monkeys were required to maintain eye/hand fixation within ±1° of FP and ±3.5° HOME and to sustain this fixation criterion for the unused effector while they indicated the choice with the other. The same effector (eye or hand) was alternated in blocks of ∼240 trials. Response effector was cued by turning off the eye or hand FP (Fig. 1). We discarded from the analyses the first trial after an effector switch.
An instructed delay task was used for screening and selecting the neurons. In this task, monkeys were required to make an eye or hand movement to the remembered location of a briefly flashed target that was either inside the RF of the neuron (Tin) or in the opposite visual field (Tout). The monkeys had to maintain the gaze and hand on their respective FPs until one or the other disappeared at the end of the delay (1000–1500 ms, truncated exponential distribution), signaling the monkey to reach or look to the remembered target. Monkeys were rewarded if the movement was sufficiently accurate (±3° visual angle) and the other modality remained at the fixation spot. Effector modality was used for a block of ∼60 consecutive trials.
Neural recording.
Recording chambers were placed on the right parietal cortex at stereotaxic coordinates 6 mm posterior to ear bar zero and 9 mm lateral to the midline. Extracellular recording electrodes (Thomas) were advanced daily along the lateral and medial banks of the intraparietal sulcus (IPS) in areas thought to correspond to the ventral LIP (LIPv) and MIP based on stereotaxic coordinates and MRI images obtained for each monkey with the chamber and electrode grid in place. Spikes were isolated online and collected with a Plexon system.
Neurons were classified as from MIP or LIP based on anatomical and physiological criteria. Neurons on the medial bank of the IPS (MIP) exhibited little if any modulation of neural activity with saccadic eye movements and strong modulation of this activity accompanying spontaneous arm movements. Neurons on the lateral bank (LIPv) were modulated strongly by spontaneous eye movements (44 neurons in monkey G and 36 in monkey T). Each of these regions occupied ∼8 mm3 within which we screened all isolated neurons for spatially selective persistent activity on an instructed delayed reach or saccade task. We selected for additional study only neurons that were clearly spatially selective by ear. This was just under 50% of the neurons we encountered, and post hoc receiver operating characteristic (ROC) analysis on memory saccades/reaches (LIP/MIP) confirmed this selectivity (mean ROC area, 0.97 and 0.93 for LIP and MIP, respectively; 90% of neurons exceeded 0.75; n = 92). Neurons in MIP (46 in monkey G and 32 in monkey T) showed weak visual responses and were only weakly modulated by saccades. They activated strongly in the periods preceding a reach movement, and this activation was spatially selective, with most RFs spanning an area between 15 and 20° in diameter centered between 10 and 25° of eccentricity in the contralateral visual field. The weak visual response of MIP neurons is consistent with our recording locations, which targeted the more anterior part of MIP. A total of 133 of 148 neurons were recorded in at least one reach and one saccade block.
Data analysis.
The psychometric functions in Figure 2a are maximum likelihood fits of a Weibull function (Quick, 1974) to data from all viewing durations:
where α and β are fitted threshold and shape parameters, respectively, and C is the motion coherence. We also fit a bounded drift-diffusion model (Ratcliff, 1978; Gold and Shadlen, 2002; Bogacz et al., 2006) to account for the variation in choice accuracy as a function of the duration, strength, and direction of random-dot motion (Fig. 2c). A putative decision variable, V, is described by the stochastic differential equation:
where C is motion strength (the sign indicates direction), and dW is the standard Wiener process. The free parameter, k, establishes the drift rate. A second free parameter, B, establishes symmetric decision bounds, such that, if |V| ≥ B, the decision terminates. For each stimulus duration, td, the choice is either determined by the probability of terminating at ±B (for t ≤ td) or by the sign of V(td) (i.e., when a bound is not reached). This two-parameter model provides an estimate of the distribution of integration times, from which we calculate the means and SDs displayed in Figure 2d. For the 0% coherence stimulus, just over 65% of the decisions would have terminated by 240 ms, and 84% would have terminated by the end of the full 360 ms epoch used in the variance analyses. Decision termination does not necessarily imply termination of evidence integration by neurons (Mazurek et al., 2003; Resulaj et al., 2009), but it invites caution against over-interpretation of the predicted variance and autocorrelation at the later time bins in Figure 9.
To estimate the psychophysical kernel (Fig. 2b), we used 0% coherence trials. For each random-dot movie, we extracted the opponent motion energy (Adelson and Bergen, 1985) using filters identical to those in the study by Kiani et al. (2008). From each of the resulting traces, we subtracted the mean of the traces for the group identified by motion strength/direction, monkey, task, and area. The sign of the resulting residual traces was then flipped for negative-direction choices so that each trace represents the residual motion energy in support of the choice on that trial. It would be zero on average if decisions were independent of the stimulus motion. The figure shows the mean ± SEM of residuals, termed the psychophysical kernel. It is important to note that the motion energy filtering imposes a delay and rise time characterized by the “motion impulse” response function shown in Figure 2b (blue curve; see legend).
The firing rate averages in Figures 7 and 8 were normalized using the average delay period activity of each neuron, excluding the first 50 ms (after motion offset) and the last 100 ms (before movement initiation). For LIP cells, the normalization constant was obtained from the saccade trials, whereas for MIP cells, the constant was obtained from the reach trials. We used a two-step regression procedure to estimate the effect of motion strength on buildup rate (see Figs. 7, 8, insets). First we fit lines to the each of the firing rate functions, using the first 150 ms from the onset of putative integration (see Figs. 7, 8, gray bars). The least squares fits use normalized rates obtained in 15 independent 10 ms bins (i.e., before the smoothing shown in the figure). The slope (i.e., buildup rate) and associated SE are depicted by the symbols and error bars in the figure insets. The lines shown in the insets are weighted least square fits to these buildup rates. To address statistical significance, we evaluated the null hypothesis that the data conform to a pair of constant buildup rates associated with the two directions:
where BU indicates buildup, C is motion strength (signed, such that positive C is toward Tin), Idir is an indicator variable, and bi are fitted coefficients. For the inset to Figure 7, Idir = 1 for motion toward Tin and 0 otherwise. For the inset to Figure 8, Idir indicates choice (1 for Tin and 0 otherwise). The null hypothesis {H0: b1 = b3 = 0} was evaluated using an F statistic. Note that we calculated the buildup rates using the first 150 ms of decision-related activity (Figs. 7, 8, gray bars), before the curves exhibit coherence-dependent saturation, although, as noted above, we think the decision evolves over a longer duration on many trials.
We analyzed the time course of the firing rate variance and autocorrelation because these measures permit a glimpse of an underlying process in which the instantaneous firing rate of parietal neurons reflect the accumulation of noisy evidence from the visual cortex. The firing rate as a function of time cannot be inferred precisely from single trials. However, across trials, it is possible to discern the hallmarks of an underlying diffusion (or random walk) process from the variance and autocorrelation of the spike counts. The approach, outlined by Churchland et al. (2011), merges the theory of doubly stochastic processes with the laws of total variance and covariance. The theory and estimation method are described in detail in that study. The basic idea is to examine the spike counts in the same short epoch over multiple trials and to subtract, from the total variance, a theoretical component, termed the point process variance (PPV), which is the variance that would be expected if the rates were identical across trials. The remaining variance, termed the variance of the conditional expectation (VarCE), captures the trial-to-trial variation in the underlying firing rates. To estimate the PPV, the mean count is multiplied by an unknown constant, φ. Churchland et al. supplied heuristics to approximate φ, but none are entirely satisfying.
For the present study, we used the theoretical values for the correlation matrix of rates (or expected counts) to estimate φ, as follows. We divided the first 360 ms of putative integration into six 60 ms spike-count bins. For each bin, we used 0% coherence trials and calculated the mean and residuals with respect to the mean (by neuron and effector modality). We then combined the residuals and their associated means using Equation 6 from the study by Churchland et al. (2011) to estimate the VarCE, S〈Ni〉2, in the six time bins under the four conditions (area by effector), given φ. We computed the 6 × 6 covariance matrix using the same residuals and substituted VarCE for the raw variances to obtain the CovCE matrix, which we convert to a correlation matrix by dividing terms by
The VarCE plotted in Figure 9d uses the fitted φ (one per area by effector). Lines are weighted least squares fits, using SEs derived from a standard bootstrap. For random walks, this should be a linear function of time, but neither the slope nor the intercept of this line are constrained by theory, at least not without additional assumptions. For this reason, the 15 predicted CorCE values furnish the stronger constraint on φ.
We used a bootstrap procedure to estimate the distribution of a sum of square deviation (SS) of the 15 fitted CorCE values (Fisher z-transformed) from their theoretical values. To compare the degree of conformance to predicted values between saccade and reach trials for LIP neurons, we compared the observed SS for reaches to the distribution of SS under saccades. For MIP neurons, we performed the bootstrap using the reach condition. This method supports what is evident by eye in the graphs, as well as many other variants of the analyses we have performed. We also performed a more sensitive bootstrap analysis that simply resamples the observed correlation values without adherence to position in the matrix (preserving symmetry). We generated 1000 valid (i.e., positive definite) correlation matrices using the bootstrap. Again, we ask how likely it would be to observe an SS statistic from our data under such a random rearrangement of the pairwise correlation values. We report the p value associated with the bootstrap (sample with replacement) but observed a similar value (p = 0.054) for a variant based on permutations of the correlation values. We also compared the SS statistic to its expected distribution from random correlation matrices conforming to uniform distribution of r values over the observed range. We mention this test in association with the MIP/saccade data, because rejection of H0 cautions against overlooking structure to the pattern of autocorrelation even if it does not conform to the expectations of drift diffusion. For the other three conditions, rejection of H0 is not surprising (p << 0.001). We also investigated the possibility that the failure to support integration in the MIP/saccade data using the variance analyses might be attributed to weak statistical power associated with low firing rates. This was confirmed by simulation of datasets that approximated the firing rates in Figures 7 and 8 (0% coherence), matched for the number of trials and distribution of trial durations, and evaluating for a significant positive slope of the VarCE versus time relationship (p < 0.05). The lower firing rates are associated with up to twice the rate of type II statistical error.
For the three analyses that focus on the epoch of putative integration (two-step regression, variance, and autocorrelation), we focused on epochs beginning where the response functions begin to separate as a function of motion strength and direction (see Fig. 7). This was ∼212 ± 8 ms after motion onset for LIP neurons and 142 ± 10 ms after motion onset for MIP neurons (see the next paragraph). We also tried a variety of approaches in which we varied the assumed start time or tailored it to the firing rate functions for each neuron and effector modality. All of these manipulations produce qualitatively similar results to those reported. We also repeated the variance analyses using theoretical correlation coefficients based on the bounded drift-diffusion model fits to the behavioral data (Fig. 2d), as noted above.
To estimate the latency at which decision-related activity emerged in each area, we compiled distributions of spike counts associated with the two strongest motion conditions in 20 ms counting windows and derived a selectivity index (the area under the ROC). This index is a nonparametric measure of the degree of separation of the two distributions associated with correct Tin and Tout choices; the index is 0.5 when the responses are overlapping. We performed this analysis by pooling standardized responses of MIP neurons in the reach condition and of LIP neurons in the saccade condition. The reported latencies are the times at which the ROC index significantly departed from 0.5 for at least 50 ms (p < 0.05; Hanley and McNeil, 1982). The SEs of the latencies were obtained by a bootstrap procedure. The full time course of this choice selectivity index is shown in Figure 6b.
Results
We recorded from single neurons in areas LIP and MIP of two rhesus monkeys that were trained to judge the direction of motion of a dynamic random-dot display (Britten et al., 1992). Task difficulty was varied by controlling stimulus viewing duration and motion strength (percentage coherence; see Materials and Methods). Monkeys communicated their decision about motion direction by looking or reaching to a left or right choice target (Fig. 1). For both effector modalities, the monkeys were required to hold both the gaze and hand on fixation spots during motion viewing and throughout an ensuing memory delay. When cued to respond, the monkeys were allowed to activate only the appropriate effector modality (i.e., saccade or reach) while maintaining continued fixation with the other. Effector modalities were alternated in blocks of ∼240 trials.
Monkeys were trained to indicate the direction of random-dot motion with either a hand or an eye movement. After eye and hand fixation, two peripheral choice targets appeared, followed by the random-dot motion display centered at the FP. The direction of motion (left or right), difficulty, and viewing duration were randomized from trial to trial. After a variable delay, either the eye or hand fixation spot was extinguished, instructing the monkey to communicate its decision by looking or reaching to the choice target. All correct trials and a random half of the 0% coherence motion trials were rewarded. The same effector was used for blocks of ∼240 trials. The monkeys had to maintain fixation with the unused effector until reward delivery. During neuronal recordings, one of the choice targets (termed Tin) was in the neuronal RF.
We first examine the monkey's accuracy and establish the epoch in which integration of sensory evidence from the random-dot display supports the monkey's choices. We then describe a neural representation of accumulating evidence in areas LIP and MIP.
Behavior
For both monkeys, discrimination performance was similar for saccade and reach trials. Psychometric functions, which plot the proportion of correct choices as a function of motion strength, are nearly superimposed for the two modalities, in both monkeys (Fig. 2a). The comparison of slopes of the psychometric curves failed to reveal an effect of motor modality on sensitivity (α term in Eq. 1; likelihood ratio test, p = 0.57 and 0.58 for monkeys T and G, respectively). Discrimination thresholds from individual sessions varied from day to day, but a pairwise comparison failed to reveal reliable differences in thresholds regardless of effector modality (p = 0.67 and 0.21, paired t test). This negative result is important because it suggests that any differences we observe in the neural responses on reach and saccade trials are unlikely to be explained by differences in behavior or decision strategy. It also justifies combining data across conditions, as we do next to infer the time window of decision formation.
Analyses of behavioral data. a, Decision accuracy depends on the strength of motion but not on the effector modality used to report the choice. Points are the proportion correct for all viewing durations. Sigmoidal curves are best fitting Weibull functions. The discrimination threshold is the motion coherence that supports a proportion correct = 0.816 (α; Eq. 1). The data are shown separately for the two monkeys but combined henceforth. b, Time course over which fluctuations in stimulus motion information guides the decision about direction. Gray curve shows the mean motion energy (±1 SEM) in support of the monkey's choice on trials in which 0% coherence motion stimuli were shown. The blue curve shows the response to a single dot appearing at t = 0 and displaced in the positive direction 40 ms later (same Δx/Δt as in the variable coherence display). Choices were affected most strongly by the first ∼250 ms of motion fluctuations in the noisy display, and information continued to affect choices for longer durations on some trials. c, Accuracy increases as a function of motion strength and stimulus duration. Thin lines are running means of the proportion correct. Smooth curves are fits to a model that assume noisy evidence is integrated to a termination threshold or bound. The behavior is best explained by perfect integration of information presented early in the trial, consistent with the analysis in c. The rate of improvement and saturation are explained by variance in the termination times across trials and motion strengths, as occurs in reaction time experiments. d, Decision times estimated from the bounded evidence accumulation model fit. Points are mean decision times predicted from the model. Error bars indicate ±1 SD.
Two analyses suggest that the monkeys base their decisions on the accumulation of motion evidence from the random-dot display. The first exploits the random fluctuations in the motion display itself. Even for 0% coherence motion, the monkeys were influenced by fluctuations in motion energy in the random dots. Although, on average, the 0% coherence stimulus supports neither direction, the evidence is not neutral from moment to moment and across trials. To quantify the influence of these fluctuations on the monkeys' choices, we extracted the motion energy profile (Adelson and Bergen, 1985; Neri, 2004; Kiani et al., 2008; Resulaj et al., 2009) as a function of time from the display of each trial, applying a sign convention such that positive motion energy represents the direction chosen by the monkey at the end of that trial. The average of these traces, termed the psychophysical kernel (Neri, 2004; Fig. 2b, gray trace), shows an influence of stimulus motion on choice. After correcting for the delay introduced by the filters that extract motion energy (Fig. 2b, blue trace), we infer that monkeys based their choices most strongly on the information in the first 250 ms of stimulus viewing, whereas later motion fluctuations were influential to a lesser degree.
This interpretation is further supported by the analysis in Figure 2c. We fit the monkeys' choices as a function of two variables: (1) viewing duration and (2) motion strength. The model, which assumes perfect integration of evidence to a threshold or bound, has been shown to account for the rate of improvement in accuracy as a function of both variables (Kiani et al., 2008; Tsetsos et al., 2012). The bound explains the curtailment in improvement in accuracy at longer viewing durations, seen as a flattening of the curves in Figure 2c. Indeed, consistent with the preceding analysis (Fig. 2b), most of the improvement explained by integration was attained by the time of the first sample of the running mean in Figure 2c. From this bounded accumulation model, we can estimate the mean integration time and its SD for each motion strength (Fig. 2d). For the lowest motion strengths, the model fit shows that mean integration times are ∼250 ms, consistent with the integration times inferred from the psychophysical kernel (Fig. 2b). The large SDs (error bars) are consistent with the prolonged tail of the kernel.
Together, these analyses establish that the monkeys integrate motion evidence to reach their decisions, and they do this based primarily on the information presented early in the trial. The results establish a window of time in which to focus our analysis of the neural activity accompanying decision formation.
Neuronal activity in MIP and LIP
As the monkeys performed the discrimination task, we recorded the activity of neurons in the medial and lateral banks of the IPS (Fig. 3). Neurons were first screened using an instructed memory delay task—either a delayed saccade or a delayed reach to the remembered location of a briefly flashed target in the periphery of the visual field. Neurons were selected for additional study if they showed spatially selective persistent activity in preparation for saccades or reaches or both, such that a preferred and nonpreferred location could be discerned easily by listening to the spiking activity (see Materials and Methods). As shown by the response averages in Figure 4, a and b, the MIP neurons selected for additional study exhibited this property during memory-guided hand and eye movements, although responses were weaker on eye-movement blocks (p = 0.0058, t test; n = 48). The absence of a visual response to the onset of visual targets is consistent with the anterior location of our recordings in MIP, possibly at the border of MIP/5v (Hwang et al., 2012). LIP neurons were approximately equally responsive on both memory-guided saccades and reaches (Fig. 4c,d), and this similarity was also representative of the population of single neurons in our sample (p = 0.32, t test; n = 51). It is this spatial selectivity, maintained over the ∼1 s of memory delay, that we exploit to study the neural correlates of decision formation.
Location of recording sites. a, The vertical line in the diagram (top) marks the approximate location of the coronal magnetic resonance image (bottom), 6 mm posterior to the interaural line. b, The recording chamber containing a saline-filled grid is visible above the IPS. The MRI was obtained on a 1.5 T scanner using a short T1 inversion-recovery sequence. Cranial screws (titanium) caused the susceptibility artifact, which appear as black indentations of the cortex on either side of the recording chamber. c, The recording location of MIP (blue) and LIP (red) neurons is plotted in stereotactic coordinates and projected to MRI slices oriented parallel to the recording chamber (top and bottom corresponds to monkeys G and T, respectively). Locations were assessed by registration and are therefore only approximate.
Activity of neurons during a delayed center-out task. Monkeys were required to shift the gaze or hand to the remembered location of a briefly flashed target. Average firing rates are shown for target locations used in the main discrimination task, inside (Tin) or outside (Tout) the RF of the neuron. a, b, Response averages from 48 MIP neurons recorded in reach and saccade blocks, respectively. c, d, Response averages from 51 LIP neurons recorded in reach and saccade blocks, respectively.
In the motion discrimination task, one of the choice targets was positioned inside the RF of the neuron and the other in the opposite hemifield (Tin and Tout, respectively). The position of the targets and the random-dot motion display were maintained through alternating blocks of trials in which the monkeys indicated their decisions by moving the hand or the eyes. Within these blocks, the different motion strengths, directions, and interevent intervals were randomized.
Responses from representative LIP and MIP neurons are shown in Figure 5. Trials are grouped by behavioral choice and ordered by motion direction and strength to facilitate visualization of important features. Spikes are aligned to three trial events: (1) display of the choice targets, (2) onset of random-dot motion, and (3) movement initiation. The example LIP neuron had a strong visual response to the onset of the choice targets, followed by a more modest sustained activity until the onset of random-dot motion. The example MIP neuron did not respond to the onset of the choice targets, and its activity remained at background level until motion onset. Approximately 150 ms after random-dot motion onset, the LIP neuron was momentarily suppressed to a level below background—a “dip”—followed by a rise in firing rate on trials in which the monkey chose Tin or followed by some degree of continued suppression on the trials in which the monkey chose Tout. Both the increase and decrease of firing rates were most evident on trials when the strongest motion was displayed, and they were similar for saccade and reach blocks. Motion also induced graded increases and decreases in the firing rate of the MIP neuron on trials in which the monkey indicated its decision with a hand movement. The pattern of responses during motion viewing was similar to that of the LIP neuron. Note that both neurons were more active during the hand-movement blocks and that the MIP neuron was less active during eye-movement trials.
Responses from two representative neurons recorded during the motion discrimination task. Each row of black dots mark the times of action potentials with respect to onset of targets (left), onset of random-dot motion (middle), and movement initiation (right). Colored dots signify the end of the motion stimulus. Trials are sorted according to stimulus strength (colored numbers), stimulus duration, and monkey's behavioral response (Tin and Tout choices). Only correct choices are shown for nonzero coherences. a, b, Responses from an MIP neuron recorded in reach and saccade blocks, respectively. c, d, Responses from an LIP neuron recorded in reach and saccade blocks, respectively.
The final epoch depicted in Figure 5 encompasses the late memory delay and movement. Here, both example neurons responded in a manner that reflected the choice for Tin or Tout. Whereas the LIP neuron responded strongly for both response modalities, the MIP neuron barely modulated its firing rate on eye-movement trials. On hand-movement blocks, both the MIP and LIP neurons sustained activity through the movement. On eye-movement blocks, the LIP neuron emitted a burst of spikes in the immediate perisaccadic epoch (−100 to 50 ms relative to saccade initiation) accompanying Tin saccades. The postsaccadic responses after Tout saccades might be related to the trace of the fixation spot, only just extinguished, which has been translated to the RF of the neuron. The MIP neuron did not modulate its response in the perisaccadic or peri-reach intervals.
The qualitative features illustrated by the two example neurons are representative of the population of neurons we recorded. Neurons in both LIP and MIP were more active during reach trials (p < 0.01 in both areas and both monkeys), and MIP neurons attenuated their firing rate during the saccade blocks (Fig. 6a). Whereas the latter observation conforms to the prediction that MIP neurons are recruited during planned reaches, the increase in the firing rate of LIP neurons in the reach block is somewhat surprising. The increase was not simply a global change in firing rate level because it was accompanied by greater selectivity. As shown in Figure 6b, LIP responses associated with Tin and Tout choices achieved greater selectivity (less overlap of the distributions of firing rates associated with the two choice types on the easiest motion conditions). In other words, they were more predictive of reach choices. Interestingly, this predictive activity evolved over a similar timeframe for saccade and reach trials (compare curves in Fig. 6b). We will examine these dynamics in greater detail below. However, it is worth noting the predictive activity in MIP on reach trials emerged 70 ms earlier than in LIP (142 ± 10 vs 212 ± 8 ms for MIP and LIP, respectively; see Materials and Methods).
Neural response selectivity in MIP and LIP as a function of effector modality and movement direction. a, Comparison of response magnitude in the delay period before saccades and reaches to Tin. Points represent mean firing rate in the 500 ms epoch ending 200 ms before movement onset. Symbol type and color indicate monkey and parietal area, respectively (n = 133). Note that the majority of LIP and MIP neurons exhibited higher firing rates during the reach trials. b, Time course of choice selectivity in MIP (blue) and LIP (red) on easy trials. The index quantifies the separation of the distributions of responses associated with Tin and Tout choices (area under the ROC). Line type indicates effector modality. Choice selectivity begins earlier in MIP but is present in both parietal areas for both effector modalities. Only correct choices on the two highest motion coherences are included in this analysis.
For the remainder of this study, we will focus on the neural correlates of decision formation in the epoch after onset of visual motion in which the monkeys are accumulating evidence from the random-dot display. We hypothesized that parietal neurons reflect this accumulation.
Representation of a decision variable in MIP and LIP
A decision variable is a quantity that tracks the accumulation of evidence leading to a decision, possibly combined with other factors, such as previous expectation of outcome, reward, and cost (Gold and Shadlen, 2007). Because of our selection criteria, we are not surprised to learn that LIP and MIP neurons represent the outcome of decisions indicated by an eye or hand movement to Tin, respectively. The question is what transpires in the epoch in which the decision is forming. To facilitate the comparison between parietal areas (LIP vs MIP) and response modalities (reaches vs saccades), we will present the results of our analyses in a 2 × 2 format, with rows designating recording area and columns designating effector modality. Figure 7 shows average firing rates for 148 neurons, normalized to the firing rate of each neuron in the delay period preceding Tin reaches or saccade. Each trace corresponds to a motion strength and direction, and each comprises trials leading to both Tin and Tout choices. These firing rates exhibit a gradual rise or decline and a clear dependence on the strength and direction of motion. We estimated the rate of this rise or decline by fitting lines to the response averages in the 150 ms epoch from the earliest point that these trends appear to emerge (see Materials and Methods). The slopes of these lines, termed buildup rates, are approximately proportional to the motion strength (Fig. 7, insets), consistent with the hypothesis that the firing rates represent the accumulated difference in motion evidence from opposing pools of motion-sensing neurons, such as those in middle temporal area MT (Shadlen and Newsome, 1996, 2001; Mazurek et al., 2003; Huk and Shadlen, 2005). The linear fits are highly reliable for all figure insets (p < 0.0003).
Evolution of neural responses during decision formation. Responses are aligned in time to the onset of random-dot motion. The strength and direction of motion are indicated by color and line type. Trials ending in either choice are included in the averages. The firing rates from each MIP neuron (a, b) were normalized to the average delay activity before reaches to Tin. The firing rates from each LIP neuron (c, d) were normalized to the average delay activity before saccades to Tin. Response averages were smoothed using a causal exponential kernel (τ = 30 ms). Inset graphs show the effect of motion strength on the rate of change (buildup rate) of the normalized average firing rates during the first 150 ms of putative integration, indicated by the gray bars. This is ∼70 ms earlier in MIP than in LIP (see Materials and Methods). Lines are weighted least square fits to these buildup rates.
The traces shown in Figure 7 are most apposite to the representation of a decision variable whose trial-by-trial instantiation could lead to either choice. This mixture of choices varies as a function of motion direction and strength, raising the concern that the effect of motion strength is explained by different mixtures of just two buildup rates, associated Tin and Tout choices, respectively. To address this, Figure 8 depicts the responses grouped by motion strength, direction, and choice. For nonzero coherences, these are the same data with the errors removed, and for 0% coherence, the data are divided into trials ending in Tin and Tout choices. This grouping allows us to ask whether motion strength affects the parietal firing rates even for trials in which the monkey makes the identical choice. This grouping of the responses shows the evolution of choice-related activity (compare solid and dashed curves), but there is still an effect of motion strength (indicated by color) that can be deduced from the ordering of the responses. The pattern is subtle for MIP on saccade blocks, but it is supported by the analysis of buildup rates (Fig. 7, insets). Were the effect of motion strength on buildup rate explained by different mixtures of Tin and Tout choices (Fig. 7), then the insets in Figure 8 should appear as a pair of flat lines. This possibility is rejected for each of the four insets (p < 0.006; Eq. 3).
Evolution of neural responses during decision formation, conditioned on choice. Conventions are the same as in Figure 7, except traces contain only those trials that end in the same choice. Only correct choices are included for nonzero motion strengths.
Together, these analyses of average firing rates support a representation of a decision variable in both LIP and MIP during both hand- and eye-movement blocks. The responses appear equally robust in LIP and MIP on the reach blocks, whereas MIP responses are weaker than those of LIP during the saccade blocks. This discrepancy led us to question whether MIP neurons actually reflect a decision variable, and, if so, why it is so much less robust. Our working hypothesis is that the decision variable is itself the accumulation of noisy evidence from direction-selective neurons in area MT and medial superior temporal area MST, which represent the momentary evidence for one direction or the other. Such an accumulation of noisy evidence has been likened to a random walk or diffusion process (Ratcliff, 1978; Gold and Shadlen, 2002). Accordingly, the firing rate traces represented by the curves in Figure 7 represent averages over trials whose underlying rates resemble the paths of diffusing particles. However, these response averages obscure the variation in firing rates across trials and as a function of time. A more stringent test of our hypothesis is that the firing rates on single trials should resemble the random path of a particle undergoing drift diffusion. Evidence bearing on this hypothesis may be discerned from an analysis of second-order statistics (e.g., variance and autocorrelation) of the firing rates.
A diffusion-like mechanism
Although it is not possible to discern the firing rate paths from the spike trains on individual trials, we can ask whether the second-order statistics of firing rates are consistent with the expectation from a diffusion process. To do so, we examined the change in spike count variance, across trials, as a function of time from the start of the putative accumulation (Churchland et al., 2011). This is because the variance of the accumulation of independent samples, as in diffusion, should increase as a function of time. Moreover, for a diffusion process, the correlation coefficient between the spike counts in time bin ti and some later time bin tj > i is rij =
Evolution of response variance and autocorrelation of firing rates during decision formation. These analyses examine the correspondence between data and predictions from a diffusion process. a, Expected autocorrelation matrix for a discrete diffusion process (upper triangular portion of the symmetric correlation matrix). The autocorrelation is between the value of a random variable formed as the cumulative sum of independent identically distributed numbers (increments) in the ith and jth time step of its evolution (i = 1–5; j = 2–6). Six time steps are shown to correspond with the same number of time bins applied to data. Heat map displays the values of the 15 predicted r values (i.e., the unique r values of the 6 × 6 correlation matrix). Notice that correlation between bins decreases as a function of the time lag between them (dashed line), and correlations separated by the same lag become increasingly correlated at later times (solid line). b, Autocorrelation matrices from MIP and LIP. The 2 × 2 arrangement is identical to the area by effector table in previous figures. To calculate these correlations, an unknown constant, φ, was derived to achieve the best fit to the 15 predicted r values (for details, see Results). Same color map as in a. c, Comparison of autocorrelation value from data (symbols) and theory (lines). These are the top row and first juxtadiagonal as indicated by corresponding line style in a. d, Variance of the rates across trials using the same six time epochs. For a discrete, unbounded diffusion process, these variances should increase linearly as a function of time. The variance estimates use the value of φ derived from the autocorrelation. Note the different scale for the LIP reach graph. Error bars in c and d are SDs obtained from a bootstrap procedure.
As explained in Materials and Methods, the pattern of autocorrelation as both a function of lag time (Fig. 9a, rows) and time itself (Fig. 9a, diagonals) poses the more stringent test of an underlying random walk or diffusion process. For both the variance and autocorrelation tests, the challenge is to factor out the component of the measured spike count variance that would be evident in MIP and LIP even if there were no trial-to-trial variance of the underlying rates. Estimation of this component of the variance, termed the PPV, depends on an unknown parameter φ. We chose the value of φ that provided the best fit to the 15 correlation coefficients expected from theory, using six 60 ms spike count windows that span the first 360 ms of putative integration. We did this for each of the parietal areas and each effector modality, as shown in Figure 9.
The pattern of autocorrelations obtained from the LIP recordings was similar to the theoretical values for both effector modalities (Fig. 9b,c; R2 = 98.1 and 98.9%, for saccade and reaches, respectively), and conformance was not reliably better for either effector (p = 0.72, bootstrap; see Materials and Methods). The pattern of autocorrelations in the MIP recordings conformed to the predictions of diffusion during hand-movement blocks but not during eye-movement blocks (Fig. 9b,c; R2 = 98.8 and 90.2% for reaches and saccades, respectively). In the latter case, there was some structure to the autocorrelation that could be distinguished from chance (p < 0.03, bootstrap; see Materials and Methods), but conformity to diffusion was clearly superior for the reach blocks (p < 0.02). Indeed, the error statistic for the MIP/saccade dataset was outside the 98% confidence interval using any of the other three conditions.
We draw a similar conclusion from the time course of the response variance. Using the best estimate of φ from the autocorrelation analysis, we extracted the observed variance in each of the six time bins. If the LIP firing rate reflects the accumulation of noisy evidence, then the variance, termed the VarCE (see Materials and Methods), should be proportional to time. As shown in Figure 9d, the pattern of variance conforms reasonably well to linear increases for saccade and reach blocks (p < 0.0005 and p < 0.02, respectively, F test), whereas MIP exhibits this pattern for reach blocks only (p < 0.0002). The pattern clearly fails to conform to a linear rise on the saccade blocks (p = 0.41). Although the patterns of variance and autocorrelation are strikingly different in MIP for the two effector modalities, this difference might simply reflect the lower firing rates of MIP neurons on saccade blocks (see Materials and Methods).
Together, these analyses allow us to appreciate that the firing rate functions plotted in Figure 7 are the means of diffusion paths, as predicted by the drift-diffusion model (Link, 1975; Ratcliff, 1978; Gold and Shadlen, 2001). Moreover, for decisions communicated by hand movements, at least two parietal regions represent the accumulation of noisy sensory evidence leading to a decision.
Discussion
To make even simple perceptual decisions, the brain must gather the appropriate evidence bearing on the alternatives and then operate on this evidence to commit to a choice. The random-dot motion task is useful for studying this process because (1) its solution requires integration of evidence across space and time and (2) a neural representation of the momentary evidence has been established in areas MT and MST (Salzman et al., 1990; Celebrini and Newsome, 1994; Ditterich et al., 2003). To make a decision, the brain accumulates this momentary evidence in time until some termination criterion is met or until the evidence runs out.
Under conditions in which a monkey communicates its decision with an eye movement, a neural representation of the accumulated evidence has been demonstrated in area LIP, the frontal eye fields, the superior colliculus, and the dorsolateral prefrontal cortex (Horwitz and Newsome, 1999; Gold and Shadlen, 2000, 2007; Ding and Gold, 2012). In particular, LIP is known to represent the accumulated evidence, combined with other factors that affect the decision, such as the previous probability, value, and costs associated with the alternatives (Seo et al., 2009; Rorie et al., 2010; Hanks et al., 2011; Drugowitsch et al., 2012). The combined quantity is termed a decision variable because application of a simple criterion to this quantity would render a commitment to a choice. A neural representation of a decision variable in LIP and other oculomotor areas suggests that the motor system is not recruited once a decision is made but instead receives a continuous flow of partial information—an evolving decision variable (Gold and Shadlen, 2007; McKinstry et al., 2008; Cisek and Kalaska, 2010; Selen et al., 2012).
The present findings extend this principle to the hand-reaching system of the parietal cortex. Our observations complement previous studies of MIP and the parietal reach region (PRR), which demonstrated choice-related activity in MIP (Snyder et al., 1997; Eskandar and Assad, 1999; Scherberger and Andersen, 2007; Pesaran et al., 2008). In light of these studies, it is not surprising that MIP neurons would represent the outcome of a decision process when the choice is indicated by a hand movement. The important finding here is that MIP activity tracks the decision variable—the partial evidence as it is being accumulated to reach a decision.
Experimental support for the representation of a decision variable is based on three related observations. First, monkeys availed themselves of the evidence in the visual display by integration of evidence (Fig. 2). The psychophysical kernel supports the use of the very earliest information—the strategy promoted by the design the random-dot display and by the exponential distribution of viewing durations—and the improvement in performance with viewing duration suggests that, for low coherence trials, integration extends to at least 300 ms on a substantial proportion of trials. Second, the mean firing rates reflect the accumulation of evidence in a corresponding epoch (Figs. 7, 8). Both LIP and MIP neurons exhibit changes in the mean firing rates that reflect the strength and direction of motion (Figs. 7, 8), and this relationship is not explained trivially by different mixtures of high and low firing rates associated with Tin and Tout choices (Fig. 8). Third, variance and within-trial autocorrelation of the firing rates evolve in a manner that is consistent with a biased random walk or drift-diffusion process. Thus, the firing rates on individual trials appear to instantiate an accumulation of both signal and noise. This third level of support is absent for the analysis of MIP responses during saccade blocks (Fig. 9).
Perhaps it is not surprising that decision-related activity is weak in MIP when decisions are communicated by eye movements, but such effector specificity did not hold for LIP, which showed greater activation for hand-movement trials. The finding contradicts the simplistic notion that the posterior parietal cortex is parceled into specific premotor domains. Indeed, careful reading of studies that support motor specialization (Snyder et al., 1997; Cui and Andersen, 2011) attests to only a relative preference for modality, as these investigators have emphasized (Andersen and Cui, 2009). The recruitment of saccade-related neurons during reach movements has been interpreted in the past as a sign of allocation of spatial attention (Robinson et al., 1978; Bushnell et al., 1981; Wardak et al., 2004) and also as reflecting covert eye movements that might occur were the monkeys not required to hold fixation away from the reach target (Snyder et al., 1997, 1998; Colby and Goldberg, 1999). After all, we often look at what we touch, but we cannot touch most of what we see. Although the present findings support a degree of modality specificity in many MIP neurons, we cannot say whether this would hold for other parts of the PRR, in which, for example, neurons are reported to exhibit stronger responses to visual targets.
The representation of a decision variable in two areas could arise several ways. Input and output connection patterns of LIP and MIP place these areas at approximately the same level in the hierarchy of visual information processing (Felleman and Van Essen, 1991; Lewis and Van Essen, 2000a,b). They both receive input from lower visual areas, enabling them to independently accumulate sensory information and compute decisions. However, they also receive input from higher cortical areas (e.g., dorsolateral prefrontal cortex), opening the possibility that they might be receiving a copy of a decision variable assembled first in these areas. Indeed, the representation of an evolving decision variable in areas LIP and MIP does not imply that the ontogeny of such signals is within those areas. An interesting possibility is that higher cortical areas help to gate the flow of evidence to MIP and LIP or control the processing of such evidence within these areas.
There is some indication in our data that decision-related activity was present in MIP earlier than in LIP (142 ± 10 and 212 ± 8 ms, respectively; Fig. 6b; see Materials and Methods). Therefore, it is possible that MIP is the source of decision-related signals in LIP on the reach trials. However, this is unlikely on the saccade trials, because MIP activity is so much weaker and lacks signatures of integration, which were evident in LIP. Other dissimilarities between LIP and MIP (e.g., visual responses to targets; Figs. 4, 5) and the similarity of LIP dynamics for both response modalities render unlikely the possibility that decision-related activity in LIP is merely a delayed replica of MIP activity. Simultaneous recordings from the two areas will resolve this, but it seems highly likely that there are at least two partially independent representations of a decision variable in the parietal cortex during reach trials.
The representation of a decision variable in two cortical areas supports the idea that analysis of sensory information occurs in circuits associated with provisional behavioral responses to stimuli (Schall, 2001; Romo et al., 2004; Rorie and Newsome, 2005; Gold and Shadlen, 2007; Lemus et al., 2007; Andersen and Cui, 2009; Cisek and Kalaska, 2010; Hernández et al., 2010; Westendorff et al., 2010), which has been termed an intentional—as opposed to representational—framework for information processing (Shadlen et al., 2008; Cisek and Kalaska, 2010). We do not want to postulate that LIP and MIP are areas specialized for representing eye- and arm-related decisions, nor that they are the only structures doing so. It is also important to note that our experiments were not designed to test effector specificity, and our results might be specific to LIPv and anterior MIP (Liu et al., 2010). However, we find noteworthy that these two parietal areas represent a computation related to those decisions, and we speculate that their activity is relevant to processing in structures to which LIP and MIP project. This is not the type of parallel processing that is often associated with neural networks (Rumelhart and McClelland, 1986) or computer systems but is instead a direct representation (Barlow, 1995) of a decision variable in the firing rates of neurons. Such parallel processing raises intriguing issues. Might it be possible for two areas to disagree on an answer or are they necessarily coordinated by the stimulus, sensory representation, or some other common input? A definitive answer awaits simultaneous recording from both areas.
Footnotes
M.N.S. is supported by the Howard Hughes Medical Institute, the National Eye Institute, and the Human Frontier Science Program. V.d.L. is supported by National Council of Science and Technology and General Directorate of Academic Staff Affairs. We thank Maria McKinley, Alicia Boulet, and Katie Ahl for technical assistance.
- Correspondence should be addressed to Dr. Michael N. Shadlen, Department of Neuroscience, College of Physicians and Surgeons, Room 16-409, Mail Box 102A, 630 West 168th Street, New York, NY 10032. shadlen{at}columbia.edu
