Abstract
Categorization is essential for interpreting sensory stimuli and guiding our actions. Recent studies have revealed robust neuronal category representations in the lateral intraparietal area (LIP). Here, we examine the specialization of LIP for categorization and the roles of other parietal areas by comparing LIP and the medial intraparietal area (MIP) during a visual categorization task. MIP is involved in goal-directed arm movements and visuomotor coordination but has not been implicated in non-motor cognitive functions, such as categorization. As expected, we found strong category encoding in LIP. Interestingly, we also observed category signals in MIP. However, category signals were stronger and appeared with a shorter latency in LIP than MIP. In this task, monkeys indicated whether a test stimulus was a category match to a previous sample with a manual response. Test-period activity in LIP showed category encoding and distinguished between matches and non-matches. In contrast, MIP primarily reflected the match/non-match status of test stimuli, with a strong preference for matches (which required a motor response). This suggests that, although category representations are distributed across parietal cortex, LIP and MIP play distinct roles: LIP appears more involved in the categorization process itself, whereas MIP is more closely tied to decision-related motor actions.
Introduction
Visual categorization is the ability to classify visual stimuli into discrete functional groups and is an essential component of recognition and decision making. Both prefrontal (PFC) and posterior parietal (PPC) cortices (Freedman et al., 2001; Freedman and Assad, 2006; Goodwin et al., 2012) are believed to contribute to the categorization process, because neurons in these areas can encode the category membership of stimuli. Recently, we showed that category signals are stronger and develop sooner in the lateral intraparietal area (LIP), a subdivision of PPC, compared with PFC during a motion-categorization task (Swaminathan and Freedman, 2012), raising the possibility that PPC is centrally involved in the categorization process. This raises two questions. (1) Are category signals during this task specific to area LIP within the PPC? (2) If categorization is distributed across PPC, what are the roles of the various subdivisions? This study examines whether activity in the medial intraparietal area (MIP) also reflects the category membership of stimuli and compares its role with LIP.
Most neurophysiological work on MIP (a subdivision of Brodmann's area 5) has focused on motor-related functions (Mountcastle et al., 1975; Johnson et al., 1996; Calton et al., 2002; Maimon and Assad, 2006; Cui and Andersen, 2007). Although non-motor cognitive signals have not been reported previously in MIP, there are several reasons that MIP could encode such signals. First, other motor-related cortical areas, such as premotor (Wallis and Miller, 2003; Hauk et al., 2004; Romo et al., 2004; Lemus et al., 2009) and motor (Wannier et al., 1989; Carpenter et al., 1999) cortices, can reflect non-motor task-relevant information. Second, MIP is interconnected with LIP (Lewis and Van Essen, 2000), which can reflect cognitive factors such as categories (Freedman and Assad, 2006) and rules (Stoet and Snyder, 2004). Third, because monkeys signaled their decisions with arm movements (which are known to drive MIP activity) in our experiment, it is possible that MIP lies along the neural pathway that transforms the stimulus into a category representation and then into a behavioral response.
We recorded from LIP and MIP neurons in two monkeys as they performed a delayed match-to-category (DMC) task. To receive a reward, monkeys were required to release a touch bar if the test category matched that of a previous sample. Interestingly, the activity of many LIP and MIP neurons reflected the category of stimuli. We found that both category and direction signals were significantly stronger and appeared with a shorter latency in LIP compared with MIP. Furthermore, during the test period, LIP encoded the category of both the remembered sample and currently visible test stimuli. In contrast, MIP more strongly differentiated between matching and non-matching test stimuli, with most neurons responding preferentially to matches (which required a motor response). Together, our results show that neuronal category signals are distributed across multiple PPC regions, but that these regions show a degree of specialization; LIP likely lies earlier on the pathway that transforms visual feature tuning into more abstract categorical representations, whereas MIP may be more involved in transforming category signals into task-appropriate motor responses.
Materials and Methods
Behavioral task and stimulus display.
The LIP data discussed in this study include neurons from a previous study. Monkeys were trained to indicate whether a test stimulus was in the same category as a previously presented sample stimulus by releasing a touch bar. Monkeys were free to respond with the hand either contralateral or ipsilateral to the recording chamber, although both monkeys had a clear preference for using the contralateral hand. We did not record which hand the monkeys used on each trial or restrain either hand during task performance. The motion stimuli were high-contrast, 9°-diameter, random-dot movies composed of 190 dots per frame, which moved at 12°/s with 100% coherence, and were displayed at a frame rate of 75 Hz. Monkeys maintained gaze fixation within a 2° radius of a fixation point. Identical stimuli, timings, and rewards were used for LIP and MIP recordings and for the two monkeys.
Monkeys grouped 360° of motion directions into two categories defined by a learned category boundary as described previously (Fig. 1A). Six evenly spaced (60° apart) motion directions were used as sample and test stimuli in addition to two directions that were on the category boundary and had ambiguous category membership. The monkeys identified the category of the motion stimulus within a DMC task (Freedman et al., 2001; Freedman and Assad, 2006; Fig. 1B). After gaze fixation was maintained for 500 ms, a sample stimulus was presented for 650 ms, followed by a 1000 ms delay and a 650 ms test stimulus. If the category of the sample and test stimuli matched, the monkey had to release a manual touch bar, using either the contralateral or ipsilateral hand to receive a juice reward. If the category of the sample and first test stimuli did not match, monkeys held the touch bar for the duration of a second delay (150 ms), after which a second test stimulus, which was always a category match to the sample, was presented and required a touch-bar release. The purpose of the second test stimulus was to ensure that the monkeys had to make a motor response on every trial. Gaze positions were measured using an EyeLink 1000 optical eye tracker (SR Research) at a sampling rate of 1 kHz and stored for offline analysis. Monkeylogic software (http://www.monkeylogic.net) was used to control task events, stimuli, and rewards and to monitor and store behavioral events (Asaad and Eskandar, 2008; Asaad et al., 2013). Stimuli were displayed on a 21-inch color CRT monitor (1280 × 1024 resolution, 75 Hz refresh rate, 57 cm viewing distance).
Electrophysiological recording.
Two male monkeys (Macaca mulatta, 8–10 kg) were implanted with a head post and recording chambers positioned over PPC and PFC. Stereotaxic coordinates for chamber placement were determined from magnetic resonance imaging (MRI) scans obtained before implantation of recording hardware. Areas LIP and MIP were accessed from a single chamber positioned over the intraparietal sulcus (IPS) centered ∼3 mm posterior to the intra-aural line. All procedures were in accordance with the University of Chicago Animal Care and Use Committee and National Institutes of Health guidelines.
LIP and MIP recordings were conducted in separate sessions using single 75 μm tungsten microelectrodes (FHC), a dura piercing guide tube, and a Kopf hydraulic microdrive system (David Kopf Instruments). There is some variability in our estimates for recording depths attributable to the location and angle of the parietal recording chambers, which led to different electrode penetration trajectories in the two monkeys. In general, LIP cells were found at more lateral locations and MIP cells were found at more medial locations within the same recording chamber. LIP was ∼2–7 mm below the surface and MIP was 1–5 mm below the surface in both monkeys. Neurophysiological signals were amplified, digitized, and stored for offline spike sorting (Plexon) to verify the quality and stability of neuronal isolations. Recordings were made from either LIP or MIP during each session.
During offline spike sorting, waveform signals corresponding to single units were identified and assigned a subjective score from 1 to 5, in which 5 indicates unequivocal recordings from a single neuron and that not a single action potential was lost and no noise was included during the session, whereas 1 indicates that the unit is likely a single neuron, but significant interference from noise and waveforms from other neurons and loss of spikes has occurred. Only well-isolated single units were included in subsequent analysis (mean score: LIP, 3.9; MIP, 4.0; units with scores <3 were not included in the population).
Receptive field mapping and stimulus placement.
All 76 of 76 LIP and 78 of 94 MIP neurons were tested with a delayed memory-saccade task. LIP neurons were identified by spatially selective visual responses during target presentation and/or delay activity during the memory-saccade task. MIP neurons were identified by responses during the animals' spontaneous movements, such as touch-bar releases, scratching, or arm movements observed before the DMC task commenced, and the absence of modulation during the memory-saccade task. LIP and MIP neurons were also differentiated based on anatomical criteria, such as the location of each electrode track relative to that expected from the magnetic resonance images, the pattern of gray–white matter transitions encountered on each electrode penetration, and the relative depths of each neuron. LIP and MIP neurons were not prescreened for direction or category selectivity before running the DMC task.
Responses during the memory-saccade task were analyzed in real time to estimate the position of LIP receptive fields (RFs) and to guide DMC stimulus placement. The typical eccentricity of stimulus placement for LIP recordings was ∼6–10°. MIP responses during the memory-saccade task were not effective in guiding DMC stimulus placement because this task evoked little modulation in these neurons (see Fig. 7B,C). Thus, the DMC-task stimulus during MIP recordings was always placed at 7° from the fixation along the horizontal axis contralateral to the recording hemisphere.
Data analyses.
The patterns of behavioral and neuronal results were similar, and similar effects were observed in both monkeys. Thus, the two datasets were combined for all population analyses.
For epoch-based analyses, firing rates were computed for each neuron in four analysis epochs used to measure sample category selectivity (Swaminathan and Freedman, 2012) and one analysis epoch used to measure test category selectivity. The fixation epoch was 500 to 0 ms before sample onset. The sample epoch was 650 ms in duration, beginning 80 ms after stimulus onset (to account for response latency). The delay epoch began 300 ms after stimulus offset and lasted for 800 ms. The test epoch was 300 ms in duration beginning at test onset. To measure test category selectivity, activity from 80 to 300 ms after test stimulus onset was used. Only neuronal activity from correct trials was analyzed.
Because neurons in LIP and MIP were selected for recording based on different criteria (delayed memory saccade modulation for LIP, arm movement modulation for MIP), it was essential to apply a common inclusion criteria for population-level neural analysis to minimize the introduction of any biases. Specifically, we were concerned with the possibility that any difference in the strength and/or latency of category selectivity was an artifact of the different criteria used to select LIP and MIP neurons for recording. This would occur if our selection criteria for recording resulted in different proportions of LIP and MIP neurons that were direction or category selective. Thus, we only included category-selective cells (as defined below) in our analysis to help mitigate this possible bias. We note that foregoing the common selection criteria for analysis and including all neurons does not qualitatively change our results (data not shown).
To compare the strength of category selectivity across areas, we only included those neurons that were sample category selective during the sample, delay, and/or test epochs, or test category selective (during the test epoch), according to a Wilcoxon's rank-sum test (p < 0.05). This relatively relaxed selection criterion was meant to capture all neurons whose activity was at all modulated by the category of the sample stimulus. Seventy-two of 76 LIP neurons and 64 of 94 MIP neurons satisfied this criterion.
To measure the latency of sample category selectivity (see Fig. 5), we focused on neurons that showed strong category selectivity in the early sample period. Thus, we only used neurons that were category selective during the early sample (80–300 ms after sample onset) according to a Wilcoxon's rank-sum test (p < 0.01). To measure the latency of test category selectivity, we included neurons selective during the test epoch (80–300 ms after test onset) according to a Wilcoxon's rank-sum test with significance level of p < 0.05. We relaxed the level of significance (compared with the sample category latency analysis) to include a sufficient number of selective neurons in each area for the analysis. Lastly, for the analysis that measured how activity aligned to stimulus onset and the behavioral response (see Fig. 6), we focused on those neurons that were test category selective or differentiated between match/non-match trials during the test epoch, both according to a Wilcoxon's rank-sum test (p < 0.05). We note that omitting the cell selection criteria and simply including all neurons in every analysis did not qualitatively affect any of the key results of this study (e.g., presence of category selectivity in both areas and the observed differences in strength and latency of selectivity between areas).
Category selectivity.
The strength and time course of category selectivity was evaluated using a receiver operating characteristic (ROC) analysis to produce a category tuning index (CTI). The ROC analysis was applied to the distribution of firing rates across trials during each analysis epoch. The area under the ROC curve represents the amount of overlap between the distributions of firing rates for the two categories and has a value between 0.5 (complete overlap) and 1.0 (complete separation). It indicates the performance of an ideal observer in assigning category membership based on the trial-by-trial firing rates of each neuron; values of 0.5 and 1.0 correspond to chance performance and perfect category assignment, respectively. To compare neuronal discrimination between versus within categories, an ROC-based CTI (rCTI) was used to measure the differences in ROC values between pairs of directions in different categories [the between-category discrimination (BCD)] and the same category [the within-category discrimination (WCD)]. Specifically, defining ROX(x,y) as the area under the ROC curve for the firing rate distributions generated by the motion stimuli with directions x and y and noting that motion directions 255°, 315°, and 15° belonged to category 1 whereas motion directions 75°, 135°, and 195° belonged to category 2, the rCTI was defined as follows:
where we subtract 0.5 and apply the absolute value (|… |) to rectify the ROC scores. To prevent any direction tuning from unequally affecting the WCD and BCD metrics, the WCD and BCD metrics contained an equal weighting of 60° and 120° ROC comparisons. Because two of six pairs of directions in the WCD metric but four of six pairs of directions in the BCD metric are 120° apart, we multiplied the 120° comparisons in the WCD and the 60° comparisons in the BCD by two. Furthermore, because there were no directions 180° apart that belonged to the same category, neither the WCD or BCD metric contained any 180° ROC comparisons.
Values of the rCTI could vary from +0.5 (strong binary-like differences in activity to directions in the two categories) to −0.5 (large activity differences between directions in the same category, no difference between categories). An rCTI value of 0 indicates the same difference in ROC values between and within categories. Very similar results were obtained using analytical approaches used in previous studies of categorization, including the ROC that compared firing rates on all trials for each category and a CTI applied to average firing rates for each direction (Freedman et al., 2001; Freedman and Assad, 2006; Swaminathan and Freedman, 2012). The rCTI has the advantage of taking into account both the differences in neuronal activity with respect to the category boundary and the reliability of category selectivity from a readout perspective.
We performed a shuffle analysis to determine whether the rCTI score of a neuron for a specified epoch was significantly above 0, indicating category selectivity. To obtain a null distribution, we shuffled the direction labels, calculated the rCTI, and repeated the process 1000 times. The rCTI value was deemed significant if it was >95% of values from the null distribution. This is equivalent to a one-tailed test, which we used because neurons could only be category selective if their rCTI value was significantly greater, but not less than, the shuffle distribution.
Calculating the latency of category selectivity using rCTI analysis.
To calculate when category selectivity developed according to an rCTI analysis, we first computed rCTI values for each neuron using a 100 ms analysis window that was stepped in 5 ms intervals. For each neuron, we defined a category selectivity threshold as three times the SD of the rCTI values calculated during the fixation period. The latency of category selectivity was defined as the first time bin at which the rCTI value exceeded this threshold for two consecutive time intervals. Because we were primarily interested in when category selectivity develops in response to the visual stimulus, latencies >500 ms after sample onset were discarded.
Using population readout to decode category and direction information from a surrogate population.
We wanted to quantify the time at which LIP and MIP began to encode the direction and the category membership of the motion stimulus after stimulus onset. To do so, we used a linear classifier to decode direction and category membership from a surrogate population of LIP and MIP neurons (Meyers et al., 2008, 2012). In this surrogate population, responses from different neurons are treated as if they occurred simultaneously although neurons were, for the most part, recorded one at a time. Because we were interested in the time at which category information first appeared in these two areas, we only included 41 LIP and 21 MIP neurons that were category selective during the early portion of sample epoch (80–300 ms after sample onset) according to a Wilcoxon's rank-sum test at p < 0.01 level of significance. We applied a more conservative threshold (p < 0.01 vs p < 0.05) compared with previous criterion to ensure that only neurons that were strongly selective during this epoch were included.
To determine whether latencies were significantly different between the two areas, we needed to generate distributions of latency values for both areas. Given the 41 LIP and 21 MIP neurons that satisfied the above criteria, we formed two neuronal populations by sampling, with replacement, 41 LIP and 21 MIP neurons (we performed a control to ensure the different population sizes did not bias our results; see Results). For each neuron, we then randomly selected 75% of trials to train the classifier, whereas the other 25% of trials would be used for testing. We then randomly sampled, with replacement, 150 trials (25 trials for each direction) from the training list and 150 trials for the testing list. Our classifier was trained and tested at 5 ms intervals, using the spike count in the proceeding 100 ms. Thus, for each time point, a trial consisted of a coordinate in a 41-dimensional (for LIP) or 21-dimensional (for MIP) space, in which the position along each axis was given by the spike count of the 41 (for LIP) or 21 (for MIP) neurons. We then randomly selected a different 75% of trials for training and repeated training and testing our linear classifier at 5 ms intervals. We repeated this step 100 times; thus, for every 5 ms time point, there were 100 measures of the classification accuracy. We defined the latency as the time relative to sample onset at which at the classifier accuracy was above chance for >99 of the 100 measures for two consecutive time points. We then sampled, with replacement, a different LIP and MIP neuronal population and repeated the latency calculation. This procedure produced 100 latency values for both areas LIP and MIP.
The linear classifier was trained using a support vector machine. Spike counts were not normalized, although similar results were obtained when normalizing spike counts (data not shown). To ensure that our results generalized to different classification methods, we also repeated our procedure using a maximum correlation coefficient classifier (Meyers et al., 2012) and found that it did not qualitatively affect our results (data not shown).
By training a linear classifier, one determines a hyperplane that best separates the trials belonging to two different classes. In the case of decoding category membership, each class corresponds to one category. However, we did not want to use all trials with motion directions from one category to form class 1 and trials with motion directions from the other category to form class 2. Similar to the rCTI analysis, we wanted to measure how accurately the neuronal populations encoded category membership while controlling for direction selectivity among the six directions. Thus, to train the classifier, for one class we used all trials with sample directions that were 60° and 120° from the sample direction of the trial to be decoded and that belonged to the same category. For the other class, we used all trials with sample directions 60° and 120° from the sample direction of the trial to be decoded and that belonged to the opposite category. Each class consists of 50 trials (25 from each direction), and only directions 30° from the boundary were decoded (only for these directions were there two other directions that were equidistant yet in opposite categories).
As an example, to decode the category of a trial with a direction of 15° (see Fig. 5A, dashed black arrow), one class of data points consisted of those trials belonging to the same category with directions of 255° and 315° (blue arrows), whereas the other class of data points consisted of trials from the opposite category with directions 75° and 135° (red arrows).
Although we trained the classifier using trials with motion directions that were equidistant (yet in opposite categories) to the direction of the trial that was to be decoded, our readout of category information could still be affected to some degree by direction tuning, in principle. For example, neurons with preferred directions perpendicular to the boundary will tend to increase the performance of the category decoder, whereas neurons with preferred directions parallel to the boundary will tend to subtract from it. However, given a population of neurons with randomly distributed preferred directions, the directional effects tend to cancel each other out, giving chance levels for reading out category information (see below).
We also wanted to measure how accurately the LIP and MIP populations encoded sample direction independent of category membership. Thus, linear classifiers were trained using only trials belonging to the same category. For example, to decode the directions of trials with an actual direction of 15° (see Fig. 5B, black dashed arrow), the classifiers were trained only using trials with motion directions of 15°, 255°, and 315° (blue arrows), which all belong to category 1.
One complication is that linear classifiers can only separate trials into one of two classes, whereas we wanted to decode the one direction of the three from that category that was most likely. Thus, we trained three different linear classifiers, in which each distinguished one direction from the other two (Vapnik, 1999). The most likely direction was then determined by which classifier produced the greatest score (the Euclidian distance from the test point to the separating boundary).
Although we designed our two classifiers to decode category and direction information independently, we wanted to confirm this property. To do so, we created a synthetic population of 100 Poisson neurons, each with baseline rates of 15 Hz. We tested our two decoders on four different conditions: (1) neurons were not tuned to direction or category; (2) neurons possessed direction, but not category, tuning; (3) neurons possessed category, but not direction, tuning; and (4) neurons possessed both direction and category tuning. Direction tuning was modeled as a cosine function with amplitude of 5 Hz and a random phase. Thus, preferred directions across the population were randomly distributed. Category tuning was modeled as an increase in spike rate by 5 Hz during presentation of the preferred category and a decrease by 5 Hz for the null category; category preference was randomly chosen.
As expected, both decoders performed at chance when no tuning was present (data not shown). When neurons were only direction tuned, the direction classifier was capable of decoding the direction while the category classifier performed at chance. Similarly, when neurons were only category tuned, the category classifier could decode category while the direction classifier performed at chance. When neurons were both direction and category tuned, both classifiers were able to decode their respective feature, although accuracy decreased slightly compared with the case in which neurons were only tuned for one feature, presumably attributable to the additional variability added by the combination of both signals. We address several caveats of this analysis in Discussion.
Assessing the link between neural activity with stimulus onset and the behavioral report.
We examined how activity in both areas was linked to stimulus onset and the behavioral responses. For the presentation of population firing rates aligned to stimulus onset and the behavioral response in Figure 6, A and B, the average firing rate of each neuron was divided by the mean rate calculated in the 500 ms before the test onset, plus 0.5 Hz to prevent noise from low firing neurons from dominating the mean response. Because we were more interested in the magnitude of change in neural activity during the test period as opposed to whether it increased or decreased, we vertically flipped all neural responses that decreased during the period from 80 to 300 ms after test onset. For the analysis in Figure 6, C and D, the firing rate of each neuron was normalized by the same value used for Figure 6, A and B.
To determine whether neural activity was more tightly linked with visual stimulus onset or the behavioral report, we performed multiple linear regression to measure how neural activity varied as a function of two factors: (1) time since test onset; and (2) time until bar release. Specifically, we modeled firing rate, fr, as follows:
where tvisual and tmotor are the time since test onset and time until bar release, respectively. We fit the coefficients Rvisual, Rmotor, and baseline on a trial-by-trial basis using only neural activity that occurred from 50 to 250 ms after test onset and from 50 to 250 ms before bar release. Only trials in which the sample category matched the first test category were used.
Characterizing neuronal activity during the memory-guided saccade task.
To map out the location of RFs of LIP neurons, we had monkeys perform a memory-guided saccade task. Briefly, monkeys had to maintain fixation on a central point for 500 ms, followed by a visual target presentation in one of eight locations for 300 ms, and finally by a delay period of 1000 ms. The central fixation point was then extinguished, and the monkey had to saccade to the location of the remembered visual target. To characterize the spatial selectivity of LIP and MIP neurons during the memory saccade task, we calculated a spatial selectivity index, which was based on a metric used previously to characterize orientation tuning (Vogels and Orban, 1994). The spatial selectivity index is a value between 0 and 1, where 0 indicates no spatial selectivity and 1 indicates perfect spatial selectivity. For each of the eight saccade target directions, characterized by the unit vector vi, where i runs from 1 to 8, we calculated the mean firing rate of the neuron from 80 to 380 ms after target onset, given by Ri. The spatial selectivity index of the neuron is then given by the following:
Results
The goal of this study is to characterize the relative contributions of LIP and MIP in visual category encoding and solving the DMC task. Analytically, this comparison is primarily based on three different metrics: (1) the strength of category selectivity; (2) the latency required for category selectivity to develop; and (3) whether neuronal activity is more closely linked to the onset of the visual stimulus or the monkeys' behavioral reports.
Behavioral paradigm
We trained two monkeys to group six directions of motion into two categories separated by a category boundary (see Materials and Methods; Fig. 1A). We assessed the monkeys' abilities to categorize motion directions in a DMC task, in which two motion stimuli (referred to as the sample and test stimuli) separated by a delay are presented. To receive a reward, the monkeys had to release a touch bar if the category of the two stimuli matched (Fig. 1B). If the two stimuli did not match, a third motion stimulus was presented whose category always matched that of the first; the monkeys were required to release a touch bar during this third stimulus to receive a reward. Because different levels of behavioral performance between sessions could confound the comparison between LIP and MIP, both monkeys were extensively trained, including directions closer to the boundary than those used during recordings, which only commenced after performance had plateaued. The monkeys correctly categorized the six motion directions in either category with >90% accuracy (Fig. 1C,D). Importantly, there was little variation in accuracy between LIP (Fig. 1C) and MIP (Fig. 1D) recording sessions for trials with sample category 1 (LIP mean accuracy = 0.954, MIP mean accuracy = 0.950, p = 0.91, Wilcoxon's rank-sum test) or for trials with sample category 2 (LIP mean accuracy = 0.930, MIP mean accuracy = 0.956, p = 0.48). Accuracy was slightly greater during MIP sessions for monkey 2, whereas there was no significant difference for monkey 1 (monkey 1: LIP = 92.9%, MIP = 92.2%, p = 0.19; monkey 2: LIP = 95.6%, MIP = 97.7%, p = 0.017, Wilcoxon's rank-sum test). Although the differences in accuracy between LIP and MIP recording sessions were small, it is possible that increased accuracy could be associated with increased neural selectivity. However, we found that selectivity was stronger in LIP, not MIP (see below). Thus, differences in selectivity between the two areas are unlikely to be related to differences in behavioral performance during LIP and MIP recording sessions.
Task design and behavioral performance. A, Monkeys were trained to categorize six directions of motion into one of two categories (denoted by the blue and red arrows). The boundary separating the two categories is shown by the dashed black line. B, Monkeys reported the category of the motion stimulus within a DMC task, releasing a touch bar if the sample and test stimuli belonged to the same category. If the category of the first test stimulus did not match the sample, a second test stimulus was shown (following a 150 ms delay) that was always a category match. The RF of the neuron is indicated by the dashed circle. C, Bar plot showing the percentage of trials for which the monkeys reported that the sample stimulus belonged to category 1 averaged across all LIP sessions. Performance was >90% for all conditions. Error bars indicate SE. D, Same as C, except for all MIP sessions.
Neural response during the DMC task
We recorded from 76 LIP and 94 MIP neurons in two monkeys during the DMC task. Given the different functional properties of LIP and MIP revealed by previous studies (see Introduction), we would expect that the two areas would respond preferentially to different aspects of the DMC task. Specifically, given the known visual responsiveness and category selectivity of LIP, we would expect modulation during the presentation of sample and test stimuli and memory delay periods. For MIP, which is known to respond to motor actions, we would expect modulation during the test epoch of match trials, when the monkeys were required to release a touch bar to indicate their “match” decisions. We confirmed this by calculating the average firing rate (normalized to the mean rate during the fixation period) across the population of all category-selective (see Materials and Methods) LIP (Fig. 2A, black curve) and MIP (gray curve) neurons for match trials. The firing rate across the LIP population increased during both the sample and test period, whereas the activity in MIP only increased during the test period. This neural response pattern was similar for non-match trials in LIP (Fig. 2B), whereas there was a much weaker increase in MIP activity during the test epoch. This was consistent with the notion that activity in MIP is linked to the motor response that occurs during the first test period of match, but not non-match, trials.
Average normalized response of LIP and MIP neurons. A, The neural activity averaged across all category-selective LIP (black curve) and MIP (gray curve) neurons, smoothed with a causal, 200 ms boxcar filter, is shown for match trials. The response is normalized so that the mean activity during the fixation period is one. Shaded area indicates SE. B, Same as A, except calculated using non-match trials.
Although the above results appear to confirm the known properties of the two areas, one question is whether MIP activity shows selectivity for non-motor aspects of the DMC task—in particular, category selectivity like that shown by several previous studies in LIP. If so, a key issue is to examine the relative encoding of category signals between LIP and MIP. Three examples of category-selective LIP neurons from the current study are shown in Figure 3A–C. The example LIP neuron in Figure 3A preferentially responded to motion directions from category 2 in an almost binary-like manner, in which the neural response to all three motion directions from category 2 (red curves) were greater than the responses to the three directions from category 1 (blue curves). Two other category-selective LIP neurons are shown in Figure 3, B and C; these neurons preferentially responded to motion directions from category 1. Across the population of 72 LIP neurons that were category selective in at least one epoch (see Materials and Methods), 74% neurons showed significant category selectivity during the sample epoch, 78% during the delay epoch, and 65% during the test epoch according to a Wilcoxon's rank-sum test at p < 0.05 significance. The fraction of category-selective cells in each area is reported in Table 1. Additionally, we found that approximately equal numbers of category-selective LIP neurons responded more strongly to either category during the sample (monkey 1: 50% of neurons preferred category 1, p = 1.00; monkey 2: 64%, p = 0.088, binomial test), delay (monkey 1: 43%, p = 0.59; monkey 2: 64%, p = 0.088), and test (monkey 1: 60%, p = 0.36; monkey 2: 43%, p = 0.44) epochs. The mean difference in firing rates between preferred and nonpreferred categories is given in Table 2.
Example responses of LIP and MIP neurons. A–C, The average neural responses of three LIP neurons are shown. The response to the three motion directions from category 1 are shown in blue, and the response to the three motion directions from category 2 are shown in red. The subpanels to the right of each plot show the activity of the neurons for match (solid traces) and non-match (dotted traces) trials. D–F, Same as A–C, except showing the response of three MIP neurons.
Sample category selectivity
Differences in firing rates between preferred and nonpreferred categories
Consistent with past studies demonstrating a role for MIP in motor actions, many MIP neurons (such as the neuron in Fig. 3D) showed strong responses during the test period, when the monkeys initiated their touch-bar-release responses. MIP neurons displayed stronger responses during match trials (Fig. 3D–F, right subpanels, red and blue solid traces), which required a manual bar release, compared with non-match trials (red and blue dotted traces). Interestingly, we also encountered a population of MIP neurons, such as the examples in Figure 3, E and F, that showed category selectivity that was qualitatively similar to that seen in LIP. Across the population of 64 MIP neurons that were category selective in at least one epoch, 55% of neurons showed significant category selectivity during the sample epoch, 64% during the delay epoch, and 52% during the test epoch (Table 1). Similar to LIP, we found that approximately equal numbers of category-selective MIP neurons responded more strongly to either category during the sample (monkey 1: 52% of neurons preferred category 1, p = 1.00; monkey 2: 54%, p = 0.74, binomial test) and delay (monkey 1: 41%, p = 0.44; monkey 2 = 49%, p = 1.00) epochs, although there did exist a bias for the test epoch (monkey 1: 70%, p = 0.052; monkey 2: 73%, p = 0.008). Overall, these results suggest that abstract category signals, which are distinct from the monkeys' planned motor responses in the DMC task, are evident in MIP as well as LIP. Next, we compare the strength of neuronal category signals in LIP and MIP.
Category selectivity across LIP and MIP
We used an rCTI that compared neuronal discrimination between pairs of directions in the same versus different categories (see Materials and Methods). Index values of +0.5 indicate strong category selectivity: perfect neuronal discriminability between pairs of directions in different categories and no discriminability between directions in the same category. Values of −0.5 indicate the opposite.
Because we were primarily interested in how category selectivity varied between the two areas, we only included the 72 of 76 LIP and the 64 of 94 MIP neurons that were sample category selective during the sample, delay, or test epoch, or selective for the test category, according to a Wilcoxon's rank-sum test with significance level of p < 0.05 (see Materials and Methods). Within this population, the mean rCTI values were significantly >0 (indicating a tendency for category selectivity) for both the population of LIP and MIP neurons for the sample (Fig. 4A, LIP = 0.069, p = 10−5; MIP = 0.038, p = 10−4, two-tailed t test), delay (LIP = 0.112, p = 10−11; MIP = 0.066, p = 10−6), and test (LIP = 0.090, p = 10−9; MIP = 0.036, p = 10−7) epochs. We also note that these results held for each monkey (data not shown). rCTI values were significantly >0 for 53% of LIP and 39% of MIP neurons during the sample, 71% LIP and 53% of MIP neurons during the delay, and 64% LIP and 44% of MIP neurons during the test epoch (p < 0.05, shuffle analysis; see Materials and Methods; Table 1). In summary, a substantial fraction of neurons in both LIP and MIP reflected the category membership of sample stimuli.
Strength of sample and test category selectivity. A, The mean sample category rCTI is shown for both LIP (black bars) and MIP (gray bars) for the fixation, sample, delay, and test epochs. The average is taken across all cells that were category selective for either the sample or test stimuli (see Materials and Methods). B, The mean test category rCTI is shown for the test epoch. Bars on the left show the mean across all cells that were category selective for the sample or test stimuli, whereas the bars to the right are the mean across all cells selective for just the test stimulus. Error bars indicate SE.
Although both areas encoded the category of sample stimuli, we found that category selectivity was greater in LIP compared with MIP. LIP rCTI values were significantly greater in LIP for the sample (p = 0.048, two sided, unpaired t test), delay (p = 0.010), and test (p = 0.001) epochs. rCTI values were greater in LIP for both monkeys across all epochs, although the differences were not significant for the sample and delay epochs for one monkey or for the sample epoch for the other (data not shown).
We next examined whether both LIP and MIP encoded the category of the test stimulus during the test epoch. Because MIP showed the strongest firing rate modulations during the test epoch (around the time of the motor response; see Fig. 2 and below), one possibility is that motor-related signals might dominate and swamp category signals during the test epoch in MIP. We calculated rCTI values for the test stimulus using spike counts from 80 to 300 ms after test onset (Fig. 4B), across the same populations of LIP and MIP used for the sample category analyses above. We found that LIP, but not MIP, showed significantly elevated test category rCTI values (LIP = 0.012, p = 0.012; MIP = 0.003, p = 0.36, two-tailed t test), although the difference in rCTI values between the two areas was not significant (p = 0.17, unpaired, two-tailed t test). We also examined the strength of test category selectivity among the subpopulation of test category-selective neurons during the test epoch (Wilcoxon's rank-sum test comparing activity with the two test categories, p < 0.05; LIP: n = 28; MIP: n = 11). Across this population, both areas showed significantly elevated rCTI values (LIP = 0.048, p = 0.0001; MIP = 0.034, p = 0.005, Wilcoxon's rank-sum test), although the difference between areas was not significantly different (unpaired t test, p = 0.38). The fraction of test category-selective cells for both areas is reported in Table 3.
Test category selectivity and match/non-match effects
If motor-related responses dominate MIP activity during the test period, we would expect MIP to be more selective to whether the trial was match/non-match (i.e., motor response/no motor response) compared with its test category selectivity. To test this, we performed a two-way ANOVA between the spike count from 80 to 300 ms after test onset with the following factors: (1) the test category; and (2) whether the test matched the sample. Consistent with the above results, the majority of MIP neurons (47 of 64) were significantly (p < 0.05) modulated by the match/non-match status of the test stimulus, whereas only 11 of 64 neurons were significantly modulated by the test category. Twenty-four of 64 showed a significant interaction between the two factors. In contrast, 37 of 72 LIP neurons significantly modulated according to the match/non-match status of the test stimulus, 33 of 72 neurons were significantly modulated by the test category, and 35 of 72 showed an interaction between the two factors. Of the 47 MIP neurons that were significantly modulated by the match/non-match status, 42 responded more strongly to match trials, whereas only 22 of 37 match/non-match-selective LIP neurons preferentially responded to match trials. This suggests that MIP activity is mostly dependent on whether the test stimulus matched the sample (which triggered the monkeys' touch-bar-release responses), whereas the LIP response was more dependent on both the match/non-match status as well as category of test stimuli.
It is possible that other factors aside from category selectivity could contribute to the greater rCTI values observed in LIP compared with MIP. For example, the ROC calculations that underlie the rCTI analysis measure the overlap in spike count distributions between trials of different conditions; low-firing rate neurons might produce spike count distributions with greater overlap, biasing the rCTI value toward 0. Thus, we needed to confirm that the observed differences in rCTI values between LIP and MIP were not because of greater LIP firing rates (sample: mean LIP firing rate = 17.3 Hz, mean MIP firing rate = 12.3 Hz; delay: LIP = 13.0 Hz, MIP = 13.3 Hz; test: LIP = 22.9 Hz, MIP = 17.5 Hz). To do so, we performed an analysis of covariance to test for a significant difference in the rCTI values between LIP and MIP while accounting for the mean firing rates. We found a significant difference in rCTI values between the two areas (sample, p = 0.048; delay, p = 0.010; test, p = 0.003), whereas there was no effect of the mean firing rates on the rCTI values (sample, p = 0.86; delay, p = 0.93; test, p = 0.56).
Although the monkeys could not predict (before test onset) whether a test stimulus would be a categorical match to the sample (requiring an immediate motor response), it is possible that the monkeys used different default motor strategies in response to the two sample categories and that these different covert motor plans might result in differential neuronal activity to the two categories (and rCTI values significantly >0). Evidence for different motor strategies could be differences in reaction times (RTs) between the two sample categories. Indeed, there was a significant difference in RTs between trials with different sample categories for 50 of 72 LIP neurons and 38 of 64 MIP neurons (p < 0.05, paired t test), and mean RTs were slightly greater for category 1 than category 2 in both monkeys (monkey 1: category 1, 234.0 ms and category 2, 241.8 ms; monkey 2: category 1, 286.3 ms and category 2, 292.4 ms). To address this, we examined the subset of recording sessions for which no significant difference in RTs was found (22 of 72 LIP neurons and 26 of 64 MIP neurons). Mean rCTI values for neurons with no significant RT difference were relatively unchanged compared with the mean rCTI values from the entire population (LIP: neurons with no difference in RT, sample rCTI = 0.079, delay rCTI = 0.085, test rCTI = 0.076; full population, sample rCTI = 0.069, delay rCTI = 0.112, test rCTI = 0.090; MIP: neurons with no difference in RT, sample rCTI = 0.040, delay rCTI = 0.076, test rCTI = 0.040; full population, sample rCTI = 0.038, delay rCTI = 0.066, test rCTI = 0.036).
The possibility that the monkeys applied different motor strategies to each of the categories might also yield different neuronal match/non-match effects for the two categories. To examine this possibility, we quantified the degree of modulation for each category by computing the area under the ROC curve that compares the spike rates between match and non-match trials during the test epoch. ROC scores were then rectified: |ROC − 0.5|. We found that the differences in neural activity between match and non-match trials were not significant between categories 1 and 2 for both LIP and MIP (LIP-rectified ROC scores: category 1 = 0.159, category 2 = 0.138, p = 0.44; MIP-rectified ROC scores: category 1 = 0.200, category 2 = 0.192, p = 0.77, paired, two-sided t test). From these two analyses, we see no evidence that category selectivity could be explained by differences in motor plans for the two sample categories.
Latency of sample category selectivity
In a previous study, we showed that category signals developed with a shorter latency in LIP compared with PFC (Swaminathan and Freedman, 2012), suggesting that the initial encoding of category signals likely does not occur in PFC during this task. Although the results from Figure 4 showed that category selectivity is weaker in MIP, it does not rule out the possibility that category signals might arise in parallel in LIP and MIP or even earlier in MIP. Thus, to further understand the neural pathways that underlie the generation of category signals, we calculated the latency required for category signals to develop in both areas.
We chose to measure the latency using two different approaches. In the first approach, we determined the time at which rCTI values, calculated using spike counts in the preceding 100 ms window, crossed above a set threshold (see Materials and Methods). We found that the mean latency of category selectivity in LIP was significantly less than in MIP (mean LIP = 196.0 ms, mean MIP = 247.2 ms, p = 0.024, two-sided t test). We note that it is difficult to compare the precise latency values with those from our previous study (Swaminathan and Freedman, 2012) given the difference in the methods used. However, applying the same methods used in that study produces qualitatively similar results (data not shown). Although this result suggests that category selectivity first develops in LIP, we wanted to confirm this result using a population decoding approach. A population decoding approach provides two advantages over the rCTI approach. First, it can be difficult to accurately determine the time at which the firing rate of a single neuron changes, especially for low-firing cells. By combining the spiking activity across a population, one reduces the noise and improves accuracy when calculating the latency. Second, it gives a better estimate of the time at which category or direction selectivity first develops within each neuronal population as opposed to the average time neurons within each area become selective. For example, selectivity may quickly develop within a subpopulation but take a longer time to spread to other neurons. In this case, the average time at which selectivity develops for single neurons would be much greater than the time at which selectivity first appears within the cortical area. Last, our population decoding approach allowed us to measure direction and category selectivity in a relatively independent manner (see Materials and Methods).
In this method, we extracted category and direction information using liner classifiers trained on a surrogate population of neurons (see Materials and Methods; Meyers et al., 2008, 2012). To measure category selectivity while controlling for direction selectivity, a different linear classifier was trained for each direction. Only trials with directions 60° and 120° away from the direction we were classifying, both from the same and opposite category, were used to train the classifier. For example, to classify trials for which the direction was 15° (Fig. 5A, dashed black arrow), the classifier was trained using trials with directions 255° and 315° from the same category (blue arrows) and trials with directions 75° and 135° from the opposite category (red arrows). Thus, a population of neurons with cosine-like direction tuning and randomly distributed preferred directions would provide no category information (and would yield chance performance from the classifier).
Time course of category and direction selectivity. A, Diagram depicting how we classified the motion direction independent of any category selectivity. As an example, to classify the motion direction from the neural response to motion direction of 15° (dashed black arrow), the classifier was trained using trials with motion directions 15°, 255°, and 315° (blue arrows), which all belong to category 1. B, Diagram depicting how we classified the stimulus category independent of any direction selectivity. As an example, to classify the category from the neural response to motion direction of 15° (dashed black arrow), the classifier was trained using trials with motion directions 225° and 315° from the same category (blue arrows) and trials with motion directions 75° and 135° from the opposite category (red arrow). C, The accuracy in classifying the sample motion direction for LIP (black curve) and MIP (gray curve) as a function of time relative to sample onset. Black and gray horizontal bars indicate when decoding accuracy is significantly (p < 0.01) above chance according to a bootstrap analysis. Chance accuracy is 0.33. D, The accuracy in classifying the sample category direction for LIP (black curve) and MIP (gray curve) as a function of time relative to sample onset. Horizontal black and gray bars indicate when decoding accuracy is significantly above chance (p < 0.05, bootstrap). Chance accuracy is 50%. E, The cumulative latency distributions for direction selectivity to develop in LIP (dashed black curve) and category selectivity to develop in LIP (solid back curve) and MIP (solid gray curve). Because direction selectivity in MIP was weak, we could not obtain an accurate latency distribution. F, Same as D, except for the test category.
Given that direction selectivity in the middle temporal area (MT) may be transformed into the abstract category representations observed in LIP and MIP, we were interested in the time course of direction selectivity in both areas. In a similar vein, we also used a population decoding approach that measured direction selectivity independent of category selectivity (see Materials and Methods). To do so, we only classified motion directions from among the three directions belonging to the same category (see Materials and Methods). For example, to classify trials in which direction was 15° (Fig. 5B, dashed black arrow), the classifier was trained using trials with directions 15°, 255°, and 315° (blue arrows), all belonging to category 1. Thus, a neuron with binary-like category selectivity would provide no direction information (and chance performance from the classifier).
Classifiers were trained and tested in 5 ms intervals to obtain the time course of category and direction selectivity. Classifiers were trained on a randomly selected 75% of the trials and were tested on the remaining 25% of trials. This procedure was repeated 100 times to obtain a distribution of decoding accuracies for every time point. As with the rCTI latency calculation above, spike counts were smoothed with a causal 100 ms window. Because we are primarily interested in when category selectivity first developed in each area, we only included 41 LIP and 21 MIP neurons that showed significant category selectivity during the initial part of the sample epoch (80–300 ms after stimulus onset) according to a Wilcoxon's rank-sum test (p < 0.01). Different neuron-selection criteria or step sizes yielded qualitatively similar results.
This analysis revealed that direction selectivity (independent of category selectivity) was present in LIP (Fig. 5C, black curve, horizontal black bar denotes significance at p < 0.01 according to a bootstrap), reaching significance 85 ms after sample onset. Direction selectivity also reached significance in MIP (gray curve and horizontal bar) but at a much later time (320 ms after sample onset) and remained above significance only intermittently. Category selectivity was present in both LIP (Fig. 5D, black curve and horizontal bar) and MIP (gray curve and horizontal bar), reaching significance 135 and 160 ms after sample onset in each area, respectively.
The latencies stated above were obtained by resampling the trials used to train the classifier. However, it is not possible to determine whether these latencies are significantly different because they are only single values. Thus, we used another bootstrap procedure in which we also resampled the neurons used in the classifier to obtain a distribution of 100 latency values for direction and category selectivity for both areas (see Materials and Methods; Fig. 5E). Using this method, direction selectivity first developed in LIP at 115 ms, followed by category selectivity in LIP at 150 ms, and finally category selectivity in MIP at 186 ms. Because direction selectivity was only weakly significant in MIP, we could not obtain an accurate latency estimate using this method. The difference between direction and category selectivity latencies in LIP was not significant (p = 0.069, bootstrap), but category selectivity developed with a shorter latency in LIP compared with MIP (p = 0.013).
We note that the above criteria only included a single MIP neuron from monkey 1 for the classifier analysis. However, relaxing our criteria from p < 0.01 to p < 0.05, which includes five MIP neurons from monkey 1, did not qualitatively affect our results (LIP latency = 147 ms, MIP latency = 182 ms, p = 0.030, bootstrap). The latency was shorter in LIP for each monkey, although the difference was only significant for one monkey (data not shown). Furthermore, the results of the classifier-based latency analysis are qualitatively similar to those from the rCTI-based approach (above), which was computed across a larger population of neurons from both monkeys.
Controlling for possible biases in the sample category latency
We wanted to know whether the shorter latency observed for LIP was indeed because category information develops sooner in this area or because there were other differences between the LIP and MIP populations that biased our results. Specifically, we examined whether differences in category selectivity, firing rates, population sizes, or direction tuning biased our results.
As shown in Figure 4, the mean strength of sample category selectivity (rCTI) was greater in LIP compared with MIP (Fig. 4A). However, for the smaller population of 41 LIP and 21 MIP neurons included in the latency analysis and restricting our analysis to the initial part of the sample epoch (80–300 ms after stimulus onset), mean rCTI values were very similar between the two areas (LIP mean rCTI = 0.103, MIP mean rCTI = 0.113, p = 0.66, unpaired two-tailed t test), although category selectivity emerged with a shorter latency in LIP. Thus, the shorter latency observed for LIP is not because of stronger category selectivity during the early sample. However, the mean firing rate during the same window was greater in LIP (24.0 Hz) compared with MIP (13.9 Hz). To test that the difference in mean firing rate cannot explain the observed difference in latency, we removed six LIP neurons with the greatest firing rates so that the mean rates were approximately equal between the two populations (LIP mean spike rate of remaining neurons = 13.9 Hz). Latency was still shorter in LIP after removing the six neurons (p = 0.032, bootstrap). To confirm that the larger population of LIP neurons (41 LIP neurons were included compared with 21 MIP neurons) did not affect our result, we sorted LIP neurons based on their sample rCTI value, and only included every second neuron. This left us with 21 neurons in both populations with approximately equal rCTI values (LIP rCTI = 0.108, MIP rCTI = 0.113), and category selectivity latency was still significantly shorter in LIP based on these populations of neurons (LIP latency = 158 ms, MIP latency = 186 ms, p = 0.025, bootstrap). Finally, we examined whether differences in direction tuning biased our latency results. As stated in Materials and Methods, our category decoder was invariant to directional signals assuming that the preferred directions of neurons were uniformly distributed. To determine whether this was a valid assumption, we modeled the firing rate of each neuron as a linear combination of cosine direction tuning with binary category tuning:
We found that preferred directions in LIP, and to a lesser extent in MIP, tended to cluster in directions perpendicular to the category boundary (number of neurons with preferred directions in each of the four quadrants, in which quadrants 2 and 4 contain directions perpendicular to the boundary: LIP = 4, 11, 9, 17; MIP = 4, 6, 3, 8). Thus, we tested the hypothesis that the shorter latency of LIP category signals compared with MIP was the result of this bias in preferred directions. We repeated our bootstrap analysis with one change: each time we randomly resampled from the 41 LIP neurons, we ensured that the number of neurons with preferred directions in quadrants perpendicular to the boundary was less than or equal to the number of neurons with preferred direction in quadrants parallel to the boundary. If our resampled group did not meet this condition, it was discarded and we resampled again. We found that equalizing the distribution of preferred directions did not greatly affect our latency results: mean LIP latency was 155 ms (compared with 150 ms in the original analysis), which was still significantly less than the 186 ms latency for MIP (p = 0.020, bootstrap).
Latency of test category selectivity
We examined whether the selectivity latency difference between LIP and MIP in the sample period was also present during the first test period. To do so, we applied our category classifier on the 28 LIP and 11 MIP neurons whose responses during the early test epoch (80–300 ms after test onset) were significantly different between test categories according to a Wilcoxon's rank-sum test (p < 0.05). Test category selectivity was significantly greater than chance in both LIP (Fig. 5F, black curve, horizontal black bar denotes significance at p < 0.01 according to a bootstrap) and MIP (gray curve and horizontal bar). Test category selectivity emerged in LIP with a latency of 142 ms (bootstrap; see Materials and Methods), whereas test category selectivity in MIP was too weak for us to calculate a latency based on our methods.
Relationship between neural activity, stimulus onset, and the behavioral report
Although category signals are present in MIP (Fig. 4), the longer latency for category signals in MIP compared with LIP (Fig. 5E), weaker sample category selectivity in MIP during the sample, delay, and test, and the relatively weak direction selectivity in MIP suggests that the transformation of visual stimuli into category signals does not occur in MIP. Thus, what role (if any) does MIP play in this categorization task? Given the known role of MIP in motor planning, as well as its modulation during the test period of match trials (Fig. 2), we hypothesized that MIP was more involved in the monkeys' motor responses. Conversely, the direction and category selectivity of LIP in addition to its shorter latency (Figs. 4, 5) was consistent with it having a more prominent role in visual processing. Thus, we compared how neural activity in both areas related to the stimulus onset and the behavioral response (a manual touch-bar release). Because the responses of LIP neurons to visual stimuli can be short latency and temporally precise (Bisley et al., 2004) and because MIP is known to respond to arm and hand movements, we would expect activity in LIP to be more tightly temporally locked to the onset of visual stimuli, whereas we would expect activity in MIP to be closely linked to the behavioral response.
Because we were interested in how test period activity related to both stimulus, decision, and motor processing, we included neurons that were selective for the test categories or neurons with a significant difference in firing rate between match and non-match trials from 80 to 300 ms after test onset (both according to a Wilcoxon's rank-sum at p < 0.05). Forty-five of 76 LIP and 71 of 94 MIP neurons met this criteria.
To visualize how test period activity in both areas was temporally aligned with both the visual stimulus onset and the behavioral response, we calculated the average, normalized neuronal response (see Materials and Methods) aligned to test onset (Fig. 6A) and aligned to the behavioral response (Fig. 6B) for the fastest one-third, middle third, and slowest third of match trials based on RTs (LIP mean RTs for each set of trials grouped by RTs = 230.6, 255.7, 290.9 ms; MIP mean RTs = 235.9, 262.9, 303.5 ms). For area LIP (left column), neural activity relative to stimulus onset did not appear to vary between trials with different RTs. This suggests that LIP responses are more tightly aligned with the stimulus onset than the motor response. In contrast, neural activity in MIP relative to the behavioral report did not appear to vary between trials with different RTs, suggesting that MIP is more tightly aligned with the monkeys' manual responses.
Relating neural activity to the visual stimulus and the manual bar release. A, Normalized neural activity averaged across LIP (left column) and MIP (right column) neurons relative to the test onset. For each neuron, trials were divided based on whether their RT was among the fastest third (blue curve), middle third (red curve), or slowest third (green curve). The colored asterisks indicate the average RT for each trial set. B, Same as A, except neural activity relative to the manual bar release is shown. C, The trial-by-trial correlation coefficient between the spike count in the preceding 200 ms with the RT of the trial averaged across LIP (left column) and MIP (right column) neurons. D, Histogram showing the normalized difference between the coefficients linking neural activity to the time since test onset and the time until bar release, for LIP (left column) and MIP (right column) neurons.
We quantified these qualitative observations from Figure 6, A and B, in two ways. First, we calculated the sliding trial-by-trial correlation coefficient between the RT and spike count in the preceding 200 ms window in 5 ms intervals (Fig. 6C). Negative values indicate that greater neural activity is associated with shorter RTs. After test stimulus onset, the average MIP correlation coefficient became more negative than the average LIP correlation coefficient; using neural activity from 50 to 250 ms after test onset, the mean MIP correlation coefficient was −0.160 compared with −0.032 for LIP (p = 10−4, unpaired two-tailed t test), suggesting that MIP activity is more strongly linked to the behavioral report compared with LIP. We also note that the correlation was slightly negative well before test onset. Given the known role of MIP in movement planning and because monkeys could anticipate the time of test stimulus onset, the negative correlation before test onset is likely related to motor preparation.
Second, we regressed neural activity against two factors: (1) the time since test onset; and (2) the time until bar release (Fig. 6D). The outputs of this regression, Rvisual and Rmotor, give the linear coefficient relating neural activity to these two factors. The greater the coefficient (either positive or negative), the stronger the link between neural activity and the visual stimulus (for Rvisual) or the behavioral report (for Rmotor). We computed Rvisual and Rmotor for all neurons and then calculated the normalized difference between the coefficients,
Identifying LIP and MIP recording locations
The results of this study obviously depend on the correct identification of the cortical area of our recording sites. For example, it could be argued that the category selectivity observed in MIP was the result of misidentifying LIP sites as belonging to MIP. However, anatomical and functional properties of the two areas lead us to conclude that this was not the case. First, LIP and MIP are located on opposite banks on the IPS (Fig. 7A), which facilitated targeting each area for neural recording because our vertical recording approach typically encounters MIP first, followed by a period of silence (the sulcus) and then LIP.
Anatomical location and neuronal activity during delayed memory saccade task. A, MRI scan from one monkey showing the location of the two cortical areas, LIP and MIP, targeted in this study. Area LIP is located on the lateral bank of the IPS, whereas area MIP is located on the medial bank of the IPS. B, The average neural response during the delayed memory saccade task for all LIP (black) and MIP (gray) neurons in response to the saccade target in the preferred direction of the neuron (i.e., the direction that elicited the greatest spike rate). Error bars denote one SE. C, Histogram showing saccade direction tuning index for all LIP (black) and MIP (gray) neurons.
Second, past studies have shown that LIP, but not MIP, activity increases when planning or executing saccades toward the RF of a neuron (Snyder et al., 2000). We confirmed this by recording activity in both areas while the monkeys performed a delayed memory-guided saccade task (see Materials and Methods). As expected, average LIP activity increased after a target appeared in its RF (defined as the target location that elicited the greatest response from 80 to 380 ms after target onset) (Fig. 7B, black trace), whereas MIP activity (gray trace) showed little modulation.
To quantify this visuospatial selectivity, we calculated the spatial selectivity index (see Materials and Methods) for all LIP (Fig. 7C, black curve) and MIP (gray curve) neurons. A score of 1 indicates that the neuron fired in response to one saccade target direction and remained silent for all other targets, whereas a score of 0 indicates that the neuron fired equally for all target directions. We found that the average spatial tuning index was significantly greater for LIP (mean LIP = 0.319, mean MIP = 0.128, p = 10−10, unpaired two-tailed t test). The memory saccade task was performed for 51 of 64 MIP neurons included in the rCTI analysis (Fig. 4). To rule out the possibility that the 13 MIP neurons for which we did not run the memory saccade task were responsible for the significant category selectivity (Fig. 4), we re-performed the significance test without these 13 neurons. We found that the average rCTI values remained significantly >0 for the sample (mean = 0.029, 10−4, two-tailed t test), delay (mean = 0.053, p = 10−5), and test (mean = 0.029, p = 10−4) epochs.
Third, past studies have shown that MIP neural activity is modulated by arm and hand movements. Although we did not record the monkey's arm or hand movements themselves (apart from encoding the time that the monkey release the touch bar during the DMC task), we consistently observed modulation of neural activity with such movements before all putative MIP recording sessions but did not detect any such modulation for all putative LIP recording sessions. Because of the distinct anatomical locations of LIP and MIP, the differing response properties to saccadic and arm movements and how neural activity in each area covaried with the stimulus onset and behavioral report (Fig. 6), we are confident that the cortical locations of the recorded neurons were correctly identified.
Discussion
Although we previously established that LIP is involved in visual categorization (Freedman and Assad, 2006), it was unknown whether this functionality was specific to LIP within the PPC or whether category signals were homogenous across the PPC. Here, we compared the roles of LIP and MIP, two areas within the PPC, in a category matching task. We found that activity in both areas reflected the learned category membership of the motion stimulus. The presence of category selectivity in MIP was not entirely expected given that this parietal region is primarily known for its role in goal-directed arm movements and visuomotor coordination (Mountcastle et al., 1975; Kalaska, 1996; Grefkes et al., 2004; Cui and Andersen, 2007). However, category selectivity was stronger and appeared earlier in LIP compared with MIP, suggesting that MIP is likely a recipient of category signals generated elsewhere. During the test period, MIP responded strongly on “match” trials in which monkeys released a touch bar to indicate that the sample and test categories matched. In fact, during the test period, MIP almost exclusively reflected the monkeys' motor responses, whereas LIP test period activity reflected a more diverse set of task-related information, including encoding of both the sample and test categories, and the match/non-match status of the test stimulus. This suggests that (similar to LIP) MIP activity reflects abstract cognitive information during this categorization task. Additional studies designed to dissociate match/non-match decision and motor factors are needed to precisely determine the roles of each area during the test period, when monkeys report their decision about category membership.
Recording locations
The naming conventions for area 5 and MIP vary across studies from different laboratories, as discussed by Seelke et al. (2012). They examined the recording locations across a number of studies and show that previous experiments targeted specific areas, such as the medial bank of the IPS or superficial regions within area 5 medial to the IPS, but referred to those locations broadly as “area 5.” This discrepancy in nomenclature leads to some difficulty in comparing results across studies. The majority of our neurons were located along the medial bank of the IPS, and thus we refer to our recordings as “MIP.”
Cognitive signals in LIP and MIP
Decades of work have established that LIP is centrally involved in both visuospatial and visuomotor functions, such as directing spatial attention, and planning saccadic eye movements (Andersen et al., 1985; Duhamel et al., 1992; Goldberg et al., 2006). However, more recent work has shown that LIP also encodes task-relevant information that can be distinct from spatial and motor factors. This includes category encoding for both stimuli presented inside or outside the RFs of the neurons (Freedman and Assad, 2006, 2009), learned stimulus associations (Fitzgerald et al., 2011), motivational salience (Leathers and Olson, 2012; Rao et al., 2012), and task rules (Stoet and Snyder, 2004). In addition, a recent paper from our laboratory showed that saccade-related spatial signals are encoded independently from category signals in LIP (Rishel et al., 2013). In contrast, few studies have examined whether such nonspatial and non-motor cognitive signals are present in MIP. Several past studies have shown that activity in MIP is modulated specifically by arm movements (as opposed to saccades; Cui and Andersen, 2007), that modulation does not depend on whether the movement is self-timed or reactive (Maimon and Assad, 2006), and that coherence between spiking activity and oscillations in the local field potential increases between MIP and dorsal premotor cortex when the monkey can choose between different motor sequences (Pesaran et al., 2008). None of the studies specifically examined whether MIP encodes non-motor cognitive signals.
However, in one study, the activity of parietal area 5 neurons in the superior parietal lobule (medial to the IPS) reflected the number of movements performed within an action sequence (Sawamura et al., 2002). This modulation could potentially encode the time until the monkey was to switch movement types and/or play a role in keeping track of the number of movements. However, it was difficult to conclude whether this signal was independent of motor factors. In our DMC task, we can dissociate the neuronal response during the sample and delay periods from any motor-related signals, because the monkey cannot predict (before the test period) the behavioral response required to successfully complete the task. Thus, our finding that MIP activity can reflect category information independently of motor responses expands our understanding of the possible roles of MIP.
Potential role of non-motor cognitive signals in MIP
In our study, we observed category signals in MIP that were distinct and dissociable from the monkeys' motor responses. What is the role of these category signals? One possibility is that the presence of category representations in MIP could expedite the motor response to visual stimuli: because sample category signals are encoded in MIP throughout the trial, MIP could make use of this available information to more efficiently plan or generate a motor response during the test period. Although category representations were encoded less strongly in MIP than in LIP, the presence of these signals indicated that, minimally, MIP has access to non-motor cognitive information and that MIP may be more involved in non-motor cognitive functions than previously assumed.
During the test period, when monkeys reported whether the sample and test stimuli belonged to the same category (i.e., “matched”) with a manual touch-bar release, MIP activity primarily reflected the monkeys' motor response and only weakly reflected the stimulus category. One possibility is that, during the test period, test category representations in MIP are swamped by stronger motor-related signals when an arm movement is made. An alternative is that MIP is less involved in category encoding during this stage of the task and that another brain area may instead be responsible for encoding these representations. However, in our current task, the match versus non-match and motor decisions were confounded (i.e., a “match” always required a touch-bar response). Future experiments could dissociate the match/non-match decision itself from the motor responses used to report those decisions that could more clearly reveal the relative encoding of category, match, and motor signals in MIP.
The time course of direction and category selectivity
In this study, we used a population decoder to measure the time course of direction and category selectivity. Because both direction and category can coexist in the same area (as is the case for LIP; Fig. 5C,D), we designed a classifier that attempted to separately assess these two signals. By limiting our direction classifier to only those directions belonging to a single category, we believe we were capable of measuring direction selectivity independently from category selectivity (see Materials and Methods).
Measuring category selectivity independently of direction selectivity is less straightforward. As discussed in Materials and Methods, direction-selective neurons with preferred directions perpendicular to the category boundary tend to increase category decoding performance, whereas the opposite held for neurons with preferred directions parallel to the boundary. Importantly, Although our classifier might be affected by direction selectivity at the single-neuron level, by decoding activity across a population, these affects will tend to cancel, leaving the classifier relatively unaffected. The one assumption is that preferred directions are uniformly distributed. For a motion-selective, but not category-selective, area such as MT (Freedman and Assad, 2006), our category decoder would perform at chance, as desired.
Although our population decoders revealed the presence of both direction and category selectivity in both areas, it does not address how direction signals are transformed into categorical representations. Although we cannot claim that this transformation occurs in LIP (see below), the development of direction selectivity tens of milliseconds before category selectivity is consistent with the idea that the transformation is mediated through local connections within LIP as opposed to connections between LIP and direction-selective areas (e.g., area MT). Future experiments will be required to fully understand the neural mechanisms that underlie this transformation.
Computing visual categories
A crucial remaining issue is to understand the mechanisms underlying the computation of the category signal itself. Is it generated in the PPC or is it computed elsewhere in the brain and then transmitted to PPC? We recently found that category selectivity was stronger and developed sooner in LIP compared with PFC (Swaminathan and Freedman, 2012). Thus, category signals in LIP are unlikely to result via feedback from PFC, and LIP could be a source of category information to PFC and other brain areas. Here, we find that LIP encodes category representations more strongly and with a shorter latency compared with MIP, indicating that category information may be preferentially concentrated in LIP compared with other PPC regions.
This raises the question of whether visual categories could, in fact, be computed in LIP. LIP shares reciprocal connections with visual-processing centers, such as the MT and medial superior temporal areas (Lewis and Van Essen, 2000), and is well positioned to integrate basic stimulus features (e.g., motion direction) to generate more abstract and meaningful representations of visual stimuli. During categorization tasks, LIP represented the category membership of stimuli, whereas neurons in area MT represent motion direction but not learned category representations (Freedman and Assad, 2006). Thus, LIP could compute visual categories by combining feature information from visual processing centers and rule-related information (i.e., the position of the category boundary) from another brain area (possibly PFC). Alternatively, visual categories could be computed elsewhere in the brain and conveyed to LIP early in the categorization process. The precise role of these areas could be clarified using reversible inactivation techniques and simultaneous recordings from brain areas involved in visual processing, motor functions, and decision making.
Footnotes
This work was supported by National Institutes of Health Grant R01 EY019041 and National Science Foundation CAREER Award 0955640. Additional support was provided by a McKnight Scholar award, the Alfred P. Sloan Foundation, and The Brain Research Foundation. We thank Gang Huang, Steve McClellan, Navaneethan Santhanam, and Stephanie Thomas for technical assistance and the staff of The University of Chicago Animal Resources Center for expert veterinary assistance. We also thank Sliman Bensmaia, Jamie Fitzgerald, Guilhem Ibos, and Jillian McKee for comments on a previous version of this manuscript.
- Correspondence should be addressed to David J. Freedman, Department of Neurobiology, University of Chicago, 949 East 58th Street, MC0928, AB310, Chicago, IL 60637. dfreedman{at}uchicago.edu
