Abstract
We have adapted Sternberg's context-recall task to investigate the neural mechanisms of encoding serial order information in working memory, in 2 male rhesus monkeys. We recorded from primary motor, premotor, and dorsolateral prefrontal cortex while the monkeys performed the task. In each cortical area, most neurons displayed marked modulation of activity during the list presentation period of the task, whereas the serial order of the stimuli needed to be encoded in working memory. The activity of many neurons changed in a consistent manner over the course of the list presentation period, without regard to the location of the stimuli presented. Remarkably, these neurons encoded serial position information in a relative (rather than absolute) manner across different list lengths. In addition, many neurons showed activity related to both location and serial position, in the form of an interaction effect. Surprisingly, the activity of these neurons was often modulated by the location of stimuli presented before the epoch in which the activity changes occurred. In motor and premotor areas, a large proportion of neurons with list presentation activity also showed direction-related activity during the response phase, whereas in prefrontal cortex most cells showed only list presentation effects. These results show that many neurons had a heterogeneous functionality by representing distinct task variables at different periods of the task. Finally, potential confounds could not account for the effects observed. For these reasons, we conclude that these neurons were indeed participating in sequence encoding in working memory.
SIGNIFICANCE STATEMENT Traditionally, primary motor, premotor, and prefrontal areas have been considered to be mainly engaged in motor output, visuomotor transformation, and higher cognitive functions, respectively. Here we show that neurons in all three cortical regions participate in the encoding of a sequence of spatial stimuli in working memory. Furthermore, a central question in cognitive neuroscience has been the manner in which the position of an item within a sequence is encoded in the brain. Our findings provide direct neurophysiological support for a specific hypothesis from cognitive psychology: that of relative coding of serial order.
Introduction
The encoding of information in working memory about the serial order of items is critical to many cognitive processes and behaviors (Lashley, 1951). In the 1960s, Saul Sternberg developed several paradigms for testing working memory mechanisms. These paradigms involve presenting a sequence of stimuli (e.g., letters or digits), and having the subject select a single response based on a rule applied to the memorized sequence of items. The paradigm that Sternberg designed for studying specifically working memory of serial order is called the context-recall task (sometimes referred to as the “searching-for-next” task) (Sternberg, 1967). In this task, a sequential list of items is presented, followed by a probe item from the list. The subject must then retrieve the item that immediately followed the probe in the list. Sternberg analyzed response times of subjects performing these tasks, and proposed mechanisms of how serial order information is retrieved from working memory. A different question is how the order information is encoded in working memory. Several models of serial order encoding have been proposed (Marshuetz, 2005). In some of these models, the serial position (SP) of a sequence item is encoded in an absolute manner (first, second, etc.), whereas in other models the item is encoded by its relative position in the sequence (e.g., beginning, middle, end) (Henson, 1998, 1999).
To date, little direct evidence exists on the brain structures and mechanisms involved in order information encoding. Neuroimaging studies (Henson et al., 2000; Marshuetz et al., 2000) have implicated several cortical areas in serial order memory tasks, including prefrontal, premotor, and parietal cortex. For these reasons, we chose premotor and prefrontal cortex for the current study, as well as motor cortex given its recognized role in cognitive motor functions (Tomasino and Gremese, 2016). Neurophysiological studies in behaving monkeys have found order-related neuronal activity in many different brain regions, including the areas sampled in this study: prefrontal cortex (Barone and Joseph, 1989; Funahashi et al., 1997; Ninokura et al., 2003; Berdyyeva and Olson, 2010), premotor cortex (Ohbayashi et al., 2003), and primary motor cortex (Kettner et al., 1996; Lu and Ashe, 2005). Most of these studies investigated neural activity in tasks requiring the ordering of a sequence of movements. Although movement sequences are an important example of serial order behavior, it can be difficult to dissociate neural activity related to order memory from activity related to the preparation and/or execution of sequential movements (but see Berdyyeva and Olson, 2009). In addition, most of these studies used small sets (usually 3) of sequence elements, leading to a limited number of possible sequences that are likely “over-learned,” such that the monkey relies on long-term memory of the sequence as opposed to working memory. Finally, most of these studies reported activity during the delay and/or response phases of the tasks, thus investigating activity related to maintenance or retrieval of serial order information, rather than encoding of order information. An exception to this was Ninokura et al. (2004), who found neuronal activity in prefrontal cortex related to encoding of the serial order of sequence elements, the physical properties of these elements, or both.
We have adapted Sternberg's context-recall task for use with behaving monkeys (Pellizzer et al., 1995), to investigate the neural mechanisms of encoding serial order in working memory in a task that does not confound activity related to serial order with activity related to sequential movements. In a prior report on these experiments (Carpenter et al., 1999), we demonstrated that the activity of many neurons in primary motor cortex reflected the serial order of successively presented stimuli during presentation of a sequence, and that the individual sequence elements could be retrieved from the activity of small ensembles of motor cortical cells. Here we compare activity during sequence presentation among three areas: primary motor, premotor, and dorsolateral prefrontal cortex. We also report neurophysiological evidence supporting a relative, as opposed to absolute, coding of serial order.
Materials and Methods
Animals.
The animals were 2 male rhesus monkeys (Macaca mulatta, 6–8 kg body weight). They are referred to as Monkey 1 and Monkey 2. Care and treatment of the animals during all stages of the experiments conformed to the guidelines described in the Guide for the care and use of laboratory animals (National Research Council, 1996). The animal protocol was approved by the respective Institutional Animal Care and Use Committees of the Minneapolis Veterans Affairs Medical Center and of the University of Minnesota.
Experimental apparatus.
The monkeys sat in a primate chair inside an electrically shielded enclosure. Monkey 1 used the left arm to perform the task, whereas Monkey 2 used the right arm; for both animals, the other arm was restrained. A 14 inch color computer monitor was placed 67 cm in front of the animal, at eye level. The monkeys were trained to control the position of a cursor on the monitor by exerting force on a 2D semi-isometric manipulandum (a “force joystick”; Measurement Systems). The manipulandum consisted of a vertical metal rod with a disc attached to the top, which monkeys grasped with the hand pronated. The signal of the force exerted on the manipulandum was used to control the position of the force feedback cursor (a red circle, 0.42 cm radius) on the monitor. The cursor was deflected constantly downward to simulate a force bias of 54 g; therefore, at any given moment, the cursor position on the screen reflected the vector sum of this bias force and the force exerted by the monkey. The bias force was used so that the monkey had to actively initiate a trial by exerting force to center the cursor. The signal from the manipulandum was converted by an analog-to-digital converter and recorded at a rate of 200 Hz. The video display was updated at a rate of 60 Hz. We used a personal computer and custom software to control and implement the behavioral tasks and to collect the position of the manipulandum, gaze direction, and neural and electromyographic data (see below).
Behavioral task.
The memory scanning task (Fig. 1) is a visuomotor version of the context-recall task devised by Saul Sternberg (Sternberg, 1967; Georgopoulos and Lurito, 1991; Pellizzer and Georgopoulos, 1993; Pellizzer et al., 1995; Carpenter et al., 1999). A trial began by turning on a white center-hold window in the center of the screen, which the monkey captured with the force feedback cursor. After 1 s (center-hold period), three to five circular yellow stimuli were presented successively on the screen (list presentation period). The number of stimuli in a sequence defines the list length. Monkey 1 performed the task with list lengths of three and four stimuli, whereas Monkey 2 performed the task with list lengths of three, four, and five stimuli (the list length 5 trials were added with the second monkey to explore further the issue of absolute vs relative serial order coding.) For Monkey 1, 400 ms elapsed between the onset of one stimulus and the onset of the next one in the list (epoch duration = 400 ms). For Monkey 2, the epoch duration was 650 ms. Each stimulus occupied 1 of 8 equally spaced possible locations at a radius of 6.25 cm from the center of the screen. Once a stimulus was presented, it remained on the screen for the duration of the trial. Thus, the monkeys did not have to memorize the locations of the list stimuli because the sequence items remained visible throughout the trial. Rather, the monkeys had to remember the serial order in which the list stimuli appeared.
Illustration of the memory scanning task. Left to Right, Successive views of the computer display during a trial. This example uses a sequence of five stimuli. A trial began when the monkey moved the cursor (red dot) into the center-hold window (white disc). After the 1 s center-hold period, the stimuli were sequentially illuminated during the list presentation period. At the end of the list presentation period, one of the stimuli turned blue (test stimulus). The rule of the context-recall memory scanning task is to move toward the stimulus that immediately followed the test stimulus during the sequence presentation. Right, The four possible test stimuli and their associated responses. Top, Trial where the test stimulus was the first in the sequence (SP of the test stimulus = 1). This instructed a movement to the second stimulus in the sequence (represented by the red arrow pointing toward the stimulus on the left). The following sets of panels illustrate cases where the test stimulus corresponded to SPs 2, 3, and 4, instructing movements to the third, fourth, and fifth stimulus, respectively.
During the center-hold and the list presentation periods, the monkeys had to maintain the cursor within the center-hold window. At the end of the list presentation period, one of the stimuli changed color from yellow to blue. This blue stimulus was referred to as the test stimulus: it provided the go signal to the monkey and also the information necessary to choose the correct target of response. The rule of the task was to move the cursor from the center-hold window toward the list stimulus that immediately followed the test stimulus during the list presentation period. Therefore, the rule constrains the test stimulus to be any stimulus, except the last of the sequence. A correct response was recorded when the cursor exceeded the distance between the center-hold window and the center of the target stimulus while staying within a window of ±22.5 degrees from the ideal direction. This required that the monkeys exert a net force of at least 270 g in the appropriate direction to complete the trial correctly. The onset of the motor response was defined as the time when the cursor exited from the center-hold window (i.e., when the net force exceeded 72 g). The end of the motor response was defined as the time when the cursor passed the target radius (i.e., when the net force exceeded 270 g). The reaction time was defined as the period from the onset of the go signal to movement onset, and the movement time was defined as the period between movement onset and the cursor reaching the target. A trial was aborted if any of the following errors occurred: (1) failure to maintain the cursor within the center-hold window throughout the center-hold period, (2) moving the cursor from the center-hold window during the list presentation period, (3) failure to initiate movement within 1 s (Monkey 1) or 2 s (Monkey 2) of go signal onset, (4) failure to complete the movement within 1 s of movement onset, and (5) failure to maintain the cursor within ±22.5 degrees of the target during the movement. The monkeys received a liquid reward after each correct trial. Only data from the correct trials were used in the analyses presented here. Different list lengths (3–5 stimuli) were presented in separate blocks. For Monkey 1, the block of sequences of four stimuli was always presented after the block of sequences of three stimuli. For Monkey 2, the blocks of sequences of different list lengths (3–5 stimuli) were presented in a random order.
Stimuli.
Given 8 possible stimulus locations, there are 8!/(8-n)! combinations of n stimuli (i.e., 336, 1680, and 6720 sequences of three, four, and five stimuli, respectively). In addition, for each sequence of stimuli, the test stimulus could be selected from any SP, except the last one. Therefore, there were 2 × 336 = 672, 3 × 1680 = 5040, and 4 × 6720 = 26,880 possible combinations of sequences of stimuli and test stimulus for each list length, respectively. The monkeys were trained using all possible combinations of stimuli and SP of the test stimulus. However, during the neurophysiological recording, we limited the number of combinations presented to record neural data for several repetitions of the same combination. The sequences of stimuli used during the experimental sessions were selected such that each stimulus location was represented an equal number of times in each SP across the entire set of sequences.
For Monkey 1, we selected 16 sequences of three stimuli and 4 sequences of four stimuli to be used during the neuronal recording sessions. These sequences were the same in each experimental session. Within each list length block, the sequence and SP of the test stimulus for each trial were selected at random until one correct response to each instance was made. This was repeated until 5 correct responses were made to each combination of sequence and SP of the test stimulus. Thus, to complete a block of sequences of three or four stimuli, Monkey 1 needed to perform 160 or 60 trials, respectively (number of trials = number of sequences × number of SPs of the test stimulus × 5 repetitions). For Monkey 2, the experimental sessions included up to three blocks of sequences of three, four, and five stimuli, respectively. Each block was composed of 8 different sequences of stimuli. These sequences varied in each experimental session. One sequence in each block was taken from a fixed pool of two possible sequences. The remaining 7 sequences were randomly generated under the constraint that each possible stimulus location was equally represented in each SP during the experimental session. The fixed pool of sequences was used so that some sequences would be repeated across experimental sessions. Monkey 2 needed to perform 80, 120, and 160 correct trials to complete blocks of three, four, and five stimuli, respectively.
Surgical procedures.
Once the monkeys became proficient in the task, they underwent several procedures to prepare them for neural recording. First, structural magnetic resonance images of the brain were obtained, under ketamine (10 mg/kg i.m.) and xylazine (0.5 mg/kg i.m.) sedation. From these images, we constructed maps that were used to choose the stereotaxic coordinates for the placement of the recording chambers. A head restraint system and recording chamber were surgically implanted under aseptic conditions and pentobarbital anesthesia (28 mg/kg i.v.). Monkey 1 had the recording chamber centered over the right motor cortex. Two recording chambers were used on Monkey 2: one over the left motor and premotor cortex and the second one over the left dorsolateral prefrontal cortex. Topical xylocaine (2%), and acetaminophen mixed with food plus codeine in drinking water, was given for postoperative analgesia.
Neurophysiological recordings.
The impulse activity of single cortical neurons was recorded extracellularly using a system of 7 independently movable microelectrodes (Uwe Thomas). The recording system has been described in detail (Mountcastle et al., 1991). The microelectrodes were made from flexible quartz fibers (80 μm diameter) with a core (30 μm) of platinum-tungsten alloy. During the recording sessions, the monkey's head was immobilized using a metal “halo” attached to head posts and the electrodes were driven through the dura into the cortex. The electrical signals from cortical neurons were amplified, bandpass filtered, and displayed on oscilloscopes. The electrodes were driven until one or more distinct, stable action potentials could be discriminated. The action potential waveforms were isolated using either a dual time-amplitude window discriminator (BAK Electronics) or a template-matching spike sorting system (Alpha-Omega Engineering). The series of action potentials (spike trains) from each neuron were stored on a computer as interspike intervals with a 1 μs time resolution. All well-isolated neurons were recorded, regardless of their activity during the task. We tried to isolate as many neurons as possible during each recording session, but it was relatively rare to record from all seven electrodes at once. If the quality of isolation deteriorated during recording, the data were kept only for blocks of trials that were completed before the deterioration. The average discharge rate had to be >0.1 impulses/s (averaged over all trials) for the data to be included in the analysis; otherwise, all neurons were analyzed regardless of whether or not their activity was task-related.
Table 1 reports the number of neurons that were isolated in each cortical area. In addition, the table indicates the number of neurons that were recorded across different list length conditions in the memory scanning task. Most neurons were recorded during multiple blocks of different list length. Although we tried to record each neuron in all conditions, this was not always possible because of the duration of the experimental sessions. Data from Monkey 1 during the response period (Pellizzer et al., 1995) and some aspects of motor cortex data from both monkeys during list presentation (Carpenter et al., 1999) have been presented previously.
Number of neurons recorded in each condition and cortical area per monkey
Recording of eye position.
Eye movements were not constrained during the task. We recorded eye position, and hence gaze direction (given that the head was fixed), of Monkey 1 during task performance with an infrared oculometer (Dr. Bouis, Karlsruhe, Germany). Gaze direction data from Monkey 1 were obtained in separate sessions from the neural recordings. For Monkey 2, gaze direction was recorded simultaneously with the neural recordings, using either the infrared oculometer or a scleral search coil system (Fuchs and Robinson, 1966; Judge et al., 1980). A calibration procedure was performed before the recording sessions by having the monkey fixate a series of targets on the screen. Eye position was sampled at 500 Hz for Monkey 1 and 200 Hz for Monkey 2.
EMG recordings.
EMG activity during task performance was recorded with intramuscular, multistranded, Teflon-coated wire electrodes (Schwartz et al., 1988). The EMG recordings were made during separate sessions from the neural recordings. The EMG signal was amplified 10,000–20,000-fold with a Grass 12A5 differential amplifier, notch filtered at 60 Hz, and bandpass filtered at 30–300 Hz. EMG signals were sampled at a rate of 500 Hz for Monkey 1, whereas EMG signals for Monkey 2 were rectified and sampled at a rate of 200 Hz. The following trunk and limb muscles were sampled from Monkey 1: paraspinal, infraspinatus, supraspinatus, trapezius (lower, middle, and upper), deltoid (anterior, middle, and posterior), pectoralis major, triceps (lateral and long heads), biceps, and forearm extensor (unspecified). For Monkey 2, all the above muscles were sampled as well as latissimus dorsi, rhomboids, and forearm flexor (unspecified).
Histology.
The fine microelectrodes used in the experiment typically do not leave visible tracks in the brain. We determined the cortical areas by localizing the coordinates of recording in reference to metal pins that were inserted at known coordinates within the recording chamber at the end of the experiments. The monkeys were then killed with a lethal dose of sodium pentobarbital. After several months in formalin, the brains were photographed, and blocks of brain tissue surrounding the recording sites were removed. The block from Monkey 1 was cut into serial 50 μm parasagittal sections. The blocks (motor/premotor, and prefrontal) from Monkey 2 were cut into serial 40 μm coronal sections. Every sixth section from each block was stained with cresyl violet and mounted. The border between motor and premotor cortex was determined on the basis of counts of large pyramidal cells in layer V (Weinrich and Wise, 1982). The recordings from Monkey 1 were in the arm region of motor cortex, whereas for Monkey 2 they included motor and premotor cortex (arm region) as well as dorsolateral prefrontal cortex, between the anterior part of the principal sulcus and the superior limb of the arcuate sulcus (see Fig. 4). We did not attempt to further parcellate the prefrontal region based on histology.
Experimental design and statistical analysis.
For each period of interest, the frequency of discharge of each neuron was computed using fractional interspike intervals. Statistical analyses were performed on the square-root–transformed frequency of discharge, which stabilizes its variance (Snedecor and Cochran, 1989). We used a conservative significance threshold of p < 0.01 for all analyses.
SP and location effects on neural activity.
The discharge activity of each neuron during the list presentation period was analyzed separately for each list length condition. For each neuron, we computed the discharge rate within each epoch of the list presentation period in each repetition of each sequence. Then we performed a two-way ANCOVA with the SP of the epoch and the spatial location of the stimulus for that epoch as independent variables. The discharge rate during the last 400 ms (for Monkey 1) or 650 ms (for Monkey 2) of the center-hold period was used as the control rate and included as a covariate in the ANCOVA. This analysis is referred to as the main ANCOVA in subsequent parts of the text. All ANCOVA and ANOVA analyses (including those described below) were performed using BMDP statistical software, release 7, program 4V (Dixon, 1992).
Cortical distribution of task-related neurons.
To explore the cortical distribution of task-related neurons, we combined the data from both monkeys into a common coordinate system. Photographs of the brains centered on the recording sites and including the reference pins (see Histology) were taken for each monkey. On each photograph, a line was drawn parallel to the midline (the longitudinal fissure) and lowered until it intersected the posterior end of the principal sulcus. This line formed the x axis of the new coordinate system. An orthogonal axis (y axis) was placed at the point where the x axis intersected the central sulcus. In this common coordinate system, x increases from posterior to anterior and y increases from ventral to dorsal. Then, for each monkey, the electrode penetration sites were transformed from recording chamber-based coordinates to this new common, anatomy-based coordinate system by translation and rotation.
To quantify the cortical distribution of neurons whose activity was significantly related to list presentation effects in the main ANCOVA, we performed logistic regression analyses. Separate logistic regressions were done for precentral (motor/premotor) and prefrontal cortex and, within each area, for each ANCOVA effect. For each neuron and list length, we used the x and y coordinates of the penetration site as independent variables, and the presence or absence of a significant effect in the main ANCOVA (SP, location, or their interaction, coded as 1 or 0) as the dependent variable. The significant effects are reported as the exponential of the regression coefficient B (which corresponds to the odds ratio) as well as the Wald statistic and p value. This analysis was done with SPSS (version 25, IBM).
Movement direction effect on neural activity.
The discharge activity of each neuron during the movement period was analyzed separately for each list length condition. For each neuron, we computed the discharge rate during the movement time in each repetition of each sequence. We then performed a one-way ANCOVA with the target direction as factor, and the discharge rate during the last 400 ms (for Monkey 1) or 650 ms (for Monkey 2) of the center-hold period as a covariate.
Discriminant analyses.
We tested whether the individual sequence elements could be retrieved from the activity of a population of simultaneously recorded neurons. For each ensemble of simultaneously recorded neurons, we pooled all sequences having the same list length and performed quadratic discriminant analyses. We used quadratic rather than linear discriminant analysis because the cells in a simultaneously recorded ensemble generally did not have a common covariance structure (Lachenbruch, 1975). The firing rates of all simultaneously recorded neurons during a particular SP epoch of a single trial were considered as an observation and used to predict the sequence element (defined by SP or, equivalently, by location) of that SP epoch. The ensemble sizes ranged from 2 to 9 for Monkey 1, and from 2 to 14 for Monkey 2. The accuracy of the classification was evaluated by computing the percentage of correctly classified observations for each SP, list length, and cortical area. This analysis was done using BMDP statistical software, release 7, program 5M (Dixon, 1992).
We also performed separate discriminant analyses using the leave-one-out cross-validation method (Efron and Tibshirani, 1993). The cross-validation procedure is often used for classification methods, to avoid overestimating the ability of the derived functions to accurately classify future observations. However, our purpose for using the discriminant analysis was not to attempt to classify future observations, but rather to evaluate the information about SP contained within the observed neural responses. The analysis without cross-validation more accurately reflects this. The results obtained using cross-validation were qualitatively similar to those presented in Results, although the rates of correct classification were somewhat lower.
Directional tuning.
We tested the directional tuning of the neuronal activity during periods of interest as follows. The preferred direction of a neuron was calculated using directional statistics (Mardia and Jupp, 2000) and corresponded to the direction of the mean resultant (Schwartz et al., 1988). The statistical significance of the directional tuning was determined by a nonparametric bootstrap procedure (Lurito et al., 1991). A neuron was considered directionally tuned if the length of its mean resultant exceeded the 99th percentile of the bootstrap distribution (Efron and Tibshirani, 1993). This analysis was implemented with a custom computer program. For neurons that showed a significant tuning both during the list presentation and during the movement, we compared the two preferred directions by calculating the circular correlation (Jammalamadaka and SenGupta, 2001).
Effect of oculomotor factors (gaze direction and saccade direction).
For both monkeys, the digitized coordinates of gaze direction were filtered and differentiated to compute the tangential velocity. Onset and offset of saccades were first identified using a velocity threshold of 50 deg/s. Then, the saccade onset was further adjusted by lowering the threshold until it reached a local minimum in the tangential velocity profile or, alternatively, until it was lower than the mean tangential velocity plus two SDs computed over a 50 ms interval preceding the onset. The offset was adjusted in a similar manner. Fixations were defined as periods between saccades.
For each neuron, we tested the effect of gaze direction on the neuronal activity during the list presentation period. For this purpose, we analyzed the discharge rates while the monkey fixated 1 of the 8 stimulus locations or the center of the screen. Fixations had to be within a 1.9° radius disk centered on one of these locations and were required to last >350 ms. To avoid contamination of the discharge rate by preceding or following eye movements, we excluded the initial and final 150 ms of the fixation from the period over which we computed the discharge rate. When there were fewer than three fixations on a stimulus location, we excluded this location from the analysis. Fixations that occurred during any epoch of the list presentation, except the last one, were included in the analyses reported. The last epoch was excluded from this analysis because the pattern of eye movements during that epoch was markedly different from that in the previous epochs (see Results). However, analyses including the last epoch gave similar results. The neuronal activity related to gaze direction was tested using an ANOVA with location of fixation as factor.
We tested the effect of the direction of saccade by analyzing the neuronal activity preceding saccades in different directions. Saccades had to be preceded and followed by a 250 and 100 ms fixation period, respectively, to be included in the analysis. In addition, they had to be restricted within a 7 degree radius disc centered on the middle of the screen, and had to have an amplitude >3 degrees. The presaccadic activity was computed as the mean discharge rate in a 100 ms period preceding the saccade onset. The discharge rates were grouped according to the direction of the saccade in 45 degree bins. Because saccades in different directions could start from different locations, we controlled for the possible effect of gaze direction on the presaccadic activity. To this end, we used the discharge rate in a 50 ms control period starting 200 ms before the saccade as a covariate in the ANCOVA, with direction of saccade as factor.
SP and location effect on EMG activity.
To evaluate the influence of SP and location of list stimuli on muscle activity, the same ANCOVA model used to analyze the neuronal data was applied to the rectified EMG data.
Results
Performance
For both monkeys, performance declined as the list length increased (i.e., as the task became more difficult). Monkey 1 responded correctly on 85% of list length 3 trials and 51% of list length 4 trials. Monkey 2 performed correctly on 82% of list length 3 trials, 76% of list length 4 trials, and 50% of list length 5 trials. Chance performance can be considered as either random choice among the stimuli displayed (100%/list length) or, more conservatively, as random choice among all stimuli other than the test stimulus (100%/(list length − 1)). The test stimulus had a different color (blue) than the other available stimuli and could never be the correct response; therefore, we chose the more conservative (100%/(list length − 1)) as the chance level performance (although both monkeys did sometimes make incorrect responses to the test stimulus). Each monkey performed significantly better than chance in all list lengths (binomial test, p < 1 × 10−12 in all cases).
General aspects of neuronal activity
Many neurons in all cortical areas sampled showed modulation of spike activity during the memory scanning task. This modulation occurred during the list presentation period and/or during the response period. Examples of neuronal activity during the memory scanning task are presented in Figure 2. The activity of two neurons from each cortical area is illustrated. Each raster plot shows the spiking activity of one neuron during five repetitions of one condition, where a condition represents the combination of list length, sequence, and SP of the test stimulus. The histogram represents the average time-varying neuronal activity. It can be seen that the modulation of activity during the list presentation period (when the monkey was making no limb movement) was often similar to, or even greater than, the modulation during the response period. The analyses presented in the following sections explore in more detail the neuronal activity during the list presentation period and how it was affected by the factors defining the sequence of stimuli.
Neural activity during the memory scanning task: the activity of two neurons from each of the three cortical areas sampled during the memory scanning task. For each cell, spiking activity is illustrated with a raster display (top) and a histogram (bottom). Activity is shown during the five correct trials from a particular sequence and SP of the test stimulus. Below the histograms, the location of the stimuli as they appear on the screen is indicated, both as diagrams and as numbers indicating the direction of the stimulus (in degrees) from the center-hold window. For example, top left plots represent the activity of a motor cortical neuron during a specific sequence of five stimuli. After the 1 s center-hold (CH) period, the first stimulus appeared on the bottom left of the screen (225 degrees), the second stimulus appeared on the top right of the screen (45 degrees), etc. In this example, the fourth stimulus was the test stimulus (as illustrated by the change in color of the dot corresponding to 270 degrees on the diagram from black to white), and the response in this example is toward the fifth stimulus (i.e., to the right or 0 degrees stimulus). In the raster displays, each line indicates one trial. Small tick marks indicate action potentials. Each line also has 6 large tick marks, which indicate the following (in order): the beginning of the center hold period, the start of the list presentation period, the onset of the test stimulus, the beginning of the movement, the end of the movement, and 500 ms after the end of the movement. The rasters are aligned to the presentation of the test stimulus. The vertical lines on the histograms indicate the following (in order): the onset of each list stimulus, the onset of the test stimulus, the mean onset time of the movement, and the mean end time of the movement. Each histogram bar represents the mean cell discharge rate in a 50 ms bin. Neuronal activity was modulated during the list presentation (when no limb movement occurs), as well as during the response. Top right, The list presentation period, test stimulus onset, and movement periods are labelled.
Neuronal activity associated with list presentation factors
Each stimulus in a sequence is completely defined by two factors: (1) SP, which corresponds to the ordinal position of the stimulus in the list; and (2) location, which corresponds to the position of the stimulus on the screen. We sought to determine the importance of these factors on cell activity during the list presentation period using an ANCOVA with SP and location as factors and the control rate as covariate. We categorized each neuron in terms of the factors that were statistically significant (p < 0.01) as follows: (1) no significant effect, (2) SP, (3) location, (4) SP and location, and (5) SP × location interaction. If a neuron showed one or two main effects and no interaction effect, it was counted in the corresponding category (2–4). If a neuron had a significant interaction effect, it was counted in the last category (5) regardless of whether a main effect was also significant. The results for each list length and cortical area are reported in Table 2. A summary of these results is illustrated in Figure 3, where the proportion of neurons in each category of effect was averaged across list lengths.
ANCOVA of neuronal activity during the list presentation period
Summary of the ANCOVA. Bars represent the mean proportion of neurons in each cortical area with significant effects in the main ANCOVA. The categories of effects are as follows: none (neither SP, location, nor their interaction was significant), SP (only SP was significant), location (only location was significant), SP + location (both main effects were significant, but their interaction was not), and SP × location interaction (this category contains all cases where the interaction was significant, regardless of whether the main effects were or were not significant). Each bar indicates the mean for each category and cortical area, averaged across monkeys and list lengths. Error bars indicate SD.
The results show that two effects were more often significant (i.e., the SP and the SP × location interaction). These two effects were the most common in all three cortical areas investigated. In contrast, a main effect of location was rarely found, either alone or in combination with an SP main effect. The effect of SP was the most common in the motor cortex, whereas the SP × location interaction effect occurred more frequently in the premotor and prefrontal areas. In all areas, the proportion of neurons with significant ANCOVA effects were not uniform across list lengths (χ2 test for “no effect” across list lengths: motor, χ2(2) = 53.6, p < 0.001; premotor, χ2(2) = 25.3, p < 0.001; prefrontal, χ2(2) = 22.9, p < 0.001). The larger list lengths tended to have more neurons with significant effects of SP and SP × location. This could be due to more neurons being recruited for sequence encoding in the more challenging conditions.
Cortical distribution of SP, location, and SP × location interaction effects
We examined the cortical distribution of neurons whose activity was significantly related to sequence parameters. Figure 4 shows the distribution of effects found in the main ANCOVA with respect to the recording sites. The electrode penetration sites from both monkeys were combined in a common coordinate frame, and the approximate locations of anatomical landmarks are displayed in gray. The “bubbles” on the plot represent the proportion of cases that had a significant effect in the main ANCOVA. A case represents the combination of neuron and list length. For example, if two neurons were recorded at a given penetration site and both showed a significant SP effect in all list lengths of the memory scanning task, then the size (diameter) of the bubble indicates that 100% of the cases showed an SP effect. If only one cell was recorded at a given site, and it showed an SP effect in list length 3 and 4 but not list length 5, the size of the bubble would indicate 66.7% of cases with an SP effect. The proportion of cases with a significant effect of SP, location, and SP × location interaction is plotted from top to bottom, respectively. In this analysis, the proportions represent the absolute prevalence of each effect, and a main effect of SP or location is not ignored if a significant interaction is also present (see Neuronal activity associated with list presentation factors). This provides a more detailed picture of how these effects were distributed in the frontal cortex.
Percentage of neurons with significant sequence parameter effects by recording site: the percentage of neurons recorded at each penetration site that had a significant effect of each of the three sequence parameters in the main ANCOVA. Top, SP. Middle, Location. Bottom, SP × location interaction. The middle of each bubble is the penetration site, and the diameter of each bubble represents the percentage of each combination of neuron and list length that showed significant effects in the ANCOVA (as per the scale on the right of the figure). Data from both monkeys were pooled into a common coordinate frame, and the location of the central sulcus, superior precentral sulcus, arcuate sulcus, and principal sulcus in this common frame is drawn in gray for each of the effects. Dashed line indicates the border between motor and premotor cortex. Top left, Dashed rectangle represents the approximate brain region covered by each plot. The proportions refer to the absolute prevalence of each significant effect, regardless of what other effects attained significance (see Results). CS, Central sulcus; SPS, superior precentral sulcus; AS, arcuate sulcus; PS, principal sulcus.
Figure 4 suggests that the distribution of the significant effects was not uniform across the brain regions sampled. We used logistic regression to assess whether the x and/or y coordinate of each penetration site affected the likelihood of a significant ANCOVA effect. In precentral (motor plus premotor) cortex, the likelihood of an SP effect increased with increasing (anterior) distance from the central sulcus, whereas in prefrontal cortex an SP effect was less likely with greater dorsal distance from the principal sulcus. This was evident in a significant x coordinate coefficient >1 in precentral cortex, and a significant y coordinate coefficient <1 in prefrontal cortex (precentral: x coordinate, exp(B) = 1.26, Wald test (1) = 114.3, p < 0.001; y coordinate not significant; prefrontal: x coordinate not significant; y coordinate, exp(B) = 0.77, Wald test (1) = 17.6, p < 0.001). For significant effects of location and SP × location interaction, these gradients (anteroposterior in precentral cortex, and dorsoventral in prefrontal cortex) were stronger than for the SP effect (precentral, location: x coordinate, exp(B) = 1.34, Wald test (1) = 126.2, p < 0.001; y coordinate not significant; precentral, SP × location: x coordinate, exp(B) = 1.29, Wald test (1) = 146.1, p < 0.001; y coordinate not significant; prefrontal, location: x coordinate not significant; y coordinate, exp(B) = 0.58, Wald test (1) = 45.9, p < 0.001; prefrontal, SP × location: x coordinate not significant; y coordinate, exp(B) = 0.58, Wald test(1) = 58.6, p < 0.001). Overall, neurons with an SP effect were distributed more evenly throughout frontal cortex, whereas neurons whose activity related to stimulus location (either as a main effect or SP × location interaction) were less evenly distributed.
Neuronal activity associated with SP
Figure 5 shows examples of neurons whose activity varied significantly with the SP within the sequence, but not with the location of the stimuli or the interaction of SP and location. Figure 5A shows activity from a neuron recorded in premotor cortex with sequences of three stimuli (list length 3). Here, the 10 trials for each sequence are shown as raster plots. For each sequence, the mean cell activity during each SP epoch (and during the control period) is indicated by the large dots superimposed on the rasters. In the main ANCOVA, the F test for the SP factor evaluates the null hypothesis that the mean activity is the same during each SP epoch. The plot shows that the activity in the third epoch was consistently lower than in the first two epochs, and indeed the F value for SP (F(2,211) = 176.88) was highly significant (p < 0.001). Neither the location factor nor the SP × location interaction was significant for this cell. Graphically, this is reflected by the similarity of the mean activity across sequences; for example, the activity during the third epoch was similar in each of the eight sequences and did not vary with the location of the stimulus presented during that epoch. The stimulus locations comprising each sequence are indicated by numbers below each raster plot (“0” is to the right, “90” is up, etc.) to illustrate the balanced design of the sequences. By looking vertically across sequences (i.e., at each column of numbers), it is apparent that each location occurs exactly once in each SP. Figure 5A (bottom histogram) plots the average neuronal activity in 50 ms bins, with all sequences combined. Because this neuron had a significant effect of SP but not of location (either alone or as an interaction), collapsing across sequences is justified and allows the effect of SP to be shown in a single plot, rather than split up into plots for each sequence as in the raster displays.
Examples of neurons with a significant SP effect. A, The neural activity of a premotor neuron during each correct trial of 8 sequences of list length 3. The location of each sequence stimulus is indicated numerically under the rasters (0 = stimulus to the right of the center-hold window, 90 = stimulus above the center-hold window, etc). For each sequence, the 10 correct trials (5 repetitions with the first stimulus as test stimulus and 5 with the second as test stimulus) are plotted as rasters. The mean ± SD firing rates during each epoch (center-hold period and SP epochs 1–3) are shown by the large black dots and error bars, overlying the rasters in the center of each SP epoch (separated by vertical gray lines). This neuron had a significant effect of SP in the main ANCOVA (F(2,211) = 176.88, p < 0.001), meaning that the null hypothesis (that activity during each SP epoch was equal) was rejected. The activity during the third SP epoch was consistently lower than that during the first two SP epochs. This neuron did not have a significant effect of either location (F(7,211) = 1.43, p = 0.195) of the individual stimuli, or of the interaction SP × location (F(14,211) = 1.34, p = 0.187). This can be seen in the similar raster plots for each of the eight different sequences. Because the individual stimuli did not significantly influence cell activity in this case, the activity during sequence presentation was summarized by combining all 8 sequences into one histogram (bottom). Gray vertical lines through the histogram indicate the onset of the first stimulus and the onset of the test stimulus. The scale activity in spikes/s is shown to the right of the histogram. B, Raster plots and histograms of neurons with a significant SP effect, but no significant effect of location or SP × location interaction. The top of each plot represents rasters of spike activity during individual trials. All sequences are intermixed because the location of individual stimuli did not significantly affect cell activity. Below the rasters is a histogram of spike activity for each cell. The two cells on the top row were recorded in motor cortex, the two from the middle row in premotor cortex, and the two on the bottom row in prefrontal cortex. Gray vertical lines through the rasters and histograms indicate the onset of the first stimulus and the onset of the test stimulus. There is wide variety of activity profiles observed in neurons with a significant main effect of SP. ANCOVA results (cell ID, F for SP, p value for SP): aa037.18, F(2,211) = 63.18, p < 0.001; aa045.13, F(3,443) = 110.34, p < 0.001; aa114.23, F(4,755) = 65.46, p < 0.001; aa105.18, F(3,443) = 69.31, p < 0.001; aa063.24, F(2,211) = 46.94, p < 0.001; aa063.20, F(2,211) = 35.99, p < 0.001. The scale activity in spikes/s is shown to the right of each histogram. S1, S2, S3, S4, S5, first, second, third, fourth, and fifth SP epoch, respectively.
Figure 5B shows further examples of neurons with a significant SP effect and no significant location or SP × location interaction effect in the main ANCOVA. The rasters for all trials are shown (all sequences are intermixed), as well as the histogram of mean activity. Two neurons from each cortical area are illustrated. Several notable aspects of the SP-related activity can be appreciated in Figure 5B. First, activity changes included both increases and decreases from control period activity. Second, the deviations from baseline activity could occur during the early, middle, or late stages of the list presentation period (i.e., they did not occur only at certain SPs). Finally, sometimes the activity changes appeared to be “time-locked” to particular SP epochs, whereas in other cases they spanned several SP epochs. For example, the cell in Figure 5B (bottom right) had a burst of activity during SP 2, whereas the cell at the top left had a burst of activity that began around the middle of the second SP epoch and continued through most of the third SP epoch. Other cells, such as the one in Figure 5B (middle right), showed gradual activity changes across all SPs.
Neuronal activity associated with SP × location interaction
The results of the main ANCOVA indicated that very few neurons had a significant location effect alone. However, the activity of a large proportion of neurons in each cortical area was significantly affected by the interaction between SP and location of the stimulus (Fig. 3). A significant interaction effect can be interpreted as an influence of stimulus location that is only manifest at certain SPs (or conversely, an influence of SP that occurs only for certain stimulus locations). We examined more closely the activity of these neurons by analyzing the directional effect during each SP epoch of the list presentation. More specifically, we sought to explore whether the activity during a given epoch was spatially tuned and, if so, whether the activity was tuned to the current stimulus or to a stimulus presented previously during the list presentation. To this end, we computed the mean resultant of the epoch activity associated with the current stimulus and the mean resultant of the same activity associated with the direction of the stimuli presented in the previous epochs. Then, we selected the greatest mean resultant and tested its statistical significance using the bootstrap procedure (1000 bootstraps).
We found that the activity of most SP × location neurons was directionally tuned in at least one epoch of the list presentation. In each epoch, the neuronal activity could be tuned to the current or to a previous stimulus. In addition, neurons could have directionally tuned activity in one or more epochs of the list presentation period. We sorted these different cases by classifying neurons into the following categories: (1) neurons that were tuned in one epoch to the current stimulus, (2) neurons that were tuned in more than one epoch to the current stimulus, (3) neurons that were tuned in one epoch to one of the previous stimuli, (4) neurons that were tuned to a single stimulus in more than one epoch, and (5) neurons having a more complex tuning pattern. All neurons in the last group were tuned during multiple epochs but, unlike neurons in category 4, they were tuned to different stimuli across epochs. The proportion of neurons in each of these categories is indicated in Table 3 for each cortical area and each list length. We found that many interaction neurons had a significant directional tuning to the current or a previous stimulus in only one epoch of the list presentation period. Remarkably, the proportion of neurons directionally tuned to a previous stimulus was higher than that of neurons tuned to the current stimulus.
Directional tuning of neurons with a SP × location interaction
The activity of two neurons tuned to the direction of the current stimulus is illustrated in Figure 6. Figure 6A shows the activity of a motor cortical neuron tuned to the current stimulus in a single epoch of the list presentation, whereas Figure 6B shows the activity of a prefrontal neuron tuned to the current stimulus in two successive epochs. The average activity during one SP epoch (plotted in black on the histograms) is plotted against the direction of the current stimulus for that epoch in the bottom plots. In the example illustrated in Figure 6A, the neuron had little activity during the first two epochs of the list presentation but increased activity during the third epoch. The activity during the third epoch was directionally tuned to the direction of the third stimulus, with a preferred direction near 315 degrees. Figure 6B shows a neuron whose activity was tuned to the current stimulus during each of the last two epochs. The two columns of Figure 6B show the same histograms but arranged according to the direction of the second stimulus (left column) or to the direction of the third stimulus (right column). In both epochs, the preferred direction of the neurons was close to 0 degrees.
Examples of SP × location interaction neurons: tuning to the current stimulus in one or more epochs. A, Histograms of activity and tuning curves from a motor cortex neuron that had a significant SP × location interaction in the main ANCOVA for list length 3 (F(14,211) = 3.0, p < 0.001). Histograms represent mean cell activity during the control period and list presentation period of the memory scanning task with list length 3. Vertical lines on the histograms indicate each SP epoch. The sequences of stimuli are indicated numerically under the histograms. Tuning analyses revealed that this neuron was tuned to the location of the third stimulus during the third SP epoch (preferred direction = 323 degrees, bootstrap p < 0.001). As such, the eight histograms (sequences) are arranged according to the location of the third stimulus (numbers in bold font), and the activity during the third SP epoch is black on the histograms, whereas that during other periods is gray. The tuning curve is plotted below the histograms, with mean ± SD cell activity during the third SP epoch plotted for each of the eight directions of the third stimulus. B, Example of a prefrontal neuron that had a significant SP × location interaction in the main ANCOVA for list length 3 (F(14,211) = 9.97, p < 0.001), which was tuned to the current stimulus in two SP epochs (preferred direction during epoch 2 = 6 degrees, p < 0.001, preferred direction during epoch 3 = 356 degrees, bootstrap p < 0.001). The two sets of histograms plot the same cell activity, but the histograms on the left are arranged according to the second stimulus whereas those on the right are ordered with respect to the third stimulus. In each case, the relevant cell activity is depicted in black on the histogram, and the relevant stimulus is depicted in bold font below the histogram. The tuning curves below the histograms show that the same directional tuning was maintained across both SPs. Thus, this neuron coded the location of the current stimulus in both the second and third SP epochs.
An example of a motor cortex neuron tuned to a previous stimulus is illustrated in Figure 7A. As in Figure 6, histograms for each of 8 sequences of list length 3 are shown. The activity of this neuron was tuned to the second stimulus during the third SP epoch. The activity during the third SP epoch (in black on the histograms) is plotted against the location of either the second (left column) or third (middle column) stimulus in the plots below the histograms. The two columns contain the same histograms arranged according to either the second or third stimulus locations (marked in bold font in the numerical sequence list). The figure shows that the activity of this neuron during the third SP epoch was not tuned to the location of the currently presented stimulus. Instead, it varied in an orderly manner with the location of the stimulus immediately preceding the current one (left column). As Table 3 indicates, this was among the most common forms of directional tuning found among interaction cells in all three cortical areas, and we refer to such neurons as “previous epoch cells.”
Examples of SP × location interaction neurons: tuning to a previous stimulus. A, Histograms of activity and tuning curves from a motor cortex neuron that had a significant SP × location interaction for list length 3 trials in the ANCOVA (F(14,211) = 2.73, p = 0.001). Histograms represent cell activity during 8 sequences of list length 3. This neuron had a significant directional tuning during the third SP epoch (blackened in the histograms). However, the tuning was not with respect to the current (third) stimulus but rather to the stimulus that was presented in the previous (second) SP epoch. Thus, the left histograms are ordered with respect to the second stimulus (numbers in bold font). This tuning toward the second stimulus is seen in the bottom left tuning curve (preferred direction of second stimulus during epoch 3 = 157 degrees, bootstrap p < 0.001). The histograms on the right (middle of page) show the same data, but now ordered with respect to the third stimulus (bold font). The plot below shows mean cell activity (±SD) in the third SP epoch with respect to the location of the third stimulus. There was no significant directional tuning to the current (third) stimulus, as evidenced by this plot and the directional analyses (bootstrap p > 0.05). B, Example of a prefrontal neuron with a significant SP × location interaction in the ANCOVA for list length 4 (F(21,443) = 2.51, p < 0.001). For this cell, activity during the third SP epoch was significantly tuned to the direction of the second stimulus (preferred direction of second stimulus during epoch 3 = 192 degrees, bootstrap p < 0.001).
Figure 7B shows an example of another previous epoch cell, recorded from prefrontal cortex during list length 4 trials. This cell showed activity in the third SP epoch that was tuned to the direction of the second stimulus. When SP × location neurons displayed directional tuning during the list presentation, this tuning could occur in any SP epoch but tended to occur more frequently in later SP epochs. For neurons that were tuned to the current stimulus in a single epoch (Table 3; Fig. 6A), 65.4% of motor cortical neurons showed directional tuning in the final SP epoch, whereas 34.6% were tuned in earlier epochs. In premotor and prefrontal cortex, 58.3% and 35.7% of neurons, respectively, were tuned during the final epoch. For cells tuned to a previous stimulus in a single epoch (Table 3; Fig. 7), 78.7%, 71.0%, and 64.2% of motor, premotor, and prefrontal neurons, respectively, showed tuning in the final SP epoch.
In summary, most recorded neurons in all areas were modulated by SP during the list presentation, either as a main effect or as an SP × location interaction. When cell activity reflected the location of the presented stimuli, it was almost always in the form of an SP × location interaction, with activity during certain SP epochs tuned to the location of the current stimulus or, more commonly, to a previously presented stimulus.
Absolute versus relative coding of SP
Because we recorded many neurons in sequences of different length, we were able to investigate the question of whether SP was encoded in absolute or relative terms. We restricted this analysis to neurons with a significant SP effect only (no location or SP × location interaction effect) because in such cases the individual elements of the sequences (locations) could be ignored and all sequences could be combined. We examined the question of whether the modulation of neuronal activity during the list presentation followed a fixed time pattern, or whether its evolution in time changed relative to the list length. In the first case, cells with an SP effect would be signaling absolute SP, and we would expect the activity profile for different list lengths to be similar across common SP epochs. The activity during the “new” SP epochs in the longer list lengths would be concatenated on to this common activity profile. In contrast, in the second case, the time-varying pattern of neuronal activity would have the same global shape regardless of the list length; that is, the pattern of activity would be adjusted relative to the length of the list presentation. In this case, the neural activity would reflect relative time within the list presentation, signaling relative SP. We found that the time profile of neuronal activity was consistent with the relative SP coding. Figure 8 shows three examples of the time course of activity across list lengths. Histograms represent average activity during list presentation for each of three list lengths, for three different motor cortex neurons with a significant SP effect only. One can see the “stretching” of the same general pattern of activity, such that the histogram assumes the same shape over the entire list presentation period regardless of the number of stimuli that were presented.
Relative coding of SP: examples of motor cortical neurons with an SP effect. Histograms of cell activity during list presentation are shown for three motor cortical neurons (one neuron per column). Three histograms are shown for each neuron, one each for list lengths 3 (top row) through 5 (bottom row). Each of the three neurons had a significant SP effect but no significant effect of location or SP × location interaction in the ANCOVA. The overall shape of the activity profile was present in all list lengths; thus, the activity peaks often occur in different SP epochs depending on the list length. This implies that these neurons encode SP information in a relative rather than absolute manner. ANCOVA results (cell ID, list length, F value for SP, p value for SP) are as follows: aa035.17, 3, F(2,211) = 11.51, p < 0.001; aa035.17, 4, F(3,443) = 24.51, p < 0.001; aa035.17, 5, F(4,755) = 13.19, p < 0.001; aa012.14, 3, F(2,211) = 12.12, p < 0.001, aa012.14, 4, F(3,443) = 25.47, p < 0.001; aa012.14, 5, F(4,755) = 22.15, p < 0.001; aa025.9, 3, F(2,221) = 41.66. p < 0.001; aa025.9, 4, F(3,443) = 60.71, p < 0.001; aa025.9, 5, F(4,755) = 104.04, p < 0.001. The scales of firing rate are included only for the list length 5 histograms but are the same across list lengths for each neuron.
To quantify this observation, we calculated two sets of correlations between the time series of activity for sequences of a given list length with the time series of activity in a different list length. The first correlation evaluated the similarity in actual time of the activity profiles in different list length conditions. Therefore, the length of the time series used in this analysis was limited to the length of the shortest list length selected for the comparison, to compare only common SP epochs. For example, to compute the correlation of the time-varying activity during sequences of three and five stimuli, we used the activity recorded during the list presentation of the first three stimuli in both list length conditions. The correlation of activity in actual time tested how similar the patterns of activity were across common SP epochs for different list lengths.
The second correlation evaluated the similarity of the time-varying activity across different list lengths in a relative time scale. In this case, the time series for the shortest list length was linearly interpolated to extract the same number of samples as the longest series selected for the comparison. Therefore, this correlation evaluated how similar the patterns of the time-varying activity were from the beginning to the end of the list presentation, regardless of the difference in list length. The cell activity was scaled to its minimum and maximum in each list length, to more directly compare the shape of the activity profile across list lengths. Figure 9A shows the time course of the normalized activity of a neuron from each cortical area during the presentation of sequences of three and five stimuli, using both actual (left plots) and relative (right plots) time scales. In each example, the activity profiles across list lengths were better matched in relative time than in actual time.
Comparing the neural activity profile across list lengths: actual versus relative time. A, Examples of one neuron from each cortical area. Plots of time-varying neuronal activity during the list presentation of 3 and 5 stimuli in either an actual (left plots) or a relative (right plots) time scale. Gray line indicates the mean activity profile during list length 3 trials. Black line indicates the mean activity profile during list length 5 trials. For each plot, activity is scaled between its minimum and its maximum activity. Plots in left column represent cell activity in actual time, and the abscissa is scaled in milliseconds. Vertical dotted lines indicate the onset of each list stimulus and of the test stimulus. Plots on right represent neural activity during each list length on the same (relative) time scale, and the abscissa is unit-less. The patterns of activity were better correlated in the relative time scale than in the actual time scale, as visualized by the better matching of the curves and by the correlation coefficients listed above each plot. B, The scatterplot represents the correlation coefficients between the time-varying neuronal activity during the list presentation of three and five stimuli, in actual versus relative time scales, for 376 motor cortical neurons that had an SP effect only. The distribution of correlation coefficients for each time scale is plotted as histograms next to the scatterplot. The correlations computed using the relative time scale were skewed toward higher values than those computed using the actual time scale.
We compared the two correlations for each pair of list lengths and for each cortical area. In all cases, we found that the correlations based on a relative time-varying pattern of activity were higher than the correlations of activity in actual time. However, the most informative of these analyses is the one comparing the patterns of activity during sequences of three and five stimuli. In this case, the difference between the actual and the relative time scale is the greatest; therefore, the possibility of discrimination between the two hypotheses is the most favorable. The results of this analysis are presented in Figure 9B and Table 4. Figure 9B plots the correlation of activity in actual time against the correlation of activity in relative time for motor cortical neurons. The distribution for each type of correlation is illustrated in the margins of the scatterplot in the same figure. The distribution of correlations based on relative time were more skewed toward higher values than the correlations based on actual time. Table 4 shows the average correlations, calculated using Fisher's z transformation (Snedecor and Cochran, 1989), for each cortical area. The correlation coefficients based on a relative time scale were significantly higher than those based on actual time in each cortical area investigated.
Correlation analysis of the time-varying neuronal activity during the list presentation of three and five stimuli
Next, we explored this relative coding of SP at the neuronal ensemble level. Figure 10 shows raster plots of each trial of the memory scanning task with list lengths 3, 4, and 5, from an ensemble of 13 simultaneously recorded motor cortical neurons. Each line represents one trial, and each of the 13 cells is plotted in a different color. All sequences are shown, randomly intermixed within each list length. During sequences of three stimuli (top of the figure), the three SPs are easily distinguishable by the predominant colors in that epoch (blue/mixed during epoch 1, yellow during epoch 2, and pink during epoch 3). During sequences of four or five stimuli (middle and lower plots), this pattern is similar and stretched across the entire list presentation. Thus, the relative (as opposed to absolute) coding of SP was evident at both the single-cell and cell ensemble levels.
Relative coding of SP: ensemble of 13 simultaneously recorded neurons. The neural activity of a simultaneously recorded ensemble is plotted as rasters. Data are from Monkey 2, motor cortex, during the list presentation period of the memory scanning task with list length 3 (top), 4 (middle), and 5 (bottom). The spikes from each of the 13 neurons in the ensemble are plotted with a different color. The spikes were plotted thicker than in previous raster plots, to aid the discrimination of the different colors. Each line indicates one trial. All correct trials are plotted, with the eight different sequences intermixed. The SP epochs are separated by the white vertical lines and labeled S1, S2, etc. The duration of each SP epoch is 650 ms. In list length 3 trials, the 3 SP epochs are easily distinguishable based on the predominant colors (blue/mixed for S1, yellow for S2, and pink for S3). However, this coding shifts with list length; for example, in list lengths 4 and 5, the pink-predominant SP epoch is not S3, but rather the last epoch. This is consistent with the relative coding of SP hypothesis, rather than absolute coding of SP.
Decoding of sequence elements from neuronal ensemble activity: discriminant analysis
The results of the main ANCOVA indicated that, during the list presentation, many neurons conveyed information about SP, via either an SP main effect or an SP × location interaction. Furthermore, the activity changes of these neurons took place throughout the list presentation period; that is, they did not occur only in certain SPs (Figs. 2, 5–9). As seen in Figure 10, the patterns of activity within a neuronal ensemble were often distinct in different SP epochs, raising the question of whether it is possible to accurately decode the sequence elements using the activity of a population of neurons.
To address this question, we computed quadratic discriminant functions to see how accurately SP epoch could be classified using ensembles of simultaneously recorded neurons.
In essence, this analysis evaluates how well the pattern of activity (across multiple neurons) during a given SP epoch can be distinguished from the pattern of the same ensemble during other SP epochs. At the level of a single sequence, each sequence element is uniquely identified by both its SP and its location. For example, for a sequence of 90-270-45, identifying a sequence element as “270” or “the second stimulus” is equivalent. The discriminant analysis was applied using each sequence separately, and the resulting classification rates were averaged across sequences, and thus indicated how accurately the individual sequence elements were classified (as opposed to SP or location per se). Because all neurons were included in this analysis, and the most common significant effects in the main ANCOVA were SP and SP × location interaction, activity relating to both factors likely contributes to the classification.
Figure 11A shows the percentage of correctly classified sequence elements (decoding accuracy) for each list length, cortical area, and ensemble size. Decoding accuracy was computed for each sequence within a given list length, averaged across sequences to obtain a mean classification accuracy for each SP epoch, and then averaged across ensembles of a given size to obtain the means and SDs plotted in Figure 11A. The chance levels of correct classification are 33.3%, 25%, and 20% for list lengths of 3, 4, and 5, respectively. The classification rates were above chance for all combinations of ensemble size, list length, and cortical area. We also found that the decoding accuracy increased as a function of the number of cells in the ensemble. Extrapolating the power functions fitted to the data in Figure 11A suggests that, in any of the three cortical areas, an ensemble of as few as 10 neurons could distinguish the individual elements in a sequence of three stimuli with 100% accuracy. The ensemble sizes required for perfect decoding in list lengths 4 and 5 were ∼14 and 16 neurons, respectively. Figure 11B shows the mean (±SD) decoding accuracy for each list length and cortical area, averaged across ensemble size, for ensemble sizes of 3–7 neurons (we limited this analysis to ensembles of 3–7 neurons because these ensemble sizes were obtained for nearly all combinations of list length and cortical area). The decoding accuracy tended to differ between cortical areas (F(2,34) = 4.51, p = 0.018, ANOVA). Decoding accuracy was highest in premotor cortex and lowest in motor cortex. In addition, decoding accuracy decreased with list length (Fig. 11A,B; F(2,34) = 14.42, p < 0.001, ANOVA).
Decoding sequence elements via discriminant analysis. A, Decoding accuracy as a function of the number of simultaneously recorded neurons, plotted for each cortical area and list length (indicated by different symbols, and numerals 3–5 to the right of each curve). Each symbol represents the mean ± SD classification rate for a given neuronal ensemble size, averaged across ensembles. Curves indicate least-squares fitted power functions. B, Decoding accuracy differed across list length (more accurate for shorter list lengths) and tended to be highest in premotor cortex and lowest in motor cortex. Symbols represent mean ± SD classification rate for ensembles of 3–7 neurons, averaged across ensemble size.
Comparison of list presentation period activity with movement period activity
As shown above, many neurons in each of the cortical areas investigated showed significant effects of the sequence parameters (SP, location, or their interaction) during the list presentation period. However, neurons in each of the three areas are also known to have directionally selective activity during arm movements to targets in 2D space (Georgopoulos et al., 1982; Weinrich et al., 1984; di Pellegrino and Wise, 1993). For this reason, the question arises whether the cells that showed activity related to the sequence parameters during list presentation (when no arm movements took place) also were related to movement direction during the response, or conversely are there separate populations of neurons that relate either to movement or to sequence parameters, but not both. Because the memory scanning task required a directional arm movement to the appropriate stimulus during the response period, we could evaluate the prevalence of directionally selective movement period activity using the same cells and the same trials as were used in the main ANCOVA to evaluate activity during list presentation.
To this end, we performed a one-way ANCOVA for each cell and list length with the activity during the movement period as the dependent variable, the direction of the target as the factor, and the control rate as a covariate. We omitted the reaction time period from this analysis, to separate more clearly the movement-related activity from activity during list presentation. Figure 12 shows the proportions of neurons in each cortical area that showed the following: (1) a significant effect of one or more of the sequence parameters (SP, location, or SP × location) during list presentation, (2) a significant effect of target direction during the movement, (3) both of the above, and (4) no effects in either the list presentation or movement periods. The light gray, black, dark gray, and white sections, respectively, represent these categories in the Venn diagrams of Figure 12. We calculated the proportions separately for each list length and then averaged them to produce the data in Figure 12. Activity related to sequence parameters was at least as common as activity related to the direction of movement in each cortical area investigated. Significant effects of sequence parameters were more common than significant effects of movement direction in the premotor and prefrontal areas, whereas the two types of effects were about equally prevalent in the motor cortex. Overall, motor cortex had the highest proportion of cells that was exclusively related to movement, prefrontal cortex had the highest proportion exclusively related to sequence parameters, and premotor cortex had the highest proportion related to both. In all areas, a large proportion of cells that had directionally selective activity during the movement also had a significant relation to sequence parameters during the list presentation period. That is, these neurons were remarkably protean, coding for different aspects of the task during different periods of the trial.
Proportion of neurons with list presentation and movement direction effects. Area-proportional Venn diagrams showing the combined results of two ANCOVA for each cortical area: (1) the ANCOVA applied to neural activity during the list presentation period, and (2) the ANCOVA applied to neural activity during the movement time. The category list presentation refers to neurons that had a significant effect of any of the factors SP, location, or their interaction in the main ANCOVA. The category movement direction refers to neurons that had a significant directional effect during the movement time. The area of each Venn diagram section is scaled according to the proportion of neurons in the corresponding category. Within each area, the proportions were calculated separately for each list length and averaged to give the proportions in the figure. In all areas, neurons with list presentation effects were more common than neurons with movement direction effects. Many cells in each area had both list presentation and movement direction effects, particularly in motor and premotor cortex.
To further compare the neuronal activity during list presentation with that during the movement, we selected cells with significant directional tuning during both periods and calculated the circular correlation of the two preferred directions. For this analysis, we selected neurons with a significant SP × location interaction in the ANCOVA, which were directionally tuned in only one epoch during list presentation (Table 3) and also displayed directional tuning during the response. The results are presented in Table 5. Overall, there was no systematic relation between the preferred directions during the list presentation and during the response in motor and premotor cortex. The circular correlations were higher in the few prefrontal cells that were directionally tuned in both periods (although they did not reach statistical significance).
Circular correlation between the neuronal preferred directions during list presentation and during movement
Analysis of the pattern of eye movements during list presentation
We were interested in characterizing the patterns of eye movements while the monkeys performed the task, given that the direction of gaze was not constrained. Eye movements during the list presentation period varied greatly from trial to trial, even when the same sequence of stimuli was presented. An example of the pattern of eye movements during two trials with the same sequence of stimuli is presented as spatial plots in Figure 13A. The sequence of stimuli is indicated by the labels S1-S5, and each stimulus is plotted with a different color. The trace for the direction of gaze is similarly color coded for each epoch of the list presentation. These examples show that there was often a saccade toward the currently presented stimulus, although that was not always the case. In addition, there were often saccades directed to stimuli presented in a previous epoch of the sequence. Finally, there were often exploratory eye movements toward regions of the screen that did not have a stimulus yet present.
Eye movements during the list presentation period. A, Two trials of the same sequence of five stimuli are shown. Each of the five stimuli is drawn with a different color. Red represents S1. Green represents S2. Blue represents S3. Gray represents S4. Orange represents S5. Eye position (direction of gaze) is plotted in the color corresponding to that SP epoch. For example, the red tracing plots the direction of gaze during the first SP epoch, the green tracing during the second SP epoch, etc. Black circle in the middle represents the center-hold window. B, Probability of Monkey 1 fixating the center-hold window (black) or one of the list stimuli (same color code as in A), throughout the time course of the control period and list presentation period. Top, Results for list length 3. Bottom, Results for list length 4. C, Probability of Monkey 2 fixating the center-hold window (black) or one of the list stimuli (same color code as in A), throughout the time course of the control period and list presentation period. Top, Results for list length 3. Middle, Results for list length 4. Bottom, Results for list length 5. Although there was quite a bit of trial-to-trial variability in the eye movements (A), both monkeys had a similar strategy of fixating a stimulus presented toward the middle of the sequence during the last SP epoch (B, C).
Although the eye movements were not stereotyped, we found that there were regularities in their pattern. We computed the proportion of eye fixations on each stimulus (see Materials and Methods) during the time course of the list presentation. The results are presented in Figure 13B, C for each monkey and each list length. The time-varying proportion of eye fixations on each stimulus is color coded as in Figure 13A. During the center-hold period, both monkeys tended to look at the center of the screen (black line) until the presentation of the first stimulus, which was fixated in >50% of the trials (red line). The duration of the fixation on a stimulus was generally less than the duration of the epoch. After the fixation of the first stimulus, the pattern of gaze direction differed between the 2 monkeys.
After the first stimulus, Monkey 1 (Fig. 13B) looked more frequently at the center of the screen than any of the other stimuli, except during the last epoch of the list presentation. At the very end of the list presentation, Monkey 1 looked more frequently at a stimulus that was presented in the middle of the sequence: the second stimulus (green line, 52% of trials) for sequences of three stimuli, and equally likely the second or third stimulus (green and blue line; 27% and 32% of trials, respectively) for sequences of four stimuli. In contrast, Monkey 2 (Fig. 13C) generally did not look at the center of the screen during the list presentation. Instead, eye fixations by this monkey were often made to the current stimulus; this was the most frequent for the first stimulus and less so for the following ones. Similar to Monkey 1, during the last epoch of the list presentation, Monkey 2 looked preferentially to one of the stimuli in the middle of the sequence. Eye fixations to the middle stimuli started even during the penultimate epoch for list lengths of four and five stimuli. The distribution of eye fixation at the end of the sequence was slightly different between Monkey 2 and Monkey 1. For sequences of three stimuli, Monkey 2 usually fixated the second stimulus at the end of the list presentation (73% of the trials), which is similar to the eye fixation pattern of Monkey 1. In list length 4 trials, the second stimulus was also the most common fixation target (89% of the trials) at the end of the list presentation. Finally, for sequences of five stimuli, the second (37% of trials) and third (55% of trials) stimuli in the sequence were most often fixated at the end of the list presentation for Monkey 2.
In summary, these results show that, at the end of the list presentation period, both monkeys directed their gaze toward a stimulus that was in the middle of the sequence presented. This preferential pattern of gaze direction was related to the serial order of the stimulus, and not to its spatial location.
Oculomotor-related neuronal activity
We investigated the possible effect of oculomotor factors on the neuronal activity during the list presentation period of the memory scanning task. To this end, we examined separately the effect of direction of gaze and direction of saccade. Because the criteria to select fixations and saccades (see Material and methods) were not met in all trials, only a subset of neurons from each cortical area could be used for these analyses. The results are reported in Table 6. We found that all cortical areas investigated contained neurons with significant oculomotor effects. The activity of premotor neurons was more commonly related to gaze and saccade than was the activity of motor and prefrontal neurons. Nevertheless, even in motor cortex, a surprising 15% of the neurons analyzed showed a significant effect of gaze direction.
Oculomotor effects on neuronal activity
Given the notable proportion of neurons that showed oculomotor-related modulation of activity, we sought to determine how this may have impacted the results of the main ANCOVA that tested the factors location, SP, and their interaction. Because saccade-related activity was rare, we limited this analysis to neurons that had an effect of gaze direction. We computed the proportion of neurons that had a gaze direction effect within each category of effect in the main ANCOVA. The results for each cortical area, averaged across list lengths, are illustrated in Figure 14. Only the results with no effect, an SP effect, or an SP × location effect are shown because the other effects (location or SP + location) were rarely found. The results show that only a minority of the neurons within each category also had a significant gaze direction effect. In addition, the neurons with a gaze direction effect more commonly had an SP × location interaction effect in the memory scanning task than an SP effect.
Prevalence of significant gaze direction effects among neurons with significant sequence parameter effects. For each cortical area, bars represent the proportion of neurons with a significant gaze direction effect among those that had (1) no significant effects in the main ANCOVA (None), (2) a significant effect of SP but not location or SP × location interaction (SP), or (3) a significant SP × location interaction. The other two categories from the main ANCOVA (location main effect only, or main effects of both SP and location) were excluded because very few neurons fell into these categories. Error bars indicate mean ± SD proportion, averaged across list length. For each category of sequence parameter effects and for all cortical areas, only a minority of neurons had gaze-related activity.
EMG activity during the list presentation
While the monkeys had to maintain the cursor within the center-hold window during list presentation, small movements within that window were possible. For this reason, we tested for possible modulation of EMG activity during the list presentation period by performing the same main ANCOVA on the rectified EMG as the one performed on the neuronal activity. For both monkeys, the majority of muscles did not show any significant effect of the list presentation factors, although a few muscles did show significant effects. The number of muscles in each category of ANCOVA effects is presented in Table 7.
ANCOVA of EMG activity during the list presentation period
Several observations suggest that the modulation of EMG activity during the list presentation was quite different from the modulation of neuronal activity during the same period. First, the results from the ANCOVA indicate that significant effects of list presentation factors were less frequent in the EMG activity than in the neuronal activity. Second, the modulation of EMG activity during the list presentation period was much smaller than the modulation during the response period (Fig. 15), which was not the case for the neuronal activity (Fig. 2). Figure 15 shows the activity of the muscle with the highest F statistic for SP. It can be seen that the change of activity during the list presentation was a very subtle gradual decrease over time, followed by a large increase in activity during the response.
EMG activity during the memory scanning task. A, Average EMG activity during performance of the memory scanning task with list length 3, from the muscle (middle deltoid, Monkey 2) that showed the strongest SP effect in the ANCOVA (F(2,79) = 19.95, p < 0.001). Vertical dashed lines indicate the beginning and end of the list presentation period. The modulation of EMG activity during the list presentation period was small, compared with the large burst of EMG activity around the movement. Nevertheless, a gradual decrease in average EMG activity occurred during list presentation. A horizontal line was drawn below the EMG trace to help visualize this small change in EMG activity during the list presentation. Tick marks along the abscissa indicate (in order) the following: onset of the first, second, and third stimuli, onset of the test stimulus, beginning and end of the movement. B, Mean ± SD of EMG activity for the same data as in A, for each of the 3 SP epochs. Note the difference in scale of the ordinate between A and B. Even though the difference in EMG activity between SP epochs was small, it was statistically significant. Thus, even when EMG activity differed between SP epochs, the change of EMG activity during list presentation was minor compared with modulation of neuronal activity during the same period (Figs. 2, 5). A.U., Arbitrary units; S1, S2, S3, first, second, and third SP epoch, respectively; Test, onset of the test stimulus; Mvmt, movement.
Discussion
The memory scanning task has several useful characteristics for studying the neural correlates of serial order memory. First, unlike most neurophysiological studies of serial order processing, no sequential movements are required thus avoiding the potential confounding of activity related to movement sequencing. Second, unlike classic neurophysiological studies of spatial working memory using the delayed-response task (Fuster and Alexander, 1971; Kubota and Niki, 1971; Funahashi et al., 1989), the memory scanning task requires multiple items to be kept in working memory and allows memory load to be controlled by varying the list length.
Relation of neuronal activity to sequence parameters
Most recorded neurons displayed marked modulation of activity during the list presentation. During this period, the sequence of stimuli had to be remembered (while not making any limb movement) to correctly perform the task. By dividing the list presentation period into epochs corresponding to the presentation of each new stimulus, we were able to evaluate whether the neuronal activity varied with the SP and/or location of each list item (Table 2; Fig. 3). Several notable findings resulted from this analysis. In all three cortical areas, neurons with a significant main effect of stimulus location were rare. At first glance, this may appear surprising given the well-known directional coding of delay-period activity in all three cortical areas (Funahashi et al., 1989; Georgopoulos et al., 1989; di Pellegrino and Wise, 1993). This lack of location effects implies that the neural representation of the sequence was not accomplished by sequentially activating populations of neurons that encode stimulus location. However, because the stimuli remained on the screen throughout the trials, the stimulus locations did not need to be held in memory. Rather, it was the serial order of the stimuli that had to be retained to perform the task, and neurons with a significant effect of SP were common in all three areas. Furthermore, the profile of the SP-related activity was not stereotyped; rather, it varied widely across neurons (Figs. 5, 8). These properties suggest that these neurons participate in the encoding of the serial order of the stimuli.
However, accurate coding of SP alone is not sufficient to provide a neural basis for accomplishing the task; the serial order information must be linked to the individual stimuli, which are distinguished by their position on the screen. In this regard, many neurons showed activity related to both location and SP, in the form of an interaction effect. Frequently, this entailed activity changes in one epoch that related to the location of the stimulus in the previous epoch (Table 3; Fig. 7). Such activity might reflect the “linking” between sequence elements that is crucial for sequence coding.
Absolute versus relative SP coding
We tested the nature of the serial order coding by comparing the activity of single neurons across list lengths, using either an absolute or a relative time scale (Figs. 8–10; Table 4). In each cortical area, the activity profiles were better correlated when the duration of list presentation was normalized; thus, these neurons appear to encode SP information in a relative rather than absolute manner. This evidence from single-neuron activity is consistent with findings from analyses of errors made by human subjects in serial order memory tasks, which also suggested relative SP coding (Henson, 1999).
This relative SP coding may provide an efficient system for representing serial order information, which does not require multiple populations of neurons each activated for specific ordinal positions. However, such a coding scheme poses some limitations as well. An open question is how such relative coding would work if the list length was not known in advance. In addition, relative coding of SP inherently limits the amount of information that can be held in working memory. By necessity, the neuronal activity becomes less distinctive for any particular sequence item as the same activity profiles encompass increasingly more SPs with increasing sequence length (Fig. 16). As the resolution of serial order coding declines with increasing sequence size, we would expect performance to deteriorate as well, which was observed both in the monkeys' behavior and in the discriminant analysis. Limited capacity is also a cardinal feature of human working memory (Miller, 1956; Cowan, 2001). Figure 16 also shows that, as sequence size increases, activity patterns for adjacent SPs become less distinct, which could explain the transposition errors seen in human subjects, in which sequence items closer in SP are more likely to be confused than items farther apart (known as the “locality constraint”) (Henson et al., 1996).
Relative coding of SP implies decreasing resolution of serial order coding with increasing sequence size. The activity of the same three neurons depicted in Figure 8 is shown here (depicted in magenta, green, and blue, respectively). These motor cortex neurons all showed a significant effect of SP on cell activity (with no significant effect of location or SP × location interaction), and all showed relative rather than absolute coding of SP, such that a similar activity profile was seen across the list presentation period, regardless of list length. For each neuron, activity across the three list lengths was combined, by adjusting the bin size such that there were 65 bins during the list presentation period (bin size = 30 ms for list length 3, 40 ms for list length 4, and 50 ms for list length 5). A, B, Histograms represent the activity of each of the three cells with the ordinate showing normalized activity and the abscissa showing a relative time scale of progress through the list presentation period. Histograms represent mean normalized activity ± 1 SE (shaded areas). Symbols (magenta circle, green square, blue triangle) represent the mean activity during selected SP epochs. A, List length = 3. When the activity profiles of these three cells are used to encode only three SPs (demarcated by vertical lines), each SP corresponds to a unique pattern of activity across the cells (magenta ≫ green > blue for SP 1, green ≫ magenta > blue for SP 2, etc). B, List length = 10. However, should the same activity profiles be used to encode 10 SPs, the individual (and especially adjacent) SPs become less distinct. For example, the pattern of activity across the three cells is similar between SPs 1 and 2, and between SPs 5 and 6 (see the superimposed symbols, which show identical ordering and similar mean activity in those SPs).
Comparison between cortical areas
We found neurons with activity related to sequence parameters in all three cortical areas. In each area, activity frequently varied as a function of the SP within the sequence (Table 2; Fig. 3). Each area also contained a large number of neurons whose discharge varied with both the location and SP of the stimuli, in the form of an interaction effect. Neurons whose activity reflected stimulus location (whether via a main effect of location or an interaction effect) were more common in premotor and prefrontal cortex than in motor cortex (Fig. 3). Within prefrontal cortex, there was a dorsoventral gradient, such that neurons closer to the principal sulcus were more likely to have a location or interaction effect than those recorded more dorsally (Fig. 4). These results are consistent with those of Ninokura et al. (2004) who found a similar gradient during sequence presentation, with serial order-specific cells located above (dorsal to) the principal sulcus, and object-specific cells more ventrally. Decoding SP from the activity of simultaneously recorded cells via discriminant analysis showed greatest decoding accuracy in premotor cortex, followed by prefrontal and then motor cortex.
In contrast to the list presentation effects, an effect of movement direction during the response was, as expected, more common in motor and premotor cortex than in prefrontal cortex. In both areas a large proportion of cells with sequence-related activity during the list presentation also showed direction-related activity during the response, whereas in prefrontal cortex most cells showed only list presentation effects (Fig. 12). Remarkably, these results show that many neurons had a heterogeneous functionality by representing distinct task variables at different periods of the task, that is, sequence parameters during the list presentation, and movement direction during the response.
Do the activity changes during list presentation truly represent sequence parameters?
The analyses discussed above indicate that neural activity during the list presentation period often varied with sequence parameters (SP and location). Thus, it is possible that these neurons were contributing to the neural encoding of the sequence. However, it is also possible that these activity changes were instead related to other variables that are confounded with sequence parameters (e.g., eye movements, limb movements, or visual responses). Because eye movements are known to influence neural activity in premotor (Boussaoud et al., 1998; Lebedev and Wise, 2001; Cisek and Kalaska, 2002) and dorsolateral prefrontal (Funahashi et al., 1991; Boussaoud et al., 1993) areas, we evaluated the dependence of neural activity upon both gaze direction and saccade direction. We found that only a minority of neurons with sequence-related activity were modulated by oculomotor factors (Fig. 14). Thus, the relation between neuronal activity and sequence parameters is not an epiphenomenon of oculomotor-related activity. In addition, in a prior publication (Carpenter et al., 1999), we showed that visual responses likewise cannot account for the neuronal sequence-related activity.
Furthermore, because we recorded from the arm area of motor and premotor cortex, one must consider whether the activity during list presentation was related to limb movements. However, no overt limb movements were made during the list presentation period, and EMG activity was essentially constant during the list presentation period, as would be expected from the task requirements. Although SP effects were seen infrequently in the muscle activity (Fig. 15; Table 7), this effect resulted from small gradual changes in the EMG signal as the trial progressed and contrasted sharply with the large modulations in neural activity during list presentation. Another possibility is that the observed activity reflected intended, but not executed, limb movements to the stimuli (Snyder et al., 1997). If an intended movement was prepared in response to the presentation of each stimulus, then cells with directional movement-related activity would have a main effect of location, which was rarely found (Table 2). If an intended movement was prepared only during some epochs, then such cells would have a significant SP × location effect, which was relatively common (Table 2). However, there was no systematic relation between the preferred directions of these neurons during list presentation and during movement (Table 5). The other common sequence parameter found was an SP main effect, which cannot be explained by directional movement-related activity. Finally, in all areas, many neurons showed list presentation effects and no motor response effects, thus making it unlikely that modulation during sequence presentation was related to movement (Fig. 12).
In conclusion, we found that neuronal activity in motor, premotor, and prefrontal cortex was frequently associated with sequence parameters. Potential confounds could not account for the effects observed. For these reasons, we conclude that these neurons were indeed participating in sequence encoding in working memory and encoded serial order in a relative rather than absolute manner.
Footnotes
This work was supported by Department of Veterans Affairs Merit Review Award CX-0004-37 to G.P. and National Institutes of Health Grant NS17413 to A.P.G.
The authors declare no competing financial interests.
- Correspondence should be addressed to either Dr. Giuseppe Pellizzer or Dr. Adam F. Carpenter, Brain Sciences Center (11B), 1 Veterans Drive, Minneapolis, MN 55417, pelli001{at}umn.edu or carpe004{at}umn.edu
