Abstract
Many decisions, from crossing a busy street to choosing a profession, require integration of discrete sensory events. Previous studies have shown that integrative decision-making favors more reliable stimuli, mimicking statistically optimal integration. It remains unclear, however, whether reliability biases operate even when they lead to suboptimal performance. To address this issue, we asked human observers to reproduce the average motion direction of two suprathreshold coherent motion signals presented successively and with varying levels of reliability, while simultaneously recording whole-brain activity using electroencephalography. By definition, the averaging task should engender equal weighting of the two component motion signals, but instead we found robust behavioral biases in participants' average decisions that favored the more reliable stimulus. Using population-tuning modeling of brain activity we characterized tuning to the average motion direction. In keeping with the behavioral biases, the neural tuning profiles also exhibited reliability biases. A control experiment revealed that observers were able to reproduce motion directions of low and high reliability with equal precision, suggesting that unbiased integration in this task was not only theoretically optimal but demonstrably possible. Our findings reveal that temporal integration of discrete sensory events in the brain is automatically and suboptimally weighted according to stimulus reliability.
SIGNIFICANCE STATEMENT Many everyday decisions require integration of several sources of information. To safely cross a busy road, for example, one must consider the movement of vehicles with different speeds and trajectories. Previous research has shown that individual stimuli are weighted according to their reliability. Whereas reliability biases typically yield performance that closely mimics statistically optimal integration, it remains unknown whether such biases arise even when they lead to suboptimal performance. Here we combined a novel integrative decision-making task with concurrent brain recording and modeling to address this question. While unbiased decisions were optimal in the task, observers nevertheless exhibited robust reliability biases in both behavior and brain activity, suggesting that reliability-weighted integration is automatic and dissociable from statistically optimal integration.
- computational modeling
- decision making
- electroencephalography
- forward encoding analyses
- signal integration
Introduction
Decision-making is a ubiquitous cognitive process involved in all forms of choice behavior. While decision-making in relation to single stimuli—for example, monitoring traffic movement on just one side of the road—has been well characterized at the computational (Smith and Ratcliff, 2009; Forstmann et al., 2016; Ratcliff et al., 2016; Summerfield and Blangero, 2017; O'Connell et al., 2018) and neural levels (Shadlen and Newsome, 1996, 2001; Philiastides and Sajda, 2006; Churchland et al., 2008; O'Connell et al., 2012; Kelly and O'Connell, 2013), much less is known about how observers integrate two (or more) distinct sources of evidence into one decision. To safely cross a busy street, for example, one should monitor traffic movement on both sides of the road when deciding whether it is safe to cross.
In recent years, there has been increased interest in the cognitive and neural mechanisms underpinning such “integrative” decision-making (Churchland et al., 2008; Leite and Ratcliff, 2010; Raposo et al., 2012; Kanitscheider et al., 2015; Spitzer et al., 2017). Integrative decision-making involves combining several uncorrelated sensory inputs from either a single stimulus attribute (e.g., motion) or several stimulus attributes (e.g., motion and color), separated in time or space, in support of a single choice. Studies have shown that, when presented with two stimuli, one of higher reliability than the other, the choice behavior of humans and rodents is biased in favor of the more reliable stimulus (Raposo et al., 2012, 2014; Brunton et al., 2013; Kanitscheider et al., 2015; Boyle et al., 2017). In fact, biases in integrative decision-making closely resemble statistically optimal signal integration (Ernst and Banks, 2002; Seilheimer et al., 2014), in which the contributions of individual signals are weighted by their reliability so as to yield statistically optimal decisions. It remains unclear, however, whether reliability biases in integrative decision-making operate only when they improve performance (i.e., are strategic for the task at hand) or even when they lead to suboptimal behavior (i.e., are automatic, and thus cannot be suppressed).
One ubiquitous finding in the literature is that some observers do not integrate discrete sensory events at all, but rather rely exclusively on signals of higher reliability (Raposo et al., 2012, 2014; Kanitscheider et al., 2015; Boyle et al., 2017). This finding suggests that integrated decisions may be subject to higher-order, strategic influences. Of particular note, studies on integrative decision-making have typically used tasks in which signal integration was helpful, but not necessary for accurate responses. It is unknown whether integrated decision-making automatically favors sources of higher reliability in tasks where the integration is essential, rather than opportunistic. To address this issue, we developed a task in which the integration of two simple visual decisions was essential. On every trial, participants saw two brief periods of coherent motion (containing signals in two different directions) and had to report the average motion direction of the two epochs. To illustrate, if a trial contained successive motion directions toward 10 o'clock and 2 o'clock, participants should indicate an average motion direction of 12 o'clock. Throughout the task, we recorded neural activity using electroencephalography (EEG) to characterize motion direction tuning to the physically presented motion signals, as well as to observers' internally generated integrated decision about the average motion direction in the trial.
To manipulate stimulus reliability, motion coherence in the first and second epoch could either be low (40% of coherently moving dots) or high (80%), both of which were well above motion discrimination thresholds (∼6–7%; Scase et al., 1996). While not very likely, the low-reliability stimuli might yield noisier sensory representations. Consequently, unbiased averaging of these more and less noisy representations could have yielded reliability biases. In a separate experiment, we verified that the motion signals of high and low reliability could be reproduced with similar precision. It follows, therefore, that optimal (i.e., unbiased) performance was possible in our task.
Materials and Methods
Participants.
Twenty-six neurotypical adult humans (mean age, 21 years; 15 females) took part in the main experiment, and another group of 24 neurotypical adults (mean age, 22 years; 14 females) participated in the control experiment. All had normal or corrected-to-normal visual acuity and normal color vision confirmed by Ishihara color plates. The sample size was selected to achieve high power (β = 0.9 at α = 0.05) to detect a medium to large effect size (Cohen's dz = 0.65) for a one-tailed, one-sample t test between response error magnitude for low- and high-motion coherence. Based on behavioral performance and the EEG signals, four participants were identified as outliers in the main experiment (see below). Based on behavioral performance, one was identified as an outlier in the control experiment. The final sample comprised 22 and 23 participants in the main and control experiments, respectively. The study was approved by the Human Research Ethics Committee of The University of Queensland (approval #2016001247) and was conducted in accordance with the Human Subjects Guidelines of the Declaration of Helsinki. All participants provided written informed consent before experimental testing.
Stimuli, task, and procedure.
Every trial started with a colored cue, indicating the target color (Fig. 1). Two of three easily discernible colors [HSL (hue, saturation, and lightness) values: pink, 0, 75, 50; yellow, 90, 75, 50; cyan, 270, 75, 50] served as target and distractor colors. The target–distractor color pairs (e.g., pink target, and blue distractor) were fixed per participant and counterbalanced between participants. After the cue, a circular patch [diameter, 15.6° of visual angle (dva)] of 160 gray, randomly moving dots (diameter, 0.6 dva; speed, 2.5 dva/s; infinite dot life) appeared. To prevent a stimulus onset-evoked response from influencing electrophysiological measures of decision-making, the gray patch remained on screen for 1 s. Thereafter, the color saturation of the dots increased gradually (>250 ms), revealing two overlapping patches of colored dots (80 dots/patch) in the target and distractor colors. Presenting two overlaid patches—a target and a distractor—in different colors prevented participants from simply accumulating all motion signals throughout a trial. Rather, they had to selectively process two discrete targets while ignoring concurrently presented distractor signals.
The color saturation remained at maximum for 1 s and then gradually returned to gray. During maximum saturation, coherent motion signals were presented briefly (500 ms) in both patches. The onset of coherent motion was jittered (250–500 ms) relative to the maximum color saturation, so that neural signals relating to motion and color information could be effectively separated. The motion coherence was pseudorandomly selected for every epoch of colored dots, with low (40%) and high (80%) coherences presented equally often. Participants had to monitor for target motion signals and ignore distractors. A feedback stimulus was presented after every trial indicating response accuracy in that trial. Response accuracy rather than speed was emphasized, and participants were given ample time to respond (maximum, 6 s).
In the main experiment, two epochs of colored dots were presented in every trial and participants had to reproduce the average motion direction of the two target signals while ignoring distractor motion events. The target motion in the first epoch was selected randomly from a range of 0–360° in 1° steps. The target motion in the second epoch was selected from a range of ±30–150° (in 1° steps) relative to the first target. The distractor motion was within a range of ±30–150° (in 1° steps) relative to the presented target motion. The dot coherence within each epoch was always the same for target and distractor motions. The dot coherence across the two epochs was selected pseudorandomly so that all four combinations (low/high in the first epoch × low/high in the second) were presented equally often. Participants completed 384 trials, with 96 trials per combination of coherences. Both behavioral and EEG data were recorded.
In the control experiment, only one epoch of colored dots was presented per trial, and participants had to reproduce the target motion direction by adjusting the orientation of a response dial. At the beginning of every trial, a colored circle (the cue) indicated the task-relevant color. Then, a field of randomly moving gray dots was shown for 1 s, followed by a 250 ms period in which the color saturation of the dots gradually increased. Over this period, two differently colored patches of dots became discernible, one in the target color and the other in a distractor color. After a brief interval (250–750 ms, randomly jittered between trials), the dots in the target and distractor patches moved in a specific direction, and at the same level of coherence, for 500 ms. The coherence levels varied pseudorandomly across trials. Participants completed 256 trials, with 128 trials per coherence level. Only behavioral data were recorded.
The control experiment was part of a different study, which varied experimental conditions not reported here. A notable difference between the main experiment and the control was in the number of possible target motion directions. Per participant, two pairs of motion directions (i.e., four in total) were selected as possible target motions from a range of directions (0–360° in 15° steps). Within a pair, the directions were 30° apart from each other, and the two pairs were 180° apart (e.g., 15° and 45°, 195° and 225°). Different combinations of the four motion directions were counterbalanced across participants. On every trial, the target motion direction was pseudorandomly sampled from the set of four possible target motions. The distractor motion was within a range of ±30–150° (in 1° steps) relative to the presented target motion. All other details were as in the main experiment.
Apparatus.
The experiments were conducted in a dark, acoustically and electromagnetically shielded room. The stimuli were presented on a 24 inch monitor with 1920 × 1080 resolution and a refresh rate of 144 Hz. The experimental software was custom coded in Python using the PsychoPy toolbox (Peirce, 2007, 2009). EEG signals were recorded using 64 Ag-AgCl electrodes (ActiveTwo, BioSemi) arranged in the 10–20 layout and sampled at 1024 Hz.
Behavioral analyses.
To identify outlier participants, the distributions of error magnitudes (i.e., the angular difference between the response and the correct answer) were compared with a uniform distribution (i.e., pure guessing) using the Kolmogorov–Smirnov test. Participants for whom the probability of the null hypothesis (i.e., a uniform distribution of error magnitudes) exceeded 0.001 were removed from further analyses. The remaining distributions per experimental condition and per participant were fitted to a theoretical model (Schneegans and Bays, 2016), and responses were separated into noisy target responses and random guesses. To quantify decision weights, a multiple-regression [ordinary least squares (OLS)] model with a term for each of the presented motion directions, expressed as complex numbers, was fitted to the responses, separately per participant and experimental condition. The absolute value of the resulting regression coefficients reflects the influence of each of the presented coherent motion signals on the response (i.e., its decision weight; Rangelov and Mattingley, 2020).
To compare the results of the main experiment with those of the control experiment, the observed circular variance of error magnitudes per condition and participant in the averaging task (the main experiment) was compared with the expected variance on the basis of a simple summation model estimated using the circular variance (Σ) observed in the single-target motion reproduction task (the control experiment). First, the expected variance was estimated for each participant in the control experiment by summing the circular variance for the four different combinations of motion coherence (Σlow + Σlow, Σlow + Σhigh, Σhigh + Σlow, Σhigh + Σhigh). Next, the distribution of the sums across participants was characterized by computing the mean (MΣ+Σ) and the SD (σΣ+Σ) of the sums for each of the four conditions. Finally, the standardized deviations (z) between the observed circular variance in the main experiment per participant (i) and condition (X) and the expected variance were computed using the following equation:
EEG analyses.
EEG signals were analyzed using the MNE-Python toolbox (Gramfort et al., 2013). The data were rereferenced offline to the average electrode, low-pass filtered at 99 Hz and notch filtered at 50 Hz to eliminate line noise. The recorded signal was preprocessed using the FASTER algorithm for automated artifact rejection (Nolan et al., 2010). The preprocessed signal was downsampled to 256 Hz, segmented into 4 s periods between the onset of the first epoch and the response-display onset, baseline-corrected relative to −100 to 0 ms pretrial and linearly detrended. Outlier trials and participants were identified using the FASTER algorithm and removed from further analyses. Between 9 and 49 trials per participant (median, 21, or ∼5%) were identified as outliers and removed from further analyses.
Previous research has shown that the time course of the central-parietal positivity (CPP) closely resembles the time course of evidence accumulation: its amplitude builds up gradually, the buildup slope is proportional to stimulus quality, and it is observed even in the absence of overt responses (O'Connell et al., 2012; Kelly and O'Connell, 2013; Twomey et al., 2016). In the literature, the CPP slope and peak amplitude have been interpreted as neural correlates of the rate of evidence accumulation and decision threshold, respectively (Rungratsameetaweemana et al., 2018). To characterize the temporal dynamics of evidence accumulation, we quantified the time course of the CPP. Visual inspection of the ERP topographies time locked to the beginning of the trial revealed a positive deflection in a cluster of central-medial electrodes (FCz, C1, Cz, C2, and CPz) consistent with the CPP ERP; to improve signal-to-noise-ratio, the average of these electrodes was used in further analyses. Whole-trial time traces were further segmented into shorter (500 ms) periods time locked to the onset of color saturation increase and the onset of coherent motion in the first and second epoch, and baseline corrected relative to −100 to 0 ms pre-onset interval. Using a jackknife-based scoring method (Ulrich and Miller, 2001), we estimated the CPP onset latency, the slope, and the peak amplitude and latency. First, the peak CPP response was identified as the maximum positive deflection in the 0–500 ms time window. Next, the CPP onset latency was identified as the time at which 50% of the peak amplitude was reached (Luck, 2012). Finally, we estimated the slope as the amplitude gradient within ±25 ms period around the CPP onset (Rangelov and Mattingley, 2020).
To recover feature-specific information about motion direction from the EEG signals, we used two different analytical approaches. First, we characterized the sensitivity of individual electrodes to different motion directions. In Equation 2, we modeled the voltage V from electrode i and time sample t as a linear combination of the sine and cosine transform of the analyzed motion direction θ. Using OLS, we estimated the weights β associated with the sine and cosine terms. As an aggregate measure of electrode sensitivity S (Eq. 3), we computed the length of a vector defined by the estimated weights of the sine and cosine terms, as follows:
As the sensitivity estimate (Eq. 3) was always positive, we bootstrapped its empirical null distribution by estimating sensitivity using randomly shuffled vectors of the motion directions (1000 times). We estimated the percentile of the empirical sensitivity (Sit) against the empirical null distribution and converted the percentile to a z-score using the inverse of a normal distribution. These analyses were conducted separately per participant and experimental condition.
In the second analysis (Fig. 2), we used a population tuning curve model (Brouwer and Heeger, 2009; Myers et al., 2015; Kok et al., 2017) to characterize feature-specific patterns of responses across the whole electrode array. To that end, the first and the second epochs (1 s segments time locked to the onset of coherent motion) from all trials were concatenated (∼700 segments per participant), temporally smoothed by convolving the time series with a Gaussian window (SD, 16 ms), shuffled, and split into 10 testing sets (
The profile of channel responses would reflect motion tuning: a uniform (i.e., flat) response profile would correspond to no tuning, whereas a prominent peak at channels close to the presented motion would reflect strong tuning.
The motion tuning analyses were conducted per time sample per participant. The tuning to target and distractor motion signals were analyzed separately. We also estimated tuning to the average motion direction (i.e., the expected, average response) using the first and the second epoch data in two separate analyses. To characterize the overall tuning strength across different conditions, the vectors of channel responses were centered on the actual presented motion direction, temporally smoothed using a Gaussian window (SD, 16 ms) and averaged across trials. As an aggregate index of the channel response profile, we computed the dot product (θ) between the channel responses and the complex-valued preferred motion directions for respective channels. To quantify the tuning strength, we used the following equation:
Equation 7 yields a good, nonparametric descriptor of the overall tuning shape: (1) the
The neural error magnitude was computed using channel response profiles for the average motion direction. These analyses were undertaken using decoding results from the second epoch only. To increase signal-to-noise-ratio, we first averaged channel responses in each trial over the 500–1000 ms interval during which there was robust tuning to the average motion direction. The averaging yielded a vector of channel responses, one for each of the 16 hypothetical motion channels. Then, we recoded the preferred motion direction of the 16 channels so that negative directions were closer to the target motion presented in the first epoch and positive values were closer to the motion in the second epoch. Finally, we quantified the neural error magnitude for each trial (1) as the location of the recoded tuning profile using the following equation:
To characterize a neural order bias independently of a potential reliability bias, we estimated
Experimental design and statistical analyses.
The main independent variables were epoch (first vs second), signal reliability (low vs high), and, where appropriate, stimulus type (target vs distractor). The data were analyzed using parametric and nonparametric inferential statistics (repeated-measures ANOVA, one-sample t test, mixed-effects linear models, and bootstrapping).
Results
In the main experiment, we presented two successive epochs of coherent motion on every trial (Fig. 1). Each epoch contained both target and distractor signals. Participants (N = 22) were asked to reproduce the average motion direction of two target signals, while ignoring two concurrently presented distractor signals. Inspection of distributions of the observed error magnitudes (correct response – actual response) revealed a continuous, relatively narrow, unimodal distribution in all experimental conditions (Fig. 3a dark bars). We used mixture-distribution modeling (Bays and Husain, 2008; Zhang and Luck, 2008) of error magnitude to independently quantify the proportion of trials on which participants guessed randomly (Pg) and the response precision (K) on the remaining, target response trials. Model fitting separately per participant and experimental condition yielded close fits to the observed error magnitudes (Fig. 3a, pink lines). Participants guessed on ∼10% of trials (guess rate PgM/SEM = 0.09/0.01) and the response precision of target responses was relatively high (K = 8.16/0.34, FWHM = 48°), indicating that participants were able to perform the task well.
Overview of the experimental paradigm. A typical display sequence per trial (right) together with the time course of different events and possible colors (left). A colored cue indicating the task-relevant color (fixed per participant) was followed by a patch of gray dots moving randomly. After 1 s buffer periods, the color saturation increased gradually to reveal two intermingled fields of distinctly colored target and distractor dots (here shown in blue and brown for target and distractor patches, respectively). Coherent motion signals were presented for 500 ms in both fields, jittered relative to the maximum saturation onset (250–500 ms). After the second of the two epochs in a trial, participants reproduced the average motion direction of the two target motion stimuli by adjusting the orientation of the response dial (black) without speed constraints (maximum, 6 s). Response feedback was shown for 500 ms following the response (white dial contour). In the control experiment, only one epoch was shown, and participants had to reproduce the target motion direction.
Illustration of the population-tuning analysis pipeline. a, A set of hypothetical motion-selective channels is created, each of which is preferentially tuned to 1 of 16 equidistantly spaced motion directions spanning the full circle (–π to π). b, The array of presented motion directions per trial in the training set of trials is convolved with the response profiles of the motion channels to yield trial-specific responses (left). The trial-specific responses are used as predictors to model the EEG time traces from the respective trials (right). The forward encoding yields a weight matrix (ωC×S) relating channel responses to the EEG. c, The inverted weight matrix (ωS×C) is applied to the EEG time traces from the testing set of trials to retrieve the profiles of the motion channel responses.
Overview of the behavioral results in the main experiment. a, Distributions of the observed error magnitudes (correct minus actual response, bars) together with the predicted distributions (pink lines) on the basis of mixture distribution modeling (Bays and Husain, 2008; Zhang and Luck, 2008). Separate models were fitted across different combinations of motion coherence (low/high) in the first and second epoch (first epoch coherence → second). b, Left, Decision weights across different experimental conditions showing the degree to which coherent motion signals presented during a trial influenced the response. Right, Difference in respective decision weights for the first and second epochs separately per condition and stimulus type. Dots represent individual participants. Error bars denote within-participant SEM (Morey, 2008).
We then analyzed the effect of target–dot coherence in the different epochs on response precision and guessing rates. While coherence in the first epoch did not influence response precision (K = 8.12 and 8.19 for low and high coherence, respectively, F < 1), low coherence in the second epoch yielded significantly worse precision than high coherence (7.61 and 8.70, respectively; F(1,21) = 14.27; p = 0.001;
To quantify the extent to which the motion signals presented in the two epochs influenced behavioral responses, we conducted linear regression analyses separately per participant and experimental condition using complex-valued response angles as a dependent variable and the presented motion directions as independent variables (target/distractor signals in the first epoch/second epoch; see Materials and Methods). The absolute values of the regression weights (i.e., the decision weights) reflect how much the respective motion signals influenced the response. Weights for targets were much larger than weights for distractors (0.56 and 0.09, respectively; F(1,21) = 2028; p < 0.001;
Most importantly, the order bias was further qualified by two-way interactions with motion coherence in both the first and second epochs (Fig. 3b, right). Specifically, whereas increasing the coherence in the first epoch increased the primacy bias (F(1,21) = 16.30, p < 0.001,
To characterize the neural correlates of integrative decision-making, we recorded brain activity using EEG. We quantified a well documented neural correlate of simple perceptual decision-making (O'Connell et al., 2012; Kelly and O'Connell, 2013; Twomey et al., 2016), the CPP (see Materials and Methods; Fig. 4). Visual inspection of the ERP topographies (Fig. 4a) revealed a positive deflection over centroparietal electrodes, consistent with the typical CPP topography (Loughnane et al., 2016). Visual inspection of the CPP time course over the trial (Fig. 4b) revealed a phasic modulation that closely followed the trial sequence. During periods when only gray, randomly moving dots were shown, the CPP amplitude was at baseline. Two sharp deflections closely followed the onset of color modulation in the first and the second epochs, and two broader deflections coincided with periods of coherent motion. (Recall that the motion signals were delivered with jittered timing only after the dot patches had reached full saturation.)
ERP results from the main experiment. a, ERP topography for color-locked responses, showing the average of time samples at which the color saturation reached a maximum (750 and 2750 ms for the first and the second epoch, respectively); and for motion-locked responses, showing the average of time samples at which the motion signal was most likely to be presented (at 1250 and 3250 ms). Marked electrodes (white circles) were used for computing the CPP time course. b, The CPP time course (top) together with the time course of different events within a trial (bottom). c, Color-locked CPP during the first 500 ms following the onset of color modulation across different experimental conditions. Only random motion directions were presented during this period. d, Motion-locked CPP. The colors were at maximum saturation during the analyzed 500 ms. Conventions as in c. For the purpose of presentation, all time traces were low-pass filtered at 10 Hz using a Butterworth fourth-order infinite impulse-response filter. The analyses were performed on unfiltered data. Shaded areas denote ±1 within-participants SEM.
It might seem surprising that we observed a CPP-like response to the modulation of color saturation, given that only random motion was presented during this time within the trial. As participants could not have accumulated any information related to the direction of coherent motion over this period, it is natural to ask what information—if any—was accumulated? The answer is that participants most likely accumulated evidence in support of a choice as to which set of concurrently presented colored dots was task relevant on that trial, based on the color cue shown at the start of the trial (Fig. 1). Unlike in the present study where color and motion onsets were separated in time, in a previous study (Rangelov and Mattingley, 2020), the color and motion onsets coincided within the trial, and there we observed a single (“unimodal”) CPP. Across the two studies, the unimodal and bimodal CPPs aligned well with behavioral decision weights for distractors. Specifically, the distractor weights were high for concurrent onsets of color and motion signals, and low for sequential onsets. When color and motion onsets are separated in time, observers can accumulate evidence for task-relevant color and motion independently. They are, therefore, better able to suppress the influence of distractor motion, as evidenced in the low behavioral decision weights. Overall, the present findings are consistent with the notion that a single task may involve several decisions, such as deciding what is the task-relevant color and what is the task-relevant motion direction. Each decision triggers a separate evidence accumulation process, and the relative timing of these decisions will determine whether a unimodal or bimodal CPP is found.
To characterize the responses of the brain to the color saturation modulation and the coherent motion onsets, 500 ms segments of the EEG data were extracted, time locked to either the color or the motion onsets (Fig. 4c,d, respectively). For both color- and motion-locked epochs, the estimated onset latencies were ∼250 ms (249 ms for color locked CPP, and 233 ms for motion-locked CPP), consistent with the notion that the CPP is not merely an early, sensory-evoked response, but rather reflects higher-level processes following sensory encoding (Loughnane et al., 2016; Twomey et al., 2016). The color-locked CPP slope was 24.7 µV/s on average, and it did not vary substantially across experimental conditions (all F values, <1.17, all p values, >0.29). Its peak latency was longer in the first epoch than in the second (319 and 300 ms, respectively; F(1,21) = 18.48, p < 0.001). The peak amplitude, by contrast, was almost twice as high in the first epoch as in the second (2.44 and 1.51 µV, respectively; F(1,21) = 48.32, p < 0.001). In the absence of slope differences, the peak amplitude effects can be accounted for by the peak latency differences: the CPP in the first epoch relative to the second took longer to reach the peak, resulting in a robust amplitude effect. No other main effects or interactions reached significance (all F values, <3.01, all p values, >0.095). Together, the color-locked CPP was primarily sensitive to the epoch order (first vs second) within the trial, exhibiting a robust primacy effect.
The motion-locked CPP, by contrast, was not sensitive to epoch order for any of the analyzed parameters (all F values, <2.10; all p values, >0.16). There was, however, a robust effect of motion coherence. Specifically, there was a steeper rise, a shorter peak latency, and a higher peak amplitude for the high-coherence signal relative to the low-coherence signal (20.96 µV/s, 307 ms, and 2.13 µV vs 10.22 µV/s, 324 ms, and 1.23 µV, respectively, Fig. 4d). Importantly, the CPP peak amplitude in each epoch was affected only by the coherence of the motion stimulus in that epoch, as evidenced by a significant interaction between epoch (first/second) and motion coherence (high/low) in both the first epoch (F(1,21) = 38.23, p < 0.001) and the second (F(1,21) = 25.83, p < 0.001). Despite the observed numerical trends for the CPP slope and peak latency, however, these effects were only marginally significant (all F values, <3.69; all values, >0.06). Overall, analyses of the motion-locked CPP parameters suggest that the motion-locked CPP reflects evidence accumulation in support of discriminating the currently presented motion target.
Whereas analyses of behavioral decision weights revealed robust interactions between the primacy bias and motion coherence across the two epochs, the ERP analyses suggest that the order effect and the coherence effect might have separable neural correlates. The primacy bias was evident only in the color-locked CPP, suggesting that processes related to discerning target and distractor patches give rise to the order effect. The reliability bias, by contrast, was evident only in the motion-locked CPP, and this component reflected the strength of the presented motion signals.
In the following set of analyses, we attempted to recover motion-specific information from the whole-scalp EEG signals time locked to the onset of coherent motion. In the first analysis (Fig. 5), we investigated whether different motion directions correlated with a focal activity at single scalp electrodes. To that purpose, we regressed the EEG voltage on the first target motion, the second target motion, and the average motion (see Materials and Methods). As Figure 5 shows, there was no evidence that single electrodes were statistically significantly sensitive to any of the analyzed motion directions implying that different motion directions were not represented as an isolated focal activity.
Electrode sensitivity to different motion directions time-locked to the onset of coherent motion signals. a, Electrode sensitivity estimated using the motion-locked EEG time traces during the first epoch. Top, Sensitivity to the first target motion. Bottom, Sensitivity to the average motion. Different lines denote different electrodes. The y-axis shows log-transformed p values from a one-sample t test against 0, and the horizontal dotted line denotes the critical p level. b, Electrode sensitivity estimated using motion-locked time traces during the second epoch. Conventions are as in a. Inset, Locations of different electrodes.
In the second analysis, we used population-tuning modeling of the motion-locked EEG signals (Brouwer and Heeger, 2009; Myers et al., 2015; Kok et al., 2017) to characterize patterns of brain activity across the scalp that were specific for different motion directions. We characterized motion tuning to both target and distractor signals in both the first and second epochs (Fig. 6). Inspection of tuning to target signals revealed a robust and sustained motion-specific response. This result stands in sharp contrast to the analyses of individual electrode sensitivity (Fig. 5) and suggests that task-relevant sensory representations are supported by a broad network of different neuronal populations. Further, comparisons between the two analyses suggest that the population-tuning results cannot be accounted for by EEG artifacts induced by eye movements in response to motion signals. If that were the case, then individual electrodes in the frontal scalp regions, which are most sensitive to eye movement artifacts, should have exhibited a similar sensitivity to target motion, like we found using the population-tuning approach. Finally, we found no significant motion tuning to distractor dot patches, although these signals were physically identical to the targets in terms of their brightness and coherence. This finding supports the notion that participants engaged in integrative decision-making, rather than passive accumulation of all presented motion signals.
Population-tuning modeling of neural activity in response to target and distractor signals. a, Time-resolved responses to target and distractor signals (top and bottom panels, respectively) of 16 hypothetical motion-specific channels spanning the full circle (from –π to +π) and centered on the actual presented motion signal. The trial-averaged profile of channel responses per participant and time sample was z-scored across channels. White contours denote areas that were significantly different from 0 (FDR corrected across all time samples and channels). Significant modulation of channel-response profiles with yellow areas ∼0 and blue areas around –π and +π indicate robust motion tuning to the presented signal. b, Time-resolved tuning strength to target signals across first and second epochs (top and bottom panels, respectively). The tuning strength is an aggregate index of the channel-response profile (for details, see Materials and Methods), with 0 indicating no tuning. The bottom panel shows -log10(pFDR-corrected) for different terms in the stepwise linear mixed-effects (LME) model of motion tuning. Values above the LME α are significant at p < 0.05 (see Materials and Methods).
Tuning to the target motion direction was sustained well after motion offset (Fig. 6, dotted vertical lines), suggesting that representations of individual signals were not merely sensory-evoked responses. Inspection of the time-resolved tuning strength for target signals (Fig. 6b) across different epochs revealed comparable tuning to high- and low-coherence signals in the first epoch. This is consistent with the observation that dot coherence in the first epoch did not have an effect on behavioral response precision (K parameter) and supports the notion that the two coherence levels afforded comparable precision (since both were well above threshold). The tuning strength in the second epoch was larger overall for high-coherence targets than for low-coherence targets; this dot coherence effect was statistically significant at ∼450 and 700 ms (Fig. 6b, bottom). This is consistent with the significant effect of dot coherence in the second epoch on behavioral response precision.
In a final analysis, we characterized the temporal dynamics of integrative decision-making by estimating neural tuning to the average motion direction (Fig. 7a). Note that this quantity was internally computed by participants, whose task was to integrate the directions of the two target signals. For the first epoch, there was no significant tuning to the average motion direction, as would be expected since average motion direction can only be determined after presentation of the second motion target within a trial. By contrast, there was robust and sustained tuning to the average motion direction in the second epoch, starting from the offset of the second coherent-motion signal. Note that during this period only random motion was present on the screen, so tuning to the average motion direction could not have been stimulus driven.
Population-tuning modeling of neural activity associated with the average motion. a, Time-resolved average-motion tuning in the first and the second epoch (top and bottom, respectively) using the forward-encoding model trained on the true average motion. Conventions as in Figure 6. b, Average-motion tuning in the first and the second epoch decoded on the basis of forward-encoding model trained on the first and the second target signals (top and bottom, respectively). c, Expected (theoretical) distributions of the neural error magnitudes on the basis of the behavioral results. The vertical dotted line shows the true average motion. The x-axis denotes recoded motion directions such that the directions on the left are closer either to the first target motion (red line) or to the low coherence motion (purple). The primacy bias predicts a leftward shift toward the first target motion, whereas the reliability bias predicts a rightward shift toward the motion of higher coherence. d, Bootstrapped distributions of the actual observed neural error magnitudes averaged across trials and participants (see Materials and Methods). Same-coherence trials (High → High and Low → Low trials) were used to estimate the neural primacy bias (red line). Different-coherence trials (High → Low and Low → High) were used to estimate the neural reliability bias (purple line). For this analysis, the tuning profiles in the High → Low trials were flipped so that the motion directions for the high coherence signals were to the right of the true average motion in both High → Low and Low → High trials. Small inset, The neural primacy bias and reliability bias averaged across participants. The error bars denote ±1 SEM.
Together, we found a robust and statistically significant motion tuning to the two target motion directions presented in each trial, as well as to their average. An important outstanding question is whether the neural representations of these three signals are supported by the same underlying processes. To address this question, we trained two forward-encoding models using either the first or the second target motion, and then used these models to estimate tuning to the average motion direction. If the neural mechanisms are shared for all three signals (first, second, average), then forward-encoding models trained on either of the component motion signals should generalize to the average motion direction. In fact, we found only weak tuning to the average motion direction for the forward model trained on motion signals in the second epoch of the trial, and no tuning for the model trained on signals from the first epoch (Fig. 7b). These findings suggest that the neural mechanisms responsible for integrating the two visual motion signals, as required in our task, are distinct from those involved in regulating the sensory encoding of individual component motion signals.
To investigate how the brain integrates two discrete decisions, we used tuning profiles to the average motion direction per trial to quantify the neural error magnitude in that trial (Fig. 7c; see Materials and Methods). This quantity represents the difference between the neural estimate of the average motion direction and the true average of the two target motions; as such, it is a neural analog of the behavioral error magnitude. Just as for behavioral error magnitudes, a distribution of neural error magnitudes across trials can be characterized by its location and width. Any shift in the location of the distribution would indicate a neural bias in integrative decision-making, and so we quantified the location of the distribution of neural error magnitudes. Motivated by the observed primacy and reliability biases in behavior, we expected to observe a shift toward the motion direction of the first target, as well as a shift toward the motion target with higher reliability, respectively (Fig. 7c, left).
To test these predictions, we first estimated the order bias using the tuning profiles from the same-coherence trials (High → High and Low → Low trials; Fig. 7c, right, red line). The profiles for every trial were recoded such that the negative angles were closer to the first presented motion in that trial, and the neural error magnitude was then estimated (see Materials and Methods). The average neural error magnitude across all trials and participants was −28° (bootstrapped pshift ≥ 0 = 0.023; see Materials and Methods), indicating a strong, statistically significant primacy bias. Next, we estimated the reliability bias using the tuning profiles from the different-coherence trials (High → Low and Low → High trials; Fig. 7c, right, purple line). The profiles were recoded such that the negative angles were closer to the low-coherence motion, which required flipping the tuning profiles in the High → Low trials. The average neural error magnitude was 8°, indicating a numerical bias in the expected direction. This bias, however, did not prove statistically significant (bootstrapped pshift ≤ 0 = 0.717).
Overall, our behavioral results have shown that averaging two target motions favored more reliable stimuli. It is possible, albeit not very likely, that stimuli of lower reliability resulted in noisier sensory representations. Consequently, unbiased averaging of these more and less noisy representations could have yielded the observed reliability biases. To empirically confirm that the reliability levels we used in our task afforded responses of comparable precision, we conducted a control experiment in which a single epoch of coherent motion was presented and participants had to reproduce the target motion direction while ignoring a concurrently presented distractor (see Materials and Methods). Unlike in the main experiment, here participants (N = 23) were shown only a single target/distractor epoch, and were thus not required to perform any motion averaging. The dot coherence varied randomly across trials between low (40%) and high (80%), as in the main experiment. We recorded only behavioral responses.
The estimated response precision (Fig. 8a, top) was high and statistically indistinguishable between low and high coherence (KM/SEM = 14.86/0.71 and 13.73/0.71, respectively; paired-sample t(22) = 1.12; pKlow < Khigh = 0.863). Similarly, the estimated guessing rates were very low and comparable between the two coherence levels (both Pg = 0.03/0.01, t(22) < 1). Analyses of target and distractor decision weights (Fig. 8a, bottom) revealed that participants successfully focused on target signals and ignored concurrently presented distractors, as indicated by target weights which were around nine times larger than those of concurrently presented distractors (0.92 vs 0.10, F(1,22) = 1693, p < 0.001,
Behavioral results of the control experiment and comparison between experiments. a, Distributions of error magnitudes (bars) together with the predictions of the mixture distribution model (pink lines) for low- and high-coherence signals (top) and decision weights for target and distractor signals (bottom). Conventions as in Figure 3. b, Distributions of deviations between the observed circular variance of error magnitudes in the main experiment and the expected circular variance on the basis of the simple summation model estimated from the control experiment (see Materials and Methods). The horizontal white lines denote the median deviation across participants per experimental condition (e.g., Low → Low).
In the main experiment, participants reproduced the average of two sequentially presented target motions, whereas in the control experiment, participants reproduced a single presented target motion. The task differences between the two experiments pose a risk that the results from one task (e.g., reproduction) may not generalize to the other task (e.g., averaging). To test whether the performance in the two tasks is directly comparable, we used the variance of error magnitudes for single-target reproduction in the control experiment to predict the variance of error magnitudes in the averaging task of the main experiment. On the simple summation model, the variance in the averaging task should be a sum of variances from the single-target task. For every participant in the main experiment, we computed a standardized difference between the observed and the expected variance (Fig. 8b, deviations from the simple summation model; see Materials and Methods) across experimental conditions. The median deviations ranged between −0.23 and 0.10 across conditions (Fig. 8b, white lines) with no statistically significant shifts from zero in any of the conditions (all Wilcoxon signed-rank tests, pFDR-corrected ≥ 0.213; false discovery rate (FDR)]. These findings demonstrate that a simple summation model can account well for performance in the averaging task and suggest that the two tasks are comparable.
Discussion
Here we demonstrated that integration of two discrete decisions is biased in favor of a more reliable stimulus even when it is detrimental for performance. By virtue of the task requirements, and our use of suprathreshold stimuli, unbiased responses were both optimal and possible. The control experiment provided independent verification that the motion coherence levels used in the main experiment afforded comparable accuracy. These results suggest that the brain encodes the reliability of sensory inputs, and that the encoded reliability is automatically used to weight respective inputs during integrative decision-making.
The relatively narrow distributions of error magnitudes, and a large difference between decision weights for targets and distractors suggest that participants were able to successfully select target signals and ignore concurrently presented distractors. Perhaps most interestingly, the population-tuning modeling of distractor motion signals revealed no distractor-specific neural activity. By contrast, motion decoding for target signals was robust and sustained well after signal offset, suggesting that the population-tuning model primarily captured decision-making processes as opposed to purely sensory-evoked activity patterns.
During the last half of the second epoch, after both targets had been shown, we observed robust tuning to the average motion direction despite the fact that this signal was never physically presented in the display. Given that the average motion direction was a function of the first and the second target motion, it is conceivable that tuning to the average motion direction simply reflects tuning to the second target. This seems unlikely, however, because comparable tuning to the average motion should also have been observed in the first epoch, but this was clearly not the case. Moreover, if tuning to the average motion direction was driven solely by the second target, then the time course of average motion tuning should have been similar to that of the second target, which, again, clearly was not the case. We therefore conclude that the robust tuning to the average target motion reflects the temporal dynamics of integrative decision-making.
When we trained the forward-encoding models on the motion directions of the two component-motion signals within each trial, there was a marked decrease in the strength of tuning to the average motion direction. This finding suggests that the neural mechanisms responsible for perceptual decisions and the mechanisms underpinning integrative decision-making are distinct. It seems reasonable to expect that perceptual decisions would primarily engage sensory brain regions dedicated to representing the current input. In the case of visual motion signals, these regions would most likely be located in the human equivalent of area MT/V5 (Shipp and Zeki, 1985; Newsome et al., 1989; Malikovic et al., 2007). Posterior parietal brain areas have also been implicated in perceptual decision-making (Shadlen and Newsome, 1996, 2001). At present, it remains unclear which processes and neural substrates might support integrative decisions of the kind required by our novel paradigm, but we can speculate on two possibilities. One is that the same neuronal populations are active in support of both “simple” perceptual decisions and more complex “integrative” decisions, with different decision types evoking different profiles of activity within a common neural network. A second possibility is that entirely different neuronal populations underpin different decision types. Associative brain regions including parietal and prefrontal cortex, which are known to combine sensory inputs with knowledge stored in memory, would be well suited to support integrative decisions. To adjudicate between these alternatives, future studies could use neuroimaging methods with a higher spatial resolution than EEG, such as fMRI, magnetoencephalography, or intracranial EEG.
As in a previous study by our group (Rangelov and Mattingley, 2020), here we observed a reliable primacy bias in behavioral responses. Further, the CPP analyses suggested that the primacy bias had little to do with the real-time processing of the motion signals. Finally, we found robust shifts in neural error magnitudes in favor of the first target–motion direction. A simple memory decay explanation would predict a recency effect (Wyart et al., 2012, 2015) rather than the primacy bias we observed, thus ruling out memory as the likely source of the order effect. Overall, the primacy bias we observed here and in previous work (Rangelov and Mattingley, 2020) adds to recent literature suggesting that the primacy bias originates at postperceptual, decision-related stages (Fritsche et al., 2017).
The reproduction task we devised enabled us to probe the nature of the representations underlying integrated decision-making. In typical decision-making paradigms (O'Connell et al., 2018), the response is a categorical decision, for example, whether motion direction is to the left or to the right. While forced-choice paradigms lend themselves to speeded responding and permit the use of computational modeling to characterize different aspects of decision-making, they do not capture the precision of the sensory and memory representations that underlie evidence accumulation processes. A well known property of the responses of the brain to sensory input is that they are graded (Livingstone and Hubel, 1984, 1988; Hubel and Livingstone, 1985; Kanitscheider et al., 2015), forming a probabilistic stimulus representation in feature space. In the case of motion signals, the large-scale neural representation of a given motion direction should resemble a bell-shaped curve with a peak over the actual direction, which gradually decreases for motion directions further away from the peak. Classical decision-making paradigms would be sensitive to the location of the peak, but would have difficulty characterizing the variability of the probabilistic representation—and that variability seems to play a critical role in integrated decision-making. Using mixture distribution modeling for behavioral measures, and population-tuning modeling for neural measures, we were able to characterize both the peak and the variance of the underlying probabilistic representations.
A key question regarding reliability-weighted integrated decision-making concerns how two discrete representations get combined in support of an integrated decision. Previous research on signal integration for sensory encoding (Ernst and Banks, 2002; Seilheimer et al., 2014) and decision-making (Raposo et al., 2012; Sheppard et al., 2013; Boyle et al., 2017) has suggested that a simple multiplication of two probabilistic representations could drive reliability-weighted integration. Such a multiplication, however, predicts lower variability (Ernst and Banks, 2002; Kanitscheider et al., 2015) of the integrated representation relative to the variability of individual sources. Alternatively, the two discrete representations could be summed, rather than multiplied, predicting higher variability in the integrated representation. Our results show that the simple summation model using single-target reproduction can account well for the averaging task, suggesting that the integration of two decisions in our paradigm relied on summation, rather than multiplication. Moreover, this summation process appears to be biased, with larger weights for high-reliability representations.
There are at least two ways the brain could encode summation weights. On the one hand, the weights could reflect the variability of the probabilistic representation of the target signal. On this account, a simple combination of two probabilistic representations of differing variances would be shifted in favor of the representation of higher reliability. The absence of strong coherence effects on the precision of behavioral responses suggests that representations of the high- and low-coherence signals were comparably reliable, speaking against this account. On the other hand, the weights could be encoded separately from signal representations as a belief about the reliability of the respective representations. In this scenario, although the low- and high-reliability signals afford comparable accuracy, the strong perceptual differences between the two reliability levels would result in different beliefs about signals with different levels of coherence. An important question arising from the beliefs-as-weights account is what neural mechanism might generate different beliefs. In the present study, we found a numerically steeper slope and shorter peak latency, as well as a statistically significantly higher peak amplitude, for motion-locked CPPs in response to high- versus low-reliability signals. These differences suggest that evidence accumulation dynamics are sensitive to the reliability of sensory input. Assuming that the brain monitors the quality of evidence accumulation, it is possible that these estimates of quality lead to different beliefs about the reliability of respective decisions. Some evidence suggests that the brain might indeed monitor the dynamics of evidence accumulation. For example, research has shown that human observers seem to terminate evidence accumulation on the basis of the instantaneous rate of accumulation rather than on the basis of a fixed accumulation threshold: a decision is triggered when the rate falls below a critical level (Nelli et al., 2018). We speculate that, in the present study, observers estimated the quality of evidence accumulation—possibly informed by the processes that generate the motion-locked CPPs—and used that estimate to weight different components of their integrative decisions. Further studies, most likely in combination with hierarchical computational modeling (Mathys et al., 2014; Smith, 2016; Ratcliff, 2018), are necessary to address this issue conclusively.
In summary, we have shown that combining two discrete, temporally separated signals in support of a single, integrated decision is biased in favor of higher-reliability signals. Unlike previous studies in which reliability-weighted integration was statistically optimal, in the present study biased integration was suboptimal. These findings suggest that reliability-weighted integrated decision-making is automatic, taking place even when it is detrimental for performance.
Footnotes
This work was supported by the Australian Research Council (ARC; www.arc.gov.au) Center of Excellence for Integrative Brain Function (ARC Center Grant CE140100007). J.B.M. was supported by an ARC Australian Laureate Fellowship (FL110100103). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
The authors declare no competing financial interests.
- Correspondence should be addressed to Dragan Rangelov at d.rangelov{at}uq.edu.au
