Abstract
Categorization is an essential cognitive and perceptual process, which happens spontaneously. However, earlier research often neglected the spontaneous nature of this process by mainly adopting explicit tasks in behavioral or neuroimaging paradigms. Here, we use frequency-tagging (FT) during electroencephalography (EEG) in 22 healthy human participants (both male and female) as a direct approach to pinpoint spontaneous visual categorical processing. Starting from schematic natural visual stimuli, we created morph sequences comprising 11 equal steps. Mirroring a behavioral categorical perception discrimination paradigm, we administered a FT-EEG oddball paradigm, assessing neural sensitivity for equally sized differences within and between stimulus categories. Likewise, mirroring a behavioral category classification paradigm, we administered a sweep FT-EEG oddball paradigm, sweeping from one end of the morph sequence to the other, thereby allowing us to objectively pinpoint the neural category boundary. We found that FT-EEG can implicitly measure categorical processing and discrimination. More specifically, we could derive an objective neural index of the required level to differentiate between the two categories, and this neural index showed the typical marker of categorical perception (i.e., stronger discrimination across as compared with within categories). The neural findings of the implicit paradigms were also validated using an explicit behavioral task. These results provide evidence that FT-EEG can be used as an objective tool to measure discrimination and categorization and that the human brain inherently and spontaneously (without any conscious or decisional processes) uses higher-level meaningful categorization information to interpret ambiguous (morph) shapes.
- categorical perception
- categorization
- discrimination
- electroencephalography
- frequency-tagging
- shape perception
Significance Statement
Every time when we encounter a new image or object, we will automatically relate it to previously stored categories. This categorization process allows us to efficiently react on new information and it influences our perception. The behavioral hallmark of categorical perception entails that we perceive differences between categories more distinctly than within a category. Previous research has mainly investigated categorical processing using explicit tasks. Here, we use carefully controlled morphs with implicit, neural electroencephalography measures to assess spontaneous categorical processing. We found higher neural amplitudes for “between”- than “within”-category morph pairs, establishing a neural correlate of categorical perception. This provides evidence that the brain inherently and automatically uses higher-level meaningful categorization information to interpret ambiguous shapes.
Introduction
The ability to categorize is an essential cognitive function
Categorization allows one to respond quickly and adaptively to new exemplars of a known category, by treating these new exemplars as somehow equivalent (Mervis and Rosch, 1981). This process enables us to interact efficiently with the world, reducing its complexity and variability. It involves abstraction using relevant shape similarities disregarding minor or irrelevant differences (Wagemans, 2013). Of course, not all differences should be ignored as one also needs to be able to discriminate between different exemplars of a known category (e.g., different faces). The ability to differentiate between exemplars (i.e., discrimination) presents the counterpart of categorization, and both processes interact (Medin et al., 1993). A well-documented interaction is categorical perception or the reduced discrimination for exemplars within a category and the enhanced discrimination (“discrimination peak”) across the category boundary (Goldstone, 1998).
Most studies only use explicit behavioral measures when investigating this spontaneous, automatic process
A disadvantage of behavioral categorization experiments is that responses may be confounded by various cognitive processes and biases, such as decisional or motivational processes. A neuroimaging approach can solve this; however, most fMRI studies focused on the key brain areas involved in categorization as well as the spatial distribution of categories (Kanwisher et al., 1997; Kriegeskorte et al., 2008). Only adaptation fMRI and visual mismatch negativity approaches have looked into a complementary neural measure to categorical behavioral hallmarks. However, earlier adaptation fMRI studies mainly found evidence for enhanced shape selectivity after category learning (Jiang et al., 2007; Gillebert et al., 2008, 2009; Folstein et al., 2013) and visual mismatch studies mainly focused on categorical perception of lexical characters or dot patterns (Yu et al., 2017a,b; Beck et al., 2021). In addition, these methods typically require many trial repetitions, they have low power, and in most of the studies mentioned above, participants also performed an explicit task. Accordingly, they are less suitable to investigate perceptual processes in an implicit way.
Opportunities for innovative implicit neural measures
A more direct and implicit neural index of perceptual discrimination and categorization can be frequency-tagging (FT) during scalp electroencephalography (EEG) recording. The principle of FT-EEG is that fast periodic visual stimulation of the human brain at a constant frequency rate (e.g., 6 Hz) leads to an EEG response on the scalp exactly at that frequency [i.e., steady-state visual evoked potential; see Norcia et al. (2015) for a review]. Using FT-EEG, the detection of periodically introduced oddball images in a series of base images will be signaled by an EEG response at the oddball frequency, which makes it an objective and implicit measure for change detection. FT-EEG paradigms have been validated in the context of low-level (e.g., contrast sensitivity; Norcia et al., 2015) and mid-level visual processing (e.g., Gestalt formation; Alp et al., 2016), as well as higher-level face processing (e.g., face identity; Rossion et al., 2012), but not yet for shape discrimination and categorization.
We investigate spontaneous visual discrimination and categorization via FT-EEG in morph sequences
In morph sequences, schematic natural objects are gradually morphed into each other in physically equal steps, allowing us to study discrimination and categorization while retaining control over physical differences. To integrate performance on standard behavioral tasks and FT-EEG measures, participants performed both types of experiments in the same session. Yet, behavioral tasks were administered at the end of the session, safeguarding the implicit nature of the FT-EEG measures.
To the extent that FT-EEG is a direct index of categorical perception, we expect roughly similar findings at a behavioral and neural level. Consistent with the behavioral categorical perception effect, we predict stronger neural oddball responses when contrasting “between”-category base–oddball pairs with “within”-category base–oddball pairs in a FT-EEG oddball experiment. Likewise, consistent with the presence of an abrupt behavioral category boundary, a FT-EEG oddball paradigm where the oddball stimulus is swept along the morph sequence (while the base stimulus remains the same) may index the crossing of the category boundary by a sudden increase in neural oddball response.
In this way, we aim to develop and validate complementary behavioral and neural FT-EEG measures for discrimination and categorization of shapes and secondly, we want to use these direct FT-EEG measures to directly assess spontaneous and implicit categorical processing.
Materials and Methods
Participants
Twenty-two healthy volunteers (12 male, 10 female) took part in the study. Participants were between 18 and 35 years old (mean age, 22.74 ± 3.00 years) and had a corrected-to-normal vision, no history of psychiatric disorders, and no usage of neuroleptics. All participants (except one) were right-handed and were native Dutch speakers. The study was approved by the Ethical Committee of the University Hospital of Leuven and University of Leuven (KU Leuven). Volunteers were recruited by flyers and posters that were distributed throughout the KU Leuven and social media. Before the start of the study, all participants signed the informed consent. After completion of the study, they received a monetary compensation for their participation.
Apparatus and acquisition
The study was performed in a quiet room where light and environmental sounds were reduced. The experiments were programmed in PsychoPy2 (Peirce et al., 2019). Stimuli were presented on a gray background of a 27 inch LCD monitor with a screen resolution of 2,560 × 1,440 pixels and 60 Hz refresh rate. Participants were positioned at a distance of 80 cm using a chin rest.
EEG was recorded using a BioSemi ActiveTwo amplifier system with 64 Ag/AgCl electrodes. During recording, the system used two additional electrodes for reference and ground (common mode sense and driven right leg). Horizontal and vertical eye movements were recorded using four electrodes placed at the outer canthi of the eyes and above and below the right orbit. The EEG was sampled at 512 Hz and electrode impedances were kept above −30 µV and under 30 µV.
Stimuli
Stimuli were derived from the stimulus set used in the studies of Hartendorp et al. (2010). The original stimulus set consists of 40 continual sequences of black-and-white schematic natural stimuli, morphed from one entity to another in equally spaced steps resulting in 20 levels.
Folstein et al. (2012) made multiple remarks on the morph sequences used in previous neuroimaging studies on categorical perception. In particular, they indicated that previously used stimulus sets or sequences were not tested and controlled for behavioral categorical perception outcomes. To overcome this issue, here, we only selected morph sequences from Hartendorp et al. (2010) that yielded similar outcomes in both their forced choice and their free-naming task and with relatively similar position, configuration and pixel-wise variation between the consecutive levels. We tested the five preselected morph sequences in terms of categorical perception in pilot experiments.
Based on additional data acquired in these pilot experiments (see Results, Addendum: pilot experiments for stimulus selection and paradigm validation) two morph sequences were ultimately selected and applied throughout the current study: Peacock–Truck (PT) and Church–Duck (CD; Fig. 1a). The original stimuli were slightly adjusted: we reduced the number of levels from 20 to 11 levels, each morph level was positioned in the background center, and the background was adjusted to gray. Stimuli extended 7 × 3 and 5 × 4 (horizontal × vertical) visual degrees for the PT and CD morph sequences, respectively. The relative (pixel) size difference between each consequent step along the morph sequence was not significantly different (PT: Χ2(9) = 0.98, p = 0.99 and CD: Χ2(9) = 1.19, p = 0.99).
Morph sequences and pairs. a, Each stimulus contains morphs from one entity to another in 11 equally spaced steps. b, Only a subset of morph pairs along the morph sequence were used to study categorical perception: levels 4–7 and 5–8 as “between-category” pairs and levels 1–4 and 8–11 as “within-category” pairs. These “between”- and “within”-category pairs are indicated by orange and blue rectangles, respectively.
To investigate categorical processing along the morph sequence, behavioral and neural experiments used all morph levels along the morph sequence. To investigate discrimination (i.e., categorical perception), we used only a subset of morph pairs along the morph sequence in both experiments: levels 4–7 and 5–8 as “between-category” pairs and levels 1–4 and 8–11 as “within-category” pairs (Fig. 1b, based on stimulus selection). The relative curvature difference [as calculated by Li and Bonner (2020)] between the stimuli of each pair along the morph sequence was not significantly different (PT: Χ2(3) = 0.29, p = 0.96 and CD: Χ2(3) = 0.02, p = 0.99). Additionally, we observed no significant difference in the relative curvature change between the “between-category” and “within-category” pairs (PT: t(1)=−0.64, p = 0.64 and CD: t(1) = 5.36, p = 0.12). Accordingly, this ensures that the categorization processes are not simply driven by differences in mid-level stimulus characteristics, such as rectilinearity and curvature.
Experimental paradigm
All participants performed one testing session at the University Hospital of Leuven (Gasthuisberg). The procedure of the session was as follows: (1) preparing the EEG and giving instructions (1 h), (2) EEG experiments (1 h), and (3) behavioral experiments (1 h).
During EEG acquisition, neural measures of discrimination and categorical processing were obtained via implicit FT-EEG oddball and sweep oddball paradigms, respectively (Table 1, a and b). Importantly, these measures involved no explicit categorical task, except during the last FT-EEG sweep block (Table 1, c). Afterward, participants performed a behavioral same–different discrimination task (Table 1, d). The session finished with a behavioral classification categorization task (Table 1, e).
Study session
Neural measures
The principle of the FT-EEG oddball paradigm is the detection of periodically introduced oddball images in a series of base images by an EEG response at the oddball frequency (B, B, B, B, O, B), which makes it an objective and implicit measure for change detection (Fig. 2a). The implementation of the FT-EEG paradigms was especially tailored to the standard behavioral psychophysical discrimination and categorization tasks. Typically, in a discrimination task, two instances of objects along the stimulus continuum are presented, and discrimination sensitivity is calculated based on the accuracy of same–different responses. Analogous to this task, here, we implemented an oddball FT-EEG paradigm to quantify the spontaneous neural sensitivity for discriminating pairs of stimuli belonging either to the same category (within-category) or to different stimulus categories (between-category). If categorical perception occurs automatically and spontaneously, we expect that “between”- versus “within”-category pairs result in systematically larger oddball responses (indicating a more prominent discrimination; Fig. 2b). In a categorization task, on the other hand, a single instance is selected from the stimulus continuum and presented, and participants have to classify whether it belongs to one category or the other, allowing to estimate the position and slope of the category boundary (by logistic fitting). Analogous to this task, we designed a FT-EEG sweep paradigm to be able to pinpoint the category boundary. In the FT-EEG sweep paradigm, the oddball stimulus is systematically swept along each morph sequence while the base stimulus remains the same (i.e., one of the endpoints of each respective morph sequence) to determine the perceptual threshold signaling categorical processing (i.e., to derive an implicit “category boundary”, see example for the peacock to truck sweep in Fig. 2c). While the base image in the sweep oddball paradigm remains the same throughout the entire trial, the oddball image varies across the morph sequence. In particular, every 10 s, the oddball image changes to the next morph level. Unlike the oddball paradigm in which we contrast pairs with an identical physical distance (which may either fall within one category or across the category boundary), in the sweep oddball trial, the base–oddball combination entails a linearly larger step size for each consequent sweep step (i.e., level 1 as base with level 1 as oddball followed by level 1 as base with level 2 as oddball etc.). Note that we checked quantitatively that all the steps of the morph stimuli along the morph sequence span an identical distance in terms of pixel size (see above, Stimuli). Accordingly, if spontaneous discrimination would largely be driven by low-level stimulus features, we may thus expect a gradual (linear) increase of the neural response in the sweep oddball paradigm, which would mirror the increasing physical difference between the stimulus pairs. Yet, if spontaneous discrimination is influenced by higher-level meaningful (categorization) processes, we may expect a sudden and disproportional increase of the neural response, implying the crossing of the categorical boundary. Using the sweep oddball paradigm (compared with the regular oddball paradigm), the neural differences would become apparent in one trial. In comparison, the oddball paradigm uses different trials for different pairs within or across the category boundary to pick up possible neural discrimination differences across the morph sequence.
Illustration of the FT-EEG paradigms. a, Principle of the oddball paradigm. Top panel, Base stimuli are presented at 6 Hz base rate while oddball stimuli are inserted at every fifth cycle (i.e., 1.2 Hz oddball rate). Bottom panel, Base stimuli elicit responses at the base frequency (6 Hz). Left, When oddball stimuli are not perceived as different from the base stimuli, the stimulus presentation will only elicit responses at the base frequency (6 Hz) and harmonics (n × 6 Hz). Right, When oddball stimuli are perceived as different from the base stimuli, they elicit an additional oddball response at 1.2 Hz and harmonics (n × 1.2 Hz). b, Oddball paradigm for stimulus discrimination. Top panel, During EEG recording in the implicit oddball paradigm, different “between” (e.g., L4–L7) and “within” (e.g., L1–L4) category trials with their respective reversed base–oddball stimulus pairs (e.g., L7–L4 and L4–L1) are randomly presented. Size variations of 20% (20% smaller, normal size, 20% larger size) are randomly implemented with a different relative size at every consecutive presentation to abolish the impact of low-level (pixel) confounds. The base and oddball stimuli are presented using a sinusoidal contrast modulation at the presentation frequency. Bottom panel, For each oddball pair, the oddball response is calculated, and each corresponding response can be plotted along a spectrum. Note that we expect that “between”- versus “within”-category pairs result in systematically larger oddball responses (indicating a more prominent discrimination). This indicates the presence of spontaneous categorical perception. c, Sweep oddball paradigm for category classification. Left panel, Throughout a trial, the endpoint of a morph continuum is systematically presented as base stimulus (at 6 Hz), and in 10 s segments all different morph levels are systematically presented (from each endpoint) at the oddball frequency of 1.2 Hz. Right panel, For each sweep step, the oddball response is calculated (top), and each corresponding response can be plotted (bottom). Note that we expect that from a certain threshold (deviance from linearity) physical differences (on the x-axis) will result in systematically larger oddball responses (on the y-axis). We test this by investigating whether there is a distinct increase in the baseline-subtracted oddball amplitude between consecutive sweep steps. This threshold would represent the implicit category boundary.
In sum, the EEG part of the study consisted of seven blocks (of 8 min each). Even blocks (2, 4, 6) involved the administration of the oddball paradigm in which we compared the neural discrimination for morphs comprising “between”- and “within”-category pairs. Odd blocks (1, 3, 5, 7) used the sweep oddball paradigm to assess the presence of a neural category boundary across the morph sequence.
In the oddball and sweep oddball paradigm, base stimuli were presented at a frequency of 6 Hz, periodically interleaved every fifth cycle with the respective oddball stimuli (i.e., 1.2 Hz). Size variations of 20% (20% smaller, normal size, 20% larger size) were randomly implemented with a different relative size at every consecutive presentation (B, B, B, B, O, B) to abolish the impact of low-level (pixel) confounds. During the presentation of the oddball and sweep oddball FT-EEG trials (except for the last block), participants were instructed to fixate on a white cross positioned in the center of the stimuli while flickering stimuli were presented. They were instructed to press the mouse whenever they detected brief changes (500 ms) in the color of the fixation cross (i.e., white to red). This task was orthogonal to the manipulation of interest and was aimed to ensure that the participants had a constant level of attention throughout the entire experiment. Only during the last block of the sweep oddball FT-EEG paradigm, participants were instructed to perform an explicit perceptual and categorical change detection task on the stimuli that were presented during the sweep sequence. After every trial, the participant had a short break of 10 s. We included a self-paced break between each block.
Within- versus between-category oddball discrimination paradigm (Table 1, a; Fig. 2b)
For both morph sequences (i.e., PT and CD), the two “within”-category pairs and two “between”-category pairs (see above, Stimuli) were used in the oddball paradigm. In addition, each stimulus of the pair was presented once as base and once as oddball, leading to a total of 16 different oddball trials. Each type of trial was presented once, in randomized order, in each of the three different (even: 2, 4, 6) blocks.
Each trial started with the presentation of a fixation cross (jittered between 2 and 5 s) in the center of the screen, after which the stimuli were presented in the center of the screen using a sinusoidal contrast modulation. Each oddball trial lasted for 30 s with a fade-in and fade-out of 1.67 s, slowly increasing and decreasing the stimulus contrast. During each oddball trial, the color change occurred 10 times. An example of an implicit oddball FT-EEG trial (with orthogonal task) for PT with level 4 (as base stimulus) and level 7 (as oddball stimulus) can be found in Movie 1. Note that this involves a “between”-category pair contrast.
Sweep along-the-morph-sequence oddball categorization paradigm (Table 1, b and c; Fig. 2c)
Stimuli of each of the morph sequences were used in two sweep trials each, starting from each of the endpoints of the stimulus continuum (i.e., progressing from peacock to truck and vice versa and progressing from church to duck and vice versa), giving rise to four different trial types. Each type of trial was presented once, in randomized order, in each of the four different (odd: 1, 3, 5, 7) blocks. At the beginning of the peacock–truck sweep trial, for example, the base and oddball stimulus were identical (i.e., 100% peacock), and after every 10 s (or 12 presentations of the same oddball stimulus), the oddball stimulus systematically changed to the next morph level (i.e., “90% peacock–10% truck,” “80% peacock–20% truck,” etc.), reaching the “100% truck–0% peacock” after 11 steps. Hence, with 11 morph levels, the total duration of a sweep trial was 110 s (10 s per sweep step) with a fade-in and fade-out of 1.67 s. For the truck–peacock, church–duck, and duck–church sweep trials, we similarly swept through the continuum space in 11 steps (Fig. 2c).
Identical to the oddball paradigm, each sweep trial started with the presentation of a fixation cross (jittered between 2 and 5 s) in the center of the screen, after which the stimuli were presented (in the center of the screen) using a sinusoidal contrast modulation. During the first three administrations of a particular sweep trial (odd blocks: 1, 3, 5), participants were instructed to perform the orthogonal fixation cross color change detection task, with 30 color changes during each trial. An example of an implicit sweep FT-EEG trial (with orthogonal task) for morphing the peacock to the truck can be found in Movie 2.
During the last and fourth administration (7th block), however, participants were instructed to perform an explicit task during which they now had to focus on the stimuli and click twice: (1) first, as soon as they observed any difference in the shape of the morph and (2) as soon as they observed a new object in the morph (i.e., when the category switch occurred). To ensure that participants understood the perceptual and categorical change detection task correctly, block 7 started with two practice trials showing a sweep from an Apple–Heart morph sequence (Hartendorp et al., 2010). Importantly, this explicit sweep oddball block and the corresponding instructions and practice trials were intentionally administered at the end of the EEG experiment to safeguard the implicit nature of the previous FT-EEG measures.
Behavioral measures
Same–different discrimination task (Table 1, d;Fig. 3a)
To assess behavioral discrimination along the morph sequences, participants performed a 2-Alternative Forced Choice (2-AFC) same–different task. During the task, morph stimuli appeared simultaneously (left and right) on the screen and participants had to indicate whether the shape of the morphs was same or different, regardless of the size and the meaning of the morph. Throughout the task, participants were presented with 14 different morph pairs for each morph sequence. These pairs comprised the four “different pairs,” which were presented with each stimulus either on the left or on the right side of the screen, thus totaling up to 8 instances. In addition, six “same pairs” were added, in which each stimulus was compared with itself.
Illustration of the behavioral paradigms. a, To assess behavioral discrimination along the morph sequences, participants performed a 2-AFC same–different task. During the task, morph stimuli appeared simultaneously (left and right) on the screen and participants had to indicate whether the shape of the morphs was same or different. A trial consisted of a 1 s presentation of a fixation cross, a 0.2 s presentation of the target stimulus pair, and a 0.2 s presentation of the mask stimuli. b, During the category classification task, participants saw a single instance of the morph sequences and labeled it in a 2-AFC task with the word that described each morph figure best (e.g., “Duck” or “Church” and “Peacock” or “Truck”) as fast and accurately as possible. The structure of a trial was similar to the discrimination task, except that only one stimulus and mask stimulus was presented in the center of the screen.
A trial consisted of a 1 s presentation of a fixation cross, a 0.2 s presentation of the target stimulus pair, and a 0.2 s presentation of the mask stimuli (Fig. 3a). Mask stimuli consisted of a random squared grayscale pattern, independent of the stimulus properties and were displayed after stimulus presentation to avoid after-image effects. In addition, to ensure that participants would not base their judgments on irrelevant pixel level stimulus differences, the size of the stimuli varied by 10% (10% smaller, normal size, 10% larger) with a different relative size at every consecutive presentation. From the moment of stimulus presentation, participants were instructed to respond as fast and as accurately as possible (maximum time limit of 10 s). With keys 1 and 3, counterbalanced across participants, participants had to indicate whether the stimuli were same or different. These options were also presented on the left (corresponding with key 1) and right (corresponding with key 3) at the bottom of the screen after mask presentation. No direct feedback on the performance was offered, but change in color (e.g., green) of the labels indicated a registered response.
Each pair of each morph sequence was presented 20 times, giving a total of 560 trials. Five breaks were included throughout the task, and in each block, each pair was presented five times. The trial order was pseudorandomized to prevent consecutive presentation of identical trials. To ensure that participants understood the task, they performed 16 practice trials on the Apple–Heart morph sequence (Hartendorp et al., 2010).
Category classification task (Table 1, e;Fig. 3b)
During the category classification task, participants saw a single instance of the morph sequences and labeled it in a 2-AFC task with the word that described each morph figure best (i.e., “Duck” or “Church” and “Peacock” or “Truck”) as fast and accurately as possible (maximum time limit of 10 s). The 11 morph instances of the different morph sequences were presented in eight blocks: alternating between blocks containing morph figures from CD and blocks containing PT morph figures. In each block, each morph instance of a morph sequence was presented five times in a random order (with a different morph instance every consecutive presentation), thus summing up to 20 presentations per instance (440 trials). Between each block, participants could take a self-paced break and view the description labels for the next block. Whether participants started the experiment with the CD or PT block was randomly determined.
The structure of a trial was similar to the discrimination task, except that only one stimulus and mask stimulus was presented in the center of the screen (Fig. 3b). Response options were again presented at the bottom of the screen after mask presentation and related to key responses 1 and 3 (counterbalanced across participants).
EEG data analysis
Preprocessing
All EEG processing steps were carried out using Letswave 6 and 7 (http://nocions.webnode.com/letswave) in Matlab R2018a (MathWorks). EEG data were segmented in 36 s and 116 s segments for oddball and sweep trials, respectively (2 s before and 4 s after each sequence), bandpass filtered (0.1–100 Hz) using a fourth-order Butterworth filter, and down sampled to 256 Hz. For three participants who blinked excessively [>2 standard deviations (SDs) above the sample mean, M = 0.18 times/s across all EEG paradigms], blinks were corrected by means of independent component analysis (ICA) using the runica algorithm (Makeig et al., 1996) as implemented in EEGLAB. For these three participants, the component accounting for most of the variance and representing vertical eye movements was removed. Next, noisy electrodes were linearly interpolated from the three spatially nearest electrodes for two participants (one and two electrodes, respectively, outside of the proposed region of interest). All data segments were rereferenced to a common average reference. Finally, data segments were further cropped to contain an integer number of 1.2 Hz cycles (oddball frequency), beginning after fade-in until 30 s (36 cycles, 7,473 time bins in total) for the oddball trials and into the sweep steps of 10 s (12 cycles, 2,349 time bins in total) for the sweep trials.
Frequency–domain processing
The resulting segments were averaged for each paradigm (implicit oddball, implicit sweep, and explicit sweep), each morph sequence and each oddball type and sweep step separately (i.e., separately for the 16 different oddball trials and 11 steps of the different sweep trials) and transformed into the frequency domain using a fast Fourier transform (FFT). The amplitude spectrum was computed with a spectral resolution of 0.033 Hz (1/30 s) and 0.1 Hz (1/10 s) for the oddball and sweep paradigm, respectively.
In the EEG paradigms, the recorded EEG contains signals at frequencies that are integer multiples (harmonics) of the frequency at which images are presented (base stimulation frequency: 6 Hz) and at the frequency at which a dimension of interest is manipulated in the sequence (1.2 Hz) if the oddball and base stimuli are perceived as different by the neural system(s). Since the EEG response at (harmonics of) these frequencies reflects both the overall noise level and the signal unique to the stimulus presentation, we used a baseline-corrected approach to describe the response in relation to the noise level (Liu-Shuang et al., 2014; Dzhelyova et al., 2017). Most importantly, unlike ERP or fMRI research in which the baseline is referred to as the signal before stimulus presentation, here, we quantify and report the baseline-subtracted oddball amplitude which uses the baseline noise in each respective step of the sweep or each respective oddball trial. Therefore, we can quantify the responses at each sweep step or oddball trial in an unbiased way and compare the relative baseline-subtracted oddball amplitudes at each respective sweep step or oddball trial with each other. For the oddball paradigm, baseline-corrected amplitude was computed at each frequency bin as the amplitude value at a given bin subtracted by the average amplitude of the 20 surrounding frequency bins (12 bins on each side, i.e., 24 bins, but excluding the 2 bins directly adjacent and the 2 bins with the most extreme values). For the sweep paradigm with its lower spectral resolution, only 12 surrounding frequency bins (8 bins on each side, and again excluding the 2 bins directly adjacent and the 2 bins with the most extreme value) were used to compute the baseline-corrected amplitude.
Afterward, for each sweep step, oddball trial, paradigm, and morph sequence separately, we quantified the response by summing these baseline-corrected amplitudes across all consecutive significant harmonics and by regions of interest (ROIs).
Determination of harmonics
For each paradigm separately (FT-EEG oddball and FT-EEG sweep oddball), we determined the harmonics for which the amplitude was significantly above noise using a z-score approach (Liu-Shuang et al., 2014; Dzhelyova et al., 2017). Especially, by considering the EEG segments for which we expected the highest oddball activity, that is, step 11 contrasting the two original endpoint stimuli of the morph sequences in the FT-EEG sweep paradigm. For all segments, FFT amplitude spectra were averaged across subjects and then pooled across all electrodes and across electrodes in the relevant ROIs, and the resulting FFTs were then transformed in z-scores computed as the difference between the amplitude at each frequency bin and the mean amplitude of the corresponding bins divided by the SDs of amplitudes in these surrounding bins (with the number of bins equal to the ones used for baseline-subtraction). Significant harmonics corresponded with a z-score above Z > 1.64 or p < 0.05, one-tailed. Based on this criterion, for both the oddball and sweep oddball paradigm, we quantified oddball responses by summing five harmonics: harmonics 1 (1.2 Hz) to 6 (7.2 Hz) excluding the harmonic corresponding to the base stimulation frequency (6 Hz). In addition, for both paradigms, the general visual response was quantified as the sum of the response at the base rate (6 Hz) and three consecutive harmonics (12, 18, and 24 Hz).
Determination of ROIs
As in Vos et al. (2023), we wanted to select the ROIs objectively, based on the data of all the subjects. We determined the ROIs separately for the base frequency (6 Hz) and the oddball frequency (1.2 Hz), as we expected different patterns of activation for the different frequencies. Hence, we calculated the baseline-subtracted amplitude across all subjects, all stimuli, and each electrode per paradigm and per sweep step, and we summed across the significant harmonics. All electrodes for which the baseline-subtracted amplitude of the response was significantly higher than the mean response (Bonferroni corrected) were retained and grouped in an ROI based on their location on the scalp. In line with the visual inspection of the topographical maps, EEG amplitude was quantified in three ROIs. Similar to multiple studies assessing face categorization via FT-EEG (Dzhelyova and Rossion, 2014a,b; Liu-Shuang et al., 2014, 2016; Rossion et al., 2015), for both paradigms, the analysis of the general visual response at base rate frequency (6 Hz and its harmonics) focused on a medial occipital ROI (MO: Oz, Iz, O1, O2), and the analysis at the oddball frequency (1.2 Hz and harmonics) focused on a left occipitotemporal ROI (LOT: P7, PO3, PO7) and a right occipitotemporal ROI (ROT: P8, PO4, PO8). This is in accordance with the medial occipital region as most responsive for base rate stimulation (Dzhelyova et al., 2017; Vettori et al., 2019).
Statistical data analysis
All data preprocessing and the consequent statistical analyses were performed using R software (https://www.R-project.org/; version 4.2.1). Linear mixed models (LMMs) were applied using the package afex in R (Singmann and Kellen, 2019). In each LMM, a random intercept per participant was implemented to account for repeated testing. Post hoc contrasts were tested with Bonferroni’s correction. Extreme outlying data points (i.e., values above Q3 + 3xIQR or below Q1 − 3xIQR) were removed. All assumptions in terms of linearity, normality, and constant variance of residuals were verified and met for all LMMs.
Neural measures
Base activity
We applied an LMM on the summed baseline-corrected base amplitudes for the MO region. For the implicit oddball paradigm (Table 1, a), type of pair (within vs between category), and morph sequence (PT and CD) were implemented as fixed within-subject factors. For the sweep paradigm (Table 1, b and c), sweep step (11 steps), morph sequence (PT and CD), and type of sweep paradigm (implicit vs explicit) were implemented as fixed within-subject factors. Difference in the direction of the sweep and/or reversal in base and oddball stimulus was not considered.
Oddball activity
We applied an LMM on the summed baseline-subtracted oddball amplitudes across LOT and ROT regions. For the implicit oddball paradigm (Table 1, a), type of pair (within vs between category), morph sequence (PT and CD), and ROI (LOT and ROT) were implemented as fixed within-subject factors. For each sweep paradigm (implicit and explicit separately; Table 1, b and c), sweep step (11 steps), morph sequence (PT and CD), and ROI (LOT and ROT) were implemented as fixed within-subject factors. Again, difference in the direction of the sweep and/or reversal in base and oddball stimulus was not considered. Specifically, for the sweep EEG data (implicit and explicit separately), the “category boundary” (at group-level) was determined by defining at which morph level the transition from each step to its consecutive step resulted in a significant difference in baseline-subtracted oddball amplitude along the sweep morph sequence (using post hoc tests). Given the higher variability for explicit sweep EEG data (only 1 presentation for each trial!), we additionally used a segmented regression analysis combined with Davies test to determine the breakpoint or “category boundary” (at group level; Davies, 2002).
Task performance during FT-EEG assessment
Orthogonal task
In case participants responded within 1 s after the onset of the fixation cross color change, this answer was scored as correct, and reaction time (RT) was included. We report the average accuracy and RT with standard error for the implicit oddball and sweep paradigm (Table 1, and b). An LMM with type of pair (within vs between category) and morph sequence (PT and CD) as fixed factors was applied on both accuracy and RT.
Explicit task
During the last block, in which participants performed an explicit change detection task during the FT-EEG sweep (Table 1, c), their two presses in seconds were converted to the corresponding discrete sweep step. We report the average sweep step with standard error. For the correlation with neural measures, we used a Spearman correlation.
Behavioral measures
Same–different discrimination task
For the 2-AFC discrimination task, d-prime was calculated as a bias-free index of accuracy for each discrimination pair (hits corresponding with the percentage of different responses for the different pairs and false alarms corresponding with the percentage of different responses for the same pairs; Stanislaw and Todorov, 1999) per morph sequence and participant. Afterward, to both d-prime and RT, an LMM was applied, with pair (within vs between category) and morph sequence (PT and CD) as fixed within-subject factors. Given that preliminary analyses showed no difference in presentation position and block order, these factors were not taken into account.
Category classification task
For the 2-AFC categorization task, the percentage of perceived category (i.e., either peacock or truck) was fitted via a psychometric curve (logistic, using the quickpsy package in R; Linares and López-Moliner, 2016) to derive an individual threshold (i.e., category boundary) and slope of this category boundary per morph sequence and for each participant and across group level (95% confidence interval). For RTs, an LMM was applied with level (11 levels) and morph sequence (PT and CD) as fixed within-subject factors. Specifically, the increased RT at “category boundary” was determined by establishing at which morph level the transition from one step to its consecutive step resulted in a significant difference in RT along morph sequence (using post hoc testing).
Results
No difference in orthogonal task performance in the between-within category oddball trials
Participants successfully performed the orthogonal fixation-cross color task during the FT-EEG paradigms. For one participant, no data for the orthogonal task was recorded due to a technical problem. For the sweep paradigm, average accuracy and RTs were M = 0.92 ± 0.02 and M = 0.43 ± 0.01, respectively. For the oddball paradigm, average accuracy and RTs were M = 0.97 ± 0.01 and M = 0.41 ± 0.01, respectively. For accuracy, no significant main effect of morph sequence (F(1,54.50) = 1.27; p = 0.27) and “between”- versus “within”-category oddball trials (F(1,54.50) = 0.23; p = 0.64) nor any interaction effect (F(1,54.50) = 3.06; p = 0.09) was observed. As well as for RTs, no significant main effect of morph sequence (F(1,60) = 2.43; p = 0.12) and “between”- versus “within”-category oddball trials (F(1,60) = 0.05; p = 0.83) nor any interaction effect (F(1,60) = 0.17; p = 0.68) was observed. This suggests that attention remained the same no matter whether the participant perceived “between”- versus “within”-category oddball discrimination pairs.
No difference in amplitude of base rate EEG responses in the between-within category oddball trials and the implicit sweep steps
Base rate brain synchronization responses for the MO ROI for the oddball discrimination experiment are displayed in Fig. 4a. Base activity was comparable between the “between”- versus “within”-category oddball trials (F(1,123.97) = 0.09; p = 0.77) but note the significant main effect between the morph sequences (F(1,20.48) = 14.76; p < 0.001), with generally higher responses for the PT sequence. No interaction effect was present (F(1,123.76) = 0.03; p = 0.85). This suggests that attention was similar along the whole experiment; that is, attention remained the same no matter whether the participant perceived “between”- versus “within”-category oddball discrimination pairs.
Attention remained stable throughout the different oddball FT-EEG trials and the implicit FT-EEG sweep. a, No significant difference in baseline-subtracted base amplitude between the different oddball FT-EEG trials (between vs within category). b, Baseline-subtracted base amplitude remained stable throughout the implicit FT-EEG sweep paradigm. No significant differences were detected across the implicit sweep. In contrast, a significant drop in base synchronization was found in the explicit FT-EEG sweep between steps 5 and 7. *Error bars correspond to standard errors of the mean.
Base rate brain synchronization responses for the MO ROI for the sweep oddball experiment are displayed in Fig. 4b. LMM showed a significant main effect of experiment type (implicit vs explicit; F(1,882.09) = 248.87; p < 0.001), a significant effect of sweep step (F(10,881.93) = 5.37; p < 0.001), and a significant interaction between experiment type and sweep step (F(10,881.93) = 2.51; p = 0.006). Post hoc testing showed that the effect of sweep step is only present in the explicit and not in the implicit experimental condition. In particular, a significant decrease in base rate amplitude is present between steps 5 and 7 (t(882) = −2.88; p = 0.04) in the explicit sweep paradigm. Finally, note a significant main effect of the morph sequences (F(1,881.93) = 175.25; p < 0.001), with generally higher responses for the PT sequence. No significant interactions of sweep step × morph sequence (F(10,881.93) = 0.4; p = 0.95), experiment type × morph sequence (F(1,881.93) = 2.51; p = 0.11), and sweep step × morph sequence × experiment type (F(10,881.93) = 0.39; p = 0.95) were present. This suggests that attention was similar along the whole implicit FT-EEG sweep, no matter what stimulus step of the morph sequence (as oddball stimulus) was contrasted with the endpoint (as base stimulus). This is in contrast to a drop in base synchronization along the explicit FT-EEG sweep (between steps 5 and 7) which suggests a drop in attention after that particular sweep step.
Classical categorical perception effect in the behavioral data
Figure 5 displays the results for the behavioral 2-AFC tasks. For the explicit 2-AFC same–different discrimination task, we observed a robust discrimination peak pattern for both morph sequences (Fig. 5a). More specifically, the d-prime for the discrimination pairs that cross the category boundary was significantly higher than for the discrimination pairs within the category (F(1,151) = 230.85; p < 0.001). No main effect of morph sequence (F(1,151) = 2.93; p = 0.09) nor morph sequence × pair interaction was observed (F(1,151) = 0.06; p = 0.8). In addition, the RTs reflected this effect (Fig. 5b), with the RTs being significantly lower for the discrimination pairs across the category boundary compared with the pairs within the category (F(1,149.98) = 125.43; p < 0.001). No main effect of morph sequence was present (F(1,149.98) = 0.57; p = 0.45). Post hoc testing of the morph sequence × pairs interaction effect (F(1,149.98) = 5.42; p = 0.02) revealed a significant higher RT for pairs across the category boundary in CD compared with PT (t(150) = 2.19; p = 0.03).
Behavioral discrimination and categorization task results. a, The same–different task can give an indication of categorical perception, while (c) the explicit categorization task is necessary to indicate the location of the category boundary: CD, 6.32 with a 95% CI of [6.24–6.41], and PT, 5.52 with a 95% CI of [5.46–5.59]. b,d, RTs are in line with the effect found for both tasks (i.e., show the reversed pattern). *Error bars correspond to standard errors of the mean.
For the explicit categorization task (Fig. 5c), we observed a clear and steep categorical boundary for both morph sequences (for each individual subject: goodness-of-fit > 0.05). At group level, the category boundary for the CD morph sequence was estimated at 6.32 with a 95% CI of [6.24–6.41] and for the PT sequence at 5.52 with a 95% CI of [5.46–5.59]. Moreover, we also found a significant main effect around the category boundary for RTs (Fig. 5d; F(10,434.86) = 47.07; p < 0.001), with a marginal interaction effect of morph sequence × step (F(10,434.86) = 1.85; p = 0.05). Post hoc testing revealed highest RTs around the category boundary (CD: t(435)5-6 = 5.01, p < 0.0001 and t(435)6-7 = −6.73, p < 0.0001; PT: t(435)4-5 = 7.31, p < 0.0001 and t(435)6-7 = −7.61, p < 0.0001). No significant main effect of morph sequence (F(1,434.86) = 1.06; p = 0.30) was observed.
Implicit FT-EEG results do also index categorical perception
Figure 6 displays the results for the implicit FT-EEG paradigms. For the oddball paradigm, we observed a robust discrimination peak pattern in the oddball activity for both morph sequences (Fig. 6a). More specifically, the baseline-subtracted oddball amplitude for oddball trials containing discrimination pairs that crossed the category boundary was significantly higher than the oddball trials containing discrimination pairs that were situated within the same category (F(1,20.80) = 16.78; p < 0.001). There was a main effect of ROI (F(1,300.17) = 8.56; p = 0.004), but not for morph sequence (F(1,300.17) = 2.83; p = 0.09). No significant interaction of morph sequence × ROI (F(1,299.91) = 0.02; p = 0.90), morph sequence × pair (F(1,299.91) = 0.03; p = 0.87), and ROI × pair (F(1,299.91) = 0.31; p = 0.58) was found. A significant interaction effect between morph sequence × ROI × pair was present (F(1,300.17) = 4.80; p = 0.03). Post hoc testing revealed that the categorical perception was significantly present in the LOT region for the CD sequence (t(137) = 3.12; p = 0.002) and in the ROT region for the PT sequence (t(137) = 3.79; p = 0.0002). This effect is also reflected in the head topographies with stronger oddball activity for the oddball trials containing “between”-category pairs in the ROT for PT and in the LOT for CD.
Implicit FT-EEG results. a, The baseline-subtracted oddball amplitude derived from the FT-EEG oddball paradigm can be used to indicate categorical perception in CD in the LOT cortex and in PT in the ROT cortex. b, EEG measures from the implicit sweep trials give a linear increase in baseline-subtracted oddball amplitude along the morph sequence. In addition, a distinctive increase between consecutive sweeps steps, indicative of a spontaneous category boundary (see dashed line), can be derived between steps 6 and 7 for CD (nonsignificant) and between steps 5 and 6 for PT (significant, mainly driven by ROT). *Error bars correspond to standard errors of the mean.
For the sweep paradigm, we observed a linear increase in oddball activity along the different sweep steps with a clear and steep category boundary for both morph sequences (Fig. 6b). More specifically, we can observe an increase in baseline-subtracted oddball amplitude along the sweep steps (F(10,903.00) = 71.53; p < 0.001). Post hoc testing showed that the significant increase in baseline-subtracted oddball activity between consecutive sweep steps is situated between steps 5 and 6 (t(903) = 4.28; p = 0.0002). When we investigated this for each morph sequence separately, the most distinct increase in baseline-subtracted oddball activity between consecutive sweeps steps was located between steps 6–7 for the CD morph sequence (nonsignificant: t(903) = 2.61; p = 0.09) and steps 5–6 for the PT sequence (significant: t(903) = 3.61; p = 0.003). There was also a significant main effect of ROI (F(1,903.00) = 9.34; p = 0.002), a significant interaction effect of ROI × morph sequence (F(1,903.00) = 10.65; p = 0.001), and sweep step × ROI × morph sequence (F(10,903.00) = 1.92; p = 0.04). Post hoc testing showed that for the PT sequence, the baseline-subtracted oddball amplitude is significantly higher in the ROT region (t(903)LOT-ROT = −4.47; p < 0.0001). More specifically, the significant increase in baseline-subtracted oddball activity between steps 5 and 6 for the PT morph sequence was driven by the ROT region (t(903) = 3.54; p = 0.004). This effect is also reflected in the head topographies with clear lateralized oddball activity from step 5 onward for the PT morph (Fig. 6b, ROT). No significant main effect of morph sequence (F(1,903.00) = 2.22; p = 0.14), nor an interaction effect of sweep step × ROI (F(10,903.00) = 1.44; p = 0.16), and sweep step × morph sequence (F(10, 903.00) = 1.05; p = 0.40) was present.
Performing an explicit discrimination and categorization task while sweeping through a morph continuum
Figure 7a (bottom panel) displays the behavioral results of the explicit task during the sweep paradigm. For both morph sequences, participants reported that they perceived a first difference between the morph levels (i.e., perceptual change detection) between steps 3 and 4 in the morph sequence (CD: M = 3.52 ± 0.13; PT: M = 3.45 ± 0.12). Furthermore, the first step in which they perceived a different object (i.e., switch in category) in the morph sequence was closely after step 6 for both the CD sequence (M = 6.41 ± 0.21) and the PT sequence (M = 6.39 ± 0.14). Note that these perceived category boundaries roughly align with those that were estimated (fitted) during the classification task.
Explicit sweep FT-EEG results. a, Top panel, EEG measures from the explicit sweep FT-EEG trial give a peak in baseline-subtracted oddball amplitude along the morph sequence. This would correspond to the category boundary (indicated by a dashed line). *Error bars correspond to standard errors of the mean. Bottom panel, Histogram of participants’ behavioral responses along the sweep paradigm with a maximum of 44 responses per step (i.e., 22 participants responded once along the original direction and once along the opposite direction). Perceptual change detection occurred around step 3 and categorical change detection occurred around steps 6–7 for both CD and PT. b, Frontal activation is present at behavioral change detection.
Figure 7a (top panel) displays the baseline-subtracted oddball amplitudes for the explicit sweep paradigm. Here, we observe a significant main effect of sweep step (F(10,891.00) = 16.90; p < 0.001), of morph sequences (F(1,891.00) = 6.43; p = 0.01), an interaction effect of sweep step × morph sequence (F(10,891.01) = 2.34; p = 0.01), and ROI × morph sequence (F(1,890.95) = 4.21; p = 0.04). Post hoc testing showed that the baseline-subtracted oddball activity first increased significantly above zero at step 3 (t(86.9) = 2.10; p = 0.02) in line with the behavioral perceptual change detection. Afterward, baseline-subtracted oddball activity significantly increased between steps 4–5 (t(891) = 2.87; p = 0.04) and steps 5–6 (t(891) = 3.05; p = 0.02). Afterward, the baseline-subtracted oddball activity drops between steps 7 and 8 (t(891) = −3.52; p = 0.005). This effect is significant for the PT sequence: an increase in steps 4–5 (t(891) = 3.53; p = 0.004) and steps 5–6 (t(891) = 2.82; p = 0.05) with a drop between steps 7–8 (t(891) = −2.80; p = 0.05) and an increase at steps 9–10 (t(891) = 3.27; p = 0.01). For the CD sequence, we did not find any significant difference in baseline-subtracted oddball activity between consecutive steps. But a segmented regression analysis with Davies test could pinpoint the breakpoint or category boundary between steps 6 and 7 (6.56; p = 0.03). In addition, post hoc testing showed that the baseline-subtracted oddball amplitude is significantly higher in the LOT region for the CD sequence compared with the PT sequence (t(891)CD-PT = 3.25; p = 0.001). This effect is also reflected in the head topographies with clear lateralized oddball activity from step 6 for the CD morph (Fig. 7a, LOT). Finally, neural activity was also present in the frontal areas at behavioral change detection (Fig. 7b). No significant main effect of ROI (F(1,890.96) = 0.15; p = 0.7) nor interaction effect of sweep step × ROI (F(10,890.97) = 0.29; p = 0.98) and sweep step × ROI × morph sequence (F(10,890.97) = 0.67; p = 0.76) was present.
Using z-scores of the oddball amplitude at each sweep step of each participant, we could detect the neural perceptual detection (first step in which Z > 1.64) at an individual level. Using the Davies test, we could detect neural category detection at an individual level. We found a positive correlation between the behavioral categorical detection and neural categorical detection (r = 0.32; p = 0.03). Closer inspection revealed no clear indication of precedence of neural detection. We found no correlation between behavioral perceptual detection and neural perceptual detection (r = −0.06; p = 0.7). Probably this is due to the higher variability resulting from only collecting behavioral and neural data during one trial.
Addendum: pilot experiments for stimulus selection and paradigm validation
Folstein et al. (2012) suggested that the category-selective contrast may not have been found in previous adaptation fMRI studies because the morph stimuli did not yield behavioral evidence for changes in perceptual discriminability. Consequently, in order to preselect the most optimal stimuli, prior to our main study, we investigated categorical perception throughout a series of different morph sequences. Accordingly, we tested five morph sequences in pilot categorical perception experiments.
First, eight participants performed both behavioral (Fig. 8a) and neural (Fig. 8b) discrimination paradigms involving these five potential morph sequences. Results indicated that the PT and CD were the only morph sequences that showed a significant behavioral categorical perception effect (PT: t(72)between-within = 2.81, p = 0.006 and CD: t(72)between-within = 2.26, p = 0.03). The neural categorical perception effect was also significant for the PT morph sequence (t(203) = 3.40; p = 0.008). Neither the Watch–Seahorse (WS) nor the Candle–Bottle (CB) morph sequence showed a behavioral (WS: t(72)between-within = 1.51, p = 0.14 and CB: t(72)between-within = −1.51, p = 0.14) nor a neural (WS: t(203)between-within = −0.62, p = 0.54 and CB: t(203)between-within = 0.042, p = 0.97) categorical perception effect. The Penguin–Child (PC) morph sequence did not show a categorical perception effect in behavior (t(72)between-within = 1.46; p = 0.15) but showed a marginal neural (t(203)between-within = 1.77; p = 0.08) categorical perception effect.
Stimulus selection and paradigm validation. a, Using a behavioral same–different discrimination task, we investigated the categorical perception effect across five different morph sequences. Only in the PT and CD morph sequences, a clear difference in behavioral discrimination sensitivity was present in the “between”- versus “within”-category pairs. b, Using the implicit oddball EEG discrimination paradigm, we investigated the neural categorical perception effect across these same morph sequences. Here, morph sequences PT and PC showed a difference in neural discrimination sensitivity between the “between”- versus “within”-category pairs. c, In the ordered FT-EEG sweep, the oddball stimulus is swept along the morph sequence in a sequential way from one endpoint of the morph sequence to the other. A linear increase of the subtracted oddball activity is clear, with a sudden increase at the category boundary. d, In the random FT-EEG paradigm, every stimulus along the morph sequences (as oddball stimulus) was contrasted with the endpoint (as base stimulus) by randomly presenting each oddball–base pair in an oddball paradigm. When post hoc ordering the baseline-subtracted oddball activity along the morph sequence, we get a similar result as the ordered sweep paradigm. *Error bars correspond to standard errors of the mean.
An example of an implicit oddball FT-EEG trial. An illustration of an implicit oddball FT-EEG trial (with orthogonal task) for the PT sequence with level 4 as base stimulus and level 7 as oddball stimulus. Note that this involves a “between”-category pair contrast. [View online]
An example of an implicit sweep FT-EEG trial. An illustration of an implicit sweep FT-EEG trial (with orthogonal task) for the PT sequence with the peacock as base stimulus while the oddball stimulus gradually varies from the peacock toward the truck during the sweep trial. Note that the resolution of the movie is a bit reduced compared to the original presentation. [View online]
Second, these same eight participants performed the ordered sweep FT-EEG paradigm using the same five morph sequences (Fig. 8c). Third, an additional eight participants performed a random FT-EEG paradigm using the same five morph sequences in which the 11 oddball–base pairs of the different sweep steps were presented in different oddball trials in a random order (Fig. 8d). When post hoc ordering the baseline-subtracted oddball activity of the random FT-EEG design along the morph sequence, we got a similar result as the ordered sweep FT-EEG paradigm (although somewhat less pronounced psychometric curves). This indicates that perceived neural sensitivity in the FT-EEG sweep paradigm is mainly measuring a perceptual process rather than an induced expectation.
The results of these last two pilot experiments showed that the morph sequences WS and CB elicited lower (i.e., random FT-EEG paradigm: t(381)CB-CD = −2.66, p = 0.08; t(381)CB-PC = −3.80, p = 0.002; t(381)CB-PT = −3.10, p = 0.02; t(379)CB-WS = −1.50, p = 1.00) and less stable neural signals (WS: M = 0.38 ± 0.08 and CB: M = 0.45 ± 0.09), while CD and PT elicited highly correlated neural signals across the different participants in the sweep paradigm (PT: M = 0.69 ± 0.05 and CD: M = 0.60 ± 0.07).
Based on these pilot results, we used the PT and CD morph sequences in the current study.
Discussion
Here, for the first time, we investigate spontaneous and automatic categorical processing with novel FT-EEG paradigms in morph sequences and relate these results to standard behavioral psychophysical tasks.
FT-EEG measures spontaneous categorical processing and perception in visual morph sequences
“Between”- and “within”-category pairs were included in a FT-EEG oddball paradigm to obtain an implicit discrimination index (given by the baseline-subtracted oddball amplitude). Using the FT-EEG oddball paradigm, we find a similar pattern as in the behavioral results: a significantly higher baseline-subtracted oddball amplitude when contrasting pairs across the category boundary in comparison with pairs within a category.
In addition, to derive a more fine-grained implicit indication of the category boundary, we also incorporated each morph sequence in a FT-EEG sweep paradigm. In this paradigm, the base stimulus was one of the two endpoints of the morph sequence (peacock or truck and church or duck), and the oddball stimulus was gradually swept across the morph sequence from one endpoint to the other (e.g., peacock to truck). If discrimination would largely be driven by low-level stimulus features, we may expect a gradual (linear) increase of the neural response, which mirrors the increasing physical difference between the stimuli. Yet, if discrimination is supported by higher-level meaningful categorization processes, we may expect a sudden increase of the neural response, indexing the crossing of the categorical boundary.
For both morph sequences, the baseline-subtracted oddball amplitude increased linearly along the sweep continuum, with a distinct deviation from linearity at a particular step along the continuum (i.e., indicating the crossing of the category boundary). These EEG results nicely mirror the observed pattern in the 2-AFC behavioral tasks, given that the implicit discrimination peaks and category boundaries are located on the same position along the morph sequence as those explicitly identified by means of the behavioral 2-AFC tasks (steps 5–6 for PT and steps 6–7 for CD). This is rather remarkable considering that the corresponding measures were obtained in different tasks conducted in separate blocks. Moreover, the lack of significant differences in terms of orthogonal task performance and base rate synchronization amplitude clearly confirm that significant changes and trends found in the baseline-subtracted oddball amplitude are due to spontaneously perceived stimulus differences by the neural system(s) and not to systematic changes in attention or brain synchronization.
The implicit oddball paradigm results are in line with adaptation fMRI research (from Folstein et al., 2013) who showed a significant “between”- versus “within”-category pair contrast in a trained dimension while people performed an unrelated task. In addition, using the sweep oddball paradigm, we show for the first time an implicit fine-grained indication of categorical processing (i.e., implicit category boundary). We thus provide evidence that the brain inherently and automatically uses higher-level meaningful categorization information to interpret ambiguous (morph) shapes.
Quite to our surprise, baseline-subtracted oddball activity for “between”- versus “within”-category pairs was significantly higher in the left occipital cortex region for the CD sequence and in the right occipital cortex region for the PT sequence. This aligns with the significantly higher baseline-subtracted oddball activity in the right occipital cortex for the PT sequence during the sweep paradigm. Post hoc stimulus inspection shows that the two stimuli sequences differ systematically in their horizontal image orientation. More specifically, their discriminative feature (i.e., the head of the peacock and front of the truck for PT and the church tower and the head of the duck for CD) is more left and right oriented, respectively, and could therefore have caused a bias in the corresponding visual fields as well as in the respective hemispheric homologs (i.e., right for PT and left for CD). Additionally note that the higher baseline-subtracted base amplitude of the PT (in comparison with the CD) could be caused by its larger visual angle.
In this paper, we use morph sequences crossing the animate–inanimate boundary, which may imply an additional categorical difference which may be rooted in spatially different neural substrates (Grill-Spector and Weiner, 2014; Ritchie et al., 2021). However, in the stimulus selection (i.e., pilot results reported in Fig. 8b,c), we show similar results for morph sequences which do not cross the animate versus inanimate boundary, thereby confirming that our findings do not depend on this particular stimulus contrast.
It may be argued that multiple presentations of this implicit FT-EEG sweep paradigm (i.e., three blocks with an identical structure and identical order of systematically sweeping through the oddball continuum) might have produced some predictability or expectancy. However, Quek and Rossion (2017) provided no evidence for the influence and/or necessity of predictability in the FT-EEG sweep paradigm. Furthermore, results from our pilot experiments where we swept across the oddball stimulus space using a random presentation order instead of a systematically progressing order (Fig. 8d) revealed a similar pattern as in our main findings.
The spontaneous nature of categorical processing: using an implicit versus explicit approach
To investigate the spontaneous nature of categorical processing, the FT-EEG sweep paradigm was not only performed implicitly (i.e., with an orthogonal task) but also during an explicit change detection task. To the extent that categorical processing is a spontaneous process or rather a task-driven process, results may differ between the implicit and the explicit FT-EEG sweep paradigms.
Behavioral performance on the explicit task during the explicit FT-EEG sweep shows (1) a perceptual change detection at step 3 for both CD and PT and (2) a similar positioning of the category boundary (categorical change detection) as the 2-AFC task (i.e., step 6 for both CD and PT). This is in line with the EEG results recorded during this explicit FT-EEG sweep, the first oddball activity to be significantly above zero is at step 3 and the category boundary appears to be at a similar position as the implicit FT-EEG sweep paradigm (i.e., a clear peak on the category boundary appears in this explicit paradigm, because of a drop in activity after crossing this boundary). More specifically note the significant positive correlation between behavioral and neural categorical change detection during the explicit FT-EEG sweep. Interestingly, also the baseline-subtracted base amplitude (at 6 Hz and harmonics) shows a similar significant drop in activity after crossing the category boundary in this explicit FT-EEG sweep trial (between steps 5 and 7), which seems to suggest that attention drops after participants performed their behavioral duty. Yet, for the oddball activity, this decrease in amplitude lasts only for a short timeframe, after which it resumes again (Fig. 7a, PT).
Additional evidence for the impact of conscious decisional processes in this explicit FT-EEG sweep paradigm is provided by activation at the frontal regions for the explicit FT-EEG sweep paradigm at behavioral change detection (Fig. 7b). This is in line with results of Jiang et al. (2007), who found a category-selective effect in the prefrontal cortex only when performing an explicit categorization task.
Taken together, the results on the explicit FT-EEG sweep are largely similar to those on the implicit FT-EEG sweep, but they are also influenced by motivational and decisional processes.
Methodological reliability and future perspectives
The methodological choices for the currently developed FT-EEG paradigms were based on previous face categorization research choices and our pilots. We take note of the plethora of methodological experiments (Rossion et al., 2020) in FT-EEG face categorization literature. For example, previous research (Alonso-Prieto et al., 2013) indicated that maximal responses for facial identity discrimination were found at a base rate stimulation of 5.88 Hz (170 ms per stimulus), thus corresponding with the classical N170 face processing peak (Liu-Shuang et al., 2016). The determination of the optimal stimulation frequency is thus dependent of the specific duration of the perceptual processing of the stimuli. Here, a similar frequency as for faces was used, based on similar rates of processing for objects (160–170 ms; Rousselet and Pernet, 2011).
FT-EEG research has flourished due to the short recording times and the highly reliable data. Due to these advantages and the low cost, this technique has been explored more and more as a clinical tool, for instance, in autism research (Van der Donck et al., 2019; Vettori et al., 2019, 2020; Sapey-Triomphe et al., 2023). Here, we show for the first time that it can be used as a direct measure for spontaneous categorical processing. Our approach may be able to shed light on fields such as categorization research in autism vexed by inconsistent findings (Van Overwalle et al., 2023).
The use of artificial stimuli would address concerns or limitations linked to the currently used stimuli. Artificial stimuli with the absence of linguistic labels would enable a purer demonstration of emerging categorization processes in the brain via sensory processing, irrespective of prior (conscious or unconscious) semantic associations (Van Overwalle et al., 2023). Looking at category learning could be an additional interesting step. In this way, we can compare FT-EEG measures on a trained stimulus set (after category learning) versus an untrained stimulus set to unambiguously relate them to the category process itself (see earlier research in adaptation fMRI: Jiang et al., 2007; Gillebert et al., 2008, 2009). The shape of stimuli could also be more controlled which would likely elicit more comparable neural activation (i.e., base synchronization) and topographical location of the oddball. Finally, artificial stimuli would also enable us to manipulate mid-level properties more easily (such as curvature or rectilinearity, Nasr et al., 2014; Long et al., 2018; Li and Bonner, 2020) to investigate the perceptual basis of categorical processing.
Conclusion
In this paper, (1) we demonstrate that FT-EEG can provide a direct measure of perceptual discrimination and categorization, (2) we validate outcome measures in explicit standard psychophysical tasks, and (3) we apply this technique to demonstrate the spontaneous nature of categorical processing.
Availability of Data and Materials
Preprocessed EEG and behavioral data with analyses scripts necessary to reproduce the statistical analyses and figures in this manuscript are available at https://osf.io/rcxfa/.
Footnotes
We thank Charlotte Buhre and Lise Desplenter with their help recruiting and testing the participants and Christophe Bossens for his help in programming the experiment. We thank Joan Liu-Shuang and Bruno Rossion for the discussions at the early stage of the project. We thank Pieter Moors, Sofie Vettori, Eline Van Geert, and Yaniv Morgenstern for sharing analysis tips and scripts. Finally, we thank all the participants for their time.
This work was supported by a personal fellowship awarded to J.V.O. (Fonds Wetenschappelijk Onderzoek; grant number 1177920N) and by a Methusalem grant awarded to J.W. (METH/14/02), the Excellence of Science (EOS) grant (G0E8718N; HUMVISCAT), and an Internal KU Leuven grant for small research infrastructure (KA/20/080) awarded to B.B.
↵*B.B. and J.W. share last authorship.
The authors declare no competing financial interests.
- Correspondence should be addressed to Jaana Van Overwalle at jaana.vanoverwalle{at}kuleuven.be.
