Article Figures & Data
Figures
- Figure 1.
Task design. A, Example scene category pairings for 1 participant. Three of 12 categories were assigned to Condition A. Each A category was reliably followed by one of three other categories assigned to Condition B to create pairs. The remaining six categories assigned to Condition X were not paired. Participants viewed the A and B (Structured) and X (Random) categories in separate blocks of the task. B, Example stimuli from the Structured block. Participants passively viewed a continuous stream of scenes. Each scene was shown for 267 ms, followed by an ISI of 267 ms with only a fixation cross on the screen. The stream was segmented into subblocks. The same exemplar of each category was presented 4 times per subblock, and new exemplars were introduced for the next subblock. For the Structured block, the category pairs remained consistent across subblocks. Colored frame represents category pairs, corresponding to the A-B pairs (and colored arrows) in subpanel A.
- Figure 2.
Electrode coverage. The contact locations on the grid, strip, and/or depth electrodes for each participant are plotted as circles in standard brain space. Contacts colored in blue were localized to the visual cortex mask.
- Figure 3.
Category evidence analysis pipeline. A, A Morlet wavelet approach was used to extract time-frequency information from contacts in visual cortex. This resulted in contact by frequency vectors for every time point of encoding phase and memory phase trials, which served as the neural patterns for subsequent analysis steps. B, To identify the neural patterns that distinguished between categories, we ran a series of binary classifiers for every pair of categories from the memory phase trials. These classifiers were trained on the contact by frequency vectors for a single time point or set of time points. The classifiers were then tested on time points from held-out data. C, After a series of feature selection steps, we chose the per-participant top-N time point set that produced the best classification accuracy, and then averaged contact by frequency vectors across those time points (across all exemplars of a given category) to create a “template” of neural activity for each category. D, We then correlated the template for each category from the memory phase with the contact by frequency vector at each time point of each trial/exemplar from that category during the (independent) encoding phase, yielding a time course of pattern similarity reflecting neural category evidence.
- Figure 4.
Behavioral results. A, Overall memory performance collapsed across conditions. Circle represents A′ (a sensitivity measure for recognition memory) for each participant. All participants were above chance (0.5). B, Hit rate as a function of condition (A: predictive; B: predictable; X: control). Bars represent group means. Errors bars indicate bootstrapped 95% CI across participants. Individual participant data are overlaid with the gray circles and lines.
- Figure 5.
Neural frequency tagging analysis. A, Schematic of analysis and hypothesized neural oscillations. We expect entrainment of visual contacts at the frequency of images in both blocks. In the Structured block, we also expect entrainment at the frequency of category pairs. B, These hypotheses were confirmed, with reliable peaks in coherence at the image and pair frequencies in Structured blocks but only at the image frequency in Random blocks. C, We examined the emergence of entrainment over time by measuring the difference in coherence at the frequency of interest, relative to the two neighboring frequencies, as we iteratively increased the number of subblocks from the start of the run included in the analysis. Left, Coherence at the pair frequency emerged over time in the Structured block (reaching significance by the 13th subblock and beyond) but not in the Random block. Right, Coherence at the image frequency was high in both blocks, regardless of how many subblocks were included. Error bands indicate the 95% bootstrapped CIs across participants.
- Figure 6.
Category decoding and feature selection. A, To establish overall category decoding accuracy, we trained and tested binary category classifiers separately for all individual time points, yielding a temporal generalization matrix. B, As a first feature-selection step, we computed the average classification accuracy (across pairwise classifiers) for each training time point and participant (colored lines). We then ranked the time points by classification accuracy. C, To select the set of time points that produced the best classification for a given participant, we trained and tested the category classifiers on an increasing number of time points, starting with the best-performing time point identified in B and iteratively adding time points by rank. We then computed the per-participant average classification accuracy for each set of time points. D, Histogram depicting which training time points were selected for template creation for all participants (e.g., count = 3 indicates that that time point was included for 3 of the 7 participants).
- Figure 7.
Neural category evidence. A, Time course of similarity between patterns of neural activity in visual contacts evoked by exemplars from A (predictive), B (predictable), and X (control) categories and category template patterns for A, B, and X, respectively, baselined to average evidence for the other categories of the same condition. Inset, Raw pattern similarity before baseline subtraction for the category template of interest (dark) and the average of the other category templates from the same condition (light). Error bands were removed for ease of visualization. Current, the trial when the item was presented; Pre, the trial before the item was presented; Post, the trial after the item was presented. For each row/condition, the Pre, Current, and Post trials are compared with the same category template (Current). Error bands represent the bootstrapped 95% CIs across participants (i.e., any time point whose band excludes 0, p < 0.05). B, Average pattern similarity collapsed across time points within ON (stimulus on screen) and ISI (fixation between stimuli) epochs. Each dot represents an individual participant. Bars represent the means across participants. Error bars indicate the bootstrapped 95% CIs. *p < 0.05. **p < 0.01. ***p < 0.001.
- Figure 8.
Subsequent memory analysis. A, Left, Time course of pattern similarity in visual contacts between the encoding of A items and the category templates for A (Perceived, A during A) and B (Predicted, B during A), as a function of whether the A items were subsequently remembered or forgotten. Right, Pattern similarity averaged within the ISI period, the epoch in which we observed overall evidence of prediction, as a function of subsequent memory for A items (filled bars represent remembered; empty bars represent forgotten). B, Left, Time course of pattern similarity in visual contacts between the category template for B and the encoding of A items (Predicted, B during A) and B items (Perceived, B during B), as a function of whether the B items were subsequently remembered or forgotten. Right, Pattern similarity averaged within the ISI period, as a function of subsequent memory for B items. Error shading/bars represent the bootstrapped 95% CI across participants. Each dot represents an individual participant. *p < 0.05. **p < 0.01. ***p < 0.001.
Tables
ID Age (yr) Sex nElec (vis) Implant Data collected Notes 1 19 F 203 (21) R G/S/D 2S, 2R R2 mem data not usable (D) 2 26 F 163 (59) L G/S/D 2S, 2R — 3 43 F 172 (10) Bi D 1S, 2R — 4 61 F 136 (0) Bi D 1S, 1R neural mem data not usable (T) 5 31 M 152 (31) L G/S/D 2S, 2R R1 encoding data not usable (T) 6 69 F 92 (7) L D 2S, 2R — 7 33 M 232 (22) Bi D 1S, 1R — 8 31 F 192 (20) Bi D 2S, 2R no mem data collected (C) 9 56 F 192 (36) Bi D 2S, 2R R1 encoding data not usable (T) 10 53 M 148 (0) Bi D 2S, 2R — ↵aDescription of patient participation. nElec (vis), the total number of electrode contacts, followed by the number of visual electrode contacts. Implant: R, right-sided implant; L, left-sided implant; Bi, bilateral implant; G, grid; S, strip; D, depth. Data collected: the number of runs for each condition collected (S, Structured; R, Random). Notes: which runs (if any) were excluded from given analyses and why: D, patient distraction (e.g., a clinician coming in and disrupting testing); T, trigger issue (i.e., an error with the equipment such that we could not align individual trials to our neural signal); C, computer error (e.g., the computer crashed).
