Causal Influence of Linguistic Learning on Perceptual and Conceptual Processing: A Brain-Constrained Deep Neural Network Study of Proper Names and Category Terms

Experimental design used for instance learning and conceptual grounding. A, Training phase with 30 object instances from ten categories. The categories are illustrated with images of robots and cat faces, but note that this is for illustrative purposes. The actual input to the model was not images, but grounding patterns consisting of sets of activated neurons (see main text for details). For each trained instance, the grounding pattern (i) was either presented to the network on its own (no symbol) or combined with a “word form pattern” of type (ii, category term) or type (iii, proper name). B, Testing phase with a collection of the initially trained 30 instances and 30 novel instances from the 10 original categories, resulting in 60 testing instances (i.e., 6 per category). C, Training conditions in the main simulations (top) and control simulations (bottom) differ in the number of training trials (tt) to match the number of instance representations and the number of word form representations, respectively.

A, Schematic extraction of a 60×60 RDM, which represents 12 instances from two different categories and the similarities between any instance pair. For illustration, we once again use the categories of robots and cat faces. The schematic dissimilarity matrix illustrates how between-category (cells outside the red boundaries) and within-category dissimilarities (cells within the red boundaries) were calculated. Of interest are the (1) within-category dissimilarity among trained instances (DissimW−TT , lightest blue shade), (2) within-category dissimilarity between a trained and a novel instance (DissimW−TN , intermediate blue shade), and (3) between-category dissimilarity of two trained instances (DissimB−TT , darkest blue shade). The RDM is symmetric about its diagonal (gray) of zeros (representing the nondissimilarity of each of the instances to itself). Only the upper half of the RDM is used for analysis, and the lower half could be abandoned (black). B, RDMs for each of the twelve model areas in three main simulations: no symbol (top row), category term (middle row), and proper name (bottom row). The squares indicate the degrees to which network activity in the 12 network areas elicited by (12 out of 60) grounding patterns in the three learning conditions differed between each other within and between categories and are color-coded from turquoise (no dissimilarity, Dissim=0 ), blue, pink, and to dark red (high dissimilarity, Dissim>3 ).

Bar charts depicting dissimilarities between network activity elicited by trained grounding patterns after learning for each of the three training conditions. A, Main simulation: within-category (W-TT) and between-category (B-TT) dissimilarity values across all 30 trained activity patterns were averaged for each of the twelve model areas. B,C, Within-category (W-TT) and between-category (B-TT) dissimilarities across the 30 trained items were averaged for extrasylvian model areas. The three training conditions of the main simulations (B) were no symbol (NoS, gray), category term (CT, blue), and proper name (PN, pink). The four training conditions of the control simulation (C) were no symbol with each instance presented over 1,000 (NoS_1x, blue-striped gray) or 3,000 trials (NoS_3x, pink-striped gray), Category term where each instance presented over 1,000 trials (CT_1x, blue) and proper name where each instance presented over 3,000 trials (PN_3x, pink). The error bars represent 95% confidence intervals of the mean. The circles above the bars represent post hoc pairwise comparisons between a reference (circles with filled colors) and a corresponding mean (unfilled circles) after Bonferroni’s correction (critical p value = 0.005). Ten comparisons relevant to the main effects of training condition and dissimilarity type and their interaction are illustrated. The asterisks represent two-tailed p values: **p < 0.005, and ***p < 0.001, ns, not significant. The results were replicated in the whole model architecture (six extrasylvian and six perisylvian model areas); see Extended Data Figure 4-1 and Extended Data Table 4-1.

Bar charts depicting dissimilarities between network activity elicited by trained novel grounding patterns after learning for each of the three training conditions. A, Main simulation: within-category dissimilarity values between any two trained instances (W-TT) and between trained and novel instances were averaged for each of the twelve model areas. B,C Within-category dissimilarities between any two trained instances (W-TT) and between trained and novel instances (W-TN) were averaged for extrasylvian model areas. The three training conditions of the main simulations (B) were no symbol (NoS, gray), category term (CT, blue), and proper name (PN, pink). The four training conditions of the control simulation (C) were NoS_1x (blue-striped gray) or NoS_3x (pink-striped gray), CT_1x (blue) and PN_3x (pink). For further explanation, see Figure 4. The results were replicated in the whole model architecture (six extrasylvian and six perisylvian model areas); see Extended Data Figure 5-1 and Extended Data Table 5-1.