Concepts are composed of features related to different sensory and motor modalities such as vision, sound, and action. It is a matter of controversy whether conceptual features are represented in sensory-motor areas reflecting the specific learning experience during acquisition. In order to address this issue, we assessed the plasticity of conceptual representations by training human participants with novel objects under different training conditions. These objects were assigned to categories such that for one class of categories, the overall shape was diagnostic for category membership, whereas for the other class, a detail feature affording a particular action was diagnostic. During training, participants were asked to either make an action pantomime toward the detail feature of the novel object or point to it. In a categorization task at test, we assessed the neural correlates of the acquired conceptual representations by measuring electrical brain activity. Here, we show that the same object is differentially processed depending on the sensory-motor interactions during knowledge acquisition. Only in the pantomime group did we find early activation in frontal motor regions and later activation in occipito-parietal visual-motor regions. In the pointing training group, these effects were absent. These results show that action information contributes to conceptual processing depending on the specific learning experience. In line with modality-specific theories of conceptual memory, our study suggests that conceptual representations are established by the learning-based formation of cell assemblies in sensory-motor areas.