Skip to main content
SpringerLink
Log in

Composition of brain oscillations and their functions in the maintenance of auditory, visual and audio–visual speech percepts: an exploratory study

  • Research Report
  • Published:
Cognitive Processing Aims and scope Submit manuscript

Abstract

In the present exploratory study based on 7 subjects, we examined the composition of magnetoencephalographic (MEG) brain oscillations induced by the presentation of an auditory, visual, and audio–visual stimulus (a talking face) using an oddball paradigm. The composition of brain oscillations were assessed here by analyzing the probability-classification of short-term MEG spectral patterns. The probability index for particular brain oscillations being elicited was dependent on the type and the modality of the sensory percept. The maintenance of the integrated audio–visual percept was accompanied by the unique composition of distributed brain oscillations typical of auditory and visual modality, and the contribution of brain oscillations characteristic for visual modality was dominant. Oscillations around 20 Hz were characteristic for the maintenance of integrated audio–visual percept. Identifying the actual composition of brain oscillations allowed us (1) to distinguish two subjectively/consciously identical mental percepts, and (2) to characterize the types of brain functions involved in the maintenance of the multi-sensory percept.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5

References

  • Alsius A, Navarra J, Campbell R, Soto-Faraco S (2005) Audiovisual integration of speech falters under high attention demands. Curr Biol 15(9):839–843

    Article  PubMed  CAS  Google Scholar 

  • Arroyo S, Lesser RP, Gordon B, Uematus S, Jackson D, Webber R (1993) Functional significance of the mu rhythm of human cortex: an electrophysiological study with subdural electrodes. Electroencephalogr Clin Neurophysiol 87:76–87

    Article  PubMed  CAS  Google Scholar 

  • Basar E (1999) Brain function and oscillations. Vol. II. Integrative brain function. Neurophysiology and cognitive processes. Springer, Berlin

    Google Scholar 

  • Basar E, Basar-Eroglu C, Karakas S, Schurmann M (2001a) Gamma, alpha, delta, and theta oscillations govern cognitive processes. Int J Psychophysiol 39:241–248

    Article  PubMed  CAS  Google Scholar 

  • Basar E, Schurmann M, Sakowitz O (2001b) The selectively distributed theta system: functions. Int J Psychophysiol 39:197–212

    Article  PubMed  CAS  Google Scholar 

  • Basar E, Schurmann M, Demiralp T, Basar-Eroglu C, Ademoglu A (2001c) Event-related oscillations are ‘real brain responses’—wavelet analysis and new strategies. Int J Psychophysiol 39:91–127

    Article  PubMed  CAS  Google Scholar 

  • Basar E, Özgören M, Karakas S, Basar-Eroglu C (2004) Super-synergy in the brain: the grandmother percept is manifested by multiple oscillations. Int J Bifurcat Chaos 14:453–491

    Article  Google Scholar 

  • Barry RJ, Rushby JA, Janette L, Smith JL, Clarke AR, Rodney J, Croft RJ (2006) Dynamics of narrow-band EEG phase effects in the passive auditory oddball task. Eur J Neurosci 24:291

    Article  PubMed  Google Scholar 

  • Bullock TH (1997) Signals and signs in the nervous system: the dynamic anatomy of electrical activity. Proc Natl Acad Sci U S A 94:1–6

    Article  PubMed  CAS  Google Scholar 

  • Calvert GA (2001) Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cereb Cortex 11:1110–1123

    Article  PubMed  CAS  Google Scholar 

  • Calvert GA, Thesen T (2004) Multisensory integration: methodological approaches and emerging principles in the human brain. J Physiol (Paris) 98:191–205

    Article  Google Scholar 

  • Colin C, Radeau M, Soquet A, Demolin D, Colin F, Deltenre P (2002) Mismatch negativity evoked by the McGurk–MacDonald effect: a phonetic representation within short-term memory. Clin Neurophysiol 113:495–506

    Article  PubMed  CAS  Google Scholar 

  • Courtney SM, Ungerleider LG, Keil K, Haxby JV (1997) Transient and sustained activity in a distributed neural system for human working memory. Nature 386:608–611

    Article  PubMed  CAS  Google Scholar 

  • Doppelmayr M, Klimesch W, Schwaiger J, Auinger P, Winkler T (1998) Theta synchronization in the human EEG and episodic retrieval. Neurosci Lett 257:41–44

    Article  PubMed  CAS  Google Scholar 

  • Duffy F, Hughes JR, Miranda F, Bernard P, Cook P (1994) Status of quantitative EEG (QEEG) in clinical practice. Clin Electroencephalogr 25:VI–XXII

    PubMed  CAS  Google Scholar 

  • Ernst MO, Bulthoff HH (2004) Merging the senses into a robust percept. Trends Cogn Sci 8:162–169

    Article  PubMed  Google Scholar 

  • Felleman DJ, Van Essen DC (1991) Distributed hierarchical processing in the primate cerebral cortex. Cereb Cortex 1:1–47

    Article  PubMed  CAS  Google Scholar 

  • Fingelkurts AnA, Fingelkurts AlA (2004) Making complexity simpler: multivariability and metastability in the brain. Int J Neurosci 114:843–862

    Article  Google Scholar 

  • Fingelkurts AnA, Fingelkurts AlA (2005) Mapping of the brain operational architectonics. In: Chen FJ (ed) Focus on brain mapping research. Nova Science Publ, New York, pp 59–98

    Google Scholar 

  • Fingelkurts AnA, Fingelkurts AlA (2006) Timing in cognition and EEG brain dynamics: discreteness versus continuity. Cogn Process 7:135–162

    Article  Google Scholar 

  • Fingelkurts AlA, Fingelkurts AnA, Krause CM, Sams M (2002) Probability interrelations between pre-/post-stimulus intervals and ERD/ERS during a memory task. Clin Neurophysiol 113:826–843

    Article  Google Scholar 

  • Fingelkurts AlA, Fingelkurts AnA, Kaplan AYa (2003a) The regularities of the discrete nature of multi-variability of EEG spectral patterns. Int J Psychophysiol 47:23–41

    Article  Google Scholar 

  • Fingelkurts AlA, Fingelkurts AnA, Krause CM, Kaplan AYa (2003b) Systematic rules underlying spectral pattern variability: experimental results and a review of the evidences. Int J Neurosci 113:1455–1481

    Google Scholar 

  • Fingelkurts AnA, Fingelkurts AlA, Krause CM, Möttönen R, Sams M (2003c) Cortical operational synchrony during audio–visual speech integration. Brain Lang 85:297–312

    Article  Google Scholar 

  • Fingelkurts AlA, Fingelkurts AnA, Kaplan AYa (2006a) Stability, reliability and consistency of the compositions of brain oscillations. Int J Psychophysiol 59(2):116–126

    Article  Google Scholar 

  • Fingelkurts AlA, Fingelkurts AnA, Kaplan AYa (2006b) Interictal EEG as a physiological adaptation. Part I: composition of brain oscillations in interictal EEG. Clin Neurophysiol 117:208–222

    Article  Google Scholar 

  • Fingelkurts AA, Fingelkurts AA, Kivisaari R, Autti T, Borisov S, Puuskari V, Jokela O, Kähkönen S (2006c) Reorganization of the composition of brain oscillations and their temporal characteristics in opioid dependent patients. Prog Neuropsychopharmacol Biol Psychiatry 30(8):1453–1465

    Article  PubMed  CAS  Google Scholar 

  • Friederici AD (2002) Towards a neural basis of auditory sentence processing. Trends Cogn Sci 6:78–84

    Article  PubMed  Google Scholar 

  • Funahashi S, Bruce CJ, Goldman-Rakic PS (1989) Mnemonic coding of visual space in the monkey’s dorsolateral prefrontal cortex. J Neurophysiol 61:331–349

    PubMed  CAS  Google Scholar 

  • Fuster JM, Alexander GE (1973) Firing changes in cells of the nucleus medialis dorsalis associated with delayed response behavior. Brain Res 61:79–91

    Article  PubMed  CAS  Google Scholar 

  • Gazzaley A, Rissman J, D’Esposito M (2004) Functional connectivity during working memory maintenance. Cogn Affect Behav Neurosci 4(4):580–599

    Article  PubMed  Google Scholar 

  • Grossberg S (2000) The complementary brain: unifying brain dynamics and modularity. Trends Cogn Sci 4:233–246

    Article  PubMed  Google Scholar 

  • Gundel A, Wilson GF (1992) Topographical changes in the ongoing EEG related to the difficulty of mental tasks. Brain Topogr 5:17–25

    Article  PubMed  CAS  Google Scholar 

  • Hari R, Salenius S (1999) Rhythmical corticomotor communication. Neuroreport 10:R1–R10

    Article  PubMed  CAS  Google Scholar 

  • Kanwisher N, Wojciulik E (2000) Visual attention: insights from brain imaging. Nat Rev Neurosci 1:91–100

    Article  PubMed  CAS  Google Scholar 

  • Kaplan AYa (1998) Nonstationary EEG: methodological and experimental analysis. Uspehi Physiologicheskih Nayk (Success Physiol Sci) 29(3):35–5 (in Russian)

  • Kaplan AYa, Fingelkurts AlA, Fingelkurts AnA, Grin’ EU, Ermolaev VA (1999) Adaptive classification of dynamic spectral patterns of human EEG. J VND (J High Nerve Act) 49:416–426 (In Russian)

    Google Scholar 

  • Karakas S, Erzengin O, Basar E (2000) A new strategy involving multiple cognitive paradigms demonstrates that ERP components are determined by the superposition of oscillatory responses. Clin Neurophysiol 111:1719–1732

    Article  PubMed  CAS  Google Scholar 

  • King AJ, Calvert GA (2001) Multisensory integration: perceptual grouping by eye and ear. Curr Biol 11:R322-R325

    Article  PubMed  CAS  Google Scholar 

  • Klimesch W (1996) Memory processes, brain oscillations and EEG synchronization. Int J Psychophysiol 24:61–100

    Article  PubMed  CAS  Google Scholar 

  • Klimesch W (1999) Event-related band power changes and memory performance. Event-related desynchronization and related oscillatory phenomena of the brain. In: Pfurtscheller G, Lopez da Silva FH (eds) Handbook of electroencephalography and clinical neurophysiology, revised edn, vol 6. Elsevier, Amsterdam, pp 151–78

  • Knudsen EI, Brainard MS (1995) Creating a unified representation of visual and auditory space in the brain. Annu Rev Neurosci 18:19–43

    Article  PubMed  CAS  Google Scholar 

  • Krause CM, Möttönen R, Jensen O, Lampinen J, Sams M (2001) Brain oscillations during audiovisual speech perception–a MEG study. Paper was presented at the Human Brain Mapping Conference, June 10–4, England

  • Kutas M, Hillyard SA (1980) Reading senseless sentences: brain potentials reflect semantic incongruity. Science 207:203–205

    Article  PubMed  CAS  Google Scholar 

  • Large M-E, Aldcroft A, Vilis T (2005) Perceptual continuity and the emergence of perceptual persistence in the ventral visual pathway. J Neurophysiol 93:3453–3462

    Article  PubMed  Google Scholar 

  • Laskaris NA, Ioannides AA (2001) Exploratory data analysis of evoked response single trials based on minimal spanning tree. Clin Neurophysiol 112:698–712

    Article  PubMed  CAS  Google Scholar 

  • Lebib R, Papo D, Douiri A, de Bode S, Gillon Dowens M, Baudonnière P-M (2004) Modulations of ‘late’ event-related brain potentials in humans by dynamic audiovisual speech stimuli. Neurosci Lett 372(1–2):74–79

    Article  PubMed  CAS  Google Scholar 

  • Lehmann D (1989) From mapping to the analysis and interpretation of the EEG/EP maps. In: Maurer K (eds) Topographic brain mapping of EEG and evoked potentials. Springer, Heidelberg, pp 53–75

    Google Scholar 

  • Levy WJ (1987) Effect of epoch length on power spectrum analysis of the EEG. Anesthesiology 66(4):489–495

    Article  PubMed  CAS  Google Scholar 

  • Libet B (1993) The neural time factor in conscious and unconscious events. Experimental and theoretical studies of consciousness (CIBA Foundation Symposium 174). Wiley, Chichester, pp 123–46

  • Logan GD (1992) Attention and preattention in theories of automaticity. Am J Psychol 105:317–339

    Article  PubMed  CAS  Google Scholar 

  • Lopes da Silva FH (1991) Neural mechanisms underlying brain waves: from neural membranes to networks. Electroencephalogr Clin Neurophysiol 79:81–93

    Article  PubMed  CAS  Google Scholar 

  • Lopes da Silva FH, Vos JE, Mooibroek J, Van Rotterdam A (1980) Relative contributions of intracortical and thalamo-cortical processes in the generation of alpha rhythms, revealed by partial coherence analysis. Electroencephalogr Clin Neurophysiol 50:449–456

    Article  PubMed  CAS  Google Scholar 

  • McCollough AW, Machizawa MG, Vogel EK (2007) Electrophysiological measures of maintaining representations in visual working memory. Cortex 43:77–94

    Article  PubMed  Google Scholar 

  • McEvoy LK, Smith ME, Gevins A (2000) Test–retest reliability of cognitive EEG. Clin Neurophysiol 111:457–463

    Article  PubMed  CAS  Google Scholar 

  • McGurk H, MacDonald J (1976) Hearing lips and seeing voices. Nature 264:746–748

    Article  PubMed  CAS  Google Scholar 

  • Mesulam M (1998) From sensation to cognition. Brain 121:1013–1052

    Article  PubMed  Google Scholar 

  • Möttönen R, Krause CM, Tiipana K, Sams M (2002) Processing of changes in visual speech in the human auditory cortex. Brain Res Cogn Brain Res 13:417–425

    Article  PubMed  Google Scholar 

  • Mukamel R, Harel M, Hendler T, Malach R (2004) Enhanced temporal non-linearities in human object-related occipito-temporal cortex. Cerebr Cortex 14:575–585

    Article  Google Scholar 

  • Muller UA (1993) Statistics of variables observed over overlapping intervals. Working paper from Olsen and Associates No 1993-6-8. November 30, p 10. File URL: http://www.olsen.ch/research/403_intervalOverlap.pdf

  • Näätänen R, Winkler I (1999) The concept of auditory stimulus representation in cognitive neuroscience. Psychol Bull 125:826–859

    Article  PubMed  Google Scholar 

  • Neuper C, Pfurtscheller G (1999) Motor imagery and ERD. Event-related desynchronization and related oscillatory phenomena of the brain. In: Pfurtscheller G, Lopes da Silva FH (eds). Handbook of electroencephalography and clinical neurophysiology, Vol. 6, revised ed. Elsevier, Amsterdam, pp 303–26

  • Nuwer MR, Comi C, Emerson R, Fuglsang-Frederiksen A, Guerit JM, Hinrichs H, Ikeda A, Luccas FJ, Rappelsberger P (1998) IFCN standards for digital recording of clinical EEG. Electroencephalogr Clin Neurophysiol 106:259–261

    Article  PubMed  CAS  Google Scholar 

  • O’Hare JJ (1991) Perceptual integration. J Wash Acad Sci 81:44–59

    Google Scholar 

  • Perneger TV (1998) What is wrong with Bonferroni adjustments. Br Med J 136:1236–1238

    Google Scholar 

  • Pfurtscheller G, Klimesch W (1990) Topographical display and interpretation of event-related desynchronization during visual–verbal task. Brain Topogr 3:85–93

    Article  PubMed  CAS  Google Scholar 

  • Pfurtscheller G, Stancak JrA, Neuper C (1996) Post-movement beta synchronization: a correlate of an idling motor area? Electroencephalogr Clin Neurophysiol 98:281–293

    Article  PubMed  CAS  Google Scholar 

  • Pöppel E (1994) Temporal mechanisms in perception. Int Rev Neurobiol 37:185–202

    Article  PubMed  Google Scholar 

  • Pöppel E (2004) Lost in time: historical foundations and the 3-second-window of temporal integration. Acta Neurobiol Exp 64:295–301

    Google Scholar 

  • Pulvermuller F, Keil A, Thomas E (1999) High-frequency brain activity: perception or active memory? Trends Cogn Sci 3:250–252

    Article  PubMed  Google Scholar 

  • Raij T (1999) Patterns of brain activity during visual imagery of letters. J Cogn Neurosci 11:282–299

    Article  PubMed  CAS  Google Scholar 

  • Riley WJ (2003) Techniques for frequency stability analysis. IEEE International Frequency Control Symposium, Tampa, FL, 4 May

  • Rosenblum LD, Saldana HM (1996) An audiovisual test of kinematic primitives for visual speech perception. J Exp Psychol Hum Percept Perform 22:318–331

    Article  PubMed  CAS  Google Scholar 

  • Rothman KJ (1990) No adjustments are needed for multiple comparisons. Epidemiology 1:43–46

    Article  PubMed  CAS  Google Scholar 

  • Sakowitz O, Schurmann M, Basar E (2000) Oscillatory frontal theta responses are increased upon bisensory stimulation. Clin Neurophysiol 111:884–893

    Article  PubMed  CAS  Google Scholar 

  • Sakowitz O, Quiroga R, Schurmann M, Basar E (2001) Bisensory stimulation increases gamma-responses over multiple cortical regions. Brain Res Cogn Brain Res 11:267–279

    Article  PubMed  CAS  Google Scholar 

  • Salmelin R, Hämälinen M, Kajola M, Hari R (1995) Functional segregation of movement-related rhythmic activity in the human brain. Neuroimage 2:237–243

    Article  PubMed  CAS  Google Scholar 

  • Sams M, Aulanko R, Hämäläinen M, Hari R, Lounasmaa OV, Lu ST, Simola J (1991) Seeing speech: visual information from lip movements modifies activity in the human auditory cortex. Neurosci Lett 127:141–145

    Article  PubMed  CAS  Google Scholar 

  • Schacter DL (1977) EEG theta waves and psychological phenomena: a review and analysis. Biol Psychol 5:47–82

    Article  PubMed  CAS  Google Scholar 

  • Singer W (1994) Putative functions of temporal correlations in neocortical processing. In: Koch C, Davis JL (eds). Large-scale neuronal theories of the brain, MIT Press, Cambridge, pp 201–237

    Google Scholar 

  • Soto-Faraco S, Navarra J, Alsius A (2004) Assessing automaticity in audiovisual speech integration: evidence from the speeded classification task. Cognition 92:B13–B23

    Article  PubMed  Google Scholar 

  • Stampfer HG, Basar E (1985) Does frequency analysis lead to better understanding of human event related potentials? Int J Neurosci 26:181–196

    Article  PubMed  CAS  Google Scholar 

  • Steriade M, Gloor P, Llinas RR, Lopes da Silva FH, Mesulam MM (1990) Basic mechanisms of cerebral rhythmic activities. Electroencephalogr Clin Neurophysiol 76:481–508

    Article  PubMed  CAS  Google Scholar 

  • Taylor JL, McCloskey DI (1990) Triggering of preprogrammed movements as reactions to masked stimuli. J Neurophysiol 63:439–446

    PubMed  CAS  Google Scholar 

  • Teder-Sälejårvi WA, McDonald JJ, Russo FDi, Hillyard SA (2002) An analysis of audio–visual crossmodal integration by means of event-related potential (ERP) recordings. Cogn Brain Res 14:106–114

    Article  Google Scholar 

  • Thatcher RW (2001) Normative EEG databases and EEG biofeedback. J Neurother 2:8–39

    Article  Google Scholar 

  • Tiippana K, Andersen TS, Sams M (2004) Visual attention modulates audiovisual speech perception. Eur J Cogn Psychol 16:457–472

    Article  Google Scholar 

  • Treisman AM, Sato S (1990) Conjunction search revisited. J Exp Psychol Hum Percept Perform 16:459–478

    Article  PubMed  CAS  Google Scholar 

  • Tononi G, Edelman GM, Sporns O (1998) Complexity and coherency: integrating information in the brain. Trends Cogn Sci 2:474–484

    Article  Google Scholar 

  • Zarahn E, Aguirre GK, D’Esposito M (1999) Temporal isolation of the neural correlates of spatial mnemonic processing with fMRI. Cogn Brain Res 7:255–268

    Article  CAS  Google Scholar 

  • Yordanova J, Kolev V, Rosso OA, Schurmann M, Sakowitz OW, Özgören M, Basar E (2002) Wavelet entropy analysis of event-related potentials indicates modality-independent theta dominance. J Neurosci Methods 117:99–109

    Article  PubMed  Google Scholar 

  • Usher M, Niebur E (1997) Active neural representations: neurophysiological data and conceptual implications. In: Peschl M, Riegler A (eds) New trends in cognitive science. University of Vienna, Austria

    Google Scholar 

  • Varela F (1995) Resonant cell assemblies: a new approach to cognitive functions and neuronal synchrony. Biol Res 28:81–95

    PubMed  CAS  Google Scholar 

Download references

Acknowledgments

The authors thank Riikka Möttönen for the MEG registration. MEG recordings were made in the Brain Research Unit at the Low Temperature Laboratory at the Helsinki University of Technology. The authors wish to thank Carlos Neves, computer science specialist, for the software development and technical support. Special thanks to Simon Johnson for skilful language editing. AAFs were supported by Research Fellowship from CIMO, Finland. Preparation of this article was supported by BM-SCIENCE. The authors state that experiment of this study complied with the current laws of Finland.

Author information

Authors and Affiliations

  1. BM-SCIENCE–Brain and Mind Technologies Research Centre, P.O. Box 77, 02601, Espoo, Finland

    Alexander A. Fingelkurts & Andrew A. Fingelkurts

  2. Department of Psychology, Cognitive Science Unit, University of Helsinki, PB 9, 00014, Helsinki, Finland

    Christina M. Krause

Authors
  1. Alexander A. Fingelkurts
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrew A. Fingelkurts
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Christina M. Krause
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Alexander A. Fingelkurts.

Additional information

Brain Oscillations and Auditory, Visual and Audio–Visual Speech Percepts.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Fingelkurts, A.A., Fingelkurts, A.A. & Krause, C.M. Composition of brain oscillations and their functions in the maintenance of auditory, visual and audio–visual speech percepts: an exploratory study. Cogn Process 8, 183–199 (2007). https://doi.org/10.1007/s10339-007-0175-x

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10339-007-0175-x

Keywords

Search

Navigation