1932

Abstract

A major challenge for systems neuroscience is to break the neural code. Computational algorithms for encoding information into neural activity and extracting information from measured activity afford understanding of how percepts, memories, thought, and knowledge are represented in patterns of brain activity. The past decade and a half has seen significant advances in the development of methods for decoding human neural activity, such as multivariate pattern classification, representational similarity analysis, hyperalignment, and stimulus-model-based encoding and decoding. This article reviews these advances and integrates neural decoding methods into a common framework organized around the concept of high-dimensional representational spaces.

Loading

Article metrics loading...

/content/journals/10.1146/annurev-neuro-062012-170325
2014-07-08
2024-03-28
Loading full text...

Full text loading...

/deliver/fulltext/neuro/37/1/annurev-neuro-062012-170325.html?itemId=/content/journals/10.1146/annurev-neuro-062012-170325&mimeType=html&fmt=ahah

Literature Cited

  1. Abdi H, Dunlop JP, Williams LJ. 2009. How to compute reliability estimates and display confidence and tolerance intervals for pattern classifiers using the Bootstrap and 3-way multidimensional scaling (DISTATIS). NeuroImage 45:89–95 [Google Scholar]
  2. Abdi H, Williams LJ, Conolly AC, Gobbini MI, Dunlop JP, Haxby JV. 2012a. Multiple Subject Barycentric Discriminant Analysis (MUSUBADA): how to assign scans to categories without using spatial normalization. Comp. Math. Methods Med. 2012:634165 [Google Scholar]
  3. Abdi H, Williams LJ, Valentin D, Bennani-Dosse M. 2012b. STATIS and DISTATIS: optimum multi-table principal component analysis and three way metric multidimensional scaling. Wiley Interdiscip. Rev. Comput. Stat. 4:124–67 [Google Scholar]
  4. Aflalo TN, Graziano MSA. 2006. Possible origins of the complex topographic organization of motor cortex: reduction of a multidimensional space onto a two-dimensional array. J. Neurosci. 26:6288–97 [Google Scholar]
  5. Aflalo TN, Graziano MSA. 2011. Organization of the macaque extrastriate cortex re-examined using the principle of spatial continuity of function. J. Neurophysiol. 105:305–20 [Google Scholar]
  6. Brants M, Baeck A, Wagemans J, Op de Beeck H. 2011. Multiple scales of organization for object selectivity in ventral visual cortex. NeuroImage 56:1372–81 [Google Scholar]
  7. Carlin JD, Calder AJ, Kriegeskorte N, Nili H, Rowe JB. 2011. A head view-invariant representation of gaze direction in anterior superior temporal cortex. Curr. Biol. 21:1817–21 [Google Scholar]
  8. Carlson TA, Hogendoorn H, Kanai R, Mesik J, Turret J. 2011. High temporal resolution decoding of object position and category. J. Vis. 11:1–17 [Google Scholar]
  9. Carlson TA, Schrater P, He S. 2003. Patterns of activity in the categorical representations of objects. J. Cogn. Neurosci. 15:704–17 [Google Scholar]
  10. Casey M, Thompson J, Kang O, Raizada R, Wheatley T. 2012. Population codes representing musical timbre for high-level fMRI categorization of music genres. Machine Learning and Interpretation in Neuroimaging Ser 7263 G Langs, I Rish, M Grosse-Wentrup, B Murphy 36–41 Berlin/Heidelberg: Springer-Verlag [Google Scholar]
  11. Chen Y, Namburi P, Elliott LT, Heinzle J, Soon CS. et al. 2011. Cortical surface-based searchlight decoding. NeuroImage 56:582–92 [Google Scholar]
  12. Connolly AC, Gobbini MI, Haxby JV. 2012a. Three virtues of similarity-based multi-voxel pattern analysis: an example from the human object vision pathway. Understanding Visual Population Codes (UVPC): Toward A Common Multivariate Framework for Cell Recording and Functional Imaging N Kriegeskorte, G Kreiman 335–55 Cambridge, MA: MIT Press [Google Scholar]
  13. Connolly AC, Guntupalli JS, Gors J, Hanke M, Halchenko YO. et al. 2012b. The representation of biological classes in the human brain. J. Neurosci. 32:2608–18 [Google Scholar]
  14. Conroy BR, Singer BD, Guntupalli JS, Ramadge PJ, Haxby JV. 2013. Inter-subject alignment of human cortical anatomy using functional connectivity. NeuroImage 81:400–11 [Google Scholar]
  15. Cortes C, Vapnik V. 1995. Support-vector networks. Mach. Learn. 20:273–97 [Google Scholar]
  16. Cox DD, Savoy RL. 2003. Functional magnetic resonance imaging (fMRI) “brain reading”: detecting and classifying distributed patterns of fMRI activity in human visual cortex. NeuroImage 19:261–70 [Google Scholar]
  17. DiCarlo JJ, Cox DD. 2007. Untangling invariant object recognition. Trends Cogn. Sci. 11:333–41 [Google Scholar]
  18. Durbin R, Mitchison G. 1990. A dimension reduction framework for understanding cortical maps. Nature 343:644–47 [Google Scholar]
  19. Edelman S, Grill-Spector K, Kushnir T, Malach R. 1998. Toward direct visualization of the internal shape space by fMRI. Psychobiology 26:309–21 [Google Scholar]
  20. Formisano I, De Martino F, Bonte M, Goebel R. 2008. “Who” is saying “what”? Brain-based decoding of human voice and speech. Science 322:970–73 [Google Scholar]
  21. Freiwald WA, Tsao DY. 2010. Functional compartmentalization and viewpoint generalization within the macaque face-processing system. Science 330:845–51 [Google Scholar]
  22. Friston K, Kiebel S. 2009. Predictive coding under the free-energy principle. Phil. Trans. R. Soc. B. 364:1211–21 [Google Scholar]
  23. Graziano MSA, Aflalo TN. 2007a. Mapping behavioral repertoire onto the cortex. Neuron 56:239–51 [Google Scholar]
  24. Graziano MSA, Aflalo TN. 2007b. Rethinking cortical organization: moving away from discrete areas arranged in hierarchies. Neuroscientist 13:138–47 [Google Scholar]
  25. Hanke M, Halchenko YO, Sederberg PB, Hanson SJ, Haxby JV, Pollman S. 2009. PyMVPA: A Python toolbox for multivariate pattern analysis of fMRI data. Neuroinformatics 7:37–53An integrated software system based on Python for performing neural decoding. [Google Scholar]
  26. Hanson SJ, Toshihiko M, Haxby JV. 2004. Combinatorial codes in ventral temporal lobe for object recognition: Haxby (2001) revisited: is there a “face” area?. NeuroImage 23:156–67 [Google Scholar]
  27. Harrison SA, Tong F. 2009. Decoding reveals the contents of visual working memory in early visual areas. Nature 458:632–35 [Google Scholar]
  28. Haxby JV. 2012. Multivariate pattern analysis of fMRI: the early beginnings. NeuroImage 62:852–55 [Google Scholar]
  29. Haxby JV, Gobbini MI, Furey ML, Ishai A, Schouten JL, Pietrini P. 2001. Distributed and overlapping representations of faces and objects in ventral temporal cortex. Science 293:2425–30Initial paper on multivariate pattern classification of fMRI. [Google Scholar]
  30. Haxby JV, Guntupalli JS, Connolly AC, Halchenko YO, Conroy BR. et al. 2011. A common, high-dimensional model of the representational space in human ventral temporal cortex. Neuron 72:404–16Introduces hyperalignment for building a common high-dimensional model of a neural representational space. [Google Scholar]
  31. Haynes JD, Rees G. 2005. Predicting the orientation of invisible stimuli from activity in human primary visual cortex. Nat. Neurosci. 8:686–91 [Google Scholar]
  32. Haynes JD, Rees G. 2006. Decoding mental states from brain activity in humans. Nat. Rev. Neurosci. 7:523–34 [Google Scholar]
  33. Haynes JD, Sakai K, Rees G, Gilbert S, Frith C, Passingham RE. 2007. Reading hidden intentions in the human brain. Curr. Biol. 17:323–28 [Google Scholar]
  34. Horikawa T, Tamaki M, Miyawaki Y, Kamitani Y. 2013. Neural decoding of visual imagery during sleep. Science 340:639–42 [Google Scholar]
  35. Hung CP, Kreiman G, Poggio T, DiCarlo JJ. 2005. Fast readout of object identity from macaque inferior temporal cortex. Science 310:863–66 [Google Scholar]
  36. Kamitani Y, Tong F. 2005. Decoding the visual and subjective contents of the human brain. Nat. Neurosci. 8:679–85First paper to show that MVPA can decode an early visual feature, namely edge orientation. [Google Scholar]
  37. Kanwisher N. 2010. Functional specificity in the human brain: a window into the functional architecture of the mind. Proc. Natl. Acad. Sci. USA 107:11163–70 [Google Scholar]
  38. Kanwisher N, McDermott J, Chun MM. 1997. The fusiform face area: a module in human extrastriate cortex specialized for face perception. J. Neurosci. 17:4302–11 [Google Scholar]
  39. Kay KN, Naselaris T, Prenger RJ, Gallant JL. 2008. Identifying natural images from human brain activity. Nature 452:352–55Introduced stimulus-model-based encoding and decoding of natural images. [Google Scholar]
  40. Kiani R, Esteky H, Mirpour K, Tanaka K. 2007. Object category structure in response patterns of neuronal population in monkey inferior temporal cortex. J. Neurophysiol. 97:4296–309 [Google Scholar]
  41. Kohler PJ, Fogelson SV, Reavis EA, Meng M, Guntupalli JS. et al. 2013. Pattern classification precedes region-average hemodynamic response in early visual cortex. NeuroImage 78:249–60 [Google Scholar]
  42. Kohonen T. 1982. Self-organizing formation of topologically correct feature maps. Biol. Cybern. 43:59–69 [Google Scholar]
  43. Kohonen T. 2001. Self-Organizing Maps Berlin: Springer
  44. Kriegeskorte N, Goebel R, Bandettini P. 2006. Information-based functional brain mapping. Proc. Natl. Acad. Sci. USA 103:3863–68 [Google Scholar]
  45. Kriegeskorte N, Kievet RA. 2013. Representational geometry: integrating cognition, computation, and the brain. Trends Cogn. Sci. 17:401–12 [Google Scholar]
  46. Kriegeskorte N, Mur M, Bandettini P. 2008a. Representational similarity analysis—connecting the branches of systems neuroscience. Front. Syst. Neurosci. 2:4Introduces RSA as a common format for the geometry of representational spaces. [Google Scholar]
  47. Kriegeskorte N, Mur M, Ruff DA, Kiani R, Bodurka J. et al. 2008b. Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron 60:1126–41First demonstration that RSA affords comparison of representational spaces across species. [Google Scholar]
  48. Kriegeskorte N, Simmons WK, Bellgowan PSF, Baker CI. 2009. Circular analysis is systems neuroscience: the dangers of double dipping. Nat. Neurosci. 12:535–40 [Google Scholar]
  49. Mahon BZ, Anzellotti S, Schwarzbach J, Zampini M, Caramazza A. 2009. Category-specific organization in the human brain does not require visual experience. Neuron 63:397–405 [Google Scholar]
  50. Martin A. 2007. The representation of object concepts in the brain. Annu. Rev. Psychol 58:25–45 [Google Scholar]
  51. Mitchell TM, Shinkareva SV, Carlson A, Chang K-M, Malave VL. et al. 2008. Predicting human brain activity associated with the meanings of nouns. Science 320:1191–95This paper introduced decoding of words and concepts based on semantic feature models. [Google Scholar]
  52. Miyawaki Y, Uchida H, Yamashita O, Sato M, Morito Y. et al. 2008. Visual image reconstruction from human brain activity using a combination of multiscale local image decoders. Neuron 60:915–29 [Google Scholar]
  53. Naselaris T, Kay KN, Nishimoto S, Gallant JL. 2011. Encoding and decoding in fMRI. NeuroImage 56:400–10Reviews methods for stimulus-model-based encoding and decoding. [Google Scholar]
  54. Naselaris T, Prenger RJ, Kay KN, Oliver M, Gallant JL. 2009. Bayesian reconstruction of natural images from human brain activity. Neuron 63:902–15 [Google Scholar]
  55. Nishimoto S, Vu AT, Naselaris T, Bejamini Y, Yu B, Gallant JL. 2011. Reconstructing visual experience from brain activity evoked by natural movies. Curr. Biol. 21:1641–46 [Google Scholar]
  56. Norman KA, Polyn SM, Detre GJ, Haxby JV. 2006. Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends Cogn. Sci. 10:424–30 [Google Scholar]
  57. Oosterhof NN, Wiestler T, Downing PE, Diedrichsen J. 2011. A comparison of volume-based and surface-based multi-voxel pattern analysis. NeuroImage 56:593–600 [Google Scholar]
  58. Op de Beeck H, Brants M, Baeck A, Wagemans J. 2010. Distributed subordinate specificity for bodies, faces, and buildings in human ventral visual cortex. NeuroImage 49:3414–25 [Google Scholar]
  59. O'Toole AJ, Jiang F, Abdi H, Haxby JV. 2005. Partially distributed representations of objects and faces in ventral temporal cortex. J. Cogn. Neurosci. 17:580–90 [Google Scholar]
  60. O'Toole AJ, Jiang F, Abdi H, Pénard N, Dunlop JP, Parent MA. 2007. Theoretical, statistical, and practical perspectives on pattern-based classification approaches to the analysis of functional neuroimaging data. J. Cogn. Neurosci. 191:735–52 [Google Scholar]
  61. Pasley BN, David SV, Mesgarani N, Flinker A, Shamma SA. et al. 2012. Reconstructing speech from human auditory cortex. PLoS Biol. 10:e1001251 [Google Scholar]
  62. Pereira F, Mitchell T, Botvinick M. 2009. Machine learning classifiers and fMRI: a tutorial overview. NeuroImage 45:Suppl. 1S199–209 [Google Scholar]
  63. Raizada RDS, Connolly AC. 2012. What makes different people's representations alike: neural similarity space solves the problem of across-subject fMRI decoding. J. Cogn. Neurosci. 24:868–77 [Google Scholar]
  64. Sabuncu M, Singer BD, Conroy B, Bryan RE, Ramadge PJ, Haxby JV. 2010. Function-based intersubject alignment of human cortical anatomy. Cereb. Cortex 20:130–40 [Google Scholar]
  65. Schönemann PH. 1966. A generalized solution of the orthogonal procrustes problem. Psychometrika 31:1–10 [Google Scholar]
  66. Schreiner CE. 1995. Order and disorder in auditory cortical maps. Curr. Opin. Neurobiol. 5:489–96 [Google Scholar]
  67. Shinkareva SV, Malave VL, Mason RA, Mitchell TM, Just MA. 2011. Commonality of neural representations of words and pictures. NeuroImage 54:2418–25 [Google Scholar]
  68. Shinkareva SV, Mason RA, Malave VL, Wang W, Mitchell TM, Just MA. 2008. Using fMRI brain activation to identify cognitive states associated with perception of tools and dwellings. PLoS ONE 1:e1394 [Google Scholar]
  69. Soon CS, Brass M, Heinze HJ, Haynes JD. 2008. Unconscious determinants of free decisions in the human brain. Nat. Neurosci. 5:543–45 [Google Scholar]
  70. Staeren N, Renvall H, De Martino F, Goebel R, Formisano E. 2009. Sound categories are represented as distributed patterns in the human auditory cortex. Curr. Biol. 19:498–502 [Google Scholar]
  71. Sudre G, Pomerleau D, Palatucci M, Wehbe L, Fyshe A. et al. 2012. Tracking neural coding of perceptual and semantic features of concrete nouns. NeuroImage 62:451–63 [Google Scholar]
  72. Tong F, Pratte MS. 2012. Decoding patterns of human brain activity. Annu. Rev. Psychol. 63:483–509 [Google Scholar]
  73. Weiner K, Grill-Spector K. 2011. The improbable simplicity of the fusiform face area. Trends Cogn. Sci. 16:251–54 [Google Scholar]
  74. Yamashita O, Sato M-A, Yoshioka T, Tong F, Kamitani Y. 2008. Sparse estimation automatically selects voxels relevant for the decoding of fMRI activity patterns. NeuroImage 42:1414–29 [Google Scholar]
/content/journals/10.1146/annurev-neuro-062012-170325
Loading
/content/journals/10.1146/annurev-neuro-062012-170325
Loading

Data & Media loading...

  • Article Type: Review Article
This is a required field
Please enter a valid email address
Approval was a Success
Invalid data
An Error Occurred
Approval was partially successful, following selected items could not be processed due to error