From What Scale of Representation Does Multivariate Pattern Analysis Decode Information?

Functional magnetic resonance imaging (fMRI) is severely limited in the extent of detail that it can reveal. In practice, most fMRI studies investigate large areas subtending centimeters, such as the primary visual cortex or face-selective regions. Electrophysiological recordings, however, have demonstrated much smaller scales of organization, for example, columns coding only for specific orientations within the primary visual cortex. These orientation columns are substantially smaller than the conventionally available fMRI resolution of 3 × 3 mm, so it was assumed that fMRI would be insensitive to these finely spaced orientation columns.

Nonetheless, by using multivariate analysis techniques, Kamitani and Tong (2005) demonstrated that it is possible to decode orientation information in the human brain even with conventional fMRI resolution. These multivariate techniques combine the information of multiple voxels (Haxby et al., 2001), thereby revealing a sensitivity to different orientations, even though individual voxels provide only a very weak sensitivity to this distinction. This sensitivity was argued to result from an unequal distribution of different orientation columns within each voxel, providing a subtle bias that multivariate techniques can exploit to discriminate between orientations [for review, see Norman et al. (2006) or Haynes and Rees (2006)].

However, given that there are also larger scales of organization within the primary visual cortex, such as those deriving from a preference for radial orientations (Sasaki et al., 2006) or the overrepresentation of cardinal orientations (Furmanski and Engel, 2000), is it plausible to suppose that the demonstrated decoding depends on fine-scale columnar organization rather than these larger structures? A possible role for such larger scales of organization was suggested by the demonstration by Op de Beeck (2010) that the decoding of orientation was unaffected by large-scale (8 mm) smoothing. If orientation decoding really depends on fine-scale variability in the distribution of columns, it is unclear how this signal could still be detected when blurred via smoothing. Swisher et al. (2010) attempted to resolve this question regarding the scale of representation that contributes to a successful orientation classification by testing classification performance with high-resolution fMRI in cats (9.4 T) and humans (7 T) who viewed oriented gratings. In the fMRI data, the authors considered classification performance by applying high- or low-pass spatial filters, enabling them to determine the contribution of different spatial scales to successful multivariate classification analysis.

On one level, the results of Swisher et al. (2010) are clear: when scanning at very high spatial resolution, both human and cat orientation decoding performance appear to depend on a small, millimeter scale of organization. In particular, the authors found that including all spatial scales up to 1.2 mm in cats and 4–5 mm in humans results in a saturation point, beyond which adding even larger spatial scales does not improve multivariate pattern classification [Swisher et al. (2010), their Fig. 1B and Fig. 2B, red lines]. This result therefore rules out the possibility that orientation decoding relies on larger scales of organization in this case and suggests that the critical information for classification performance is included within these millimeter scales of activation. There are, of course, large-scale orientation maps present as well, such as radial bias and differential sensitivity to cardinal versus oblique orientations, but, crucially, the authors show that these organizations only manifested at a centimeter scale and were undetected at the fine scales where successful orientation decoding was achieved. These large-scale maps could be decoded when no fine-scale information was present, but this decoding ability increased as more of this fine-scale information was included [Swisher et al. (2010), their Fig. 1B and Fig. 2B, blue lines].

Thus, Swisher et al. (2010) successfully demonstrated that orientation columns can, in principle, contribute to multivariate orientation decoding. However, as the authors suggest, this evidence should not be taken as a confirmation that decoding necessarily results from columns. To illustrate this point, consider the fact that even at scales substantially larger than orientation columns (greater than a few millimeters), the decoding performance remains significantly above chance. It is possible that there is some information of orientation columns remaining at these scales, which enables successful decoding. But it is also possible that such decoding results from some millimeter-scale organization, based on some local orientation grouping. Indeed, several alternatives have already been described in the literature, including pinwheel organization at ∼1.5–2 mm and a relationship between the ocular dominance columns and the orientation columns (Yacoub et al., 2008). In fact, other scales of organization might also be present, yet unknown at this point. Therefore, understanding the levels of orientation column organization and its strength might be interesting not only for the field of multivariate pattern analysis, but also in a broader context of investigating the organizational principles underlying the primary visual cortex. Potentially, such scales of organization could be observed in spatial autocorrelations of the orientation map data. Consider, for example, supplementary Figure 2 in Yacoub et al. (2008), where such autocorrelation reveals a curious increase at the scale of 1.5 mm (as compared to shuffled maps), presumably indicating some columnar grouping at a few millimeter scale. At different scanning resolution, these millimeter scales might become more important for decoding than the columns themselves.

A comparison study between the present results and the commonly used 3 × 3 mm resolution scanning could address these questions. Although Swisher et al. (2010) found orientation columns to contribute most to the decoding at 1 × 1 mm resolution, this result might not generalize to lower-resolution scans. In particular, at lower resolutions fine spatial frequency signals are greatly diminished, while millimeter-scale organization might become more prominent. Thus, at this lower resolution (typically used in decoding studies), the decoding might actually reflect ways in which orientation columns are organized in groups rather than inhomogeneous biases in columnar placement. Swisher et al. (2010) provide data consistent with this possibility when reanalyzing Kamitani and Tong's (2005) original data (collected at a conventional 3 × 3 mm resolution), in which large amounts of smoothing diminished classification performance only slightly, as if there was a stronger influence of larger-scale maps. A comparison of classification performance of the data collected at a lower resolution and smoothed data of high-resolution data on the same subjects could clarify this issue. If similar levels of organization are used in decoding orientation, both cases should elicit similar classification performance. Such results would be a strong indication that even in conventional fMRI, orientation decoding is closely linked to columnar organization and not millimeter-scale maps. However, until this evidence is obtained, at a conventional resolution orientation decoding can only be regarded as related to any organization from subvoxel-size columns to local groupings of a few millimeters.

Another possibility for addressing this question stems from the observation by Swisher et al. (2010) of a correspondence between the average distance of columns with a similar orientation preference and the saturation point in cats. The saturation point reflects the stage at which adding even larger-scale information stops contributing to increased classification performance, which is achieved at 1.2–1.4 mm for cats [Swisher et al. (2010), their Fig. 1B]. If column-level structures really underpin classification performance, there should be a link between a saturation point and a column size. While for cats this notion is intuitive, as an average orientation column size matched the saturation point, it is less obvious for humans, where the saturation point was ∼4–5 mm (much greater than the average spacing of columns). There are many reasons for such a difference, for example, the increased head motion in humans (compared to anesthetized cats) or different voxel sizes used (0.3125 × 0.3125 mm in cats, and 1 × 1 mm in humans), since noise will be diminished more rapidly when adding together information from smaller voxels, resulting in earlier saturation. On the other hand, the difference could in fact reflect millimeter-scale organization, and not columns, contributing mostly to the decoding. So if the orientation columns were indeed underlying decoding performance in this case, in principle, such a relationship would be reflected in correlations between the observed individual differences in column size (Yacoub et al., 2008) and individual differences in the saturation point.

In summary, the observation that orientation decoding could be achieved by multivariate techniques within the primary visual cortex was originally assumed to reflect sensitivity to random biases in the placement of different orientation columns. However, the existence of larger scales of organization in the primary visual cortex suggested another scale of organization that could underpin successful decoding. Swisher et al. (2010) provide an important advance on this debate by demonstrating that, although such larger scales of organization exist, they do not contribute to classification performance. Challenges lie ahead in trying to dissociate the source of such information between subvoxel scale (columns) and millimeter-scale (local clusters of columns) organization. We have highlighted several steps that could be taken to further our understanding of the scale of organization exploited with conventional fMRI resolutions. In particular, we proposed several ways to test the possibility that there may exist scales of organization smaller than the cardinal and radial bias ruled out by the reviewed paper that might reflect some means by which columns are organized to group together.