A network that uses few active neurones to code visual input predicts the diverse shapes of cortical receptive fields

Abstract

Computational models of primary visual cortex have demonstrated that principles of efficient coding and neuronal sparseness can explain the emergence of neurones with localised oriented receptive fields. Yet, existing models have failed to predict the diverse shapes of receptive fields that occur in nature. The existing models used a particular “soft” form of sparseness that limits average neuronal activity. Here we study models of efficient coding in a broader context by comparing soft and “hard” forms of neuronal sparseness.

As a result of our analyses, we propose a novel network model for visual cortex. The model forms efficient visual representations in which the number of active neurones, rather than mean neuronal activity, is limited. This form of hard sparseness also economises cortical resources like synaptic memory and metabolic energy. Furthermore, our model accurately predicts the distribution of receptive field shapes found in the primary visual cortex of cat and monkey.

Appendices

Appendix A: Derivation of the energy functions for hard-sparseness

Inserting $b_i = a_i y_i, f(x) = ||x||_{L_0}$, and the definitions $c=\Psi x, C = \Psi \Psi^T$ and $P^y = \mbox{diag}(y)$, one can rewrite Eq. (2) (up to the constant $\left\langle x,x\right\rangle\!/2$) as

$$ E(y,a)=\frac{1}{2}\left\langle a,P^{y}CP^{y}a - 2P^{y}c\right\rangle + \theta \,\mbox{Tr}(P^y)\label{energy1} $$

(12)

where $\mbox{Tr}(P^y) = \sum_i y_i = ||y||$. Equation (12) requires interleaved optimisation in both variable sets a and y. For fixed y the optimal analogue coefficients are $a^* = \mbox{argmin}_a ||P^y C P^y a - P^y c||$. The solution can be computed as Eq. (4) using the pseudoinverse. If one inserts Eq. (4) in Eq. (12), the resulting energy function is

$$ E(y) = - \frac{1}{2} \left\langle c, [P^{y}CP^{y}]^{+}c\right\rangle + \theta ||y||\label{fullmodenergydup} $$

(13)

which solely depends on the binary variables since the coefficients are optimised implicitly.

Using the identities for the pseudoinverse: $A^+ = [A^T A]^+ A^T$ and $[A^T]^+ = (A^+)^T$ (see pseudoinverse in Wikipedia), the inner product in Eq. (13) can be written

$$\displaylines{ \left\langle c, [P^{y}CP^{y}]^{+} c\right\rangle = \left\langle x, (P^y \Psi)^T [P^{y}\Psi (P^{y}\Psi)^T]^{+} P^y \Psi x\right\rangle\cr = \left\langle x, [P^y \Psi]^+ P^y \Psi \;x\right\rangle\label{projectors} }$$

(14)

The operator in the inner product on the RHS of Eq. (14) is a projection operator:

$$ [P^y \Psi]^+ P^y \Psi =: P_{\Psi}^y = \left(P_{\Psi}^y\right)^2\label{projector} $$

(15)

which projects into the subspace spanned by the receptive fields of the active units $\{i: y_i=1\}$. Equation (14) and definition 15 yield Eq. (5).

Another way to rewrite Eq. (13) is to insert $P^y C P^y = [P^y (C-1\!\!1) P^y +1\!\!1] P^y =: C^y P^y$. $C^y$ is a full rank matrix if the selected set of basis functions is linearly independent. This is guaranteed in the complete case and very likely to be fulfilled for sparse selections in overcomplete bases. Thus, for sparse y vectors we can replace the pseudoinverse in Eq. (13) by the ordinary inverse and use the power series expansion: $[C^y]^{-1} = 1\!\!1 - P^y (C-1\!\!1) P^y + [P^y (C-1\!\!1) P^y]^2 - \cdots$ Using the expansion up to the first order yields the approximations for Eqs. (5) and (4), respectively

$$ E(y) \simeq \frac{1}{2} \left\langle c, P^y (C-2 \!\cdot\!1\!\!1) P^y c\right\rangle + \theta \,\mbox{Tr}(P^y)\label{recenergyxfo} $$

(16)

$$ a^* \simeq [1\!\!1 - P^y (C-1\!\!1) P^y] c\label{approxcoeffs} $$

(17)

With the definition $T_{ij}:= - c_i C_{ij} c_j + 2 \delta_{ij} c_i^2$, Eq. (7) follows from Eq. (16).

Appendix B: Fitting of receptive fields

In the sparse regime, each basis function can be well fitted with a two-dimensional Gabor function in the image coordinates $u, v$: $h(u',v') = A\; \exp [-( \frac{u'}{\sqrt 2 \sigma_{u'}})^2 - ( \frac{v'}{\sqrt 2 \sigma_{v'}})^2] \cos ( 2 \pi f u' + \Phi )$, where $u'$ and $v'$ are translated and rotated image coordinates, $\sigma_{u'}$ and $\sigma_{v'}$ represent the widths of the Gaussian envelope, and f and Φ are the spatial frequency and phase of the sinoidal grating. Notation in Fig. 6: $\mbox{width}: = \sigma_{u} f$ and $\mbox{length}: = \sigma_{v} f$. To measure asymmetry of a Gabor function we split h along the $v'$ axis into $h_-$ and $h_+$ and use: $\mbox{Asym} := |\int h_+ ds - \int h_- ds |/ \int |h| ds$.