Homeostatic synaptic scaling in self-organizing maps

Abstract

Various forms of the self-organizing map (SOM) have been proposed as models of cortical development [Choe Y., Miikkulainen R., (2004). Contour integration and segmentation with self-organized lateral connections. Biological Cybernetics, 90, 75–88; Kohonen T., (2001). Self-organizing maps (3rd ed.). Springer; Sirosh J., Miikkulainen R., (1997). Topographic receptive fields and patterned lateral interaction in a self-organizing model of the primary visual cortex. Neural Computation, 9(3), 577–594]. Typically, these models use weight normalization to contain the weight growth associated with Hebbian learning. A more plausible mechanism for controlling the Hebbian process has recently emerged. Turrigiano and Nelson [Turrigiano G.G., Nelson S.B., (2004). Homeostatic plasticity in the developing nervous system. Nature Reviews Neuroscience, 5, 97–107] have shown that neurons in the cortex actively maintain an average firing rate by scaling their incoming weights. In this work, it is shown that this type of homeostatic synaptic scaling can replace the common, but unsupported, standard weight normalization. Organized maps still form and the output neurons are able to maintain an unsaturated firing rate, even in the face of large-scale cell proliferation or die-off. In addition, it is shown that in some cases synaptic scaling leads to networks that more accurately reflect the probability distribution of the input data.

Introduction

The self-organizing map (SOM), in its various forms, has been a useful model of cortical development (Choe and Miikkulainen, 2004, Kohonen, 2001, Obermayer et al., 1992, Sirosh and Miikkulainen, 1997). Sirosh and Miikkulainen (1997) showed the simultaneous development of receptive field properties and lateral interactions in a realistic model. The usefulness of the developed lateral connections was shown by Choe and Miikkulainen (2004) for contour integration and segmentation. It is this lateral connectivity that ensures the neighboring neurons come to respond to similar stimuli and form a good map.

In these models, Hebbian learning is used to strengthen associations between stimuli and winning neurons. This type of associative learning has been well documented in the experimental literature (Bi and Poo, 2001, Bliss and Lomo, 1973), but our understanding has remained incomplete. It is well known that the most straightforward implementations of Hebbian learning lead to unconstrained weight growth. To counteract this problem, typical SOM algorithms use weight normalization: after each learning iteration all the weights converging onto a neuron are divided by the sum of the incoming weights (or the square root of the sum of the squared weights). It has been argued that this type of weight normalization is biologically plausible. For example, a neuron might have a finite resource necessary for maintaining incoming synapses. This might keep an upper limit on the total summed size of the incoming synapses. While this sounds within the realm of biological possibility, and is obviously helpful in keeping Hebbian learning in check, little evidence from the experimental literature is available for support.

More plausible mechanisms for controlling the Hebbian process based on maintaining an average output firing rate have recently emerged. Two different types of these internal mechanisms have been found. One type controls the intrinsic excitability of the neuron (reviewed by Zhang and Linden (2003)). The molecular causes underlying this mechanism are still being investigated, but some of the behavior has been documented. In a typical experiment, a neuron would be excited repeatedly at high frequency. Then the output firing rate would be measured when current is injected. The intrinsic excitability (the ratio of firing rate to injected current) is higher after the stimulation. This makes the neuron even more sensitive to its inputs.

Two models have proposed that neurons modify their excitability to maintain a high rate of information transfer. In the first of these models (Stemmler & Koch, 1999), individual neurons change their ion channel conductances in order to match an input distribution. The neurons are able to maintain high information rates in response to a variety of distributions. The second model (Triesch, 2004) proposes that neurons adjust their output nonlinearities to maximize information transfer. There it is assumed that the neuron can keep track of its average firing rate and average variance of firing rate. Given this limited information, it does the best it can by adjusting the slope and offset of an output sigmoid function.

In the second type of internal neuronal mechanism it was shown (Maffei et al., 2004, Turrigiano et al., 1998, Turrigiano and Nelson, 2004) that neurons in the cortex actively maintain an average firing rate by scaling their incoming weights. The mechanism has been examined in cultures and in other experiments using in vivo visual deprivation. It has been shown that the incoming synapses are altered by a multiplicative factor, which presumably preserves the relative strengths of the synapses. The underlying mechanisms are not yet known, but there is ongoing research looking at intracellular chemical factors such as calcium and brain-derived neurotrophic factor (BDNF) (Turrigiano & Nelson, 2004). The levels of these factors are related to firing rates, so integrating them over time could lead to an estimate of average firing rate and produce a chemical signal for synaptic change. Another interesting finding is that a neuron with high average firing rate will decrease the strength of incoming excitatory synapses, but increase the strength of incoming inhibitory neurons (Maffei et al., 2004), thus altering the excitatory/inhibitory balance.

Homeostatic mechanisms have been implemented in two models. In one study, the molecular underpinnings of homeostasis were explored (Yeung, Shouval, Blais, & Cooper, 2004). It was suggested that LTP, LTD, and synaptic homeostatic scaling are all related through intracellular calcium levels. The homeostatic mechanism may influence the level of calcium, thereby changing under what conditions LTP and LTD are induced (since calcium levels play a major role in LTP and LTD). The other model concerns a particular class of associative memories (Chechik, Meilijson, & Ruppin, 2001). It is shown that the storage capacity can be increased if the neuron’s weights are controlled with either weight normalization or homeostatic synaptic scaling.

There have also been attempts to change neuron thresholds within neural networks. One network that was described (Horn & Usher, 1989) was a Hopfield network with neurons that were always producing one of two possible outputs (+1 or −1). The neuronal thresholds for determining which state to be in, acting on the total amount of input activation, was adjusted according to the recent activation. They reported that interesting periodic dynamics emerged. Another adjustable threshold network (Gorchetchnikov, 2000) was a winner-take-all associative memory. In that work, the thresholds on output sigmoid functions were adjusted. It was shown that on problems like XOR classification, the network would learn the appropriate mapping faster than without an adjustable threshold. While these approaches are computationally interesting, there is not yet evidence that neurons adjust their threshold based on their average firing rate.

It has previously been suggested that homeostatic synaptic scaling might form a basis for keeping Hebbian learning in check (Miller, 1996, Turrigiano and Nelson, 2004). This possibility is explored here. It may very well be that this mechanism is one of several that constrain synapse strength, but it is examined here in isolation to get a better understanding of its capability.