Abstract
In the 1960s, Evarts first recorded the activity of single neurons in motor cortex of behaving monkeys (Evarts, 1968). In the 50 years since, great effort has been devoted to understanding how single neuron activity relates to movement. Yet these single neurons exist within a vast network, the nature of which has been largely inaccessible. With advances in recording technologies, algorithms, and computational power, the ability to study these networks is increasing exponentially. Recent experimental results suggest that the dynamical properties of these networks are critical to movement planning and execution. Here we discuss this dynamical systems perspective and how it is reshaping our understanding of the motor cortices. Following an overview of key studies in motor cortex, we discuss techniques to uncover the “latent factors” underlying observed neural population activity. Finally, we discuss efforts to use these factors to improve the performance of brain–machine interfaces, promising to make these findings broadly relevant to neuroengineering as well as systems neuroscience.
- motor control
- motor cortex
- dynamical systems
- brain-machine interfaces
- neural population dynamics
- machine learning
Introduction
Our knowledge of the motor cortices (MC) is rapidly evolving. Traditional models of motor cortical activity held that the firing rates of individual neurons “represent” externally measurable movement covariates, such as hand or joint kinematics, forces, or muscle activity. Much effort in related studies was devoted to finding the “correct” coordinate system. However, the increased ability to record from many neurons simultaneously has revealed many features of population activity that are difficult to reconcile with a purely representational viewpoint. First, much of the observed, high-dimensional activity of neural populations in MC can be explained as a combination of a modest number of “latent factors”; abstract, time-varying patterns that cannot be observed directly, but represent the correlated activity of the neural population. Second, during movements, these factors appear to evolve in time by obeying consistent dynamic rules, much like the lawful dynamics that govern physical systems. Through this lens, the complex, often-puzzling responses of individual neurons are naturally explained as minor elements in a coordinated underlying dynamical system. These findings have provided a new framework for evaluating neural activity during many of the functions that are ascribed to MC, such as motor preparation and execution, motor learning, bimanual control, and the production of muscle activity.
Beyond its application to the motor cortices, the dynamical systems framework and related computational methods may have broad applicability throughout the brain. Over the past decade, the ability to record from large populations of neurons has increased exponentially (Stevenson and Kording, 2011; Sofroniew et al., 2016, Jun et al., 2017; Stringer et al., 2018). These data collection tools promise to further transform our understanding of the brain, but only if we can process and interpret the coming wave of massive datasets. Trying to interpret the “tuning” of 10,000 neurons is not only onerous but a missed opportunity; much of the brain's computation is inaccessible from the activity of individual neurons, but is instead instantiated via population-level dynamics. Fortunately, modeling neural populations as low-dimensional dynamical systems is providing new insights in many cortical areas, including areas that mediate cognitive processes such as decision-making (Mante et al., 2013; Raposo et al., 2014; Carnevale et al., 2015), interval timing (Remington et al., 2018), and navigation (Harvey et al., 2012; Morcos and Harvey, 2016). This has deep implications for systems neuroscience: moving forward, the central thrust in understanding how brain areas perform computations and mediate behaviors may be through uncovering their population structure and underlying dynamics. MC is a critical model for studying these phenomena, as its activity appears strongly governed by internal dynamics, yet is well related to observable behavior. These characteristics make MC an excellent “proving ground” for tools that may be useful in a wide variety of brain areas.
Furthermore, our increasing knowledge of latent factors and dynamics in MC creates new opportunities to harness cortical activity to build high-performance and robust brain–machine interfaces (BMIs) to restore mobility to people with paralysis. BMIs aim to restore function by directly interfacing with the brain and reading out neural activity related to a person's movement intent. To date, the vast majority of BMIs that use MC activity have been based on a representational viewpoint, with the assumption that individual neurons represent external movement covariates. Incorporating knowledge of the latent structure and dynamics of MC population activity potentially offers the means to develop BMIs whose performance and long-term stability are greatly improved.
Our review is divided to cover three broad areas: (1) an overview of the dynamical systems view of MC, including key studies that have tested its applicability and demonstrated new insight into the structure of population activity in MC; (2) current techniques to uncover latent structure and dynamics from the activity of neural populations; and (3) recent efforts to use latent factors and dynamics to improve BMI performance.
The dynamical systems view and evidence in motor cortex
Early work to understand the relationship between MC activity and movements drew inspiration from studies in sensory areas, such as the experiments of Hubel and Wiesel (1959) in visual cortex. In those experiments, the response of a neuron was modeled as a function of carefully controlled features of the presented stimuli (Hubel and Wiesel, 1959). Similarly, studies in MC revealed that the responses of individual neurons (e.g., spike counts over hundreds of milliseconds) could be reasonably well modeled as a function of kinetic or kinematic movement parameters (Evarts, 1968; Georgopoulos et al., 1982; Schwartz et al., 1988). A complication of the motor domain is that these movement covariates could only be studied by training animals to produce highly stereotypic movements, replete with many correlations across limb segments and measurement systems. Over the next decades, a long-simmering debate that had originated perhaps with Hughlings Jackson over which parameters of movement were represented (Jackson, 1873; Phillips, 1975) was given new fuel. Anatomical considerations argue for a strong, direct link between primary motor cortex and muscle activity (Landgren et al., 1962; Jankowska et al., 1975; Cheney and Fetz, 1985), supported by many studies which found that neural activity covaries with muscle activation and kinetics (Evarts, 1968; Hepp-Reymond et al., 1999; Gribble and Scott, 2002; Holdefer and Miller, 2002). Yet correlates of higher-level parameters such as endpoint position (Riehle and Requin, 1989), velocity (Georgopoulos et al., 1982), speed (Churchland et al., 2006a), and curvature (Hocherman and Wise, 1991) could all be found as well. As this list became longer, some began to notice that these representations could also break down quite badly (Fu et al., 1995; Churchland and Shenoy, 2007b), and that such correlations could be spurious (Mussa-Ivaldi, 1988). This led many to wonder whether viewing MC as a representational system is appropriate (Fetz, 1992; Scott, 2008; Churchland et al., 2010).
Rather than asking which parameters constitute the output of MC, one might instead view the system from a generative perspective: how does MC generate its output? From this perspective, MC is seen as a computational engine whose activity translates high-level movement intention into the complex patterns of muscle activity required to execute a movement (Todorov and Jordan, 2002; Scott, 2004; Shenoy et al., 2013). If so, how might this computation be performed?
For decades, theoreticians have posited that brain areas may perform computation through network-level phenomena in which information is distributed across the activity of many neurons, and processed via lawful dynamics that dictate how the activity of a neural population evolves over time (for review, see Yuste, 2015). We formalize this dynamical view in Fig. 1A. We assume that at a given time point t, the activity of a population of D neurons can be captured by a vector of spike counts n(t) = [n1(t), n2(t), … nD(t)]. The neural population acts as a coordinated unit, with a K-dimensional internal “state” x(t) = [x1(t), x2(t), … xK(t)]. In many brain areas, x(t) has been observed to be much lower-dimensional than the total number of observed neurons (i.e., K ≪ D; Fig. 1B; Cunningham and Yu, 2014). This dimensionality is likely somewhat constrained due to the recurrent connectivity of the network, which restricts the possible patterns of coactivation that may occur (for review, see Gallego et al., 2017a). A notable consideration, however, is that the observed dimensionality is often lower than might be expected due to network constraints alone; this particularly low dimensionality may be further induced by the simplicity of common behavioral paradigms (Gao and Ganguli, 2015; Gao et al., 2017).
The dynamical systems view posits an additional constraint: the evolution of the population's activity in time is largely determined by internal rules (dynamics). In the limit of an autonomous dynamical system (i.e., a system that operates independently of any external factors), and without noise, the system's evolution follows the equation ẋ = f(x); that is, its future state changes are completely dependent upon (and predicted by) the current state. A conceptual example of a low-dimensional system with simple rotational dynamics (a 1-D pendulum), and its related dynamical flow-field, is presented in Figure 1, C and D. We note that MC clearly cannot be autonomous. It must receive and process inputs, such as sensory information, to produce responsive behaviors. However, as discussed below, the model of an autonomous dynamical system is reasonable for MC activity during the execution of well prepared movements. During behaviors that are unprepared, or where unpredictable events necessitate corrections (such as responding to task perturbations), MC activity may be well modeled as an input-driven dynamical system, analogous to a pendulum started from particular initial conditions, and subject to external perturbations (Pandarinath et al., 2018).
The dynamical systems framework makes testable predictions about the nature of MC activity. First, it predicts that the initial conditions of the system, such as those observed during movement preparation, largely determine the subsequent evolution of activity. Second, the activity of neurons in MC should relate not only to the inputs and outputs of the system, but also to the computations being performed. Finally, distinct computations may be appropriated into different, non-overlapping neural dimensions. Here, we explore experimental evidence related to each of these predictions.
Early studies exploring the dynamical systems hypothesis in MC examined whether preparatory activity served as an “initial condition” for the subsequent dynamics. In tasks with delay periods, where a subject has knowledge of the movement condition before execution, neural activity in MC approaches distinct “preparatory states” for distinct movements (Tanji and Evarts, 1976). In a dynamical system, initial conditions determine subsequent activity patterns, so the same dynamical “rules” can give rise to different activity patterns and behaviors if the initial condition is different. Similarly, in MC, an altered preparatory state relates to altered movement execution. If neural preparatory activity is not in the right state at the time of the go cue, either due to natural fluctuations (Churchland et al., 2006b; Afshar et al., 2011; Michaels et al., 2015, 2018), subthreshold microstimulation during the delay period (Churchland and Shenoy, 2007a), or a change in the location of the target (Ames et al., 2014), the reaction time is delayed compared with well prepared trials. This suggests that, if motor preparation is incorrect, subjects do not move until their preparation has been corrected. Furthermore, motor adaptation to a visuomotor rotation (Vyas et al., 2018), visuomotor scaling (Stavisky et al., 2017a), or a curl field (Perich and Miller, 2017; Perich et al., 2017) has been shown to modify the motor preparatory state. These modifications correspond to altered execution trajectories, and the associated changes in preparation states and execution trajectories transfer from covert settings (BMI tasks without movement) to overt movements (normal reaching movements; Vyas et al., 2018).
Additional work has tested whether dynamical system models continue to predict observed neural activity during the transition from preparation to movement. A simple model in which preparatory activity seeds the initial condition for rotational dynamics during movement generation fits neural activity well in nonhuman primates (Fig. 2A; Churchland et al., 2012; Elsayed et al., 2016; Michaels et al., 2016; Pandarinath et al., 2018) and humans (Pandarinath et al., 2015, 2018). Furthermore, recurrent neural networks trained to generate muscle activity after receiving preparatory input display dynamics similar to those recorded in MC (Hennequin et al., 2014; Sussillo et al., 2015; Kaufman et al., 2016), suggesting that a dynamical system that uses preparatory activity as the initial condition for subsequent movement dynamics may be a natural strategy for generating muscle activity during reaching.
Another important prediction of the dynamical systems model is that not all of the activity in MC must directly relate to task parameters or muscle activity, but may instead relate to internal processes that subserve the current computation. For example, the switch from movement preparation to movement generation is accompanied by a substantial change in dynamics (Kaufman et al., 2014; Elsayed et al., 2016). Recent work has posited that this change is accomplished by a large, condition-invariant translation in state-space, which triggers the activation of movement-generation dynamics. Indeed, this condition-invariant signal is not only present at the switch from preparation to generation, but is also the largest aspect of the motor cortical response (Kaufman et al., 2016). Furthermore, during movement generation itself, the dominant patterns of neural activity may also play a role in supporting neural dynamics, rather than directly encoding the output. One challenge for a dynamical system results when the flow-field is highly “tangled”: when there are points in the space in which very similar states lead to very different future behavior. If two nearby points lead to different paths, then small amounts of noise in the system can lead to dramatic differences in the evolution of the neural state (Fig. 2B). A robust dynamical system, therefore, must ensure that the tangling is low, potentially by adding in additional dimensions of activity whose job is to “pull apart” points of high tangling (Fig. 2C). In MC, although some components of neural activity resemble muscle-like signals during movement generation, the largest patterns of neural activity during movement generation appear to function to reduce tangling (Fig. 2D; Russo et al., 2018). Thus, within MC, evidence has been found that some signals primarily support and drive dynamics, rather than directly encoding input or output.
Finally, the dynamical systems framework predicts that to perform different computations, neural activity may use different dimensions (Mante et al., 2013). Although this need not be a property of every possible dynamical system, using different dimensions for different functions allows a system to better maintain independence between its different roles. It has long been observed that many neurons are active both during movement preparation and movement generation. How then does preparatory activity avoid causing movement? Traditional views held that preparatory activity lies below a movement-generation threshold (Tanji and Evarts, 1976; Erlhagen and Schöner, 2002) or is under the influence of a gate (Bullock and Grossberg, 1988; Cisek, 2006). However, subthreshold activation fails to explain why preferred directions are minimally correlated between preparation and movement (Churchland et al., 2010; Kaufman et al., 2010), and there is little evidence for gating within MC, as inhibitory neurons in MC are not preferentially activated during motor preparation (Kaufman et al., 2013). The dynamical systems model, by contrast, makes a different prediction: that unwanted movements can be avoided by avoiding specific neural dimensions. Some dimensions, termed “output-potent”, correspond to activation patterns that are output to the muscles, whereas others, termed “output-null”, do not (Fig. 2D; Kaufman et al., 2014). In MC, different dimensions are activated during movement preparation and generation (Fig. 2E; Elsayed et al., 2016). Furthermore, the dimensions that best correlate with muscle activity are preferentially active during movement generation, suggesting that output-potent dimensions are selectively avoided during preparation (Kaufman et al., 2014). Similarly, distinct dimensions may be explored during cortically-dependent movement versus non-cortically-dependent movement (Miri et al., 2017), and sensory feedback initially enters MC in different dimensions from those of the muscle-potent activity (Stavisky et al., 2017b).
This division of different functions into different neural dimensions is not limited to muscle-related activity. In BMIs, where output-potent dimensions can be specified explicitly, activity that is informative about experimentally induced perturbations is initially orthogonal to the corresponding corrective responses (Stavisky et al., 2017b). During long-term BMI use, activity in output-potent dimensions is more stable than output-null dimensions (Flint et al., 2016). Neural activity also tends to occupy different dimensions for different movement categories: for example, pedaling in a forward versus reverse direction (Russo et al., 2018), isometric force production versus limb movement (Gallego et al., 2017b), or moving with the contralateral versus ipsilateral arm (Ames and Churchland, 2018). Using different neural dimensions for different functions may give the MC flexibility to generate activity patterns that support a wide variety of functions without interfering with one another (Perich et al., 2017).
Methods for estimating and evaluating motor cortical dynamics
As detailed above, much of the activity of MC neurons is naturally explained as a reflection of a low-dimensional dynamical system. Studying such dynamic processes requires techniques that can infer latent structure and its dynamics from observed, high-dimensional data. Related techniques have been applied to a wide variety of model systems and brain areas over the last two decades (for review, see Cunningham and Yu, 2014). However, MC holds particular value for testing these techniques, as its activity is closely tied to observable behavior (e.g., movement conditions, arm or hand kinematics, reaction times), which provides a key reference for validating the inferred state estimates. In this section, we review common techniques for estimating latent state and dynamics that have been applied in MC. We first present a general framework for discussion. Next, we review techniques that are applied to time points independently (i.e., do not explicitly model neural dynamics). Finally, we review techniques that do explicitly model neural dynamics, thereby resulting in better latent state estimates.
Techniques for latent state estimation typically view spiking activity as being “generated” by an underlying state x(t) (Fig. 3A). A common assumption is that for any given trial, the observed high-dimensional spike counts n(t) reflect a noisy sample from each neuron's underlying firing rate distribution r(t), a distribution that is itself derived from the latent state x(t). For motor cortical data, the distinction between observations n(t) and underlying rates r(t) captures the empirical observation that the spiking of any given neuron across multiple repeats (trials) of the same movement is highly variable.
A standard approach to de-noising n(t) and approximating r(t) is trial-averaging. Trial-averaging assumes all trials of a given movement condition are identical, and reduces single-trial noise in the estimate of r(t) by averaging n(t) across repeated trials. r(t) is often further de-noised by convolving it with a smoothing kernel. A common approach to estimate the lower dimensional x(t) is to perform principal component analysis (PCA). Performing PCA on r(t) rather than n(t) is preferred; if performed on n(t), PCA often results in poor latent factor estimation, because it simply maximizes the variance captured by the low-dimensional space, without separating variance that is shared among neurons from variance that is independent across neurons (for review, see Yu et al., 2009). When performed on r(t), PCA is typically accompanied by firing rate normalization, so that neurons with high rates (and thus higher variability) do not dominate the dimensionality reduction. Further, PCA can be extended by integrating some supervision into the dimensionality reduction step, e.g., by integrating information about task conditions to identify dimensions that capture neural variability related to particular task variables, using de-mixed PCA (dPCA; Fig. 3B; Kobak et al., 2014; Kaufman et al., 2016; Gallego et al., 2017b). The strategy of trial-averaging followed by PCA has led to several insights into latent structure and dynamics in MC (Churchland et al., 2012; Ames et al., 2014; Kaufman et al., 2014; Pandarinath et al., 2015; Elsayed et al., 2016; Kaufman et al., 2016; Gallego et al., 2017b; Russo et al., 2018).
However, circumventing the need to average over trials is critical for elucidating inherently single-trial phenomena, such as the trial-to-trial variability of real movements (and their corresponding error corrections), non-repeated behaviors such as natural movements, random target tasks, and tasks involving learning. Likewise, studying “internal” processes that vary substantially across trials and have limited behavioral correlates, such as decision-making, vacillation, and internal state estimates (Golub et al., 2015; Kaufman et al., 2015), is also impossible with trial-averaged data. Factor analysis (FA; Everitt, 1984) is often favored for analyzing single-trial phenomena (Santhanam et al., 2009; Sadtler et al., 2014; Golub et al., 2015, 2018; Athalye et al., 2017). A key assumption of FA is that activity that is correlated across neurons represents “signal” [comprising the latent factors x(t)], and activity that is not correlated across neurons represents “noise”. This assumption matches the graphical model in Figure 3A. A recent, complementary approach to capturing trial-dependent variability in n(t) without corrupting the latent factors is to integrate information regarding trial ordering into the dimensionality reduction step, and introduce a set of “trial factors” that accommodate variability across trials, as in tensor components analysis (TCA; Williams et al., 2018).
A key limitation of the above techniques (PCA, FA) is that they treat neighboring time points as though they are independent. However, as discussed, a core assumption of the dynamical systems framework is that time points are intimately related, and in particular, previous states are predictive of future states. Therefore, methods that simultaneously infer latent states and dynamics should provide more accurate state estimation by leveraging the interdependencies of data points that are close in time. Two well developed families of models are Gaussian process-based approaches (Fig. 3C; Yu et al., 2009; Lakshmanan et al., 2015; Zhao and Park, 2017; Duncker and Sahani, 2018) and linear dynamical systems (LDS)-based approaches (Macke et al., 2011; Buesing et al., 2012; Gao et al., 2015, 2016; Kao et al., 2015, 2017; Aghagolzadeh and Truccolo, 2016). Gaussian Process approaches assume that the latent state x(t) is composed of factors that vary smoothly and independently in time, with each factor having its own characteristic time constant. In comparison, LDS-based approaches assume that the latent state at a given time point is a linear function of the previous state (i.e., ẋ = Ax), which incorporates linear interactions between latent dimensions. One issue with the LDS approach is that the matrix A is time-invariant, yet must capture the dynamics at all time points. In MC, this is potentially problematic, because activity during different behavioral phases (e.g., preparation and movement) is governed by very different dynamics (Kaufman et al., 2014, 2016; Elsayed et al., 2016). A promising approach to address this challenge is switching LDS (SLDS), which assumes that at any given time point, the system's evolution obeys one of a discrete set of possible dynamics, each of which must be learned (Fig. 3D; Petreska et al., 2011; Linderman et al., 2017; Wei et al., 2018).
An alternative approach to uncovering single-trial population dynamics uses recurrent neural networks (RNNs). Known as latent factor analysis via dynamical systems (LFADS; Sussillo et al., 2016a; Pandarinath et al., 2018), the approach trains an RNN as a generative model of the observed spiking activity. RNNs are powerful nonlinear function approximators, capable of modeling complex, highly nonlinear dynamics through adjustment of their recurrent connectivity (Fig. 4A). LFADS uses a sequential autoencoder (SAE) framework (Fig. 4B), allowing the potentially nonlinear dynamics to be learned from noisy, single-trial neural population activity using stochastic gradient descent. This allows LFADS to accurately infer dynamics on a single-trial, moment-to-moment basis (Fig. 4C). A critical confirmation that these dynamics are accurate and meaningful is that they lead to dramatic improvements in the ability to predict behavior. As shown (Fig. 4D), the LFADS-inferred latent representations were considerably more informative about subjects' reaching movements than was the population activity that was directly observed. These findings reinforce that population states, rather than the activity of individual neurons, may be a key factor in understanding how brain areas mediate behaviors, and further, that SAEs provide a powerful new avenue toward linking the activity of neural populations to the behaviors they mediate.
Leveraging latent factors and dynamics for BMIs
BMIs aim to help compensate for lost motor function by directly decoding movement intent from neuron spiking activity to control external devices or restore movement (Taylor et al., 2002; Carmena et al., 2003; Hochberg et al., 2006, 2012; Ethier et al., 2012; Collinger et al., 2013; Sadtler et al., 2014; Gilja et al., 2015; Ajiboye et al., 2017; Pandarinath et al., 2017). BMIs have largely decoded neural activity into movement through a representational viewpoint: each neuron represents a reach direction and if the neuron fires, it votes for movement in that direction (Georgopoulos et al., 1982; Taylor et al., 2002; Gilja et al., 2012; Hochberg et al., 2012; Collinger et al., 2013). Efforts to decode EMG activity have been essentially similar, though in a higher-dimensional, more abstract space (Pohlmeyer et al., 2007; Ethier et al., 2012). However, as previously discussed, this representational model has important limitations in describing MC activity. In this section, we review recent studies that have asked whether representational assumptions also limit BMI performance, and if so, whether performance and robustness can be increased by incorporating MC latent factors and dynamics.
Using latent factors and dynamics to increase BMI performance
The dynamical systems view holds that movement-related variables (such as kinematics or EMG activity) are not the only factors that influence the activity of MC neurons. However, BMI decoders based on the standard representational model do not take other factors into account when relating observed activity to movement intention. Recent work introduced a decoding architecture (graphically represented in Fig. 5A) that incorporates latent factors and their dynamics (modeled as a simple linear dynamical system; Kao et al., 2015; Aghagolzadeh and Truccolo, 2016). One advantage of this architecture is that modeling latent factors can account for the multiple, diverse influences on observed neural activity to better uncover movement-related variables. A second advantage is that latent factors may be more easily de-noised than the observed high-dimensional activity, resulting in higher BMI performance. Briefly, the dynamical systems view assumes that the temporal evolution of MC states is largely predictable. If so, deviations from this prediction may primarily correspond to noise. To de-noise, MC dynamics can be used to adjust the latent factors so that they are more consistent with the dynamic predictions (Fig. 5B). In closed-loop BMI experiments, decoding the dynamically de-noised latent factors significantly increased performance over previous approaches (Fig. 5C), including previous representational decoders that de-noise activity by (1) smoothing using an experimenter-chosen filter (optimal linear estimator; Velliste et al., 2008; Collinger et al., 2013), (2) incorporating prior knowledge about kinematic smoothness (kinematic Kalman filter; Wu et al., 2003; Kim et al., 2008; Gilja et al., 2012, 2015; Hochberg et al., 2012), and (3) learning filtering parameters via least-squares regression (Wiener filter; Carmena et al., 2003; Hochberg et al., 2006).
BMI performance may also be increased through the use of non-movement signals that become apparent by examining latent factors. Recently, Even-Chen et al. (2017) exploited this idea to identify factors that reflect errors made during BMI control. The motivation for this work is that errors inevitably happen when controlling a BMI; however, instead of the user having to correct an error explicitly, it is possible to detect (or predict) its occurrence and automatically correct it (or prevent it). They applied PCA to identify an error-related signal in MC and found dimensions where projected neural data reflected task errors (example latent factors observed during errors are shown in Fig. 5D). In real-time experiments with monkeys, these latent factors were decoded to both prevent and autocorrect errors, improving BMI performance.
Using dynamics and latent factors to increase BMI longevity
Ideally, a BMI's performance would be maintained indefinitely. However, neural recording conditions frequently change across days, and even within-day in pilot clinical trials, e.g., due to neuron death or electrode movement and failure (Barrese et al., 2013; Perge et al., 2013; Sussillo et al., 2016b; Downey et al., 2018), which can lead to decoding instability. Current approaches to solve this problem include decoding more stable neural signals (e.g., threshold crossings and local field potentials; Flint et al., 2013; Nuyujukian et al., 2014; Gilja et al., 2015; Stavisky et al., 2015), gradually updating decoder parameters using a weighted sliding average (Orsborn et al., 2012; Dangi et al., 2013), automated decoder recalibration by updating “tuning” estimates daily (Bishop et al., 2014), continuous recalibration by retrospectively inferring the user's intention among a set of fixed targets (Jarosiewicz et al., 2015), and training robust neural network decoders on a diversity of conditions using large data volumes (Sussillo et al., 2016b).
A separate class of approaches aims to exploit the underlying neural latent space, which, as a property of the neural population, should have a stable relationship with the user's intention that is independent of the specific neurons observed at any moment (Gao and Ganguli, 2015; Dyer et al., 2017; Gallego et al., 2017b; Kao et al., 2017; Pandarinath et al., 2018). However, it is challenging to relate the observed neurons from a given recording condition to the underlying latent space. Recent studies using supervised alignment strategies have demonstrated the potential of latent dynamics to maintain BMI performance. Kao et al. (2017) exploited historical information about population dynamics (Fig. 5E,F), finding that even under severe neuron loss, aligning the remaining neurons to previously learned dynamics could partially rescue closed-loop performance, effectively extending BMI lifetime. Alternatively, Pandarinath et al. (2018) trained a single LFADS model using data from 44 independently recorded neural populations spanning many millimeters of MC and 5 months of recording sessions (Fig. 5G,H). They then used a single linear decoder to map these latent dynamics onto kinematics (Fig. 5I). This work demonstrated that, in the absence of learning, a single, consistent dynamical model describes neural population activity across long time periods and large cortical areas, and yields improved off-line decoding performance for any given recording session than was otherwise possible.
Some settings lack data for supervised alignment, i.e., directly linking neural activity from new recording conditions to motor intent may be challenging (settings without structured behaviors, or where intent is less clear on a moment-by-moment basis). In these settings, unsupervised techniques may be useful for aligning data. Recently, Dyer et al. (2017) introduced a semi-supervised approach called distribution alignment decoding (DAD; Fig. 6A–C). This approach aims to map neural data from a new recording condition (new data) onto a previously recorded low-dimensional movement distribution. To do this, DAD first reduces the dimensionality of the neural data (using PCA or a nonlinear manifold learning technique), and then searches for an affine transformation to match the low-dimensional neural data to movements, by minimizing the Kullback–Leibler (KL)-divergence between the two datasets (Fig. 6B). Their results demonstrate that DAD can achieve similar performance to that of a supervised decoder that has access to corresponding measurements of neural state and movement, if the underlying data distribution contains asymmetries that facilitate alignment (Fig. 6C). Although a powerful approach for aligning neural and movement data, DAD solves a non-convex optimization problem with many local minima (Fig. 6B) by using a brute force search. To improve alignment and avoid having to first perform dimensionality reduction, neural network architectures such as generative adversarial networks (GANs; Goodfellow et al., 2014; Molano-Mazon et al., 2018) provide a potential method to learn nonlinear mappings from one distribution to another (Fig. 6D). By leveraging the fact that low-dimensional representations of neural activity are consistent across days and even subjects, distribution alignment methods like DAD or GANs provide a strategy for decoding movements without labeled training data from new recording conditions.
Conclusions
The increasing ability to monitor large numbers of neurons simultaneously will present new opportunities to study neural activity at the population level. Mounting evidence shows that this provides a qualitatively different window into the nervous system from that of single-neuron recordings, and that population-level dynamics likely underlie neural population activity across a wide range of systems. Here we reviewed recent evidence that such dynamics shape activity and drive behavior in MC, outlined key methods for inferring latent factors and dynamics that have been applied to MC activity, and showed how uncovering latent factors and dynamics can yield higher-performing and more robust BMIs. Continuing advances in recording technologies, algorithms, and computational power will enable studies of dynamics that were not previously possible, and further, may open new avenues for neural prostheses to address a wide variety of disorders of the nervous system.
Footnotes
This work was supported by a Burroughs Wellcome Fund Collaborative Research Travel Grant (C.P.) and NIH NINDS R01NS053603 (L.E.M.).We thank Steven Chase, Chandramouli Chandrasekaran, Juan Gallego, Matthew Kaufman, Daniel O'Shea, David Sussillo, Sergey Stavisky, Xulu Sun, Eric Trautmann, Jessica Verhein, Saurabh Vyas, Megan Wang, and Byron Yu for their feedback on the paper.
The authors declare no competing financial interests.
- Correspondence should be addressed to Chethan Pandarinath, Emory University, 101 Woodruff Circle Northeast, Atlanta, GA 30322-0001. chethan{at}gatech.edu