Neural spiking for causal inference and learning — part 2/5

Cleaned-text code points [15686, 31650). Bibliography omitted here; source.md includes it.

The Effect of Interventions: when a variable is intervened upon, forced to take a given value, the edge between that variable and its parents is severed, changing the data-generating distribution. That is, if we intervene on a node j , then the interventional distribution is Where node j has been forced to take a value, y j , and the dependence on its parents has been severed.

To use this theory, first, we describe a graph such that ρ is compatible with the conditional independence requirement of the above definition. Consider the ordering of the variables Φ that matches the feedforward structure of the underlying dynamic feedforward network ( Fig 1A ). From this ordering we construct the graph over the variables Φ ( Fig 1B ). This graph respects the order of variables implied in Fig 1A , but it is over-complete, in the sense that it also contains a direct link between X and R . This direct link between X and R , though absent in the underlying dynamical model, cannot be ruled out in a distribution over the aggregate variables, so must be included. The graph is directed, acyclic and fully-connected. Being fully connected in this way guarantees that we can factor the distribution ρ with the graph and it will obey the conditional independence criterion described above.

PNG larger image

TIFF original image

(A) The dynamic spiking network model. The state at time bin t depends on both the previous state and a hierarchical dependence between inputs x t , neuron activities h t , and the reward signal r . Omitted for clarity are the extra variables that determine the network state ( v ( t ) and s ( t )). (B) To formulate the supervised learning problem, these variables are aggregated in time to produce summary variables of the state of the network during the simulated window. We consider these variables being drawn IID from a distribution ( X , Z , H , S , R ) ∼ ρ . Intervening on the underlying dynamic variables changes the distribution ρ accordingly. E.g. severing the connection from x t to h t for all t renders H independent of X . Thus the graphical model over ( X , Z , H , R ) has the same hierarchy (ordering) as the underlying dynamical model. However, the aggregate variables do not fully summarize the state of the network throughout the simulation. Therefore, unlike the structure in the underlying dynamics, H may not fully separate X from R —we must allow for the possibility of a direct connection from X to R . (C) and (D) are simple examples illustrating the difference between observed dependence and causal effect. We have omitted the dependence on X for simplicity. Violin plots show reward when H 1 is active or inactive, without (left subplot) and with (right) intervening on H 1 . (C) If H 1 and H 2 are independent, the observed dependence matches the causal effect. (D) If H 2 causes H 1 then H 2 is an unobserved confounder, and the observed dependence and causal effects differ.

https://doi.org/10.1371/journal.pcbi.1011005.g001

The second criterion is that the graph can be used to describe what happens when interventions are made. Fully spelling this out is beyond the scope of this study, so here we assume the interventional distributions on nodes of factor the distribution ρ as expected in the definition above. This is reasonable since, for instance, intervening on the underlying variable h i ( t ) (to enforce a spike at a given time), would sever the relation between Z i and H i as dictated by the graph topology. Taken together, this means the graph describes a causal Bayesian network over the distribution ρ .

Given this causal network, we can then define a neuron’s causal effect. In causal inference, the causal effect can be understood as the expected difference in an outcome R when a ‘treatment’ H i is exogenously assigned. The causal effect of neuron i on reward R is defined as: where do represents the do-operator, notation for an intervention [ 27 ]. Because of the fact that the aggregated variables maintain the same ordering as the underlying dynamic variables, there is a well-defined sense in which R is indeed an effect of H i , not the other way around, and therefore that the causal effect β i is a sensible, and not necessarily zero, quantity ( Fig 1C and 1D ). That is to say, it makes sense to associate with each neuron, for each stimulus, what its causal effect is on the output and thus reward.

#### Result II: Causal effects and finite difference approximation of gradients.

The causal effect β i is an important quantity for learning: if we know how a neuron contributes to the reward, the neuron can change its behavior to increase it. More specifically, in a spiking neural network, the causal effect can be seen as a type of finite difference approximation of the partial derivative (reward with a spike vs reward without a spike). That is, since we assume that R is function of some filtered neural output of the network, then it can be shown that the causal effect, β i , under certain assumptions, does indeed approximate the reward gradient . That is, there is a sense in which: (1) More specifically: where D i R is a random variable that represents the finite difference operator of R with respect to neuron i ’s firing, and Δ s is a constant that depends on the spike kernel κ and acts here like a kind of finite step size. Refer to the methods section for the derivation. This result establishes a connection between causal inference and gradient-based learning. It suggests that methods from causal inference may provide efficient algorithms to estimate reward gradients, and thus can be used to optimize reward. In this way the causal effect is a relevant quantity for learning.

### Estimating a neuron’s causal effect using the spiking discontinuity

Having understood how the causal effect of a neuron can be defined, and how it is relevant to learning, we now consider how to estimate it. The key observation of this paper is to note that a discontinuity can be used to estimate causal effects, without randomization, but while retaining the benefits of randomization. We will refer to this approach as the Spiking Discontinuity Estimator (SDE). This estimator is equivalent to the regression discontinuity design (RDD) approach to causal inference which is popular in economics [ 28 , 29 ].

As outlined in the introduction, the idea is that inputs that place a neuron close to its spiking threshold can be used in an unbiased causal effect estimator. For inputs that place the neuron just below or just above its spiking threshold, the difference in the state of the rest of the network becomes negligible, the only difference is the fact that in one case the neuron spiked and in the other case the neuron did not. Any difference in observed reward can therefore only be attributed to the neuron’s activity. Statistically, within a small interval around the threshold spiking becomes as good as random [ 28 – 30 ]. In this way the spiking discontinuity may allow neurons to estimate their causal effect.

To demonstrate the idea that a neuron can use its spiking non-linearity to estimate causal effects, here we analyze a simple two neuron network obeying leaky integrate-and-fire (LIF) dynamics. The neurons receive a shared scalar input signal x ( t ), with added separate noise inputs η i ( t ), that are correlated with coefficient c . Each neuron weighs the noisy input by w i . The correlation in input noise induces a correlation in the output spike trains of the two neurons [ 31 ], thereby introducing confounding. At the end of a trial period T , the neural output determines a reward signal R . Most aspects of causal inference can be investigated in a simple, few-variable model such as this [ 32 ], thus demonstrating that a neuron can estimate a causal effect in this simple case is an important first step to understanding how it can do so in a larger network.

To give intuition into how this confounding problem manifests in a neural learning setting, consider how a neuron can estimate its causal effect. A basic model of a neuron’s effect on reward is that it can be estimated from the following piece-wise constant model of the reward function: (2) That is, there is some baseline expected reward, γ i , and a neuron-specific contribution β i , where H i represents the spiking indicator function for neuron i over the trial of period T . Then denote by β i the causal effect of neuron i on the resulting reward R .

Estimating β i naively as (3) is biased if H i is correlated with other neuron’s activity ( Fig 2A ). Call this naive estimator the observed dependence .

PNG larger image

TIFF original image

(A) Graphical model describing neural network. Neuron H i receives input X , which contributes to drive Z i . If drive is above the spiking threshold, then H i is active. The activity contributes to reward R . Though not shown, this relationship may be mediated through downstream layers of a neural network, and complicated interactions with the environment. From neuron H i ’s perspective, the activity of the other neurons H which also contribute to R is unobserved. For clarity, other neuron’s Z variables have been omitted from this graph. Unobserved dependencies of X → R and Z i → R are shown here, even though not part of the underlying dynamical model, such dependencies in the graphical model may still exist, as discussed in the text. (B) The reward may be tightly correlated with other neurons’ activity, which act as confounders. However any discontinuity in reward at the neuron’s spiking threshold can only be attributed to that neuron. The discontinuity at the threshold is thus a meaningful estimate of the causal effect (left). The effect of a spike on a reward function can be determined by considering data when the neuron is driven to be just above or just below threshold (right). (C) This is judged by looking at the neural drive to the neuron over a short time period. Marginal sub- and super-threshold cases can be distinguished by considering the maximum drive throughout this period. (D) Schematic showing how spiking discontinuity operates in network of neurons. Each neuron contributes to output, and observes a resulting reward signal. Learning takes place at end of windows of length T . Only neurons whose input drive brought it close to, or just above, threshold (gray bar in voltage traces; compare neuron 1 to 2) update their estimate of β . (E) Model notation.

https://doi.org/10.1371/journal.pcbi.1011005.g002

Instead, we can estimate β i only for inputs that placed the neuron close to its threshold. That is, let Z i be the maximum integrated neural drive to the neuron over the trial period. If θ is the neuron’s spiking threshold, then a maximum drive above θ results in a spike, and below θ results in no spike. The neural drive used here is the leaky, integrated input to the neuron, that obeys the same dynamics as the membrane potential except without a reset mechanism. By tracking the maximum drive attained over the trial period, we can track when inputs placed the neuron close to its threshold, and marginally super-threshold inputs can be distinguished from well-above-threshold inputs, as required for SDE ( Fig 2C ). Let p be a window size within which we are going to call the integrated inputs Z i ‘close’ to threshold, then the SDE estimator of β i is: (4)

A one-order-higher model of the reward adds a linear correction, resulting in the piece-wise linear model of the reward function: (5) where γ i , α li and α ri are the linear regression parameters. This higher-order model can allow for larger window sizes p , and thus a lower variance estimator. Call the causal effect estimate using the piecewise constant model and the causal effect estimate using the piecewise-linear model . Both such models are explored in the simulations below.

This approach relies on some assumptions. First, a neuron assumes its effect on the expected reward can be written as a function of Z i which has a discontinuity at Z i = θ , such that, in the neighborhood of Z i = θ , the function can be approximated by either its 0-degree (piecewise constant version) or 1-degree Taylor expansion (piecewise linear). This approach also assumes that the input variable Z i is itself a continuous variable. This is the case in simulations explored here.

To summarize the idea: for a neuron to apply spiking discontinuity estimation, it simply must track if it was close to spiking, whether it spiked or not, and observe the reward signal. Then the comparison in reward between time periods when a neuron almost reaches its firing threshold to moments when it just reaches its threshold allows for an unbiased estimate of its own causal effect ( Fig 2D and 2E ). Below we gain intuition about how the estimator works, and how it differs from the naive estimate.

### A simple empirical demonstration of SDE

Simulating this simple two-neuron network shows how a neuron can estimate its causal effect using the SDE ( Fig 3A and 3B ). To show how it removes confounding, we implement both the piece-wise constant and piece-wise linear models for a range of window sizes p . When p is large, the piece-wise constant model corresponds to the biased observed-dependence estimator, , while small p values approximate the SDE estimator and result in an unbiased estimate ( Fig 3A ). The window size p determines the variance of the estimator, as expected from theory [ 33 ]. The piece-wise linear model, Eq (5) , is more robust to confounding ( Fig 3B ), allowing larger p values to be used. Thus the linear correction that is the basis of many RDD implementations [ 28 ] allows neurons to more readily estimate their causal effect. This linear dependence on the maximal voltage of the neuron may be approximated by plasticity that depends on calcium concentration; such implementation issues are considered in the discussion.

PNG larger image

TIFF original image

(A) Estimates of causal effect (black line) using a constant spiking discontinuity model (difference in mean reward when neuron is within a window p of threshold) reveals confounding for high p values and highly correlated activity. p = 1 represents the observed dependence, revealing the extent of confounding (dashed lines). Curves show mean plus/minus standard deviation over 50 simulations. (B) The linear model is unbiased over larger window sizes and more highly correlated activity (high c ). (C) Relative error in estimates of causal effect over a range of weights (1 ≤ w i ≤ 20) show lower error with higher coefficient of variability (CV; top panel), and lower error with lower firing rate (bottom panel). (D) Over this range of weights, spiking discontinuity estimates are less biased than just the naive observed dependence. (E,F) Approximation to the reward gradient overlaid on the expected reward landscape. The white vector field corresponds to the true gradient field, the black field correspond to the spiking discontinuity estimate (E) and observed dependence (F) estimates. The observed dependence is biased by correlations between neuron 1 and 2—changes in reward caused by neuron 1 are also attributed to neuron 2.

https://doi.org/10.1371/journal.pcbi.1011005.g003

The same simple model can be used to estimate the dependence of the quality of spike discontinuity estimates on network parameters. To investigate the robustness of this estimator, we systematically vary the weights, w i , of the network. This allows for an exploration SDE’s performance in a range of network states. SDE works better when activity is fluctuation-driven and at a lower firing rate ( Fig 3C ). Thus spiking discontinuity is most applicable in irregular but synchronous activity regimes [ 26 ]. Over this range of network weights, spiking discontinuity is less biased than the observed dependence ( Fig 3D ). Overall, corrected estimates based on spiking considerably improve on the naive implementation.