Neural spiking for causal inference and learning — part 4/5

Cleaned-text code points [46404, 61674). Bibliography omitted here; source.md includes it.

There are a number of ways that the simulations presented here are simplifications of true learning circuits. For instance, the LIF neural network implemented in Figs 2 – 4 has a fixed threshold. Yet cortical neurons do not have a fixed threshold, but rather one that can adapt to recent inputs [ 52 ]. We note that our exploration of learning in this more complicated case—the delayed XOR model ( S1 Text )—consists of populations of LIF and adaptive LIF neurons. The adaptive LIF neurons do have a threshold that adapts, based on recent spiking activity. The results in this model exhibit the same behavior as that observed in previous sections—for sufficiently highly correlated activity, performance is better for a narrow spiking discontinuity parameter p (cf. Fig 2A ). This suggests populations of adaptive spiking threshold neurons show the same behavior as non-adaptive ones. This makes sense since what is important for the spike discontinuity method is that the learning rule use a signal that is unique to an individual neuron. The stochastic, all-or-none spiking response provides this, regardless of the exact value of the threshold. The neuron just needs to know if it was close to its threshold or not. Of course, given our simulations are based on a simplified model, it makes sense to ask what neuro-physiological features may allow spiking discontinuity learning in more realistic learning circuits.

If neurons perform something like spiking discontinuity learning we should expect that they exhibit certain physiological properties. In this section we discuss the concrete demands of such learning and how they relate to past experiments. First, the spiking regime is required to be irregular [ 26 ] in order to produce spike trains with randomness in their response for repeated inputs. Irregular spiking regimes are common in cortical networks (e.g. [ 53 ]). Further, for spiking discontinuity learning, plasticity should be confined to cases where a neuron’s membrane potential is close to threshold, regardless of spiking. This means inputs that place a neuron close to threshold, but do not elicit a spike, still result in plasticity. This type of sub-threshold dependent plasticity is known to occur [ 54 – 56 ]. This also means that plasticity will not occur for inputs that place a neuron too far below threshold. In fact, in past models and experiments testing voltage-dependent plasticity, changes do not occur when postsynaptic voltages are too low [ 57 , 58 ]. Spiking discontinuity predicts that plasticity does not occur when postsynaptic voltages are too high. However, in many voltage-dependent plasticity models, potentiation does occur for inputs well-above the spiking threshold. To test if this is an important difference between what is statistically correct and what is more readily implementable in neurophysiology, we experimented with a modification of the learning rule, which does not distinguish between barely above threshold inputs and well above threshold inputs. For the piecewise-linear model, this modification did not adversely affect the neurons’ ability to estimate causal effects ( S2 Fig ). Thus threshold-adjacent plasticity as required for spike discontinuity learning appears to be compatible with neuronal physiology.

The learning rules presented here also need knowledge of outcomes and would thus likely be dependent on neuromodulation. Neuromodulated-STDP is well studied in models [ 51 , 59 , 60 ]. However, this learning requires reward-dependent plasticity that differs depending on if the neuron spiked or not. This may be communicated by neuromodulation. For instance, there is some evidence that the relative balance between adrenergic and M1 muscarinic agonists alters both the sign and magnitude of STDP in layer II/III visual cortical neurons [ 59 ]. To the best of our knowledge, how such behavior interacts with postsynaptic voltage dependence as required by spike discontinuity is unknown.

Overall, the structure of the learning rule is that a global reward signal (potentially transmitted through neuromodulators) drives learning of a variable, β , inside single neurons. This internal variable is combined with a term to update synaptic weights. The update rule for the weights depends only on pre- and post-synaptic terms, with the post- term getting updated over time, independently of the weight updates. Such an interpretation is interestingly in line with recently proposed ideas on inter-neuron learning, e.g., Gershman 2023 [ 61 ], who proposes an interaction of intra-cellular variables and synaptic learning rules can provide a substrate for memory. Thus, taken together, these factors show that SDE-based learning may well be compatible with known neuronal physiology.

### Conclusion

Here we presented the first exploration of the idea that neurons can perform causal inference using their spiking mechanism. There are a number of avenues for future work to develop the idea further. In particular, we primarily presented empirical results demonstrating the idea in numerous settings. Further experiments with the impact of learning window size in other learning rules where a pseudo-derivative type approach to gradient-based learning is applied can be performed, to establish the broader relevance of the insights made here. And the theoretical results that we presented were made under the strong assumption that the neural network activity, when appropriately aggregated, can be described by a causal Bayesian network. More rigorous results are needed. The problem of coarse-graining, or aggregating, low-level variables to obtain a higher-level model amenable to causal modeling is a topic of active research [ 62 , 63 ]. Further fleshing out an explicit theory that relates neural network activity to a formal causal model is an important future direction.

In this paper SDE based learning was explored in the context of maximizing a reward function or minimizing a loss function. But the same mechanism can be exploited to learn other signals—for instance, surprise (e.g. [ 64 ]) or novelty (e.g. [ 65 ]). SDE-based learning is a mechanism that a spiking network can use in many learning scenarios.

The most important aspect of spike discontinuity learning is the explicit focus on causality. A causal model is one that can describe the effects of an agent’s actions on an environment. Learning through the reinforcement of an agent’s actions relies, even if implicitly, on a causal understanding of the environment [ 37 , 66 ]. Here, by explicitly casting learning as a problem of causal inference we have developed a novel learning rule for spiking neural networks. We present the first model to propose a neuron does causal inference. We believe that focusing on causality is essential when thinking about the brain or, in fact, any system that interacts with the real world.

## Methods

### Neuron simulations and noise

We consider the activity of a simulated network of n neurons whose activity is described by their spike times, :

The neurons obey leaky integrate-and-fire (LIF) dynamics (8) where integrate and fire means simply: A refractory period of 3ms is added to the neurons. Different choices of refractory period were not shown to affect SDE performance ( S1 Fig ). Noisy input η i is comprised of a common DC current, x , and noise term, ξ ( t ), plus an individual noise term, ξ i ( t ): The noise processes are independent white noise: . This parameterization is chosen so that the inputs η 1,2 have correlation coefficient c . σ = 10 was used for the noise magnitude unless otherwise stated. Simulations are performed with a step size of Δ t = 1ms. Here the reset potential was set to v r = 0. The firing rate of a noisy integrate and fire neuron is (9) where and , σ i = σw i is the input noise standard deviation [ 21 ]. This is used in the learning rule derived below.

We define the input drive to the neuron, u i , as the leaky integrated input without a reset mechanism. That is, over each simulated window of length T : The SDE method operates when a neuron receives inputs that place it close to its spiking threshold—either nearly spiking or barely spiking—over a given time window. In order to identify these time periods, the method uses the maximum input drive to the neuron: The input drive is used here instead of membrane potential directly because it can distinguish between marginally super-threshold inputs and easily super-threshold inputs, whereas this information is lost in the voltage dynamics once a reset occurs. Here a time period of T = 50ms was used. Reward is administered at the end of this period: R = R ( s T ).

In our simulations, the dynamics are standard integrate and fire dynamics, and are given by (10) for synaptic time scale τ s = 0.02s. We thus use a standard model for the dynamics of all the neurons.

### Reward model and causal effect estimation

The simulations for Figs 3 and 4 are about standard supervised learning and there an instantaneous reward is given by . In order to have a more smooth reward signal, R is a function of s rather than h . The reward function used here has the following form: We choose a = −30, b = 20, x = −4.

The true causal effect was estimated from simulations with zero noise correlation and with a large window size—in order to produce the most accurate estimate possible. Determining the causal effect analytically is in general intractable.

### Learning causal effects through the spiking discontinuity

To remove confounding, spiking discontinuity learning considers only the marginal super- and sub-threshold periods of time to estimate β . This works because the discontinuity in the neuron’s response induces a detectable difference in outcome for only a negligible difference between sampled populations (sub- and super-threshold periods). The SDE estimates [ 28 ]: for maximum input drive obtained over a short time window, Z i , and spiking threshold, θ ; thus, Z i < θ means neuron i does not spike and Z i ≥ θ means it does.

To estimate , a neuron can estimate a piece-wise linear model of the reward function: locally, when Z i is within a small window p of threshold. Here γ i , α li and α ri are nuisance parameters, and β i is the causal effect of interest. This means we can estimate from

A neuron can learn an estimate of through a least squares minimization on the model parameters β i , α li , α ri . That is, for time period n (of length T ), if we let and , then if the neuron solves the following minimization: then that allows it to estimate its causal effect.

A neuron can perform stochastic gradient descent on this minimization problem, giving the learning rule: for learning rate η and for all time periods at which z i , n is within p of threshold θ .

When estimating the simpler, piece-wise constant model for either side of the threshold, the learning rule simplifies to: again for all time periods at which z i , n is within p of threshold θ . This causal inference strategy, established by econometrics, is ultimately what allows neurons to produce unbiased estimates of causal effects.

### The causal effect as a finite-difference operator

As a discrete event, we are interested not necessarily in the local gradient but in the finite difference between spiking and non-spiking. Importantly, this finite-difference approximation is exactly what our estimator gets at. We present a derivation here.

In particular, we show that To show this we replace with a type of finite difference operator: Here is a set of nodes that satisfy the back-door criterion [ 27 ] with respect to H i → R . By satisfying the backdoor criterion we can relate the interventional distribution to the observational distribution. When R is a deterministic, differentiable function of S and Δ s → 0 this recovers the reward gradient and we recover gradient descent-based learning.

To consider the effect of a single spike, note that unit i spiking will cause a jump in S i compared to not spiking (according to synaptic dynamics). If we let Δ s equal this jump then it can be shown that is related to the causal effect.

First, assuming the conditional independence of R from H i given S i and Q j ≠ i : (11) Now if we assume that on average H i spiking induces a change of Δ s in S i within the same time period, compared with not spiking, then: (12) Note that S i given no spike (the distribution on the RHS) is not necessarily zero, even if no spike occurs in that time period, because s i (0), the filtered spiking dynamics at t = 0, is not necessarily equal to zero. s i (0) instead has its own, non-zero distribution, representing activity from previous time periods. This approximation is reasonable because the linearity of the synaptic dynamics means that the difference in S i between spiking and non-spiking windows is simply exp(−( T − t si )/ τ s )/ τ s , for spike time t si . We approximate this term with its mean: (13) under the assumption that spike times occur uniformly throughout the length T window. These assumptions are supported numerically ( Fig 6 ).

PNG larger image

TIFF original image

(A) Simulated spike trains are used to generate S i | H i = 0 and S i | H i = 1. QQ-plot shows that S i following a spike is distributed as a translation of S i in windows with no spike, as assumed in (12) . (B) This offset, Δ s , is independent of firing rate and is unaffected by correlated spike trains. (C) Over a range of values (0.01 < T < 0.1, 0.01 < τ s < 0.1) the derived estimate of Δ s ( Eq (13) ) is compared to simulated Δ s . Proximity to the diagonal line (black curve) shows these match. (D) Δ s as a function of window size T and synaptic time constant τ s . Larger time windows and longer time constants lower the change in S i due to a single spike.

https://doi.org/10.1371/journal.pcbi.1011005.g006

Writing out the inner two expectations of (11) gives: after making the substitution s i → s i + Δ s in the first term. Writing this back in terms of expectations gives: Thus estimating the causal effect is similar to taking a finite difference approximation of the reward.

### Wider and deeper network implementations

We need to give more details for the wide network ( Fig 5A ) simulations. We use a single hidden layer of N neurons each receives inputs η . This is a mix of some fixed, deterministic signal and a noise signal, ξ : As in the N = 2 cases above, this is chosen so that the noise between any pair of neurons is related with a correlation coefficient c . This is then weighted by the vector w , which drives the leaky integrate and fire neurons to spike. The vector w was set to a vector of all ones. The output of the single hidden layer s is weighed by a second vector u . From this a cost/reward function of R = | U ⋅ s − Y | is computed. The vector U is randomly generated from a normal distribution such that each neuron has a different causal effect on the output. The target Y is set to Y = 0.1. From this setup, each neuron can estimate its effect on the function R using either the spiking discontinuity learning or the observed dependence estimator. Varying N allows us to study how the method scales with network size. Ten simulations were run for each value of N and c in Fig 5A .