Section: Neuroscience
Topic: Neuroscience

Nonlinear computations in spiking neural networks through multiplicative synapses

Nardin, Michele1  ; Phillips, James W.2, 3 ; Podlaski, William F.4 ; Keemink, Sander W.5, 6  
1 Institute of Science and Technology Austria, Klosterneuburg, Austria
2 Current affiliation: UCL Department of Science, Technology, Engineering and Public Policy (STEaPP), University College London, London, UK
3 Independent researcher
4 Champalimaud Research, Champalimaud Centre for the Unknown, Lisbon, Portugal
5 Artificial Intelligence, Donders Institute for Brain, Cognition and Behaviour, Radboud University,
6 Nijmegen, the Netherlands

Corresponding author(s): Nardin, Michele (michele.nardin@ist.ac.at); Keemink, Sander W. (sander.keemink@donders.ru.nl)

10.24072/pcjournal.69 - Peer Community Journal, Volume 1 (2021), article no. e68.

Get full text PDF Peer reviewed and recommended by PCI
article image

The brain efficiently performs nonlinear computations through its intricate networks of spiking neurons, but how this is done remains elusive. While nonlinear computations can be implemented successfully in spiking neural networks, this requires supervised training and the resulting connectivity can be hard to interpret. In contrast, the required connectivity for any computation in the form of a linear dynamical system can be directly derived and understood with the spike coding network (SCN) framework. These networks also have biologically realistic activity patterns and are highly robust to cell death. Here we extend the SCN framework to directly implement any polynomial dynamical system, without the need for training. This results in networks requiring a mix of synapse types (fast, slow, and multiplicative), which we term multiplicative spike coding networks (mSCNs). Using mSCNs, we demonstrate how to directly derive the required connectivity for several nonlinear dynamical systems. We also show how to carry out higher-order polynomials with coupled networks that use only pair-wise multiplicative synapses, and provide expected numbers of connections for each synapse type. Overall, our work demonstrates a novel method for implementing nonlinear computations in spiking neural networks, while keeping the attractive features of standard SCNs (robustness, realistic activity patterns, and interpretable connectivity). Finally, we discuss the biological plausibility of our approach, and how the high accuracy and robustness of the approach may be of interest for neuromorphic computing.

Published online:
DOI: 10.24072/pcjournal.69
Type: Research article

Nardin, Michele 1; Phillips, James W. 2, 3; Podlaski, William F. 4; Keemink, Sander W. 5, 6

1 Institute of Science and Technology Austria, Klosterneuburg, Austria
2 Current affiliation: UCL Department of Science, Technology, Engineering and Public Policy (STEaPP), University College London, London, UK
3 Independent researcher
4 Champalimaud Research, Champalimaud Centre for the Unknown, Lisbon, Portugal
5 Artificial Intelligence, Donders Institute for Brain, Cognition and Behaviour, Radboud University,
6 Nijmegen, the Netherlands
License: CC-BY 4.0
Copyrights: The authors retain unrestricted copyrights and publishing rights 
@article{10_24072_pcjournal_69,
     author = {Nardin, Michele and Phillips, James W. and Podlaski, William F. and Keemink, Sander W.},
     title = {Nonlinear computations in spiking neural networks through multiplicative synapses},
     journal = {Peer Community Journal},
     eid = {e68},
     publisher = {Peer Community In},
     volume = {1},
     year = {2021},
     doi = {10.24072/pcjournal.69},
     url = {https://peercommunityjournal.org/articles/10.24072/pcjournal.69/}
}
TY  - JOUR
AU  - Nardin, Michele
AU  - Phillips, James W.
AU  - Podlaski, William F.
AU  - Keemink, Sander W.
TI  - Nonlinear computations in spiking neural networks through multiplicative synapses
JO  - Peer Community Journal
PY  - 2021
VL  - 1
PB  - Peer Community In
UR  - https://peercommunityjournal.org/articles/10.24072/pcjournal.69/
DO  - 10.24072/pcjournal.69
ID  - 10_24072_pcjournal_69
ER  - 
%0 Journal Article
%A Nardin, Michele
%A Phillips, James W.
%A Podlaski, William F.
%A Keemink, Sander W.
%T Nonlinear computations in spiking neural networks through multiplicative synapses
%J Peer Community Journal
%D 2021
%V 1
%I Peer Community In
%U https://peercommunityjournal.org/articles/10.24072/pcjournal.69/
%R 10.24072/pcjournal.69
%F 10_24072_pcjournal_69
Nardin, Michele; Phillips, James W.; Podlaski, William F.; Keemink, Sander W. Nonlinear computations in spiking neural networks through multiplicative synapses. Peer Community Journal, Volume 1 (2021), article  no. e68. doi : 10.24072/pcjournal.69. https://peercommunityjournal.org/articles/10.24072/pcjournal.69/

PCI peer reviews and recommendation, and links to data, scripts, code and supplementary information: 10.24072/pci.cneuro.100003

Conflict of interest of the recommender and peer reviewers:
The recommender in charge of the evaluation of the article and the reviewers declared that they have no conflict of interest (as defined in the code of conduct of PCI) with the authors or with the content of the article.

[1] Abbott, L. F.; DePasquale, B.; Memmesheimer, R.-M. Building functional networks of spiking model neurons, Nature Neuroscience, Volume 19 (2016) no. 3, pp. 350-355 | DOI

[2] Alemi A et al. Learning nonlinear dynamics in efficient, balanced spiking networks using local plasticity rules. In: Thirty-second aaai conference on artificial intelligence. URL: https://ojs. aaai.org/index.php/AAAI/article/view/11320, (2018)

[3] Arandia-Romero, I.; Tanabe, S.; Drugowitsch, J.; Kohn, A.; Moreno-Bote, R. Multiplicative and Additive Modulation of Neuronal Tuning with Population Activity Affects Encoded Information, Neuron, Volume 89 (2016) no. 6, pp. 1305-1316 | DOI

[4] Augustine, G. J.; Santamaria, F.; Tanaka, K. Local Calcium Signaling in Neurons, Neuron, Volume 40 (2003) no. 2, pp. 331-346 | DOI

[5] Baker, C.; Zhu, V.; Rosenbaum, R. Nonlinear stimulus representations in neural circuits with approximate excitatory-inhibitory balance, PLOS Computational Biology, Volume 16 (2020) no. 9 | DOI

[6] Barak, O. Recurrent neural networks as versatile tools of neuroscience research, Current Opinion in Neurobiology, Volume 46 (2017), pp. 1-6 | DOI

[7] Barrett, D. G.; Denève, S.; Machens, C. K. Optimal compensation for neuron loss, eLife, Volume 5 (2016) | DOI

[8] Barth, A. L.; Poulet, J. F. Experimental evidence for sparse firing in the neocortex, Trends in Neurosciences, Volume 35 (2012) no. 6, pp. 345-355 | DOI

[9] Boerlin, M.; Machens, C. K.; Denève, S. Predictive Coding of Dynamical Variables in Balanced Spiking Networks, PLoS Computational Biology, Volume 9 (2013) no. 11 | DOI

[10] Calaim, N.; Dehmelt, F. A.; Gonçalves, P. J.; Machens, C. K. Robustness in spiking networks: a geometric perspective, (2020) | DOI

[11] Cunningham, J. P.; Yu, B. M. Dimensionality reduction for large-scale neural recordings, Nature Neuroscience, Volume 17 (2014) no. 11, pp. 1500-1509 | DOI

[12] Davies, M.; Srinivasa, N.; Lin, T.-H.; Chinya, G.; Cao, Y.; Choday, S. H.; Dimou, G.; Joshi, P.; Imam, N.; Jain, S.; Liao, Y.; Lin, C.-K.; Lines, A.; Liu, R.; Mathaikutty, D.; McCoy, S.; Paul, A.; Tse, J.; Venkataramanan, G.; Weng, Y.-H.; Wild, A.; Yang, Y.; Wang, H. Loihi: A Neuromorphic Manycore Processor with On-Chip Learning, IEEE Micro, Volume 38 (2018) no. 1, pp. 82-99 | DOI

[13] Dayan P; Abbott LF Theoretical neuroscience: computational and mathematical modeling of neural systems. Computational Neuroscience Series. URL: https://mitpress.mit.edu/ books/theoretical-neuroscience, (2021)

[14] de Branges, L. The Stone-Weierstrass theorem, Proceedings of the American Mathematical Society, Volume 10 (1959) no. 5, pp. 822-824 | DOI

[15] Denève, S.; Machens, C. K. Efficient codes and balanced networks, Nature Neuroscience, Volume 19 (2016) no. 3, pp. 375-382 | DOI

[16] Eliasmith, C. A Unified Approach to Building and Controlling Spiking Attractor Networks, Neural Computation, Volume 17 (2005) no. 6, pp. 1276-1314 | DOI

[17] Eliasmith C; Anderson CH Neural engineering: Computation, representation, and dynamics in neurobiological systems, MIT press, (2004)

[18] Eliasmith, C.; Stewart, T. C.; Choo, X.; Bekolay, T.; DeWolf, T.; Tang, Y.; Rasmussen, D. A Large-Scale Model of the Functioning Brain, Science, Volume 338 (2012) no. 6111, pp. 1202-1205 | DOI

[19] Fino, E.; Yuste, R. Dense Inhibitory Connectivity in Neocortex, Neuron, Volume 69 (2011) no. 6, pp. 1188-1203 | DOI

[20] Gabbiani, F.; Krapp, H. G.; Koch, C.; Laurent, G. Multiplicative computation in a visual neuron sensitive to looming, Nature, Volume 420 (2002) no. 6913, pp. 320-324 | DOI

[21] Harris, K. D.; Mrsic-Flogel, T. D. Cortical connectivity and sensory coding, Nature, Volume 503 (2013) no. 7474, pp. 51-58 | DOI

[22] Indiveri, G.; Linares-Barranco, B.; Hamilton, T. J.; Schaik, A. v.; Etienne-Cummings, R.; Delbruck, T.; Liu, S.-C.; Dudek, P.; Häfliger, P.; Renaud, S.; Schemmel, J.; Cauwenberghs, G.; Arthur, J.; Hynna, K.; Folowosele, F.; Saighi, S.; Serrano-Gotarredona, T.; Wijekoon, J.; Wang, Y.; Boahen, K. Neuromorphic Silicon Neuron Circuits, Frontiers in Neuroscience, Volume 5 (2011) | DOI

[23] Jaeger H The “echo state” approach to analysing and training recurrent neural networks- with an erratum note. Bonn, Germany: German National Research Center for Information Tech- nology GMD Technical Report 148, 13. URL: https://www.ai.rug.nl/minds/uploads/ EchoStatesTechRep.pdf, (2001)

[24] Kasthuri, N.; Hayworth, K. J.; Berger, D. R.; Schalek, R. L.; Conchello, J. A.; Knowles-Barley, S.; Lee, D.; Vázquez-Reina, A.; Kaynig, V.; Jones, T. R.; Roberts, M.; Morgan, J. L.; Tapia, J. C.; Seung, H. S.; Roncal, W. G.; Vogelstein, J. T.; Burns, R.; Sussman, D. L.; Priebe, C. E.; Pfister, H.; Lichtman, J. W. Saturated Reconstruction of a Volume of Neocortex, Cell, Volume 162 (2015) no. 3, pp. 648-661 | DOI

[25] Keemink, S. W.; Machens, C. K. Decoding and encoding (de)mixed population responses, Current Opinion in Neurobiology, Volume 58 (2019), pp. 112-121 | DOI

[26] Ko, H.; Hofer, S. B.; Pichler, B.; Buchanan, K. A.; Sjöström, P. J.; Mrsic-Flogel, T. D. Functional specificity of local synaptic connections in neocortical networks, Nature, Volume 473 (2011) no. 7345, pp. 87-91 | DOI

[27] Koch, C.; Poggio, T. Multiplying with Synapses and Neurons, Single Neuron Computation, Elsevier, 1992, pp. 315-345 | DOI

[28] Larkum, M. E.; Nevian, T. Synaptic clustering by dendritic signalling mechanisms, Current Opinion in Neurobiology, Volume 18 (2008) no. 3, pp. 321-331 | DOI

[29] Lefort, S.; Tomm, C.; Floyd Sarria, J.-C.; Petersen, C. C. The Excitatory Neuronal Network of the C2 Barrel Column in Mouse Primary Somatosensory Cortex, Neuron, Volume 61 (2009) no. 2, pp. 301-316 | DOI

[30] Levien, R. B.; Tan, S. M. Double pendulum: An experiment in chaos, American Journal of Physics, Volume 61 (1993) no. 11, pp. 1038-1044 | DOI

[31] Li, N.; Daie, K.; Svoboda, K.; Druckmann, S. Robust neuronal dynamics in premotor cortex during motor planning, Nature, Volume 532 (2016) no. 7600, pp. 459-464 | DOI

[32] London, M.; Häusser, M. Dendritic Computation, Annual Review of Neuroscience, Volume 28 (2005) no. 1, pp. 503-532 | DOI

[33] Lorenz, E. N. Deterministic Nonperiodic Flow, Journal of the Atmospheric Sciences, Volume 20 (1963) no. 2, pp. 130-141 | DOI

[34] Maass, W.; Natschläger, T.; Markram, H. Real-Time Computing Without Stable States: A New Framework for Neural Computation Based on Perturbations, Neural Computation, Volume 14 (2002) no. 11, pp. 2531-2560 | DOI

[35] Mancoo A et al Understanding spiking networks through convex optimization. Advances in Neural Information Processing Systems 33. URL: https://proceedings.neurips.cc/paper/ 2020/file/64714a86909d401f8feb83e8c2d94b23-Paper.pdf, (2020)

[36] Mante, V.; Sussillo, D.; Shenoy, K. V.; Newsome, W. T. Context-dependent computation by recurrent dynamics in prefrontal cortex, Nature, Volume 503 (2013) no. 7474, pp. 78-84 | DOI

[37] Mastrogiuseppe, F.; Ostojic, S. Linking Connectivity, Dynamics, and Computations in Low-Rank Recurrent Neural Networks, Neuron, Volume 99 (2018) no. 3, pp. 609-623 | DOI

[38] Merolla, P. A.; Arthur, J. V.; Alvarez-Icaza, R.; Cassidy, A. S.; Sawada, J.; Akopyan, F.; Jackson, B. L.; Imam, N.; Guo, C.; Nakamura, Y.; Brezzo, B.; Vo, I.; Esser, S. K.; Appuswamy, R.; Taba, B.; Amir, A.; Flickner, M. D.; Risk, W. P.; Manohar, R.; Modha, D. S. A million spiking-neuron integrated circuit with a scalable communication network and interface, Science, Volume 345 (2014) no. 6197, pp. 668-673 | DOI

[39] Mitchell, S. J.; Silver, R. Shunting Inhibition Modulates Neuronal Gain during Synaptic Excitation, Neuron, Volume 38 (2003) no. 3, pp. 433-445 | DOI

[40] Neftci, E. O.; Mostafa, H.; Zenke, F. Surrogate Gradient Learning in Spiking Neural Networks: Bringing the Power of Gradient-Based Optimization to Spiking Neural Networks, IEEE Signal Processing Magazine, Volume 36 (2019) no. 6, pp. 51-63 | DOI

[41] Nezis, P.; van Rossum, M. C. W. Accurate multiplication with noisy spiking neurons, Journal of Neural Engineering, Volume 8 (2011) no. 3 | DOI

[42] Peña, J. L.; Konishi, M. Auditory Spatial Receptive Fields Created by Multiplication, Science, Volume 292 (2001) no. 5515, pp. 249-252 | DOI

[43] Perin, R.; Berger, T. K.; Markram, H. A synaptic organizing principle for cortical neuronal groups, Proceedings of the National Academy of Sciences, Volume 108 (2011) no. 13, pp. 5419-5424 | DOI

[44] Poirazi, P.; Brannon, T.; Mel, B. W. Pyramidal Neuron as Two-Layer Neural Network, Neuron, Volume 37 (2003) no. 6, pp. 989-999 | DOI

[45] Popov, V. I.; Stewart, M. G. Complexity of contacts between synaptic boutons and dendritic spines in adult rat hippocampus: Three-dimensional reconstructions from serial ultrathin sections in vivo, Synapse, Volume 63 (2009) no. 5, pp. 369-377 | DOI

[46] Rigotti, M.; Barak, O.; Warden, M. R.; Wang, X.-J.; Daw, N. D.; Miller, E. K.; Fusi, S. The importance of mixed selectivity in complex cognitive tasks, Nature, Volume 497 (2013) no. 7451, pp. 585-590 | DOI

[47] Rubin, D. B.; Van Hooser, S. D.; Miller, K. D. The Stabilized Supralinear Network: A Unifying Circuit Motif Underlying Multi-Input Integration in Sensory Cortex, Neuron, Volume 85 (2015) no. 2, pp. 402-417 | DOI

[48] Salinas, E.; Abbott, L. F. A model of multiplicative neural responses in parietal cortex., Proceedings of the National Academy of Sciences, Volume 93 (1996) no. 21, pp. 11956-11961 | DOI

[49] Schiller, J.; Major, G.; Koester, H. J.; Schiller, Y. NMDA spikes in basal dendrites of cortical pyramidal neurons, Nature, Volume 404 (2000) no. 6775, pp. 285-289 | DOI

[50] Semedo, J. D.; Zandvakili, A.; Machens, C. K.; Yu, B. M.; Kohn, A. Cortical Areas Interact through a Communication Subspace, Neuron, Volume 102 (2019) no. 1, pp. 249-259 | DOI

[51] Song, S.; Sjöström, P. J.; Reigl, M.; Nelson, S.; Chklovskii, D. B. Highly Nonrandom Features of Synaptic Connectivity in Local Cortical Circuits, PLoS Biology, Volume 3 (2005) no. 3 | DOI

[52] Stringer, C.; Pachitariu, M.; Steinmetz, N.; Carandini, M.; Harris, K. D. High-dimensional geometry of population responses in visual cortex, Nature, Volume 571 (2019) no. 7765, pp. 361-365 | DOI

[53] Stringer, C.; Pachitariu, M.; Steinmetz, N.; Reddy, C. B.; Carandini, M.; Harris, K. D. Spontaneous behaviors drive multidimensional, brainwide activity, Science, Volume 364 (2019) | DOI

[54] Strogatz SH Nonlinear dynamics and chaos with student solutions manual: With applications to physics, biology, chemistry, and engineering, CRC press, (2018)

[55] Sussillo, D. Neural circuits as computational dynamical systems, Current Opinion in Neurobiology, Volume 25 (2014), pp. 156-163 | DOI

[56] Sussillo, D.; Abbott, L. Generating Coherent Patterns of Activity from Chaotic Neural Networks, Neuron, Volume 63 (2009) no. 4, pp. 544-557 | DOI

[57] Thalmeier, D.; Uhlmann, M.; Kappen, H. J.; Memmesheimer, R.-M. Learning Universal Computations with Spikes, PLOS Computational Biology, Volume 12 (2016) no. 6 | DOI

[58] Young, A. R.; Dean, M. E.; Plank, J. S.; S. Rose, G. A Review of Spiking Neuromorphic Hardware Communication Systems, IEEE Access, Volume 7 (2019), pp. 135606-135620 | DOI

[59] Zenke, F.; Vogels, T. P. The Remarkable Robustness of Surrogate Gradient Learning for Instilling Complex Function in Spiking Neural Networks, Neural Computation, Volume 33 (2021) no. 4, pp. 899-925 | DOI

[60] Zhang, D.; Li, Y.; Rasch, M. J.; Wu, S. Nonlinear multiplicative dendritic integration in neuron and network models, Frontiers in Computational Neuroscience, Volume 7 (2013) | DOI

[61] Zhou, W.; Xu, Y.; Simpson, I.; Cai, Y. Multiplicative Computation in the Vestibulo-Ocular Reflex (VOR), Journal of Neurophysiology, Volume 97 (2007) no. 4, pp. 2780-2789 | DOI

Cited by Sources:

block.super