- Access by Xinjiang University
Learning in neural networks by reinforcement of irregular spiking
Phys. Rev. E 69, 041909 – Published 30 April, 2004
DOI: https://doi.org/10.1103/PhysRevE.69.041909
Abstract
Artificial neural networks are often trained by using the back propagation algorithm to compute the gradient of an objective function with respect to the synaptic strengths. For a biological neural network, such a gradient computation would be difficult to implement, because of the complex dynamics of intrinsic and synaptic conductances in neurons. Here we show that irregular spiking similar to that observed in biological neurons could be used as the basis for a learning rule that calculates a stochastic approximation to the gradient. The learning rule is derived based on a special class of model networks in which neurons fire spike trains with Poisson statistics. The learning is compatible with forms of synaptic dynamics such as short-term facilitation and depression. By correlating the fluctuations in irregular spiking with a reward signal, the learning rule performs stochastic gradient ascent on the expected reward. It is applied to two examples, learning the XOR computation and learning direction selectivity using depressing synapses. We also show in simulation that the learning rule is applicable to a network of noisy integrate-and-fire neurons.
Article Text
References (24)
- D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Nature (London) 323, 533 (1986).
- M. Jabri and B. Flower, IEEE Trans. Neural Netw. 3, 154 (1992).
- G. Cauwenberghs, in A Fast Stochastic Error-Descent Algorithm for Supervised Learning and Optimization, edited by S. J. Hanson, J. D. Cowan, and C. L. Giles (Morgan Kaufmann, San Mateo, CA, 1993), Vol. 5, pp. 244–251.
- R. J. Williams, Mach. Learn. 8, 229 (1992).
- J. Baxter and P. L. Bartlett, J. Artif. Intell. Res. 15, 319 (2001).
- P. Mazzoni, R. A. Andersen, and M. I. Jordan, Proc. Natl. Acad. Sci. U.S.A. 88, 4433 (1991).
- W. Softky and C. Koch, J. Neurosci. 13, 334 (1993).
- A. G. Barto and P. Anandan, IEEE Trans. Syst. Man Cybern. 15, 360 (1985).
- A. G. Barto and M. I. Jordan, in IEEE First International Conference on Neural Networks, San Diego, 1987, edited by M. Caudill and C. Butler (IEEE, New York, 1987), Vol. 2, pp. 629–636.
- F. S. Chance, S. B. Nelson, and L. F. Abbott, J. Neurosci. 18, 4785 (1998).
- R. S. Zucker, Annu. Rev. Neurosci. 12, 13 (1989).
- M. Tsodyks, K. Pawelzik, and H. Markram, Neural Comput. 10, 821 (1998).
- L. F. Abbott, J. A. Varela, K. Sen, and S. B. Nelson, Science 275, 220 (1997).
- C. van Vreeswijk and H. Sompolinsky, Science 274, 1724 (1996).
- R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction (MIT Press, Cambridge, MA, 1998).
- W. Maass and A. M. Zador, Neural Comput. 11, 903 (1999).
- T. Natschlager, W. Maass, and A. Zador, Network 12, 75 (2001).
- D. V. Buonomano and M. M. Merzenich, Science 267, 1028 (1995).
- J.-S. Liaw et al., Hippocampus 6, 591 (1996).
- N. Brunel and V. Hakim, Neural Comput. 11, 1621 (1999).
- G.-Q. Bi and M.-M. Poo, J. Neurosci. 18, 10464 (1998).
- H. Markram, J. Lubke, M. Frotscher, and B. Sakmann, Science 275, 213 (1997).
- C. C. Bell, V. Z. Han, Y. Sugawara, and K. Grant, Nature (London) 387, 278 (1997).
- P. Dayan and G. E. Hinton, in Feudal Reinforcement Learning, edited by S. J. Hanson, J. D. Cowan, and C. L. Giles (Morgan Kaufmann, San Mateo, CA, 1993), Vol. 5, pp. 271–278.