- Access by Xinjiang University
On-line learning in soft committee machines
Phys. Rev. E 52, 4225 – Published 1 October, 1995
DOI: https://doi.org/10.1103/PhysRevE.52.4225
Abstract
The problem of on-line learning in two-layer neural networks is studied within the framework of statistical mechanics. A fully connected committee machine with K hidden units is trained by gradient descent to perform a task defined by a teacher committee machine with M hidden units acting on randomly drawn inputs. The approach, based on a direct averaging over the activation of the hidden units, results in a set of first-order differential equations that describes the dynamical evolution of the overlaps among the various hidden units and allows for a computation of the generalization error. The equations of motion are obtained analytically for general K and M and provide a powerful tool used here to study a variety of realizable, over-realizable, and unrealizable learning scenarios and to analyze the role of the learning rate in controlling the evolution and convergence of the learning process.
References (17)
- J. S. Denker, D. B. Schwartz, B. Wittner, S. A. Solla, R. E. Howard, L. D. Jackel and J. J. Hopfield, Complex Syst. 1, 877 (1987).
- G. Cybenko, Math. Control Signals Syst. 2, 303 (1988).
- K. Hornik, M. Stinchcombe and H. White, Neural Networks 2, 359 (1989).
- T. L. H. Watkin, A. Rau and M. Biehl, Rev. Mod. Phys. 65, 499 (1993).
- H. S. Seung, H. Sompolinsky, and N. Tishby, Phys. Rev. A 45, 6056 (1993).
- H. Schwarze, J. Phys. A 26, 5781 (1993).
- G. Parisi, Phys. Rev. Lett. 43, 1754 (1979).
- T. M. Heskes and B. Kappen, Phys. Rev. A 44, 2718 (1991).
- O. Kinouchi and N. Caticha, J. Phys. A 25, 6243 (1992).
- M. Biehl and P. Riegler, Europhys. Lett. 25, 525 (1994).
- N. Barkai, H. S. Seung, and H. Sompolinsky, in Advances in Neural Information Processing Systems, edited by G. Tesauro, D. S. Touretzky, and T. K. Leen (MIT Press, Cambridge, 1995), Vol. 7, p. 303.
- Y. Kabashima, J. Phys. A 27, 1917 (1994).
- M. Copelli and N. Caticha, J. Phys. A 28, 1615 (1995).
- M. Biehl and H. Schwarze, J. Phys. A 28, 643 (1995).
- D. Saad and S. A. Solla, Phys. Rev. Lett. 74, 4337 (1995).
- D. Saad and S. A. Solla (unpublished).
- P. Sollich, Phys. Rev. E 49, 4637 (1994).