- Access by Xinjiang University
Statistical mechanics approach to early stopping and weight decay
Phys. Rev. E 58, 833 – Published 1 July, 1998
DOI: https://doi.org/10.1103/PhysRevE.58.833
Abstract
Overtraining as a result of the difference between the empirical loss and the expected loss is a serious problem in neural network learning. It is known that methods such as early stopping, weight decay, or input noise can reduce overtraining. Here, these methods are studied in detail. We use a model that allows an analytical treatment. The treatment is based on an equilibrium statistical mechanics approach that is extended to its finite temperature solution. An unrealizable task that shows strong overtraining is examined. We find that overtraining can be completely avoided with each of the three methods if the parameters are optimally chosen. It is also shown that overtraining can appear in a realizable task, if the task is highly nonlinear. Also there overtraining can be avoided with each of the three methods.
References (20)
- Some references on-line are O. Kinouchi and N. Caticha, J. Phys. A 25, 6243 (1992); ibid.M. Biehl and H. Schwarze, 26, 2651 (1993); D. Saad and S. A. Solla, Phys. Rev. E 52, 4225 (1995); S. Bös, N. Murata, S. Amari, and K.-R. Müller, Brain Science Institute RIKEN Technical Report No. TR-BSI-1997-21.
- H. Akaike, IEEE Trans. Autom. Control. 19, 716 (1974).
- N. Murata, S. Yoshizawa, and S. Amari, in Advances in Neural Information Processing Systems 5, edited by S. J. Hanson, J. D. Cowan, and R. P. Lippmann (Morgan-Kaufmann, San Mateo, CA, 1993), p. 607.
- D. J. C. MacKay, Neural Comput. 4, 415 (1992); ibid.4, 448 (1992).
- L. Holmström and P. Koistinen, IEEE Trans. Neural Netw. 3, 24 (1992).
- J. Sjöberg and L. Ljung, report.
- R. Reed, R. J. Marks II, and S. Oh, IEEE Trans. Neural Netw. 6, 529 (1995).
- P. Sollich, in Advances in Neural Information Processing Systems 7, edited by G. Tesauro, D. Touretzky, and T. Leen (The MIT Press, Cambridge, MA, 1995), p. 207.
- A. Krogh and J. Hertz, J. Phys. A 25, 1117 (1992); ibid.25, 1135 (1992); in Advances in Neural Information Processing Systems 4, edited by J. E. Moody, S. J. Hanson, and R. P. Lippmann (Morgan-Kaufmann, San Mateo, CA, 1992), p. 950.
- J. Hertz, in Statistical Mechanics of Neural Networks, edited by L. Garrido, Lecture Notes in Physics Vol. 398 (Springer, Heidelberg, 1990), p. 137.
- R. Kühn and S. Bös, J. Phys. A 26, 831 (1993).
- J. Hertz, A. Krogh, and R. G. Palmer, Introduction to the Theory of Neural Computation (Addison-Wesley, Redwood City, CA, 1991).
- S. Bös, W. Kinzel, and M. Opper, Phys. Rev. E 47, 1384 (1993).
- Preliminary results of this work have been published by S. Bös, in International Conference on Artificial Neural Networks 95, edited by F. Fogelman Soulié and P. Gallinari (EC & Cie, Paris, 1985), p. 111; S. Bös, in Advances in Neural Information Processing Systems 8, edited by D. S. Touretzky, M. C. Mozer, and M. E. Hasselmo (The MIT Press, Cambridge, MA, 1996), p. 218; in International Conference on Artificial Neural Networks 96, edited by C. von der Malsburg, W. von Seelen, J. C. Vorbrüggen, and B. Sendhoff, Lecture Notes in Computer Sciences Vol. 1112 (Springer, Berlin, 1996), p. 551.
- K. H. Fischer and J. A. Hertz, Spin Glasses (Cambridge University Press, Cambridge, 1991).
- S. Bös and M. Opper, in Advances in Neural Information Processing Systems 9, edited by M. C. Mozer, M. I. Jordan, and T.Petsche (The MIT Press, Cambridge, MA, 1997), p. 141; S. Bös and J. Opper, J. Phys. A (to be published).
- B. Widrow and M. E. Hoff, Jr., in 1960 IRE WESCON Convention Record, Part 4 (IRE, New York, 1960), p. 96.
- H. S. Seung, H. Sompolinsky, and N. Tishby, Phys. Rev. A 45, 6056 (1992); S. Amari, N. Fujita, and S. Shinomoto, Neural Comput. 4, 605 (1992).
- S. Amari, N. Murata, K.-R. Müller, M. Finke, and H. Yang, IEEE Trans. Neural Netw. 8985 (1997).
- K.-R. Müller, M. Finke, N. Murata, K. Schulten, and S. Amari, Neural Comput. 8, 1085 (1996).