- Physical Review Journals
All JournalsPhysics Magazine
Access Provided by Xinjiang University

Physical Review E

Reuse & Permissions

It is not necessary to obtain permission to reuse this article or its components as it is available under the terms of the Creative Commons Attribution 4.0 International license. This license permits unrestricted use, distribution, and reproduction in any medium, provided attribution to the author(s) and the published article's title, journal citation, and DOI are maintained. Please note that some figures may have been included with permission from other third parties. It is your responsibility to obtain the proper permission from the rights holder directly for these figures.

Export citation

Export citation

Choose format for download:

Download Citation
  • Open Access
  • Access by Xinjiang University

Information-theoretic model selection for optimal prediction of stochastic dynamical systems from data

David Darmon*

  • Department of Military and Emergency Medicine, Uniformed Services University of the Health Sciences, Bethesda, Maryland 20814, USA and The Henry M. Jackson Foundation for the Advancement of Military Medicine, Bethesda, Maryland 20817, USA

  • *david.darmon.ctr@usuhs.edu
PDF

Phys. Rev. E 97, 032206 – Published 16 March, 2018

DOI: https://doi.org/10.1103/PhysRevE.97.032206

Abstract

In the absence of mechanistic or phenomenological models of real-world systems, data-driven models become necessary. The discovery of various embedding theorems in the 1980s and 1990s motivated a powerful set of tools for analyzing deterministic dynamical systems via delay-coordinate embeddings of observations of their component states. However, in many branches of science, the condition of operational determinism is not satisfied, and stochastic models must be brought to bear. For such stochastic models, the tool set developed for delay-coordinate embedding is no longer appropriate, and a new toolkit must be developed. We present an information-theoretic criterion, the negative log-predictive likelihood, for selecting the embedding dimension for a predictively optimal data-driven model of a stochastic dynamical system. We develop a nonparametric estimator for the negative log-predictive likelihood and compare its performance to a recently proposed criterion based on active information storage. Finally, we show how the output of the model selection procedure can be used to compare candidate predictors for a stochastic system to an information-theoretic lower bound.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (79)

  1. N. H. Packard, J. P. Crutchfield, J. D. Farmer, and R. S. Shaw, Phys. Rev. Lett. 45, 712 (1980).
  2. F. Takens, Lect. Notes Math. 898, 366 (1981).
  3. H. Kantz and T. Schreiber, Nonlinear Time Series Analysis, Vol. 7 (Cambridge University Press, Cambridge, 2004).
  4. E. Bradley and H. Kantz, Chaos 25, 097610 (2015).
  5. G. Sugihara, B. Grenfell, R. M. May, P. Chesson, H. Platt, and M. Williamson, Phil. Trans. R. Soc. Lond. B 330, 235 (1990).
  6. A. Garfinkel, M. L. Spano, W. L. Ditto, and J. N. Weiss, Science 257, 1230 (1992).
  7. Y. Jian, S. Silvestri, J. Brown, R. Hickman, and M. Marani, Ecol. Appl. 26, 2611 (2016).
  8. C. Frazier and K. Kockelman, Transp. Res. Rec. 1897, 9 (2004).
  9. P. Shang, X. Li, and S. Kamae, Chaos Solitons Fractals 25, 121 (2005).
  10. M. De Domenico, A. Lima, and M. Musolesi, Pervasive Mob. Comput. 9, 798 (2013).
  11. V. Sreekumar, S. Dennis, I. Doxas, Y. Zhuang, and M. Belkin, PLoS ONE 9, e97166 (2014).
  12. J. Garland and E. Bradley, in International Symposium on Intelligent Data Analysis (Springer, Berlin, 2013), pp. 210–222.
  13. T. Sauer, J. A. Yorke, and M. Casdagli, J. Stat. Phys. 65, 579 (1991).
  14. A. Eftekhari, H. L. Yap, M. B. Wakin, and C. J. Rozell, Phys. Rev. E 97, 022222 (2018).
  15. J. D. Farmer and J. J. Sidorowich, Phys. Rev. Lett. 59, 845 (1987).
  16. J. P. Crutchfield and B. S. McNamara, Complex Syst. 1, 121 (1987).
  17. M. Casdagli, Physica D 35, 335 (1989).
  18. G. Sugihara and R. M. May, Nature 344, 734 (1990).
  19. A. S. Weigend and N. A. Gershenfeld, in Time Series Prediction: Forecasting the Future and Understanding the Past, Santa Fe Institute Studies in the Science of Complexity, Vol. XV, edited by N. A. Gershenfeld and A. S. Weigend (Addison-Welsey, Reading, MA, 1993).
  20. J. Garland, R. G. James, and E. Bradley, Phys. Rev. E 93, 022221 (2016).
  21. M. Casdagli, S. Eubank, J. D. Farmer, and J. Gibson, Physica D 51, 52 (1991).
  22. S. Ørstavik and J. Stark, Phys. Lett. A 247, 145 (1998).
  23. J. Stark, Nonlinear Dynamics and Statistics (Birkhäuser, Boston, MA, 2001), pp. 81–103.
  24. K.-S. Chan and H. Tong, Chaos: A Statistical Perspective (Springer Science & Business Media, New York, 2013).
  25. B. Cheng and H. Tong, J. R. Statist. Soc. B 54, 427 (1992).
  26. M. Ragwitz and H. Kantz, Phys. Rev. E 65, 056201 (2002).
  27. G. Claeskens and N. Hjort, Model Selection and Model Averaging (Cambridge University Press, Cambridge, 2008), pp. 22–65 and 70–241.
  28. J. T. Lizier, M. Prokopenko, and A. Y. Zomaya, Inf. Sci. 208, 39 (2012).
  29. H. Tong, Non-Linear Time Series: A Dynamical System Approach (Oxford University Press, Oxford, 1990), pp. 152 and 178.
  30. S. Caires and J. A. Ferreira, Stat. Inf. Stoch. Process. 8, 151 (2005).
  31. K.-S. Chan and H. Tong, Bernoulli 8, 1 (2002).
  32. J. P. Crutchfield and D. P. Feldman, Chaos 13, 25 (2003).
  33. R. W. Yeung, IEEE Trans. Inf. Theor. 37, 466 (1991).
  34. A. Kraskov, H. Stögbauer, and P. Grassberger, Phys. Rev. E 69, 066138 (2004).
  35. J. T. Lizier, Front. Robot. AI 1, 11 (2014).
  36. W. Gao, S. Oh, and P. Viswanath, in 2017 IEEE International Symposium on Information Theory (ISIT) (IEEE, Los Alamitos, CA, 2017), pp. 1267–1271.
  37. E. N. Lorenz, J. Atmos. Sci. 26, 636 (1969).
  38. D. Darmon, Entropy 18, 190 (2016).
  39. Z. Lincheng and L. Zhijun, Acta Math. Sin. 1, 314 (1985).
  40. J. Fan and Q. Yao, Nonlinear Time Series: Nonparametric and Parametric Methods (Springer Science & Business Media, New York, 2003).
  41. J. A. Nelder and R. Mead, Comput. J. 7, 308 (1965).
  42. M. Box, Comput. J. 8, 42 (1965).
  43. S. G. Johnson, The nlopt nonlinear-optimization package, version 2.4.2, http://ab-initio.mit.edu/nlopt.
  44. L. Devroye, L. Gyorfi, A. Krzyzak, and G. Lugosi, Ann. Stat. 22, 1371 (1994).
  45. D. Darmon, sidpy: Python Tools for Specific Information Dynamics, https://doi.org/10.5281/zenodo.1186296.
  46. B. F. Finkenstädt, Q. Yao, and H. Tong, Stat. Comp. 11, 229 (2001).
  47. D. Holstein and H. Kantz, Phys. Rev. E 79, 056202 (2009).
  48. L. Wasserman, All of Statistics: A Concise Course in Statistical Inference (Springer Science & Business Media, New York, 2013).
  49. K.-S. Chan and H. Tong, J. R. Statist. Soc. B 56, 301 (1994).
  50. M. Korzen and S. Jaroszewicz, J. Stat. Soft. 57, 1 (2014).
  51. H. Tong and K.-S. Lim, J. R. Statist. Soc. B 42, 245 (1980).
  52. P. A. P. Moran, Aust. J. Zool. 1, 291 (1953).
  53. T. M. Cover and J. A. Thomas, Elements of Information Theory (John Wiley & Sons, New York, 2012), p. 663.
  54. H. Singh, N. Misra, V. Hnizdo, A. Fedorowicz, and E. Demchuk, Am. J. Math. Manage. Sci. 23, 301 (2003).
  55. R. Chen and R. S. Tsay, J. Am. Stat. Assoc. 88, 955 (1993).
  56. S. N. Wood, J. Am. Stat. Assoc. 99, 673 (2004).
  57. R. McGill, J. W. Tukey, and W. A. Larsen, Am. Stat. 32, 12 (1978).
  58. J. D. Hart, J. R. Statist. Soc. B 53, 173 (1991).
  59. E. N. Lorenz, J. Atmos. Sci. 20, 130 (1963).
  60. P. Grassberger and I. Procaccia, Physica D 9, 189 (1983).
  61. J. Stark, D. S. Broomhead, M. E. Davies, and J. Huke, J. Nonlinear Sci. 13, 519 (2003).
  62. A. Rößler, SIAM J. Numer. Anal. 48, 922 (2010).
  63. M. Lindner, R. Vicente, V. Priesemann, and M. Wibral, BMC Neurosci. 12, 119 (2011).
  64. M. Casdagli, J. R. Statist. Soc. B 54, 303 (1992).
  65. Q. Yao and H. Tong, Physica D 115, 49 (1998).
  66. F. Roux, M. Wibral, W. Singer, J. Aru, and P. J. Uhlhaas, J. Neurosci. 33, 17827 (2013).
  67. P. Wollstadt, M. Martínez-Zarzuela, R. Vicente, F. J. Díaz-Pernas, and M. Wibral, PLoS ONE 9, e102833 (2014).
  68. C. Gómez, J. T. Lizier, M. Schaum, P. Wollstadt, C. Grützner, P. Uhlhaas, C. M. Freitag, S. Schlitt, S. Bölte, R. Hornero et al., Front. Neuroinform. 8 (2014).
  69. G. Gómez-Herrero, W. Wu, K. Rutanen, M. C. Soriano, G. Pipa, and R. Vicente, Entropy 17, 1958 (2015).
  70. C. A. Skarda and W. J. Freeman, Behav. Brain Sci. 10, 161 (1987).
  71. W. S. Pritchard, D. W. Duke, and K. K. Krieble, Psychophysiology 32, 486 (1995).
  72. J. Theiler and P. E. Rapp, Electroencephalogr. Clin. Neurophysiol. 98, 213 (1996).
  73. R. G. Andrzejak, K. Lehnertz, F. Mormann, C. Rieke, P. David, and C. E. Elger, Phys. Rev. E 64, 061907 (2001).
  74. B. Cheng and H. Tong, Phil. Trans. R. Soc. Lond. A 348, 325 (1994).
  75. F. Hamilton, T. Berry, and T. Sauer, Phys. Rev. X 6, 011021 (2016).
  76. F. Hamilton, T. Berry, and T. Sauer, The European Physical Journal Special Topics 226, 3239 (2017).
  77. T. Schreiber, Phys. Rev. Lett. 85, 461 (2000).
  78. D. Darmon and P. E. Rapp, Phys. Rev. E 96, 022121 (2017).
  79. B. Cheng and H. Tong, in Stochastic and Spatial Structures of Dynamical Systems: Proceedings of the Colloquium, edited by S. van Strien and S. Verduyn Lunel (Koninklijke Nederlandse Akademie van Wetenschappen, Amsterdam, 1996), pp. 29–37.

Outline

Information

Phys. Rev. E 97, 032206– Published 16 March, 2018
  • Received 21 November 2017
Crossmark - Check for updates button
Creative Commons Logo

Published by the American Physical Society under the terms of the Creative Commons Attribution 4.0 International license. Further distribution of this work must maintain attribution to the author(s) and the published article's title, journal citation, and DOI.

Published by the American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation