- Open Access
- Access by Xinjiang University
Information temperature and macroscopic descriptors of additive -order Markov chains
Phys. Rev. E 114, 014116 – Published 10 July, 2026
DOI: https://doi.org/10.1103/tdml-7tk4
Abstract
Large-scale language models (LLMs) operate in extremely high-dimensional state spaces, where both token embeddings and their hidden representations create complex dependences that are not easily reduced to classical Markov structures. In this paper we explore a theoretically solvable approximation of LLM dynamics using -order additive Markov chains. Such models allow the conditional probability of the next token to be decomposed into a superposition of contributions from multiple historical depths, reducing the combinatorial explosion typically associated with high-order Markov processes. The main result of the work is the establishment of a correspondence between an additive multistep chain and a chain with a stepwise memory function. This correspondence allows the introduction of the concept of information temperature not only for stepwise but also for additive -order Markov chains.
Physics Subject Headings (PhySH)
Article Text
References (13)
- W. X. Zhao et al., A survey of large language models, arXiv:2303.18223.
- M. Phuong and M. Hutter, Formal algorithms for transformers, arXiv:2207.09238.
- O. Zekri, A. Odonnat, A. Benechehab, L. Bleistein, N. Boullé, and I. Redko, Large language models as Markov chains, arXiv:2410.02724.
- O. V. Usatenko, S. S. Melnyk, G. M. Pritula, and V. A. Yampol'skii, Information entropy and temperature of the binary Markov chains, Phys. Rev. E 106, 034127 (2022).
- O. V. Usatenko and G. M. Pritula, Information temperature as a measure of complexity of random symbolic sequences, Chaos Soliton. Fract. 202, 117452 (2025).
- O. V. Usatenko and V. A. Yampolskii, Binary -step Markov chains and long-range correlated systems, Phys. Rev. Lett. 90, 110601 (2003).
- Z. A. Mayzelis, S. S. Apostolov, S. S. Melnyk, O. V. Usatenko, and V. A. Yampol'skii, Additive -step Markov chains as prototype model of symbolic stochastic dynamical systems with long-range correlations, Chaos Soliton. Fract. 34, 112 (2007).
- O. V. Usatenko, S. S. Apostolov, Z. A. Mayzelis, and S. S. Melnik, Random Finite-Valued Dynamical Systems: Additive Markov Chain Approach, Kharkov Series in Physics and Mathematics (Cambridge Scientific, Cambridge, 2010), Vol. 1.
- S. S. Melnyk, O. V. Usatenko, and V. A. Yampol'skii, Memory functions of the additive Markov chains: Applications to complex dynamic systems, Physica A 361, 405 (2006).
- C. W. Gardiner, Handbook of Stochastic Methods for Physics, Chemistry, and the Natural Sciences (Springer, Berlin, 1985).
- S. S. Apostolov, Z. A. Mayzelis, O. V. Usatenko, and V. A. Yampol'skii, Equivalence of the Markov chains and two-sided symbolic sequences, Europhys. Lett. 76, 1015 (2006).
- C. E. Shannon and W. Weaver, The Mathematical Theory of Communication (University of Illinois Press, Urbana, 1949).
- O. V. Usatenko, V. A. Yampol'skii, K. E. Kechedzhy, and S. S. Mel'nyk, Symbolic stochastic dynamical systems viewed as binary -step Markov chains, Phys. Rev. E 68, 061107 (2003).
