- Physical Review Journals
All JournalsPhysics Magazine
Access Provided by Xinjiang University

Physical Review E

Export citation

Export citation

Choose format for download:

Download Citation
  • Access by Xinjiang University

Autotaxis of a run-and-tumble agent via reinforcement learning

Ramesh Pramanik1, Shradha Mishra2, and Sakuntala Chatterjee1

PDF

Phys. Rev. E 114, 054402 – Published 3 November, 2026

DOI: https://doi.org/10.1103/4xnp-dsn4

Abstract

We study the run-and-tumble motion of a single autochemotactic agent that secretes and senses chemical cues using a reinforcement learning (RL) framework. In a chemical environment that changes dynamically via secretion, diffusion, and degradation of attractant chemicals, we study the effectiveness of RL for a successful navigation strategy. We show that RL enhances chemotactic performance. Starting from a random initial position, the probability of finding the agent in the favorable region (near the attractant peak) increases faster with time and reaches a higher value in the long-time limit when RL is used, compared to the case when the agent performs usual run-and-tumble chemotaxis without using RL. We also characterize the chemotactic performance of the agent by varying different parameters like rates of secretion, degradation, and diffusion. Importantly, we observe a nonmonotonic dependence of the chemotactic performance on the learning and exploration rates, indicating the existence of an optimal RL strategy for the most efficient navigation. We explain these observations as a result of the interplay between the preexisting attractant environment and the agent-generated attractant profile.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (80)

  1. M. Eisenbach, Chemotaxis (Imperial College Press, London, 2004).
  2. H. C. Berg, E. coli in Motion (Springer Science & Business Media, New York, 2008).
  3. J. P. Moore and T. Emonet, Physics of bacterial chemotaxis, Curr. Biol. 34, R972 (2024).
  4. L. Y. Young and R. Mitchell, Negative chemotaxis of marine bacteria to toxic chemicals, Appl. Microbiol. 25, 972 (1973).
  5. P. Vargas, L. Barbier, P. J. Sáez, and M. Piel, Mechanisms for fast cell migration in complex environments, Curr. Opin. Cell Biol. 48, 72 (2017).
  6. B. M. Friedrich and F. Jülicher, Chemotaxis of sperm cells, Proc. Natl. Acad. Sci. USA 104, 13256 (2007).
  7. H. V. Ramírez-Gómez, I. Tuval, A. Guerrero, and A. Darszon, Analysis of sperm chemotaxis, in Echinoderms Part B, edited by A. Hamdoun and K. R. Foltz, Methods in Cell Biology (Academic Press, San Diego, CA, 2019), Vol. 151, pp. 473–486.
  8. U. B. Kaupp, N. D. Kashikar, and I. Weyand, Mechanisms of sperm chemotaxis, Annu. Rev. Physiol. 70, 93 (2008).
  9. M. Eisenbach, Sperm navigation in humans: A concerted action of multiple means, Commun. Biol. 8, 923 (2025).
  10. S. Lange and B. M. Friedrich, Sperm chemotaxis in marine species is optimal at physiological flow rates according theory of filament surfing, PLoS Comput. Biol. 17, e1008826 (2021).
  11. R. Grima, Strong-coupling dynamics of a multicellular chemotactic system, Phys. Rev. Lett. 95, 128103 (2005).
  12. A. Sengupta, S. van Teeffelen, and H. Löwen, Dynamics of a microorganism moving by chemotaxis in its own secretion, Phys. Rev. E 80, 031122 (2009).
  13. Z. Mokhtari, R. I. A. Patterson, and F. Höfling, Spontaneous trail formation in populations of auto-chemotactic walkers, New J. Phys. 24, 013012 (2022).
  14. T. M. Konijn, J. G. Van De Meene, J. T. Bonner, and D. S. Barkley, The acrasin activity of adenosine-3′,5′-cyclic phosphate, Proc. Natl. Acad. Sci. USA 58, 1152 (1967).
  15. T. Gregor, K. Fujimoto, N. Masaki, and S. Sawai, The onset of collective behavior in social amoebae, Science 328, 1021 (2010).
  16. P. V. Afonso, M. Janka-Junttila, Y. J. Lee, C. P. McCann, C. M. Oliver, K. A. Aamer, W. Losert, M. T. Cicerone, and C. A. Parent, LTB4 is a signal-relay molecule during neutrophil chemotaxis, Dev. Cell 22, 1079 (2012).
  17. E. O. Budrene and H. C. Berg, Complex patterns formed by motile cells of Escherichia coli, Nature (London) 349, 630 (1991).
  18. Y. Tsori and P.-G. de Gennes, Self-trapping of a single bacterium in its own chemoattractant, Europhys. Lett. 66, 599 (2004).
  19. J. van der Kolk, F. Raßhofer, R. Swiderski, A. Haldar, A. Basu, and E. Frey, Anomalous collective dynamics of autochemotactic populations, Phys. Rev. Lett. 131, 088201 (2023).
  20. M. B. Amar, Collective chemotaxis and segregation of active bacterial colonies, Sci. Rep. 6, 21269 (2016).
  21. E. Lushi, R. E. Goldstein, and M. J. Shelley, Nonlinear concentration patterns and bands in autochemotactic suspensions, Phys. Rev. E 98, 052411 (2018).
  22. N. Livne, A. Vaknin, and O. Agam, Pattern formation in E. coli through negative chemotaxis: Instability, condensation, and merging, Phys. Rev. Res. 7, 023095 (2025).
  23. M. Mukherjee and P. Ghosh, Growth-mediated autochemotactic pattern formation in self-propelling bacteria, Phys. Rev. E 97, 012413 (2018).
  24. M. Ma, M. Gao, C. Tong, and Y. Han, Chemotaxis-driven pattern formation for a reaction–diffusion–chemotaxis model with volume-filling effect, Comput. Math. Appl. 72, 1320 (2016).
  25. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction (MIT Press, Cambridge, MA, 2018).
  26. L. Panait and S. Luke, Cooperative multi-agent learning: The state of the art, Auton. Agents Multi-Agent Syst. 11, 387 (2005).
  27. L. Busoniu, R. Babuska, and B. De Schutter, A comprehensive survey of multiagent reinforcement learning, IEEE Trans. Syst Man Cybernetics C 38, 156 (2008).
  28. M. Kubat, An Introduction to Machine Learning (Springer, Cham, Switzerland, 2017).
  29. C. J. C. H. Watkins and P. Dayan, Q-learning, Mach. Learn. 8, 279 (1992).
  30. M. Zinkevich, A. Greenwald, and M. Littman, Cyclic equilibria in markov games, in Advances in Neural Information Processing Systems, edited by Y. Weiss, B. Schölkopf, and J. Platt (MIT Press, Cambridge, MA, 2005), Vol. 18.
  31. I. M. de Abril and R. Kanai, Curiosity-driven reinforcement learning with homeostatic regulation, in 2018 International Joint Conference on Neural Networks, IJCNN 2018, Rio de Janeiro, Brazil, July 8-13 (IEEE, New York, 2018), pp. 1–6.
  32. P. Mohan, L. Sharma, and P. Narayan, Optimal path finding using iterative SARSA, in 2021 5th International Conference on Intelligent Computing and Control Systems (ICICCS) (Institute of Electrical and Electronics Engineers (IEEE), Piscataway, New Jersey, USA, 2021), pp. 811–817.
  33. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms, arXiv:1707.06347.
  34. P. B. Sampat, A. Verma, R. Gupta, and S. Mishra, Ordering through learning in two-dimensional Ising spins, Phys. Rev. E 106, 054149 (2022).
  35. A. Kumar, P. K. Mishra, R. Singh, S. Mishra, and D. Giri, Adaptive dynamics of Ising spins in one dimension leveraging reinforcement learning, Phys. Scr. 100, 056009 (2025).
  36. E. O. Neftci and B. B. Averbeck, Reinforcement learning in artificial and biological systems, Nat. Mach. Intell. 1, 133 (2019).
  37. N. J. Treloar, N. Braniff, B. Ingalls, and C. P. Barnes, Deep reinforcement learning for optimal experimental design in biology, PLoS Comput. Biol. 18, e1010695 (2022).
  38. B. Yongacoglu, G. Arslan, and S. Yüksel, Reinforcement learning for decentralized stochastic control, in 2019 IEEE 58th Conference on Decision and Control (CDC) (Institute of Electrical and Electronics Engineers (IEEE), Piscataway, New Jersey, USA, 2019), pp. 5556–5561.
  39. X. Zhu, Y. Wang, and Z. Wu, Reinforcement learning for optimal control of stochastic nonlinear systems, AIChE J. 71, e18840 (2025).
  40. R. Munos and P. Bourgine, Reinforcement learning for continuous stochastic control problems, in Advances in Neural Information Processing Systems, edited by M. I. Jordan, M. J. Kearns, and S. A. Solla (MIT Press, Cambridge, MA, 1997), Vol. 10.
  41. M. Tejedor, A. Z. Woldaregay, and F. Godtliebsen, Reinforcement learning application in diabetes blood glucose control: A systematic review, Artif. Intell. Med. 104, 101836 (2020).
  42. T. J. Rademaker, E. Bengio, and P. François, Attack and defense in cellular decision-making: Lessons from machine learning, Phys. Rev. X 9, 031012 (2019).
  43. M. Durve, F. Peruani, and A. Celani, Learning to flock through reinforcement, Phys. Rev. E 102, 012601 (2020).
  44. H. Hou, T. Gan, Y. Yang, X. Zhu, S. Liu, W. Guo, and J. Hao, Using deep reinforcement learning to speed up collective cell migration, BMC Bioinf. 20, 571 (2019).
  45. R. Pramanik, S. Mishra, and S. Chatterjee, Run-and-tumble chemotaxis using reinforcement learning, Phys. Rev. E 111, 014106 (2025).
  46. N. Tovazzi, G. Muñoz-Gil, and M. Caraglio, Run-and-tumble particles learning chemotaxis, Soft Matter 21, 8886 (2025).
  47. H. C. Berg and D. A. Brown, Chemotaxis in Escherichia coli analysed by three-dimensional tracking, Nature (London) 239, 500 (1972).
  48. S. Dev and S. Chatterjee, Run-and-tumble motion with steplike responses to a stochastic input, Phys. Rev. E 99, 012402 (2019).
  49. P.-G. de Gennes, Chemotaxis: The role of internal delays, Eur. Biophys. J. 33, 691 (2004).
  50. S. Chatterjee, R. A. da Silveira, and Y. Kafri, Chemotaxis when bacteria remember: Drift versus diffusion, PLoS Comput. Biol. 7, e1002283 (2011).
  51. N. Vladimirov, D. Lebiedz, and V. Sourjik, Predicted auxiliary navigation mechanism of peritrichously flagellated chemotactic bacteria, PLoS Comput. Biol. 6, e1000717 (2010).
  52. S. Dev and S. Chatterjee, Optimal methylation noise for best chemotactic performance of E. coli, Phys. Rev. E 97, 032420 (2018).
  53. S. Dev Mandal and S. Chatterjee, Effect of receptor cooperativity on methylation dynamics in bacterial chemotaxis with weak and strong gradient, Phys. Rev. E 105, 014411 (2022).
  54. S. Dev Mandal and S. Chatterjee, Bacterial chemotaxis in a traveling wave attractant environment, J. Stat. Mech. (2025) 063501.
  55. S. Chatterjee, Short time extremal response to step stimulus for a single cell E. coli, J. Stat. Mech. (2022) 123503.
  56. R. Pramanik, R. K. Yadav, and S. Chatterjee, Dynamics of chemoreceptor activity with time-periodic attractant field, Eur. Phys. J. E 48, 60 (2025).
  57. S. Dev Mandal and S. Chatterjee, Effect of switching time scale of receptor activity on chemotactic performance of Escherichia coli, Indian J. Phys. 96, 2619 (2022).
  58. S. D. Mandal and S. Chatterjee, Effect of receptor clustering on chemotactic performance of E. coli: Sensing versus adaptation, Phys. Rev. E 103, L030401, (2021).
  59. M. Li and G. L. Hazelbauer, Cellular stoichiometry of the components of the chemotaxis signaling complex, J. Bacteriol. 186, 3687 (2004).
  60. H. C. Berg and P. M. Tedesco, Transient response to chemotactic stimuli in Escherichia coli, Proc. Natl. Acad. Sci. USA 72, 3235 (1975).
  61. Y. Kafri and R. A. da Silveira, Steady-state chemotaxis in Escherichia coli, Phys. Rev. Lett. 100, 238101 (2008).
  62. S. Dev and S. Chatterjee, Optimal search in E. coli chemotaxis, Phys. Rev. E 91, 042714 (2015).
  63. J. E. Segall, S. M. Block, and H. C. Berg, Temporal comparisons in bacterial chemotaxis, Proc. Natl. Acad. Sci. USA 83, 8987 (1986).
  64. S. M. Block, J. E. Segall, and H. C. Berg, Adaptation kinetics in bacterial chemotaxis, J. Bacteriol. 154, 312 (1983).
  65. D. A. Brown and H. C. Berg, Temporal stimulation of chemotaxis in Escherichia coli, Proc. Natl. Acad. Sci. USA 71, 1388 (1974).
  66. R. M. Macnab and D. E. Koshland, The gradient-sensing mechanism in bacterial chemotaxis, Proc. Natl. Acad. Sci. USA 69, 2509 (1972).
  67. M. Putzke and H. Stark, Optimal navigation of a smart active particle: Directional and distance sensing, Eur. Phys. J. E 46, 48 (2023).
  68. P. A. Monderkamp, F. J. Schwarzendahl, M. A. Klatt, and H. Löwen, Active particles using reinforcement learning to navigate in complex motility landscapes, Mach. Learn.: Sci. Technol. 3, 045024 (2022).
  69. H. Man, C. Wang, J.-R. Li, Y. Tian, and S.-G. Chen, Autonomous discovery of the Ising model's critical parameters with reinforcement learning, J. Stat. Mech. (2025) 123410.
  70. A. T. Zai, C. Lorenz, S. Srikantharajah, N. Giret, and R. H. R. Hahnloser, Estimating the motor exploration in reinforcement learning, iScience 29, 114398 (2026).
  71. A. Alonso and J. B. Kirkegaard, Learning optimal integration of spatial and temporal information in noisy chemotaxis, PNAS Nexus 3, pgae235 (2024).
  72. S. Liang, A. N. Singh, Y. Zhu, D. T. Limmer, and C. Yang, Probing reaction channels via reinforcement learning, Mach. Learn.: Sci. Technol. 4, 045003 (2023).
  73. Y. Jia and X. Y. Zhou, Q-learning in continuous time, J. Mach. Learn. Res. 24, 1 (2023).
  74. S. Paramanick, U. Pardhi, H. Soni, and N. Kumar, Spontaneous emergence of run-and-tumble-like dynamics in a robotic analog of Chlamydomonas: Experiment and theory, Phys. Rev. Lett. 135, 168301 (2025).
  75. Z. Li, Q. Cai, X. Zhang, G. Si, Q. Ouyang, C. Luo, and Y. Tu, Barrier crossing in Escherichia coli chemotaxis, Phys. Rev. Lett. 118, 098101 (2017).
  76. M. D. Lazova, T. Ahmed, D. Bellomo, R. Stocker, and T. S. Shimizu, Response rescaling in bacterial chemotaxis, Proc. Natl. Acad. Sci. USA 108, 13870 (2011).
  77. X. Zhu, G. Si, N. Deng, Q. Ouyang, T. Wu, Z. He, L. Jiang, C. Luo, and Y. Tu, Frequency-dependent Escherichia coli chemotaxis behavior, Phys. Rev. Lett. 108, 128101 (2012).
  78. M. Gerhard, A. Jayaram, A. Fischer, and T. Speck, Hunting active Brownian particles: Learning optimal behavior, Phys. Rev. E 104, 054614 (2021).
  79. S. Muiños-Landin, A. Fischer, V. Holubec, and F. Cichos, Reinforcement learning with artificial microswimmers, Sci. Rob. 6, eabd9285 (2021).
  80. Q. Wang and H. Wu, Mathematical modeling of chemotaxis guided amoeboid cell swimming, Phys. Biol. 18, 045001 (2021).

Outline

Information

Phys. Rev. E 114, 054402– Published 3 November, 2026
  • Received 24 February 2026
  • Accepted 9 October 2026
Crossmark - Check for updates button

©2026 American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation