- Physical Review Journals
All JournalsPhysics Magazine
Access Provided by Xinjiang University

Physical Review Fluids

Export citation

Export citation

Choose format for download:

Download Citation
  • Access by Xinjiang University

Turbulence control for drag reduction through deep reinforcement learning

Taehyuk Lee and Junhyuk Kim*

Changhoon Lee†

  • Department of Mechanical Engineering, Yonsei University, Seoul 03722, Korea

  • Department of Mechanical Engineering, Yonsei University, Seoul 03722, Korea and School of Mathematics and Computing, Yonsei University, Seoul 03722, Korea

  • *Present address: Korea Atomic Energy Research Institute, Daejeon 34057, Korea.
  • †clee@yonsei.ac.kr
PDF

Phys. Rev. Fluids 8, 024604 – Published 8 February, 2023

DOI: https://doi.org/10.1103/PhysRevFluids.8.024604

Abstract

Deep reinforcement learning (DRL) was applied to turbulence control for drag reduction in direct numerical simulations of turbulent channel flow. Based on the wall shear stress information only, the DRL is capable of determining the optimal distribution of wall blowing and suction, which can reduce drag by 20%, comparable with previous wall-shear-based controls such as suboptimal control [Lee et al., J. Fluid Mech. 358, 245 (1998)] and neural-network-based control [Lee et al., Phys. Fluids 9, 1740 (1997)]. However, our DRL-based control can determine the optimal amplitude of the wall actuation, which was not possible in previous controls. More importantly, from an analysis of the optimal actuation fields, two distinct types of drag reduction mechanisms are identified: the first cancels the near-wall sweep and ejection events, whereas the second mechanism suppresses the streamwise vortices near the wall, which is similar to that of the suboptimal control. This study demonstrates the successful application of DRL to turbulence control and its physical interpretation.

View figure in article

Physics Subject Headings (PhySH)

Article Text

References (47)

  1. J. Kim, Physics and control of wall turbulence for drag reduction, Philos. Trans. R. Soc. A 369, 1396 (2011).
  2. S. L. Brunton and B. R. Noack, Closed-loop turbulence control: Progress and challenges, Appl. Mech. Rev. 67, 050801 (2015).
  3. J. Rabault, M. Kuchta, A. Jensen, U. Réglade, and N. Cerardi, Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control, J. Fluid Mech. 865, 281 (2019).
  4. H. Choi, P. Moin, and J. Kim, Active turbulence control for drag reduction in wall-bounded flows, J. Fluid Mech. 262, 75 (1994).
  5. C. M. Ho and Y. C. Tai, Review: MEMS and its application to flow control, J. Fluids Eng. 118, 437 (1996).
  6. C. Lee, J. Kim, D. Babcock, and R. Goodman, Application of neural networks to turbulence control for drag reduction, Phys. Fluids 9, 1740 (1997).
  7. C. Lee, J. Kim, and H. Choi, Suboptimal control of turbulent channel flow for drag reduction, J. Fluid Mech. 358, 245 (1998).
  8. L. Löfdahl and M. Gad-el-Hag, MEMS-based pressure and shear stress sensors for turbulent flows, Meas. Sci. Technol. 10, 665 (1999).
  9. J.-I. Choi and H. J. Sung, Assessment of suboptimal control for drag reduction in turbulent channel flow, J. Turbul. 3, N29 (2002).
  10. H. Rebbeck and K. S. Choi, A wind-tunnel experiment on real-time opposition control of turbulence, Phys. Fluids 18, 035103 (2006).
  11. N. Kasagi, Y. Suzuki, and K. Fukagata, Microelectromechanical systems–based feedback control of turbulence for skin friction reduction, Annu. Rev. Fluid Mech. 41, 231 (2009).
  12. J. Kim and C. Lee, Deep unsupervised learning of turbulence for inflow generation at various Reynolds numbers, J. Comput. Phys. 406, 109216 (2020).
  13. J. Kim and C. Lee, Prediction of turbulent heat transfer using convolutional neural networks, J. Fluid Mech. 882, A18 (2020).
  14. H. Kim, J. Kim, S. Won, and C. Lee, Unsupervised deep learning for super-resolution reconstruction of turbulence, J. Fluid Mech. 910, A29 (2021).
  15. J. N. Kutz, Deep learning in fluid dynamics, J. Fluid Mech. 814, 1 (2017).
  16. M. P. Brenner, J. D. Eldredge, and J. B. Freund, Perspective on machine learning for advancing fluid mechanics, Phys. Rev. Fluids 4, 100501 (2019).
  17. K. Duraisamy, G. Iaccarino, and H. Xiao, Turbulence modeling in the age of data, Annu. Rev. Fluid Mech. 51, 357 (2019).
  18. S. L. Brunton, B. R. Noack, and P. Koumoutsakos, Machine learning for fluid mechanics, Annu. Rev. Fluid Mech. 52, 477 (2020).
  19. B.-Z. Han and W.-X. Huang, Active control for drag reduction of turbulent channel flow based on convolutional neural networks, Phys. Fluids 32, 095108 (2020).
  20. J. Park and H. Choi, Machine-learning-based feedback control for drag reduction in a turbulent channel flow, J. Fluid Mech. 904, A24 (2020).
  21. L. P. Kaelbling, M. L. Littman, and A. W. Moore, Reinforcement learning: A survey, J. Artif. Intell. Res. 4, 237 (1996).
  22. N. S. Nise, Control Systems Engineering (Wiley, Hoboken, NJ, 2020).
  23. H. Tang, J. Rabault, A. Kuhnle, Y. Wang, and T. Wang, Robust active flow control over a range of Reynolds numbers using an artificial neural network trained through deep reinforcement learning, Phys. Fluids 32, 053605 (2020).
  24. H. Ghraieb, J. Viquerat, A. Larcher, P. Meliga, and E. Hachem, Single-step deep reinforcement learning for open-loop control of laminar and turbulent flows, Phys. Rev. Fluids 6, 053902 (2021).
  25. K. Gustavsson, L. Biferale, A. Celani, and S. Colabrese, Finding efficient swimming strategies in a three-dimensional chaotic flow by reinforcement learning, Eur. Phys. J. E 40, 110 (2017).
  26. J. Viquerat, J. Rabault, A. Kuhnle, H. Ghraieb, A. Larcher, and E. Hachem, Direct shape optimization through deep reinforcement learning, J. Comput. Phys. 428, 110080 (2021).
  27. G. Beintema, A. Corbetta, L. Biferale, and F. Toschi, Controlling Rayleigh–Bénard convection via reinforcement learning, J. Turbul. 21, 585 (2020).
  28. P. Garnier, J. Viquerat, J. Rabault, A. Larcher, A. Kuhnle, and E. Hachem, A review on deep reinforcement learning for fluid mechanics, Comput. Fluids 225, 104973 (2021).
  29. G. Novati, H. L. de Laroussilhe, and P. Koumoutsakos, Automating turbulence modelling by multi-agent reinforcement learning, Nat. Mach. Intell. 3, 87 (2021).
  30. J. Kim, H. Kim, J. Kim, and C. Lee, Deep reinforcement learning for large-eddy simulation modeling in wall-bounded turbulence, Phys. Fluids 34, 105132 (2022).
  31. D. Fan, L. Yang, Z. Wang, M. S. Triantafyllou, and G. E. Karniadakis, Reinforcement learning for bluff body active flow control in experiments and simulations, Proc. Natl. Acad. Sci. USA 117, 26091 (2020).
  32. T. Sonoda, Z. Liu, T. Itoh, and Y. Hasegawa, Reinforcement learning of control strategies for reducing skin friction drag in a fully developed channel flow, arXiv:2206.15355.
  33. R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction (MIT Press, Cambridge, MA, 2018).
  34. T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra, Continuous control with deep reinforcement learning, arXiv:1509.02971.
  35. S. Fujimoto, H. Hoof, and D. Meger, Addressing function approximation error in actor-critic methods, in Proceedings of the 35th International Conference on Machine Learning (PMLR, 80, 2018), pp. 1587–1596.
  36. J. Rabault and A. Kuhnle, Accelerating deep reinforcement learning strategies of flow control through a multi-environment approach, Phys. Fluids 31, 094105 (2019).
  37. https://github.com/taehyuklee/TurbulenceControlCode.
  38. D. Silver, G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller, Deterministic policy gradient algorithms, in Proceedings of the 31st International Conference on Machine Learning (PMLR, 32(1), 2014), pp. 387–395.
  39. M. T. Wojnowicz, The Ornstein-Uhlenbeck process in neural decision-making: Mathematical foundations and simulations suggesting the adaptiveness of robustly integrating stochastic neural evidence, Ph.D. thesis, University of Washington, 2013.
  40. S. B. Pope, Turbulent Flows (Cambridge University Press, Cambridge, UK, 2000).
  41. M. Plappert, R. Houthooft, P. Dhariwal, S. Sidor, R. Y. Chen, X. Chen, T. Asfour, P. Abbeel, and M. Andrychowicz, Parameter space noise for exploration, arXiv:1706.01905.
  42. V. Belus, J. Rabault, J. Viquerat, Z. Che, E. Hachem, and U. Reglade, Exploiting locality and translational invariance to design effective deep reinforcement learning control of the 1-dimensional unstable falling liquid film, AIP Adv. 9, 125014 (2019).
  43. A. Cahill, Catastrophic forgetting in reinforcement-learning environments, Ph.D. thesis, University of Otago, 2011.
  44. J. Kim and P. Moin, The structure of the vorticity field in turbulent channel flow. part 2. study of ensemble-averaged fields, J. Fluid Mech. 162, 339 (1986).
  45. J. M. Wallace, H. Eckelmann, and R. S. Brodkey, The wall region in turbulent shear flow, J. Fluid Mech. 54, 39 (1972).
  46. R. J. Adrian, Hairpin vortex organization in wall turbulence, Phys. Fluids 19, 041301 (2007).
  47. T. R. Bewley, P. Moin, and R. Temam, DNS-based predictive control of turbulence: An optimal benchmark for feedback algorithms, J. Fluid Mech. 447, 179 (2001).

Outline

Information

Phys. Rev. Fluids 8, 024604– Published 8 February, 2023
  • Received 13 August 2022
  • Accepted 20 January 2023
Crossmark - Check for updates button

©2023 American Physical Society

Sign In to Your Journals Account

Filter

Filter

Article Lookup

Enter a citation