Reinforcement-learning-based control of convectively unstable flows. (English) Zbl 1539.76055
Summary: This work reports the application of a model-free deep reinforcement learning (DRL) based flow control strategy to suppress perturbations evolving in the one-dimensional linearised Kuramoto-Sivashinsky (KS) equation and two-dimensional boundary layer flows. The former is commonly used to model the disturbance developing in flat-plate boundary layer flows. These flow systems are convectively unstable, being able to amplify the upstream disturbance, and are thus difficult to control. The control action is implemented through a volumetric force at a fixed position, and the control performance is evaluated by the reduction of perturbation amplitude downstream. We first demonstrate the effectiveness of the DRL-based control in the KS system subjected to a random upstream noise. The amplitude of perturbation monitored downstream is reduced significantly, and the learnt policy is shown to be robust to both measurement and external noise. One of our focuses is to place sensors optimally in the DRL control using the gradient-free particle swarm optimisation algorithm. After the optimisation process for different numbers of sensors, a specific eight-sensor placement is found to yield the best control performance. The optimised sensor placement in the KS equation is applied directly to control two-dimensional Blasius boundary layer flows, and can efficiently reduce the downstream perturbation energy. Via flow analyses, the control mechanism found by DRL is the opposition control. Besides, it is found that when the flow instability information is embedded in the reward function of DRL to penalise the instability, the control performance can be further improved in this convectively unstable flow.
MSC:
| 76D55 | Flow control and optimization for incompressible viscous fluids |
| 76D10 | Boundary-layer theory, separation and reattachment, higher-order effects |
| 76E15 | Absolute and convective instability and stability in hydrodynamic stability |
| 76M99 | Basic methods in fluid mechanics |
| 68T05 | Learning and adaptive systems in artificial intelligence |
Keywords:
boundary layer control; model-free deep reinforcement machine learning; linearised Kuramoto-Sivashinsky equation; optimised sensor placementSoftware:
Nek5000References:
| [1] | Akervik, E., Hoepffner, J., Ehrenstein, U. & Henningson, D.S.2007Optimal growth, model reduction and control in a separated boundary-layer flow using global eigenmodes. J. Fluid Mech.579, 305-314. · Zbl 1175.76049 |
| [2] | Akhtar, I., Borggaard, J., Burns, J.A., Imtiaz, H. & Zietsman, L.2015Using functional gains for effective sensor location in flow control: a reduced-order modelling approach. J. Fluid Mech.781, 622-656. · Zbl 1359.76110 |
| [3] | Bagheri, S., Henningson, D.S., Hoepffner, J. & Schmid, P.J.2009Input-output analysis and control design applied to a linear model of spatially developing flows. Appl. Mech. Rev.62 (2), 020803. |
| [4] | Beintema, G., Corbetta, A., Biferale, L. & Toschi, F.2020Controlling Rayleigh-Bénard convection via reinforcement learning. J. Turbul.21 (9-10), 585-605. |
| [5] | Belson, B.A., Semeraro, O., Rowley, C.W. & Henningson, D.S.2013Feedback control of instabilities in the two-dimensional Blasius boundary layer: the role of sensors and actuators. Phys. Fluids25 (5), 054106. |
| [6] | Belus, V., Rabault, J., Viquerat, J., Che, Z., Hachem, E. & Reglade, U.2019Exploiting locality and translational invariance to design effective deep reinforcement learning control of the 1-dimensional unstable falling liquid film. AIP Adv.9 (12), 125014. |
| [7] | Blanloeuil, P., Nurhazli, N.A. & Veidt, M.2016 Particle swarm optimization for optimal sensor placement in ultrasonic SHM systems. In Nondestructive Characterization and Monitoring of Advanced Materials, Aerospace, and Civil Infrastructure 2016 (ed. T. Yu, A.L. Gyekenyesi, P.J. Shull & H.F. Wu), p. 98040E. International Society for Optics and Photonics. |
| [8] | Bogatskiy, A., et al.2022 Symmetry group equivariant architectures for physics. arXiv:2203.06153. |
| [9] | Brancher, P. & Chomaz, J.-M.1997Absolute and convective secondary instabilities in spatially periodic shear flows. Phys. Rev. Lett.78 (4), 658. |
| [10] | Brennan, G., Gajjar, J. & Hewitt, R.2021Tollmien-Schlichting wave cancellation via localised heating elements in boundary layers. J. Fluid Mech.909, A16. · Zbl 1461.76204 |
| [11] | Brunton, B.W., Brunton, S.L., Proctor, J.L. & Kutz, J.N.2016Sparse sensor placement optimization for classification. SIAM J. Appl. Maths76 (5), 2099-2122. · Zbl 1350.49037 |
| [12] | Brunton, S.L.2021 Machine learning of dynamics with applications to flow control and aerodynamic optimization. In Advances in Critical Flow Dynamics Involving Moving/Deformable Structures with Design Applications: Proceedings of the IUTAM Symposium on Critical Flow Dynamics Involving Moving/Deformable Structures with Design Applications, June 18-22, 2018, Santorini, Greece (ed. M. Braza, K. Hourigan & M. Triantafyllou), p. 327. Springer Nature. |
| [13] | Brunton, S.L. & Noack, B.R.2015Closed-loop turbulence control: progress and challenges. Appl. Mech. Rev.67 (5), 050801. |
| [14] | Brunton, S.L., Noack, B.R. & Koumoutsakos, P.2020Machine learning for fluid mechanics. Annu. Rev. Fluid Mech.52, 477-508. · Zbl 1439.76138 |
| [15] | Bucci, M.A., Semeraro, O., Allauzen, A., Wisniewski, G., Cordier, L. & Mathelin, L.2019Control of chaotic systems by deep reinforcement learning. Proc. R. Soc. Lond. A475, 20190351. · Zbl 1472.68171 |
| [16] | Burda, Y., Edwards, H., Pathak, D., Storkey, A., Darrell, T. & Efros, A.A.2018 Large-scale study of curiosity-driven learning. arXiv:1808.04355. |
| [17] | Castellanos, R., Cornejo Maceda, G., De La Fuente, I., Noack, B.R., Ianiro, A. & Discetti, S.2022Machine-learning flow control with few sensor feedback and measurement noise. Phys. Fluids34 (4), 047118. |
| [18] | Chen, K.K. & Rowley, C.W.2011H2 optimal actuator and sensor placement in the linearised complex Ginzburg-Landau system. J. Fluid Mech.681, 241-260. · Zbl 1241.76164 |
| [19] | Chen, K.K. & Rowley, C.W.2014 Fluid flow control applications of H2 optimal actuator and sensor placement. In 2014 American Control Conference, pp. 4044-4049. IEEE. |
| [20] | Choi, H., Moin, P. & Kim, J.1994Active turbulence control for drag reduction in wall-bounded flows. J. Fluid Mech.262, 75-110. · Zbl 0800.76191 |
| [21] | Colburn, C.2011 Estimation techniques for large-scale turbulent fluid systems. PhD thesis, UC San Diego. |
| [22] | Corke, T.C., Enloe, C.L. & Wilkinson, S.P.2010Dielectric barrier discharge plasma actuators for flow control. Annu. Rev. Fluid Mech.42, 505-529. |
| [23] | Cvitanović, P., Davidchack, R.L. & Siminos, E.2010On the state space geometry of the Kuramoto-Sivashinsky flow in a periodic domain. SIAM J. Appl. Dyn. Syst.9 (1), 1-33. · Zbl 1267.35027 |
| [24] | Delbende, I. & Chomaz, J.-M.1998Nonlinear convective/absolute instabilities in parallel two-dimensional wakes. Phys. Rev. Fluids10 (11), 2724-2736. · Zbl 1185.76618 |
| [25] | Dergham, G., Sipp, D. & Robinet, J.-C.2011Accurate low dimensional models for deterministic fluid systems driven by uncertain forcing. Phys. Fluids23 (9), 094101. |
| [26] | Duriez, T., Brunton, S.L. & Noack, B.R.2017Machine Learning Control - Taming Nonlinear Dynamics and Turbulence. Springer. · Zbl 1349.76001 |
| [27] | Fabbiane, N., Semeraro, O., Bagheri, S. & Henningson, D.S.2014Adaptive and model-based control theory applied to convectively unstable flows. Appl. Mech. Rev.66 (6), 060801. |
| [28] | Fan, D., Yang, L., Wang, Z., Triantafyllou, M. & Karniadakis, G.2020Reinforcement learning for bluff body active flow control in experiments and simulations. Proc. Natl Acad. Sci. USA117 (42), 26091-26098. |
| [29] | Feng, Z., Wan, D., Zhang, M. & Wang, B.-F.2022Nonlinear spatiotemporal instabilities in two-dimensional electroconvective flows. Phys. Rev. Fluids7 (2), 023701. |
| [30] | Fischer, P., Lottes, J. & Kerkemeier, S.2017 Nek5000 Verison 17.0. Argonne National Laboratory, Illinois. Available: https://nek5000.mcs.anl.gov. |
| [31] | Garnier, P., Viquerat, J., Rabault, J., Larcher, A., Kuhnle, A. & Hachem, E.2021A review on deep reinforcement learning for fluid mechanics. Comput. Fluids225, 104973. · Zbl 1521.76029 |
| [32] | Gautier, N., Aider, J.-L., Duriez, T., Noack, B.R., Segond, M. & Abel, M.2015Closed-loop separation control using machine learning. J. Fluid Mech.770, 442-457. |
| [33] | Giannetti, F. & Luchini, P.2007Structural sensitivity of the first instability of the cylinder wake. J. Fluid Mech.581, 167-197. · Zbl 1115.76028 |
| [34] | Grundmann, S. & Tropea, C.2008Active cancellation of artificially introduced Tollmien-Schlichting waves using plasma actuators. Exp. Fluids44 (5), 795-806. |
| [35] | Güemes, A., Martín, A., Castellanos, R., Oscar, F. & Discetti, S.2022 Active drag reduction in minimal flow units via deep reinforcement learning. In 2022 1st Spanish Fluid Mechanics Conference, June 19-22. |
| [36] | Guéniat, F., Mathelin, L. & Hussaini, M.Y.2016A statistical learning strategy for closed-loop control of fluid flows. Theor. Comput. Fluid Dyn.30 (6), 497-510. |
| [37] | Ha, D. & Schmidhuber, J.2018Recurrent world models facilitate policy evolution. Adv. Neural Inform. Proc. Syst.31, 2455-2467. |
| [38] | Hervé, A., Sipp, D., Schmid, P.J. & Samuelides, M.2012A physics-based approach to flow control using system identification. J. Fluid Mech.702, 26-58. · Zbl 1248.76052 |
| [39] | Hu, W., Morris, K. & Zhang, Y.2016 Sensor location in a controlled thermal fluid. In 2016 IEEE 55th Conference on Decision and Control (CDC), pp. 2259-2264. IEEE. |
| [40] | Huerre, P. & Monkewitz, P.A.1985Absolute and convective instabilities in free shear layers. J. Fluid Mech.159, 151-168. · Zbl 0588.76067 |
| [41] | Huerre, P. & Monkewitz, P.A.1990Local and global instabilities in spatially developing flows. Annu. Rev. Fluid Mech.22 (1), 473-537. · Zbl 0734.76021 |
| [42] | Jamal, R.A. & Morris, K.2015 Output feedback control of the Kuramoto-Sivashinsky equation. In 2015 54th IEEE Conference on Decision and Control (CDC), pp. 567-571. IEEE. |
| [43] | Jin, B., Illingworth, S.J. & Sandberg, R.D.2022Optimal sensor and actuator placement for feedback control of vortex shedding. J. Fluid Mech.932, A2. · Zbl 1509.76029 |
| [44] | Kar, S.K.2006A semi-implicit Runga-Kutta time-difference scheme for the two-dimensional shallow-water equations. Mon. Weath. Rev.134 (10), 2916-2926. |
| [45] | Karniadakis, G.E., Kevrekidis, I.G., Lu, L., Perdikaris, P., Wang, S. & Yang, L.2021Physics-informed machine learning. Nat. Rev. Phys.3 (6), 422-440. |
| [46] | Kennedy, J. & Eberhart, R.1995 Particle swarm optimization. In Proceedings of ICNN’95 - International Conference on Neural Networks, pp. 1942-1948. IEEE. |
| [47] | Khan, T., Morris, K. & Stastna, M.2015 Computation of the optimal sensor location for the estimation of an 1-D linear dispersive wave equation. In 2015 American Control Conference (ACC), pp. 5270-5275. IEEE. |
| [48] | Kim, J. & Bewley, T.R.2007A linear systems approach to flow control. Annu. Rev. Fluid Mech.39, 383-417. · Zbl 1296.76074 |
| [49] | Kim, J., Kim, H., Kim, J. & Lee, C.2022 Deep reinforcement learning for large-eddy simulation modeling in wall-bounded turbulence. arXiv:2201.09505. |
| [50] | Kober, J., Bagnell, J.A. & Peters, J.2013Reinforcement learning in robotics: a survey. Intl J. Robot. Res.32 (11), 1238-1274. |
| [51] | Koizumi, H., Tsutsumi, S. & Shima, E.2010 Feedback control of Kármán vortex shedding from a cylinder using deep reinforcement learning. In 2018 Flow Control Conference. AIAA Paper 2018-3691. |
| [52] | Koziel, S. & Yang, X.-S.2011Computational Optimization, Methods and Algorithms. Springer. · Zbl 1217.90006 |
| [53] | Kuramoto, Y. & Tsuzuki, T.1976Persistent propagation of concentration waves in dissipative media far from thermal equilibrium. Prog. Theor. Phys.55 (2), 356-369. |
| [54] | Lee, C., Kim, J., Babcock, D. & Goodman, R.1997Application of neural networks to turbulence control for drag reduction. Phys. Rev. Fluids9 (6), 1740-1747. |
| [55] | Lewis, F.L., Vrabie, D. & Syrmos, V.L.2012Optimal Control. John Wiley & Sons. · Zbl 1284.49001 |
| [56] | Li, J. & Zhang, M.2022Reinforcement-learning-based control of confined cylinder wakes with stability analyses. J. Fluid Mech.932, A44. · Zbl 1511.76023 |
| [57] | Lillicrap, T.P., Hunt, J.J., Pritzel, A., Heess, N., Erez, T., Tassa, Y., Silver, D. & Wierstra, D.2015 Continuous control with deep reinforcement learning. arXiv:1509.02971. |
| [58] | Ling, J., Kurzawski, A. & Templeton, J.2016Reynolds averaged turbulence modelling using deep neural networks with embedded invariance. J. Fluid Mech.807, 155-166. · Zbl 1383.76175 |
| [59] | Linot, A.J. & Graham, M.D.2020Deep learning to discover and predict dynamics on an inertial manifold. Phys. Rev. E101 (6), 062209. |
| [60] | Loiseau, J.-C., Noack, B.R. & Brunton, S.L.2018Sparse reduced-order modelling: sensor-based dynamics to full-state estimation. J. Fluid Mech.844, 459-490. · Zbl 1461.76369 |
| [61] | Manohar, K., Brunton, B.W., Kutz, J.N. & Brunton, S.L.2018Data-driven sparse sensor placement for reconstruction: demonstrating the benefits of exploiting known patterns. IEEE Control Syst. Mag.38 (3), 63-86. · Zbl 1477.93128 |
| [62] | Manohar, K., Kutz, J.N. & Brunton, S.L.2021 Optimal sensor and actuator selection using balanced model reduction. IEEE Trans. Automat. Contr.67 (4), 2108-2115. · Zbl 1537.93112 |
| [63] | Mehrabian, A.R. & Yousefi-Koma, A.2007Optimal positioning of piezoelectric actuators on a smart fin using bio-inspired algorithms. Aerosp. Sci. Technol.11 (2-3), 174-182. |
| [64] | Miranda, L.J.2018Pyswarms: a research toolkit for particle swarm optimization in Python. J. Open Res. Softw.3 (21), 433. |
| [65] | Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D. & Riedmiller, M.2013 Playing Atari with deep reinforcement learning. arXiv:1312.5602. |
| [66] | Monkewitz, P.A. & Nguyen, L.N.1987Absolute instability in the near-wake of two-dimensional bluff bodies. J. Fluids Struct.1 (2), 165-184. · Zbl 0619.76052 |
| [67] | Natarajan, M., Freund, J.B. & Bodony, D.J.2016Actuator selection and placement for localized feedback flow control. J. Fluid Mech.809, 775-792. · Zbl 1383.76408 |
| [68] | Oehler, S.F. & Illingworth, S.J.2018Sensor and actuator placement trade-offs for a linear model of spatially developing flows. J. Fluid Mech.854, 34-55. · Zbl 1415.76219 |
| [69] | Paris, R., Beneddine, S. & Dandois, J.2021Robust flow control and optimal sensor placement using deep reinforcement learning. J. Fluid Mech.913, A25. · Zbl 1461.76373 |
| [70] | Park, J. & Choi, H.2020Machine-learning-based feedback control for drag reduction in a turbulent channel flow. J. Fluid Mech.904, A24. |
| [71] | Pino, F., Schena, L., Rabault, J., Kuhnle, A. & Mendez, M.A.2022 Comparative analysis of machine learning methods for active flow control. arXiv:2202.11664v2. |
| [72] | Pivot, C., Mathelin, L., Cordier, L., Guéniat, F. & Noack, B.R.2017 A continuous reinforcement learning strategy for closed-loop control in fluid dynamics. In 35th AIAA Applied Aerodynamics Conference. AIAA Paper 2017-3566. |
| [73] | Rabault, J., Kuchta, M., Jensen, A., Réglade, U. & Cerardi, N.2019Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control. J. Fluid Mech.865, 281-302. · Zbl 1415.76222 |
| [74] | Rabault, J. & Kuhnle, A.2019Accelerating deep reinforcement learning strategies of flow control through a multi-environment approach. Phys. Rev. Fluids31 (9), 094105. |
| [75] | Rabault, J., Ren, F., Zhang, W., Tang, H. & Xu, H.2020Deep reinforcement learning in fluid mechanics: a promising method for both active flow control and shape optimization. J. Hydrodyn.32, 234-246. |
| [76] | Ren, F., Rabault, J. & Tang, H.2021Applying deep reinforcement learning to active flow control in weakly turbulent conditions. Phys. Fluids33 (3), 037121. |
| [77] | Rowley, C.W., Mezić, I., Bagheri, S., Schlatter, P. & Henningson, D.S.2009Spectral analysis of nonlinear flows. J. Fluid Mech.641, 115-127. · Zbl 1183.76833 |
| [78] | Sashittal, P. & Bodony, D.J.2021Data-driven sensor placement for fluid flows. Theor. Comput. Fluid Dyn.35 (5), 709-729. |
| [79] | Schmid, P.J.2010Dynamic mode decomposition of numerical and experimental data. J. Fluid Mech.656, 5-28. · Zbl 1197.76091 |
| [80] | Schmidhuber, J.2015Deep learning in neural networks: an overview. Neural Netw.61, 85-117. |
| [81] | Shimomura, S., Sekimoto, S., Oyama, A., Fujii, K. & Nishida, H.2020Closed-loop flow separation control using the deep Q network over airfoil. AIAA J.58 (10), 4260-4270. |
| [82] | Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D. & Riedmiller, M.2014 Deterministic policy gradient algorithms. In International Conference on Machine Learning (ed. E.P. Xing, & T. Jebara), pp. 387-395. PMLR. |
| [83] | Sipp, D., Marquet, O., Meliga, P. & Barbagallo, A.2010Dynamics and control of global instabilities in open-flows: a linearised approach. Appl. Mech. Rev.63 (3), 030801. |
| [84] | Sipp, D. & Schmid, P.J.2016Linear closed-loop control of fluid instabilities and noise-induced perturbations: a review of approaches and tools. Appl. Mech. Rev.68 (2), 020801. |
| [85] | Sivashinsky, G.I.1977Nonlinear analysis of hydrodynamic instability in laminar flames - I. Derivation of basic equations. Acta Astronaut.4 (11), 1177-1206. · Zbl 0427.76047 |
| [86] | Smidt, T.E., Geiger, M. & Miller, B.K.2021Finding symmetry breaking order parameters with Euclidean neural networks. Phys. Rev. Res.3, L012002. |
| [87] | Sonoda, T., Liu, Z., Itoh, T. & Hasegawa, Y.2022 Reinforcement learning of control strategies for reducing skin friction drag in a fully developed channel flow. arXiv:2206.15355. |
| [88] | Strykowski, P.J. & Sreenivasan, K.R.1990On the formation and suppression of vortex shedding at low Reynolds numbers. J. Fluid Mech.218, 71-107. |
| [89] | Sturzebecher, D. & Nitsche, W.2003Active cancellation of Tollmien-Schlichting instabilities on a wing using multi-channel sensor actuator systems. Intl J. Heat Fluid Flow24 (4), 572-583. |
| [90] | Sutton, S.R. & Barto, A.G.2018Reinforcement Learning: An Introduction. MIT. · Zbl 1407.68009 |
| [91] | Tang, H., Rabault, J., Kuhnle, A., Wang, Y. & Wang, T.2020Robust active flow control over a range of Reynolds numbers using an artificial neural network trained through deep reinforcement learning. Phys. Fluids32 (5), 053605. |
| [92] | Viquerat, J., Meliga, P. & Hachem, E.2021 A review on deep reinforcement learning for fluid mechanics: an update. arXiv:2107.12206. · Zbl 07513859 |
| [93] | Wagiman, K.R., Abdullah, M.N., Hassan, M.Y. & Radzi, N.H.M.2020A new optimal light sensor placement method of an indoor lighting control system for improving energy performance and visual comfort. J. Build. Engng30, 101295. |
| [94] | Xu, H., Zhang, W., Deng, J. & Rabault, J.2020Active flow control with rotating cylinders by an artificial neural network trained by deep reinforcement learning. J. Hydrodyn.32 (2), 254-258. |
| [95] | Yi, T.-H., Li, H.-N. & Gu, M.2011Optimal sensor placement for health monitoring of high-rise structure based on genetic algorithm. Math. Probl. Engng2011, 395101. |
| [96] | Zeng, K. & Graham, M.D.2021Symmetry reduction for deep reinforcement learning active control of chaotic spatiotemporal dynamics. Phys. Rev. E104, 014210. |
| [97] | Zeng, K., Linot, A.J. & Graham, M.D.2022 Data-driven control of spatiotemporal chaos with reduced-order neural ODE-based models and reinforcement learning. arXiv:2205.00579. |
| [98] | Zhang, P., Shen, H. & Zhai, H.2018Machine learning topological invariants with neural networks. Phys. Rev. Lett.120, 066401. |
This reference list is based on information provided by the publisher or from digital mathematics libraries. Its items are heuristically matched to zbMATH identifiers and may contain data conversion errors. In some cases that data have been complemented/enhanced by data from zbMATH Open. This attempts to reflect the references listed in the original paper as accurately as possible without claiming completeness or a perfect matching.
