本論文提出了一種程序化生成動作控制器的方法。
我們使用了計算流體力學 (Computational fluid dynamics)方法，模擬流體與水下游泳者間的相互作用，並使用深度強化學習，學習出真實的游泳者的運動動作。
論文中提出的目標函數(Loss function) 以及為其設計的課程 (Curriculum learning)，大幅提升了最佳化的效率和探索出的動作的有效性。
然後，我們實施了策略蒸餾，將學到的游泳策略轉移到神經網路動作控制器上。
最後產出的動作控制器能夠從學習出的游泳策略所產生的資料集中，學習到流體動力學的資訊，並且能夠透過連續當前狀態預測下一幀的狀態產生真實且具互動性的游泳運動。
我們能夠經由此學習出的動作控制器，生成出足夠敏捷且可控制的游泳動作，使其能快速應對使用者所期望的輸入。

We propose an approach to generate the motion controllers procedurally.
The simulation framework uses computational fluid dynamics (CFD) to simulate the two-way coupling interactions with the underwater swimmer and deep reinforcement learning to learn the realistic locomotion of the swimmer. Our proposed loss function and curriculum learning method significantly improve the efficiency of the learning process and the motions' effectiveness during exploration.
In the second stage, we enforce a policy distillation to transfer the learned swimming policy to a neural motion controller. The resulting motion controller learns the dynamics implicitly from the dataset generated through the learned swimming policy and can synthesize realistic interactive animations by predicting the next state at each frame according to the current state.
With the learned motion controller, we are able to generate swimming movements that are more agile and interactable with given user inputs.

[1] X. B. Peng, G. Berseth, K. Yin, and M. Van De Panne, “Deeploco: Dynamic locomotion skills using
hierarchical deep reinforcement learning,” ACM Trans. Graph., vol. 36, Jul 2017.
[2] J. Won, D. Gopinath, and J. Hodgins, “Physics-based character controllers using conditional vaes,”
ACM Trans. Graph., vol. 41, Jul 2022.
[3] L. Fussell, K. Bergamin, and D. Holden, “Supertrack: Motion tracking for physically simulated char-
acters using supervised learning,” ACM Trans. Graph., vol. 40, Dec 2021.
[4] V. Tsounis, M. Alge, J. Lee, F. Farshidian, and M. Hutter, “Deepgait: Planning and control of
quadrupedal gaits using deep reinforcement learning,” 2019.
[5] P. Fernbach, S. Tonneau, and M. Taïx, “Croc: Convex resolution of centroidal dynamics trajectories to
provide a feasibility criterion for the multi contact planning problem,” in 2018 IEEE/RSJ International
Conference on Intelligent Robots and Systems (IROS), pp. 1–9, 2018.
[6] I. Mordatch, K. Lowrey, G. Andrew, Z. Popovic, and E. Todorov, “Interactive control of diverse
complex characters with neural networks,” in Proceedings of the 28th International Conference on
Neural Information Processing Systems - Volume 2, NIPS’15, (Cambridge, MA, USA), p. 3132–
3140, MIT Press, 2015.
[7] L. Liu and J. Hodgins, “Learning basketball dribbling skills using trajectory optimization and deep
reinforcement learning,” ACM Trans. Graph., vol. 37, Jul 2018.
[8] H. Zhang, S. Starke, T. Komura, and J. Saito, “Mode-adaptive neural networks for quadruped motion
control,” ACM Trans. Graph., vol. 37, Jul 2018.
[9] K. Lee, S. Min, S. Lee, and J. Lee, “Learning time-critical responses for interactive character control,”
ACM Trans. Graph., vol. 40, Jul 2021.
[10] D. Holden, O. Kanoun, M. Perepichka, and T. Popa, “Learned motion matching,” ACM Trans. Graph.,
vol. 39, Jul 2020.
[11] S. Starke, I. Mason, and T. Komura, “Deepphase: Periodic autoencoders for learning motion phase
manifolds,” ACM Trans. Graph., vol. 41, Jul 2022.
[12] J. Tan, Y. Gu, G. Turk, and C. K. Liu, “Articulated swimming creatures,” ACM Trans. Graph., vol. 30,
Jul 2011.
[13] S. Min, J. Won, S. Lee, J. Park, and J. Lee, “Softcon: Simulation and control of soft-bodied animals
with biomimetic actuators,” ACM Trans. Graph., vol. 38, Nov 2019.
[14] P. Ma, T. Du, J. Z. Zhang, K. Wu, A. Spielberg, R. K. Katzschmann, and W. Matusik, “DiffAqua,”
ACM Transactions on Graphics, vol. 40, pp. 1–14, Aug 2021.
[15] M. Gazzola, B. Hejazialhosseini, and P. Koumoutsakos, “Reinforcement learning and wavelet adapted
vortex methods for simulations of self-propelled swimmers,” SIAM Journal on Scientific Computing,
vol. 36, no. 3, pp. B622–B639, 2014.
[16] S. Verma, G. Novati, and P. Koumoutsakos, “Efficient collective swimming by harnessing vortices
through deep reinforcement learning,” Proceedings of the National Academy of Sciences, vol. 115,
pp. 5849–5854, May 2018.
[17] D. Satoi, M. Hagiwara, A. Uemoto, H. Nakadai, and J. Hoshino, “Unified motion planner for fishes
with various swimming styles,” ACM Trans. Graph., vol. 35, Jul 2016.
[18] Q. Chen, T. Lu, Y. Tong, G. Luo, X. Jin, and Z. Deng, “A practical model for realistic butterfly flight
simulation,” ACM Trans. Graph., vol. 41, Mar 2022.
[19] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal policy optimization
algorithms,” 2017.
[20] W. Yu, G. Turk, and C. K. Liu, “Learning symmetric and low-energy locomotion,” ACM Transactions
on Graphics, vol. 37, pp. 1–12, Aug 2018.
[21] B. Ramos, F. Trost, and N. Thuerey, “Control of two-way coupled fluid systems with differentiable
solvers,” 2022.