由於現今人們對光學系統的要求提高，如輕量化、照明品質及特殊功能等，自由曲面的重要性也因此逐年增長。然而，自由曲面的設計是具有挑戰性的。首先，創建自由曲面往往無法從以前的案例中找到合適的設計起始點，因為自由曲面具有高度不對稱性，並且依賴光源、目標及曲面之間的映射關係。其次，自由曲面的設計需要專業的數學和光學背景，這使得自由曲面的設計具有挑戰性並且相當耗時。第三，創建自由曲面經常會遇到不連續和無法製造的問題。
因此，本研究將人工智慧的強化學習與自由曲面的光學環境相結合來解決這些問題。由於自由曲面的發展歷史較短，每個自由曲面系統都有其特殊性，會導致訓練資料不足的問題，所以我們使用強化學習為框架，利用它的學習特點：從學習中試誤，來獲得可訓練的資料。此外，為了克服製造問題和提供可控的參數，我們使用澤尼克多項式來描述自由曲面，因為透過澤尼克多項式各項的線性疊加，可以高度擬合大多數曲面的形狀。
在本研究裡使用了強化學習裡運用在連續動作空間的演算法，來控制自由曲面形狀的變化，並將我們所需的光學環境及光線追跡，運用OpenAI Gym的環境格式進行改寫，以達到和演算法的互動進而生成自由曲面反射罩。

Nowadays, the demand for optical systems is increasing, such as the lightweight, illumination quality, and special functions, so the importance of free-form surfaces is also growing year by year. However, the design of a free-form surface is challenging. Firstly, creating a free-form surface often can't find a suitable starting point from previous cases because the free-form surface is highly asymmetric and relies on the mapping relationship between the light source and the target surface. Secondly, the design of free-form surfaces requires a solid background in mathematics and optics. It makes the creation of free-form surfaces challenging and time-consuming to design. Thirdly, creating a free-form surface often encounters problems of discontinuity and inability to manufacture. Therefore, this research combines artificial intelligence with free-form surface optical design to solve the issues.
Because the development history of the free-form surface is short, each free-form surface system has its particularity, which will lead to the problem of insufficient training data. Therefore, we use the reinforcement learning framework and make use of its learning characteristics: trial and error in learning to obtain the training data. In addition, we use Zernike polynomials to describe free-form surfaces to overcome manufacturing problems and provide controllable parameters. By linear superposition of Zernike polynomials terms, the shapes of most surfaces can be highly fitted.
In our research, we use the continuous action space algorithm of reinforcement learning to control the change of the surface shape of the free-form surface, and the required optical environment and ray tracing are rewritten by using the environmental format of OpenAI Gym, so as to achieve the interaction with the algorithm and generate the required free-form surface reflector.

[1] T. Yang, Y. Duan, D. Cheng, and Y. Wang, “Freeform Imaging Optical System Design: Theories, Development, and Applications,” Guangxue Xuebao/Acta Optica Sinica, vol. 41, no. 1, 2021, doi: 10.3788/AOS202141.0108001.
[2] T. Yang, D. Cheng, and Y. Wang, “Designing freeform imaging systems based on reinforcement learning,” Optics Express, vol. 28, no. 20, 2020, doi: 10.1364/oe.404808.
[3] Y. Cheng and Y. B. Lu, “A design method of freeform lens for LED based on intensity distribution,” Optik (Stuttg), vol. 179, 2019, doi: 10.1016/j.ijleo.2018.10.184.
[4] Florian Fournier, “Freeform Reflector Design With Extended Sources,” College of Optics and Photonics, 2010.
[5] C. Gannon and R. Liang, “Using machine learning to create high-efficiency freeform illumination design tools,” Dec. 2018.
[6] H. Ries and J. Muschaweck, “Tailored freeform optical surfaces,” Journal of the Optical Society of America A, vol. 19, no. 3, 2002, doi: 10.1364/josaa.19.000590.
[7] R. Wu et al., “Freeform illumination design: a nonlinear boundary problem for the elliptic Monge–Ampére equation,” Optics Letters, vol. 38, no. 2, 2013, doi: 10.1364/ol.38.000229.
[8] J. Rubinstein and G. Wolansky, “Reconstruction of optical surfaces from ray data,” Optical Review, vol. 8, no. 4, 2001, doi: 10.1007/s10043-001-0281-4.
[9] J. Hou, H. Li, Z. Zheng, and X. Liu, “Distortion correction for imaging on non-planar surface using freeform lens,” Optics Communications, vol. 285, no. 6, 2012, doi: 10.1016/j.optcom.2011.12.014.
[10] J. C. Miñano, P. Benítez, W. Lin, J. Infante, F. Muñoz, and A. Santamaría, “An application of the SMS method for imaging designs,” Optics Express, vol. 17, no. 26, 2009, doi: 10.1364/oe.17.024036.
[11] F. Duerr, P. Benítez, J. C. Miñano, Y. Meuret, and H. Thienpont, “Analytic design method for optimal imaging: coupling three ray sets using two free-form lens profiles,” Optics Express, vol. 20, no. 5, 2012, doi: 10.1364/oe.20.005576.
[12] T. Yang, J. Zhu, X. Wu, and G. Jin, “Direct design of freeform surfaces and freeform imaging systems with a point-by-point three-dimensional construction-iteration method,” Optics Express, vol. 23, no. 8, 2015, doi: 10.1364/oe.23.010233.
[13] Y. Xu et al., “Artificial intelligence: A powerful paradigm for scientific research,” The Innovation, vol. 2, no. 4. 2021. doi: 10.1016/j.xinn.2021.100179.
[14] H. Wankerl, M. L. Stern, A. Mahdavi, C. Eichler, and E. W. Lang, “Parameterized reinforcement learning for optical system optimization,” Journal of Physics D: Applied Physics, vol. 54, no. 30, 2021, doi: 10.1088/1361-6463/abfddb.
[15] H. Gross et al., “Overview on surface representations for freeform surfaces,” in Optical Systems Design 2015: Optical Design and Engineering VI, 2015, vol. 9626. doi: 10.1117/12.2191255.
[16] J. Ye, L. Chen, X. Li, Q. Yuan, and Z. Gao, “Review of optical freeform surface representation technique and its application,” Optical Engineering, vol. 56, no. 11, 2017, doi: 10.1117/1.oe.56.11.110901.
[17] H. Zhang, J. Lu, R. Liu, and P. Ma, “Smoothing optimization of supporting quadratic surfaces with Zernike polynomials,” 2018. doi: 10.1117/12.2317455.
[18] E. Savio, L. de Chiffre, and R. Schmitt, “Metrology of freeform shaped parts,” CIRP Annals, vol. 56, no. 2, pp. 810–835, 2007, doi: 10.1016/j.cirp.2007.10.008.
[19] X. Jiang, P. Scott, and D. Whitehouse, “Freeform Surface Characterisation - A Fresh Strategy,” CIRP Annals, vol. 56, no. 1, pp. 553–556, 2007, doi: 10.1016/j.cirp.2007.05.132.
[20] J. S. Schruben, “Formulation of a Reflector-Design Problem for a Lighting Fixture,” J Opt Soc Am, vol. 62, no. 12, 1972, doi: 10.1364/josa.62.001498.
[21] K. P. Thompson, P. Benítez, and J. P. Rolland, “Freeform Optical Surfaces: Report from OSA’s First Incubator Meeting,” Optics and Photonics News, vol. 23, no. 9, pp. 32–37, Sep. 2012, doi: 10.1364/OPN.23.9.000032.
[22] F. Z. Fang, X. D. Zhang, A. Weckenmann, G. X. Zhang, and C. Evans, “Manufacturing and measurement of freeform optics,” CIRP Annals - Manufacturing Technology, vol. 62, no. 2, 2013, doi: 10.1016/j.cirp.2013.05.003.
[23] Z. Feng, Y. Luo, and Y. Han, “Design of LED freeform optical system for road lighting with high luminance/illuminance ratio,” Optics Express, vol. 18, no. 21, 2010, doi: 10.1364/oe.18.022020.
[24] S. Wei, Z. Zhu, W. Li, and D. Ma, “Compact freeform illumination optics design by deblurring the response of extended sources,” Optics Letters, vol. 46, no. 11, 2021, doi: 10.1364/ol.425075.
[25] P. Benitez et al., “Advanced freeform optics enabling ultra-compact VR headsets,” in Digital Optical Technologies 2017, 2017, vol. 10335. doi: 10.1117/12.2270317.
[26] L. Gu, D. Cheng, Y. Liu, J. Ni, T. Yang, and Y. Wang, “Design and fabrication of an off-axis four-mirror system for head-up displays,” Applied Optics, vol. 59, no. 16, 2020, doi: 10.1364/ao.392602.
[27] Y. Xie et al., “Optical design and fabrication of an all-aluminum unobscured two-mirror freeform imaging telescope,” Applied Optics, vol. 59, no. 3, 2020, doi: 10.1364/ao.379324.
[28] Y. Luo, Z. Feng, Y. Han, and H. Li, “Design of compact and smooth free-form optical system with uniform illuminance for LED source,” Optics Express, vol. 18, no. 9, 2010, doi: 10.1364/oe.18.009055.
[29] L. P. Kaelbling, M. L. Littman, and A. W. Moore, “Reinforcement learning: A survey,” Journal of Artificial Intelligence Research, vol. 4, 1996, doi: 10.1613/jair.301.
[30] R. S. Sutton and A. G. Barto, “Reinforcement learning: An Introduction Second edition,” Learning, vol. 3, no. 9, 2012.
[31] OpenAI, “Part 2: Kinds of RL Algorithms,” OpenAI Spinning Up. https://spinningup.openai.com/en/latest/spinningup/rl_intro2.html#citations-below
[32] C. J. C. H. Watkins and P. Dayan, “Q-learning,” Machine Learning, vol. 8, no. 3–4, 1992, doi: 10.1007/bf00992698.
[33] V. Mnih et al., “Playing Atari with Deep Reinforcement Learning,” arXiv, Dec. 2013.
[34] V. R. Konda and J. N. Tsitsiklis, “Actor-critic algorithms,” 2000.
[35] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, “Proximal Policy Optimization Algorithms,” Jul. 2017.
[36] G. Brockman et al., “OpenAI Gym,” Jun. 2016, [Online]. Available: https://doi.org/10.48550/arXiv.1606.01540
[37] OpenAI Gym, “https://www.gymlibrary.ml/.”
[38] J. Schwiegerling, “Zernike Polynomials: Table in Cartesian Coordinates,” in Field Guide to Visual and Ophthalmic Optics, 2009. doi: 10.1117/3.592975.p88.
[39] E. B. Lewis, “Control of body segment differentiation in Drosophila by the bithorax gene complex.,” Prog Clin Biol Res, vol. 85 Pt A, 1982, doi: 10.1007/978-1-4419-8981-9_15.