隨著時代的進步光學系統的尺寸也隨之變得越來越小，而隨著光學系統的縮小當中的光學元件也必須隨著一起變小，而需要在如此有限的空間中要達到與以往相同的效果就勢必要使用自由度更高的光學元件如：非球面、XY多項式、圓錐曲線等等，而在非成像光學的領域中也具有相似的問題如：特殊光型的照明、特殊能量的分布等等，要達到如此複雜的目標往往需要多個光學元件調整光型，光束在這些元件中傳遞所造成的能量損耗往往是無法忽略的，而在之中自由曲面具有近乎無限的自由度能夠提供給光學設計，使得系統能夠在有限的空間以及元件數量之中達到以往需要更多元件才能夠達到的目標。
然而自由曲面雖然有著可以縮減光學系統的元件和大小的特性，但是在自由曲面的設計以及製造上往往會遇到許多的困難，首先在自由曲面的設計上可以分為映射方法以及數學方法，其中映射法所設計出的自由曲面精確度高但是往往具有曲面不連續使得在製造與加工上十分困難，而數學方法需要建立光源與目標面的方程式這類方法需要深厚的數學以及光學知識，並且需要花費大量的時間求解複雜的數學方程式。
本研究提出了將自由曲面的光學設計與強化學習互相結合，目的為節省自由曲面光學設計所需要的時間以及降低自由曲面設計的門檻，本文利用強化學習可以自我學習並且不需要準備大量訓練資料的特性可以很好的解決自由曲面發展歷史短且每個自由曲面設計案例間相關性低難以取得大量訓練樣本的問題。並且基於澤爾尼克多項式多用於波前的擬合使用足夠多的項次可以擬和各類複雜的波前是十分適合用於描述自由曲面的工具。

Nowadays, the size of the optical system is getting smaller and smaller. As the optical system shrinks, the optical components must also become smaller. To achieve the same effect in such a limited space, it is necessary to use optical elements with a higher degree of freedom, such as aspheric surfaces, XY polynomials, conic sections, etc. To achieve such a complex goal often requires multiple optical elements to adjust the light shapes, and the energy loss caused by the beam passing through these elements is often not negligible. The freeform surface has almost unlimited degrees of freedom that can provide optical design It enables the system to achieve the goal that requires more components in the past in a limited space and the number of components.
Although the free-form surface has the characteristics of reducing the components and size of the optical system, it often encounters many difficulties in the design and manufacture of the free-form surface. There are mapping methods and mathematical methods in the design of the free-form surface. Among them, the free-form surface designed by the mapping method has high precision but often has a surface discontinuity, which makes it very difficult to manufacture and process, while the mathematical method needs to establish the equation of the light source and the target surface. Such methods require profound mathematical and optical knowledge and require to spending a lot of time solving complex math equations.
This study proposes to combine the optical design of free-form surfaces with reinforcement learning. The purpose is to save the time required for optical design of free-form surfaces and lower the threshold for free-form surface design. In this paper, reinforcement learning can be used for self-learning and does not require the large amount of training data while training the model. And based on Zernike polynomials, it is mostly used for fitting wave fronts. With enough Zernike polynomial’s terms, it can fit various complex wave fronts. It is very suitable for describing free-form surfaces.

[1] P. J. Nyirenda, M. Mulbagal, W. F. J. C.-A. D. Bronsvoort, and Applications, "Definition of freeform surface feature classes," vol. 3, no. 5, pp. 665-674, 2006.
[2] Y. Cheng and Y. J. O. Lu, "A design method of freeform lens for LED based on intensity distribution," Optik, vol. 179, pp. 266-272, 2019.
[3] Z. Feng, B. D. Froese, and R. J. A. o. Liang, "Freeform illumination optics construction following an optimal transport map," Applied Optics, vol. 55, no. 16, pp. 4301-4306, 2016.
[4] N. K. Yadav, J. ten Thije Boonkkamp, and W. J. O. C. IJzerman, "Computation of double freeform optical surfaces using a Monge–Ampère solver: Application to beam shaping," Optics Communication, vol. 439, pp. 251-259, 2019.
[5] S. Wills, "Freeform Optics: Notes from the Revolution," Optics and Photonics News, vol. 28, no. 7, pp. 34-41, 2017/07/01 2017.
[6] J. Ye, L. Chen, X. Li, Q. Yuan, and Z. J. O. E. Gao, "Review of optical freeform surface representation technique and its application," SPIE, vol. 56, no. 11, p. 110901, 2017.
[7] P. Benitez et al., "Advanced freeform optics enabling ultra-compact VR headsets," in Digital Optical Technologies 2017, 2017, vol. 10335, pp. 107-113: SPIE.
[8] Y. Luo, Z. Feng, Y. Han, and H. J. O. e. Li, "Design of compact and smooth free-form optical system with uniform illuminance for LED source," Optics Express, vol. 18, no. 9, pp. 9055-9063, 2010.
[9] Z. Feng, Y. Luo, and Y. J. O. e. Han, "Design of LED freeform optical system for road lighting with high luminance/illuminance ratio," Optics Express, vol. 18, no. 21, pp. 22020-22031, 2010.
[10] X. Mao, H. Li, Y. Han, and Y. Luo, "Polar-grids based source-target mapping construction method for designing freeform illumination system for a lighting target with arbitrary shape," Optics Express, vol. 23, no. 4, pp. 4313-4328, 2015/02/23 2015.
[11] F. Fournier, "Freeform Reflector Design With Extended Sources," 2010.
[12] V. Lakshminarayanan and A. J. J. o. M. O. Fleck, "Zernike polynomials: a guide," Journal of Modern Optics, vol. 58, no. 7, pp. 545-561, 2011.
[13] Z. Zheng, X. Sun, X. Liu, and P. J. D. Gu, "Design of reflective projection lens with Zernike polynomials surfaces," Displays, vol. 29, no. 4, pp. 412-417, 2008.
[14] H. Zhu, Q. Cui, M. Piao, and C. Zhao, Design of a dual-band MWIR/LWIR circular unobscured three-mirror optical system with Zernike polynomial surfaces (SPIE/COS Photonics Asia). SPIE, 2014.
[15] M. A. A. Schroeder, Acustica, "Modulation transfer functions: Definition and measurement," vol. 49, no. 3, pp. 179-182, 1981.
[16] A. Kendall et al., "Learning to drive in a day," in 2019 International Conference on Robotics and Automation (ICRA), 2019, pp. 8248-8254: IEEE.
[17] A. S. Polydoros, L. J. J. o. I. Nalpantidis, and R. Systems, "Survey of model-based reinforcement learning: Applications on robotics," Journal of Intelligent & Robotic Systems vol. 86, no. 2, pp. 153-173, 2017.
[18] O. Vinyals et al., "Starcraft ii: A new challenge for reinforcement learning," 2017.
[19] L. P. Kaelbling, M. L. Littman, and A. W. J. J. o. a. i. r. Moore, "Reinforcement learning: A survey," Journal of Intelligent Research, vol. 4, pp. 237-285, 1996.
[20] L. Kaiser et al., "Model-based reinforcement learning for atari," 2019.
[21] A. L. Strehl, L. Li, E. Wiewiora, J. Langford, and M. L. Littman, "PAC model-free reinforcement learning," in Proceedings of the 23rd international conference on Machine learning, 2006, pp. 881-888.
[22] K. Ciosek, Q. Vuong, R. Loftin, and K. J. A. i. N. I. P. S. Hofmann, "Better exploration with optimistic actor critic," Advances in Neural Information Processing Systems, vol. 32, 2019.
[23] J. Peters and S. Schaal, "Natural Actor-Critic," Neurocomputing, vol. 71, no. 7, pp. 1180-1190, 2008/03/01/ 2008.
[24] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. J. a. p. a. Klimov, "Proximal policy optimization algorithms," 2017.
[25] T. Yang, D. Cheng, and Y. J. O. E. Wang, "Designing freeform imaging systems based on reinforcement learning," Optics Express, vol. 28, no. 20, pp. 30309-30323, 2020.
[26] H. Wang, Z. Zheng, C. Ji, and L. J. J. a. p. a. Guo, "Automated optical multi-layer design via deep reinforcement learning," 2020.
[27] D. Dewey, "Reinforcement learning and the reward engineering principle," in 2014 AAAI Spring Symposium Series, 2014.
[28] B. C. Blasco, J. J. M. Moreno, A. P. Pol, and A. S. J. P. Abad, "Using the R-MAPE index as a resistant measure of forecast accuracy," Psicothema, vol. 25, no. 4, pp. 500-506, 2013.
[29] L. Shapiro, "New ANSI/IEC standard for projectors," in Projection Displays II, 1996, vol. 2650, pp. 209-216: SPIE.