在本論文中，我們採用端到端的深度學習應用在自動駕駛領域。近期的文獻中[1]，已經探索並驗證了深度神經網絡（DNN）模仿人類駕駛的可能性。本篇論文主要的目標為僅使用一單眼攝影機學習自動駕駛，並在非結構化的路徑(如凌亂的走廊或越野環境)中進行導航。論文中探討現有的DNN 架構並採用不同的數據處理策略，包括訓練資料集平衡，資料預處理和資料增量技術。為了客觀的評估訓練結果，我們使用Udacity 提供的開源駕駛模擬器[2]。藉由實際佈署訓練好的模型於模擬器中，來量化自動駕駛的穩定性與準確性。在一般駕駛情況下，駕駛在直線道路上的機會遠高於轉彎的情境，因此這會產生不平衡的訓練資料集。如此不平衡的訓練資料會讓學習系統傾向直行駕駛而容易在轉彎處失控。在我們的研究中，我們提出有意義的資料增量技術: 透過投影延伸中央相機的影像來模擬左右駕駛影像，進而創造出更多轉彎的駕駛數據。隨後透過簡單的車輛運動模型，來修正方向盤角度給模擬出來的左右影像。我們的實驗證實，所提出的資料增量技術有助於神經網絡模型提高自動駕駛的穩定度。在我們提出的學習策略中，我們提出一個新的目標函數用於增加車輛駕駛在急轉彎穩定度。採用新的目標函數後，神經網絡可以學習到代表性不足的急轉彎情景。這使我們的系統能夠高速行駛在賽道上，甚至可以在狹窄蜿蜒的山路上自動駕駛。最後，在本研究中我們實作了一台1 比10 的小型自動駕駛車輛系統。駕駛平台基於遙控卡車，最高時速可達40公里/小時。上面搭載NVIDIA TX2 和ROS 系統，用於感知環境和控制車輛。我們從人行道和校園走廊來收集數據。在現實世界的資料中，我們訓練了一個深度卷積神經網絡（CNN）控制器，用於展示端到端深度學習應用於自動駕駛的可行性。

In this thesis, learning methodologies for end-to-end autonomous driving is considered. Previous work has demonstrated the possibility of modern deep neural networks (DNNs) to mimic human driving [1]. We aim our study at exploring the use of a monocular camera for driving in complex and unstructured paths such as messy hallways or off-road trails. Existing driving DNN architecture is employed with different data manipulation strategies which consists of dataset balancing, data preprocessing and data augmentation. To the best of evaluation, a series of experiments are conducted on the Udacity driving simulator [2] for quantifying the course following accuracy. Usually, we driving more straight line then taking the turn. This create unbalanced driving dataset. In our study, meaningful data augmentation is considered to create more driving data by simulating left and right driving image. With appropriately assigned steering commands, augmented driving data help neural network model improve the robustness of autonomous driving. In the proposed learning methodology, the new objective function is employed to increase sharp turn stability. With new objective function, our driving neural network will learn under-represented sharp-turn scenario. This makes our system able to drive on racing track and even narrow mountain road. Finally, a 1/10 scale autonomous driving vehicle system is also considered in this study. Our driving platform is based on a short track racer that reaches a maximum speed of 40 km/h. It features a off-the-shelf hardware and ROS computer for perception and control. We collected data from outdoor tracks on the sidewalk and campus hallway. In real-world driving, we set up a reflexive Convolutional Neural Network (CNN) controller for demonstrating the platform for end-to-end training.

[1] M. Bojarski, D. D. Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. D. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang, J. Zhao, and K. Zieba, “End to end learning for self-driving cars,” Computing Research Repository (CoRR), vol. abs/1604.07316, 2016. [Online]. Available: http://arxiv.org/abs/1604.07316
[2] Udacity, “Udacity’s self-driving car simulator.” [Online]. Available: https://github.com/udacity/self-driving-car-sim
[3] R. Wallace, A. T. Stentz, C. Thorpe, H. Moravec, W. R. L. Whittaker, and T. Kanade, “First results in robot road-following,” in Proceedings of the International Joint Conference on Artificial Intelligence, January 1985.
[4] The Robotics Institute., “Navlab: CMU Navigation Laboratory.” http://www.cs.cmu.edu/afs/cs/project/alv/www/index.html, 2014.
[5] D. A. Pomerleau, “Alvinn: An autonomous land vehicle in a neural network,”
pp. 305–313, 1989. [Online]. Available: http://papers.nips.cc/paper/95-alvinn-an-autonomous-land-vehicle-in-a-neural-network.pdf
[6] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” in Advances in Neural Information Processing Systems (NIPS), F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, Eds., pp. 1097–1105. Curran Associates, Inc., 2012. [Online]. Available: http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks.pdf
[7] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” International Conference on Learning Representations (ICLR), 2014.
[8] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
[9] G. E. Hinton, S. Osindero, and Y.-W. Teh, “A fast learning algorithm for deep belief nets,” Neural Comput., vol. 18, no. 7, pp. 1527–1554, Jul. 2006. [Online]. Available: http://dx.doi.org/10.1162/neco.2006.18.7.1527
[10] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” in Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
[11] P. Sermanet, D. Eigen, X. Zhang, M. Mathieu, R. Fergus, and Y. LeCun, “Overfeat: Integrated recognition, localization and detection using convolutional networks,” Computing Research Repository (CoRR), vol. abs/1312.6229, 2013.
[12] B. Huval, T. Wang, S. Tandon, J. Kiske, W. Song, J. Pazhayampallil, M. Andriluka, P. Rajpurkar, T. Migimatsu, R. Cheng-Yue, F. Mujica, A. Coates, and A. Y. Ng, “An empirical evaluation of deep learning on highway driving,” Computing Research Repository (CoRR), vol. abs/1504.01716, 2015.
[13] F. Codevilla, M. Müller, A. Dosovitskiy, A. López, and V. Koltun, “End-to-end driving via conditional imitation learning,” in International Conference on Robotics and Automation (ICRA), 2018.
[14] M. Quigley, K. Conley, B. P. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, and A. Y. Ng, “ROS: an open-source robot operating system,” in ICRA Workshop on Open Source Software, 2009.
[15] A. Kendall, M. Grimes, and R. Cipolla, “PoseNet: A convolutional network for real-time 6-DOF camera relocalization,” in Proceedings of the International Conference on Computer Vision (ICCV), 2015.
[16] S. Brahmbhatt, J. Gu, K. Kim, J. Hays, and J. Kautz, “Geometry-aware learning of maps for camera localization,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[17] C. Hubschneider, A. Bauer, J. Doll, M. Weber, S. Klemm, F. Kuhnt, and J. M. Zöllner, “Integrating end-to-end learned steering into probabilistic autonomous driving,” in 2017 IEEE 20th International Conference on Intelligent Transportation Systems (ITSC), pp. 1–7, Oct 2017.
[18] A. Carvalho, Y. Gao, A. Gray, H. E. Tseng, and F. Borrelli, “Predictive control of an autonomous ground vehicle using an iterative linearization approach,” in 2013 IEEE 16th International Conference on Intelligent Transportation Systems (ITSC), pp. 2335–2340, 2013.
[19] C. E. Beal and J. C. Gerdes, “Model predictive control for vehicle stabilization at the limits of handling,” IEEE Transactions on Control Systems Technology, vol. 21, no. 4, pp. 1258–1269, July 2013.
[20] J. Levinson, J. Askeland, J. Becker, J. Dolson, D. Held, S. Kammel, J. Z. Kolter, D. Langer, O. Pink, V. R. Pratt, M. Sokolsky, G. Stanek, D. M. Stavens, A. Teichman, M. Werling, and S. Thrun, “Towards fully autonomous driving: Systems and algorithms,” in Intelligent Vehicles Symposium, pp. 163–168. IEEE, 2011.
[21] R. Rajamani, Vehicle Dynamics and Control, ser. Mechanical Engineering Series. Springer US, 2011. [Online]. Available: https://books.google.com.tw/books?id=cZJFDox4KuUC
[22] J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel, “Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition,” Neural Networks, vol. 32, pp. 323–332, 2012.
[23] I. P. Alonso, D. F. Llorca, M. Á. Sotelo, L. M. Bergasa, P. R. de Toro, J. Nuevo, M. Ocaña, and M. Á. G. Garrido, “Combination of feature extraction methods for SVM pedestrian detection,” IEEE Trans. Intelligent Transportation Systems, vol. 8, no. 2, pp. 292–307, 2007.
[24] A. Gurghian, T. Koduri, S. V. Bailur, K. J. Carey, and V. N. Murali, “Deeplanes: End-to-end lane position estimation using deep neural networks,” in Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 38–45. IEEE Computer Society, 2016.
[25] Y. LeCun, E. Cosatto, J. Ben, U. Muller, and B. Flepp, “DAVE: Autonomous off-road vehicle control using end-to-end learning,” Courant Institute/CBLL, Tech. Rep. DARPAIPTO Final Report, 2004.
[26] G. Williams, B. Goldfain, P. Drews, J. M. Rehg, and E. A. Theodorou, “Autonomous racing with autorally vehicles and differential games,” arXiv preprint arXiv:1707.04540, 2017.
[27] H. Robbins and S. Monro, “A stochastic approximation method,” in Herbert Robbins Selected Papers, pp. 102–109. Springer, 1985.
[28] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, “Learning representations by back-propagating errors,” nature, vol. 323, no. 6088, p. 533, 1986.
[29] N. Smolyanskiy, A. Kamenev, J. Smith, and S. Birchfield, “Toward low-flying autonomous mav trail navigation using deep neural networks for environmental awareness,” in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4241–
4247, Sept 2017.
[30] J. Kong, M. Pfeiffer, G. Schildbach, and F. Borrelli, “Kinematic and dynamic vehicle models for autonomous driving control design.” in Intelligent Vehicles Symposium, pp. 1094–1099. IEEE, 2015. [Online]. Available: http://dblp.uni-trier.de/db/conf/ivs/
ivs2015.html#KongPSB15
[31] E. Santana and G. Hotz, “Learning a driving simulator,” Computing Research Repository (CoRR), vol. abs/1608.01230, 2016.
[32] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: A simple way to prevent neural networks from overfitting,” The Journal of Machine Learning Research (JMLR), vol. 15, no. 1, pp. 1929–1958, Jan. 2014. [Online]. Available:
http://dl.acm.org/citation.cfm?id=2627435.2670313
[33] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” Computing Research Repository (CoRR), vol. abs/1412.6980, 2014. [Online]. Available: http://arxiv.org/abs/1412.6980
[34] M. Bojarski, A. Choromanska, K. Choromanski, B. Firner, L. D. Jackel, U. Muller, and K. Zieba, “Visualbackprop: visualizing cnns for autonomous driving,” Computing Research Repository (CoRR), vol. abs/1611.05418, 2016. [Online]. Available: http://arxiv.org/abs/1611.05418
[35] G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter, “Self-normalizing neural networks,” in Advances in Neural Information Processing Systems (NIPS), pp. 971–980, 2017.
[36] A. Krogh and J. A. Hertz, “A simple weight decay can improve generalization,” in Advances in Neural Information Processing Systems 4, J. E. Moody, S. J. Hanson, and R. P. Lippmann, Eds., pp. 950–957. Morgan-Kaufmann, 1992. [Online]. Available:
http://papers.nips.cc/paper/563-a-simple-weight-decay-can-improve-generalization.pdf
[37] Y. LeCun, L. Bottou, G. B. Orr, and K. Müller, “Efficient backprop,” in Neural Networks: Tricks of the Trade (2nd ed.), ser. Lecture Notes in Computer Science, vol. 7700, pp. 9–48. Springer, 2012.