Study on Compressing ConvNets Using Pruning Filters and Layers｜國立臺灣科技大學博碩士論文系統

簡易檢索 / 詳目顯示

回結果列表

研究生：	Muhammad Zulfan Azhari Muhammad Zulfan Azhari
論文名稱：	Study on Compressing ConvNets Using Pruning Filters and Layers Study on Compressing ConvNets Using Pruning Filters and Layers
指導教授：	呂政修 Jenq-Shiou Leu
口試委員:	呂政修 Jenq-Shiou Leu 陳省隆 Hsing-Lung Chen 方文賢 Wen-Hsien Fang 陳郁堂 Yie-Tarng Chen 鄭瑞光 Ray-Guang Cheng
學位類別：	碩士 Master
系所名稱：	電資學院 - 電子工程系 Department of Electronic and Computer Engineering
論文出版年：	2018
畢業學年度：	106
語文別：	英文
論文頁數：	30
中文關鍵詞：	deep learning 、convolutional neural networks 、model compression 、pruning filter 、pruning layers
外文關鍵詞：	deep learning, convolutional neural networks, model compression, pruning filter, pruning layers
相關次數：	點閱：223 下載：1
分享至:	分享至facebook 分享至twitter

查詢本校圖書館目錄查詢臺灣博碩士論文知識加值系統勘誤回報

The development of convolutional neural networks (CNNs) has been reaching state-of-the-art era. It is applied in the various field of science such as computer vision, bioinformatics, and natural language processing (NLP). However, it can’t be denied that the implementation of CNNs requires more resource for storing the CNNs model and computing ability. It becomes a critical issue for implementation CNNs in a device with limited resources such as drone, smartphone, and robot. Model compression can tackle this issue by reducing the model size of CNNs including filter and layer number. Pruning filters is one of common technique to compress the CNNs model. It removes the least important filters within a layer. In this study, we present a method which combines pruning filters and pruning layer to reduce the model of CNNs. We also proposed a method that measures the essential of a layer. We conduct the experiments on AlexNet and VGG16 as the CNNs models. Then, we also utilize CIFAR10 and Caltech-256 as benchmark dataset. Thereafter, we compare to the state-of-the-art existing method. The experimental results show that even though the accuracies are lower than that of an existing method, our proposed method is still able to compress the AlexNet until 48.04% of the original while for VGG16, we can compress it until 28.61% of the original.

ABSTRACT    i
ACKNOWLEDGEMENTS    ii
CONTENTS    iv
LIST OF FIGURES    vi
LIST OF TABLES    vii
LIST OF EQUATIONS    viii
CHAPTER 1 INTRODUCTION    1
1.1    Research Background    1
1.2    Research Objectives    3
1.3    Outline and Report    3
CHAPTER 2 LITERATURE REVIEW    5
2.1    Model Compression Methods    5
2.2    Pruning Filters    6
2.3    Deep Convolutional Neural Networks Models    7
2.3.1    AlexNet    7
2.3.2    VGG16    8
CHAPTER 3 METHODOLOGY    10
3.1    Pruning Filter Steps    10
3.2    Proposed Method    13
3.2.1    Pruning Layers on AlexNet    15
3.2.2    Pruning Layers on VGG16    16
CHAPTER 4 EXPERIMENTAL RESULTS    18
4.1    Datasets    18
4.2    Environment Setup    19
4.3    Evaluation of AlexNet    19
4.3.1    Evaluation of Pruning Layers on AlexNet    19
4.3.2    Evaluation Result of the proposed method on AlexNet    20
4.4    Evaluation of VGG16    23
4.4.1    Evaluation of Pruning Layers on VGG16    23
4.4.2    Evaluation Result of the proposed method on VGG16    24
CHAPTER 5 CONCLUSION AND FUTURE WORKS    26
5.1    Conclusion    26
5.2    Future Works    26
REFERENCES    27


                                

[1]. M. v. Gerven and S. Bothe, eds. (2018). Artificial Neural Networks as Models of Neural Information Processing. Lausanne: Frontiers Media. doi: 10.3389/978-2-88945-401-3.
[2]. Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based Learning Applied to Document Recognition”, in Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, 1998.
[3]. A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet Classification with Deep Convolutional Neural Networks”, in Neural Information Processing Systems 25th, vol. 1, pp. 1097-1105, 2012.
[4]. M. D. Zeiler and R. Fergus, “Visualizing and Understanding Convolutional Networks”, arXiv preprint arXiv: 1311.2901, 2013.
[5]. K. Simonyan and A. Zisserman, “Very Deep Convolutional for Large-Scale Image Recognition”, arXiv preprint arXiv: 1409.1556, 2014.
[6]. L. J. Ba and R. Caruana, “Do Deep Nets Really Need to be Deep?”, arXiv preprint arXiv:1312.6184, 2013.
[7]. Y. L. Cun, J. S. Denker, and S. A. Solla. “Optimal Brain Damage”, in Proceedings of the 2nd International Conference on Neural Information Processing Systems, pp. 598-605, 1989.
[8]. Z. Tong and G. Tanaka, “A Pruning Method Based on Weight Variation Information for Feedforward Neural Networks’, in IFAC-PapersOnline, vol. 48, no. 8, pp. 221-226, 2015.
[9]. S. Han, H. Mao, and W. J. Dally, “Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding”, arXiv preprint arXiv: 150.00149, 2015.
[10]. H. Li, A. Kadav, I. Durdanovic, H. Samet, H. P. Graf, “Pruning Filters for Efficient ConvNets”, arXiv preprint arXiv: 1608.08710, 2017.
[11]. Y. Cheng, D. Wang, P. Zhou, and T. Zhang, “A Survey of Model Compression and Acceleration for Deep Neural Networks”, arXiv preprint arXiv: 1710.09282, 2017.
[12]. S. Gupta, A. Agrawal, K. Gopalakrishnan, and P. Narayanan, “Deep Learning with Limited Numerical Precision”, in Proceedings of the 32nd International Conference on International Conference on Machine Learning, vol. 37, pp. 1737–1746, 2015.
[13]. Y. Choi, M. El-Khamy, and J. Lee, “Towards the Limit of Network Quantization”, arXiv preprint arXiv: 1612.01543, 2017.
[14]. S. J. Hanson and L. Y. Pratt, “Comparing Biases for Minimal Network Construction with Back-Propagation”, in Proceedings of the 1st International Conference on Neural Information Processing Systems, pp. 177-185. 1988.
[15]. R. Rigamonti, A. Sironi, V. Lepetit, and P. Fua, “Leaning Separable Filters”, in 2013 IEEE Conference on Computer Vision and Pattern Recognition. 2013.
[16]. E. Denton, W. Zaremba, j. Bruna, Y. LeCun, and Rob Fergus, “Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation”, arXiv preprint arXiv: 1404.0736, 2014.
[17]. T. S. Cohen and M. Welling, “group Equivariant Convolutional Networks”, arXiv preprint arXiv: 1602.07576, 2016.
[18]. G. Hinton, O. Vinyals, and Jeff Dean. “Distilling the Knowledge in a Neural Network”, arXiv preprint arXiv: 1503.02531, 2015.
[19]. Y. Gong, L. Liu, M. Yang, and L. Bourdev, “Compressing Deep Convolutional Network using Vector Quantization”, arXiv preprint arXiv: 1412.6115, 2014.
[20]. J. Wu, C. Leng, Y. Wang, Q. Hu, and J. Cheng, “Quantized Convolutional Neural Networks for Mobile Devices”, arXiv preprint arXiv: 1512.06473, 2015.
[21]. V. Vanhoucke, A. Senior, and M. Z. Mao, “Improving the speed of neural networks on CPUs”, in Deep Learning and Unsupervised Feature Learning NIPS Workshop, 2011.
[22]. S. Anwar, K. Hwang, and W. Sung, “Structured Pruning of Deep Convolutional Neural Networks”, arXiv preprint arXiv: 1512.08571, 2015.
[23]. V. Nair and G. E. Hinton, “Rectified Linear Units Improve Restricted Boltzmann Machines”, in Proceedings of the 27th International Conference on International Conference on Machine Learning, 2010.
[24]. X. Dong, S. Chen, and S. J. Pan, “Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon”, arXiv preprint arXiv: 1705.07565, 2017.
[25]. A. Aghasi, A. Abdi, N. Nguyen, and J. Romberg, “Net-Trim: Convex Pruning of Deep Neural Networks with Performance Guarantee”, arXiv preprint arXiv: 1611.05162, 2017.
[26]. G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov. “Improving Neural Networks by Preventing Co-Adaptation of Feature Detectors”. arXiv preprint arXiv:1207.0580, 2012.
[27]. G. Griffin, A. Holub, and P. Perona, “Caltech-256 Object Category Dataset”, CalTech Report, 2007.
[28]. M. D. Zeiler, “ADADELTA: An Adaptive Learning Rate Method,” arXiv preprint arXiv: 1212.5701, 2012.

簡易檢索 / 詳目顯示

相關論文