在雨天環境下，雨滴沾黏在影像擷取設備的鏡頭上，會造成影像的模糊和退化。因此，除雨相關研究已成為近年來機器視覺研究的重要主題之一。除雨技術不僅可以改善影像品質，還可以被應用於多個不同的領域，如自動駕駛和監控系統。
本論文採用卷積神經網路深度學習架構來消除因雨滴沾黏鏡頭而產生模糊和退化的影像。我們利用透明度(Transparency)差異來標記影像中因雨滴沾黏鏡頭而產生的模糊區塊，並透過深度光譜空間學習(Deep Spectral-Spatial Learning, DSLS)模型來學習如何去除這些區塊。在實驗中，我們使用現有開源資料庫上的合成影像，意即在原始照片上隨機生成雨痕或雨滴；並在實際雨天環境，具有高光源和低光源多種不同場景下進行影像採樣，以進行訓練及測試。採用暗通道先驗(Dark Channel Prior)的方式進行影像顏色還原，以提高影像中特徵的辨識度，使得在不同的光線環境下，各種影像皆有被還原的可行性。實驗結果顯示本方法有很好的還原效果。經過還原的影像，在生成影像測試集中，其平均峰值信噪比(Peak Signal-to-Noise Ratio, PSNR)達到31.8758 dB，其平均結構相似度指數(Structural Similarity Index, SSIM)達到0.9437。在實際採用高畫質的真實雨滴影像進行訓練測試時，其平均PSNR達到26.7055 dB，其平均SSIM達到0.8549。
本論文的主要貢獻是消除影像中因鏡頭沾黏雨滴而造成模糊和退化，將模糊影像還原成清晰影像。此方法可應用在汽車主動安全系統和自動駕駛輔助的功能。未來的研究方向，可著重於改進深度卷積神經網路模型，使其能夠處理更加複雜的影像，並提高還原效果和速度。

In rainy environments, raindrops stick to the lens of the image capture device, which will cause blurring and degradation of the image. Therefore, the research related to rain removal has become one of the important topics in machine vision research in recent years.
The rain removal technique not only can improve image quality, but also can be applied to many different fields, such as autonomous driving and surveillance systems.
This paper uses a convolutional neural network deep-learning architecture to remove blurred and degraded images caused by raindrops sticking to the lens. We use the transparency difference to label the blurred regions in the image caused by raindrops sticking to the lens, and learn how to remove these blocks through the Deep Spectral-Spatial Learning (DSLS) model. In our experiments, we use synthetic images from existing open-source databases, meaning that rain streaks or drops are randomly generated on the original photos.
Moreover, in the actual rainy environment, image sampling is carried out in various scenes with high light source and low light source for training and testing. The Dark Channel Prior method is used to restore the image color to improve the recognition of the features in the image, so that it is feasible to restore various images under different light environments.
The experimental results show that this method has a good reduction effect. For the restored image, in the generated image test set, its average Peak Signal-to-Noise Ratio (PSNR) reached 31.8758 dB, and its average Structural Similarity Index (SSIM) reached 0.9437. When actually using high-quality real raindrop images for training and testing, its average PSNR reaches 26.7055 dB, and its average SSIM reaches 0.8549.
The main contribution of this thesis is to eliminate the blurring and degradation caused by the lens sticking to the raindrops in the image, so as to restore a blurry image to a clear image. This method will be applicable to the functions of automotive active safety system and automatic driving assistance. Future research directions can be focused on improving the deep convolutional neural network model so that it can handle more complex images and improve the restoration effect and speed.

參考文獻
D. Berman, D. Levy, S. Avidan, and T. Treibitz, “Underwater single image color restoration using haze-lines and a new quantitative dataset,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 43, no. 8, pp. 2822-2837, 2021, DOI: 10.1109/TPAMI.2020.2977624.
W. Luo, S. Duan, and J. Zheng, “Underwater image restoration and enhancement based on a fusion algorithm with color balance, contrast optimization, and histogram stretching,” IEEE Access, vol. 9, pp. 31792-31804, 2021, DOI: 10.1109/ACCESS. 2021.3060947.
Y. T. Peng, K. Cao, and P. C. Cosman, “Generalization of the dark channel prior for single image restoration,” IEEE Transactions on Image Processing, vol. 27, no. 6, pp. 2856-2868, 2018, DOI: 10.1109/TIP.2018.2813092.
M. Han, Z. Lyu, T. Qiu, and M. Xu, “A review on intelligence dehazing and color restoration for underwater images,” IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 50, no. 5, pp. 1820-1832, May 2020, DOI: 10.1109/ TSMC.2017.2788902.
C. O. Ancuti, C. Ancuti, C. D. Vleeschouwer, and M. Sbert, “Color channel compensation (3C): A fundamental pre-processing step for image enhancement,” IEEE Transactions on Image Processing, vol. 29, no. 7, pp. 2653-2665, 2020, DOI: 10.1109/ TIP.2019.2951304.
X. Ding, Y. Wang, and X. Fu, “An image dehazing approach with adaptive color constancy for poor visible conditions,” IEEE Geoscience and Remote Sensing Letters, vol. 19, no. 4, pp. 1-5, 2022, DOI: 10.1109/LGRS.2021.3134020.
S. Kanti Dhara, M. Roy, D. Sen, and P. Kumar Biswas, “Color cast dependent image dehazing via adaptive airlight refinement and non-linear color balancing,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 5, pp. 2076-2081, 2021, DOI: 10.1109/TCSVT.2020.3007850.
W. Wang, Z. Li, S. Wu, and L. Zeng, “Hazy image decolorization with color contrast restoration,” IEEE Transactions on Image Processing, vol. 29, no. 3, pp. 1776-1787, 2020, DOI: 10.1109/TIP.2019.2939946.
D. Singh, M. Kaur, M. Y. Jabarulla, V. Kumar, and H. N. Lee, “Evolving fusion-based visibility restoration model for hazy remote sensing images using dynamic differential evolution,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, no. 5, pp. 1-14, 2022, DOI: 10.1109/TGRS.2022.3155765.
T. H. Park and I. K. Eom, “Sand-dust image enhancement using successive color balance with coincident chromatic histogram,” IEEE Access, vol. 9, pp. 19749-19760, 2021, DOI: 10.1109/ACCESS.2021.3054899.
T. Matsui and M. Ikehara, “GAN-based rain noise removal from single-image considering rain composite models,” IEEE Access, vol. 8, pp. 40892-40900, 2020, DOI: 10.1109/ACCESS.2020.2976761.
T. T. Gong and J. S. Wang, “Wavelet based deep recursive pyramid convolution residual network for single image rain removal,” IEEE Access, vol. 8, pp. 195870-195882, 2020, DOI: 10.1109/ACCESS.2020.3034238.
C. Y. Lin, Z. Tao, A. S. Xu, L.W. Kang, and F. Akhyar, “Sequential dual attention network for rain streak removal in a single image,” IEEE Transactions on Image Processing, vol. 29, no. 3, pp. 9250-9265, 2020, DOI: 10.1109/TIP.2020.3025402.
X. Liang and F. Zhao, “Single-image rain removal network based on an attention mechanism and a residual structure,” IEEE Access, vol. 10, pp. 52472-52480, 2022, DOI: 10.1109/ACCESS.2022.3175196.
X. Bi and J. Xing, “Multi-scale weighted fusion attentive generative adversarial network for single image de-raining,” IEEE Access, vol. 8, pp. 69838-69848, 2020, DOI: 10.1109/ACCESS.2020.2983436.
Y. T. Wang, X. L. Zhao, T. X. Jiang, L. J. Deng, Y. Chang, and T. Z. Huang, “Rain streaks removal for single image via kernel-guided convolutional neural network,” IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 8, pp. 3664-3676, 2021, DOI: 10.1109/TNNLS.2020.3015897.
K. H. Lee, E. Ryu, and J. O. Kim, “Progressive rain removal via a recurrent convolutional network for real rain videos,” IEEE Access, vol. 8, pp. 203134-203145, 2020, DOI: 10.1109/ACCESS.2020.3036680.
A. Khodja, Z. Zheng, J. Mo, D. Zhang, and L. Chen, “Rain to rain: Learning real rain removal without ground truth,” IEEE Access, vol. 9, pp. 57325-57337, 2021, DOI: 10.1109/ACCESS.2021.3072687.
Q. Wu, L. Wang, K. N. Ngan, H. Li, F. Meng, and L. Xu, “Subjective and objective de-raining quality assessment towards authentic rain image,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 30, no. 11, pp. 3883-3897, 2020, DOI: 10.1109/TCSVT.2020.2972566.
Y. Que, S. Li, and H. J. Lee, “Attentive composite residual network for robust rain removal from single images,” IEEE Transactions on Multimedia, vol. 23, no. 2, pp. 3059-3072, 2021, DOI: 10.1109/TMM.2020.3019680.
K. Zhang, D. Li, W. Luo, and W. Ren, “Dual attention-in-attention model for joint rain streak and raindrop removal,” IEEE Transactions on Image Processing, vol. 30, no. 4, pp. 7608-7619, 2021, DOI: 10.1109/TIP.2021.3108019.
W. Luo, J. Lai, and X. Xie, “Weakly supervised learning for raindrop removal on a single image,” IEEE Transactions on Circuits and Systems for Video Technology, vol. 31, no. 5, pp. 1673-1683, 2021, DOI: 10.1109/TCSVT.2020.3014267.
M. W. Shao, L. Li, D. Y. Meng, and W. M. Zuo, “Uncertainty guided multi-scale attention network for raindrop removal from a single image,” IEEE Transactions on Image Processing, vol. 30, no. 3, pp. 4828-4839, 2021, DOI: 10.1109/ TIP.2021.3076283.
W. Yan, L. Xu, W. Yang, and R. T. Tan, “Feature-aligned video raindrop removal with temporal constraints,” IEEE Transactions on Image Processing, vol. 31, no. 4, pp. 3440-3448, 2022, DOI: 10.1109/TIP.2022.3170726.
Y. Guo, J. Chen, X. Ren, A. Wang, and W. Wang, “Joint raindrop and haze removal from a single image,” IEEE Transactions on Image Processing, vol. 29, no. 2, pp. 9508-9519, 2020, DOI: 10.1109/TIP.2020.3029438.
T. Yang, X. Chang, H. Su, N. Crombez, Y. Ruichek, T. Krajnik, and Z. Yan, “Raindrop removal with light field image using image inpainting,” IEEE Access, vol. 8, pp. 58416-58426, 2020, DOI: 10.1109/ACCESS.2020.2981641.
N. M. Alsharay, Y. Chen, O. A. Dobre, and O. De Silva, “Improved sea-ice identification using semantic segmentation with raindrop removal,” IEEE Access, vol. 10, pp. 21599-21607, 2022, DOI: 10.1109/ACCESS.2022.3150969.
S. Ezumi and M. Ikehara, “Single image raindrop removal using a non-local operator and feature maps in the frequency domain,” IEEE Access, vol. 10, pp. 91976-91983, 2022, DOI: 10.1109/ACCESS.2022.3202888.
Z. Han, L. Li, W. Jin, X. Wang, G. Jiao, and H. Wang, “Convolutional neural network training for RGBN camera color restoration using generated image pairs,” IEEE Photonics Journal, vol. 12, no. 5, pp. 1-15, 2020, DOI: 10.1109/JPHOT.2020.3025088.
K. Cui, A. Boev, E. Alshina, and E. Steinbach, “Color image restoration exploiting inter-channel correlation with a 3-stage CNN,” IEEE Journal of Selected Topics in Signal Processing, vol. 15, no. 2, pp. 174-189, 2021, DOI: 10.1109/JSTSP. 2020.3043148.
S. M. Aghito and S. Forchhammer, “Efficient coding of shape and transparency for video objects,” IEEE Transactions on Image Processing, vol. 16, no. 9, pp. 2234-2244, 2007, DOI: 10.1109/TIP.2007.903902.
S. Alletto, C. Carlin, L. Rigazio, Y. Ishii, and S. Tsukizawa, “Adherent raindrop removal with self-supervised attention maps and spatiotemporal generative adversarial networks,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. Workshop (ICCVW), Seoul, Korea (South), October 27-28, 2019, pp. 2329–2338.
X. Liu, M. Suganuma, Z. Sun, and T. Okatani, “Dual residual networks leveraging the potential of paired operations for image restoration,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), Long Beach, CA, USA, June 15-20, 2019, pp. 7007–7016.
H. Zhang and Y. Li, “Spectral-spatial classification of hyperspectral imagery based on deep convolutional network,” in Proc. International Conference on Orange Technologies (ICOT), Melbourne, VIC, Australia, December 18-20, 2016, pp. 44-47, DOI: 10.1109/ICOT.2016.8278975.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278-2324, 1998, DOI: 10.1109/5.726791.
R. S. Sutton, A. G. Barto, and R. J. Williams, “Reinforcement learning is direct adaptive optimal control,” IEEE Control Systems Magazine, vol. 12, no. 2, pp. 19-22, 1992, DOI: 10.1109/37.126844.
A. Johnson, T. Moher, S. Ohlsson, and M. Gillingham, “The round earth project: Deep learning in a collaborative virtual world,” in Proc. IEEE Virtual Reality, Houston, TX, USA, March 13-17, 1999, pp. 164-171, DOI: 10.1109/VR.1999.756947.
T. Gajecki and W. Nogueira, “An end-to-end deep learning speech coding and denoising strategy for cochlear implants,” in Proc. IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), Singapore, Singapore, May 23-27, 2022, pp. 3109-3113, DOI: 10.1109/ICASSP43922. 2022. 9746963.
C. Fan, J. Yi, J. Tao, Z. Tian, B. Liu, and Z. Wen, “Gated recurrent fusion with joint training framework for robust end-to-end speech recognition,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 198-209, 2021, DOI: 10.1109/TASLP.2020.3039600.
L. Chai, J. Du, Q. F. Liu, and C. H. Lee, “A cross-entropy-guided measure (CEGM) for assessing speech recognition performance and optimizing DNN-based speech enhancement,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, no.7, pp. 106-117, 2021, DOI: 10.1109/TASLP.2020.3036783.
W. Lin and G. Chen, “Large memory capacity in chaotic artificial neural networks: A view of the anti-integrable limit,” IEEE Transactions on Neural Networks, vol. 20, no. 8, pp. 1340-1351, 2009, DOI: 10.1109/TNN.2009.2024148.
A. L. P. Tay, J. M. Zurada, L. P. Wong, and J. Xu, “The hierarchical fast learning artificial neural network (HieFLANN)—An autonomous platform for hierarchical neural network construction,” IEEE Transactions on Neural Networks, vol. 18, no. 6, pp. 1645-1657, 2007, DOI: 10.1109/TNN.2007.900231.
G. P. J. Schmitz, C. Aldrich, and F. S. Gouws, “ANN-DT: An algorithm for extraction of decision trees from artificial neural networks,” IEEE Transactions on Neural Networks, vol. 10, no. 6, pp. 1392-1401, 1999, DOI: 10.1109/72.809084.
H. Shao, “Delay-dependent stability for recurrent neural networks with time-varying delays,” IEEE Transactions on Neural Networks, vol. 19, no. 9, pp. 1647-1651, 2008, DOI: 10.1109/TNN.2008.2001265.
Y. Xia and J. Wang, “A recurrent neural network for solving nonlinear convex programs subject to linear constraints,” IEEE Transactions on Neural Networks, vol. 16, no. 2, pp. 379-386, 2005, DOI: 10.1109/TNN.2004.841779.
Z. Zuo, C. Yang, and Y. Wang, “A new method for stability analysis of recurrent neural networks with interval time-varying delay,” IEEE Transactions on Neural Networks, vol. 21, no. 2, pp. 339-344, 2010, DOI: 10.1109/TNN.2009.2037893.
Q. Wang, H. Fan, L. Zhu, and Y. Tang, “Deeply supervised face completion with multi-context generative adversarial network,” IEEE Signal Processing Letters, vol. 26, no. 3, pp. 400-404, 2019, DOI: 10.1109/LSP.2018.2890205.
Y. Shi, Q. Li, and X. X. Zhu, “Building footprint generation using improved generative adversarial networks,” IEEE Geoscience and Remote Sensing Letters, vol. 16, no. 4, pp. 603-607, 2019, DOI: 10.1109/LGRS.2018.2878486.
B. Mandal, N. B. Puhan, and A. Verma, “Deep convolutional generative adversarial network-based food recognition using partially labeled data,” IEEE Sensors Letters, vol. 3, no. 2, pp. 1-4, 2019, DOI: 10.1109/LSENS.2018.2886427.
X. Xu, T. Gao, Y. Wang, and X. Xuan, “Event temporal relation extraction with attention mechanism and graph neural network,” Tsinghua Science and Technology, vol. 27, no. 1, pp. 79-90, 2022, DOI: 10.26599/TST.2020.9010063.
X. Xiang, X. Zhang, and H. Chen, “A convolutional network with multi-scale and attention mechanisms for end-to-end single-channel speech enhancement,” IEEE Signal Processing Letters, vol. 28, pp. 1455-1459, 2021, DOI: 10.1109/LSP. 2021.3093859.
W. Cai and Z. Wei, “Remote sensing image classification based on a cross-attention mechanism and graph convolution,” IEEE Geoscience and Remote Sensing Letters, vol. 19, no.3, pp. 1-5, 2022, DOI: 10.1109/LGRS.2020.3026587.
K. Ahiska, M. K. Ozgoren, and M. K. Leblebicioglu, “Autopilot design for vehicle cornering through icy roads,” IEEE Transactions on Vehicular Technology, vol. 67, no. 3, pp. 1867-1880, 2018, DOI: 10.1109/TVT.2017.2765245.
A. Mahmood, Y. Kim, and J. Park, “Robust H^∞ autopilot design for agile missile with time-varying parameters,” IEEE Transactions on Aerospace and Electronic Systems, vol. 50, no. 4, pp. 3082-3089, 2014, DOI: 10.1109/TAES.2014.130750.
B. Panchal, K. Subramanian, and S. E. Talole, “Robust missile autopilot design using two time-scale separation,” IEEE Transactions on Aerospace and Electronic Systems, vol. 54, no. 3, pp. 1499-1510, 2018, DOI: 10.1109/TAES.2018.2796654.
R. Qian, R. T. Tan, W. Yang, J. Su, and J. Liu, “Attentive generative adversarial network for raindrop removal from a single image,” in Proc. Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, June 18-23, 2018, pp. 2482-2491, DOI: 10.1109/CVPR.2018.00263.