簡易檢索 / 詳目顯示

回結果列表
研究生： 蕭育宜
Yu-Yi Hsiao
論文名稱： 零樣本學習應用於強化低光照遙測影像之研究
Zero-Shot Learning Applied to Enhance Low-Light Telemetry Images
指導教授： 陳俊良
Jiann-Liang Chen
口試委員: 陳俊良
Jiann-Liang Chen
鄧惟中
Wei-Chung Teng
林宗男
Tsung-Nan Lin
孫雅麗
Yea-Li Sun
楊竹星
Chu-Sing Yang
學位類別： 碩士
Master
系所名稱： 電資學院 - 電機工程系
Department of Electrical Engineering
論文出版年： 2023
畢業學年度： 111
語文別： 英文
論文頁數： 85
中文關鍵詞： 低光照圖像增強 、零樣本學習 、可程式化邏輯陣列 、神經網路量化 、立方衛星
外文關鍵詞： Low-Light Image Enhancement, Zero-Shot Learning, FPGA, Neural Network Quantization, CubeSats
相關次數： 點閱：303  下載：8 
分享至:
查詢本校圖書館目錄 查詢臺灣博碩士論文知識加值系統 勘誤回報

    • 立方衛星在各領域被廣泛應用，對社會和民生產生重要影響。然而，由於地球的自轉和衛星相機的限制，衛星拍攝的影像面臨低光問題，影響觀看體驗和影像品質。為了解決這個問題，研究者開展了低光圖像增強相關技術的研究，並提出了Zero-DCE一基於零樣本學習的模型。與傳統機器學習方法不同，該模型不需大量衛星相關資料，適用於衛星應用中難以取得有效資料的限制。
    然而，由於立方衛星的軟硬體資源有限，將人工智慧應用於衛星上是一個具有挑戰性的任務。為使Zero-DCE模型能夠被部署於立方衛星上，本研究提出一新模型Zero-DCE QCS，以改進模型原有不足之處，降低模型大小，在提升推理速度下強化效果。提出之模型採用量化與轉換技術處理圖像資料，透過降採樣降低圖像大小並減少卷積層數，同時引入深度分離卷積及線性擬合方法替換指數操作。最後，通過雙線性插值法將增強的圖像恢復到原始大小。
    此外，亦設計新的損失函數以改善原損失函數於帶有紅外線濾鏡的影像在經過強化後所產生的色彩過度平衡現象以提高影像品質。為更好地適應立方衛星的硬體環境，本研究還提出了MCU和FPGA的異質計算架構，利用FPGA進行加速計算，並由MCU負責系統控制。
    根據多個資料集上的評估結果，本研究提出的模型在信噪比、圖像誤差比、訊息保真度和結構相似性等方面優於先前的研究。此外，新模型在參數量、記憶體使用率和浮點運算方面均優於原模型。研究使用Microchip SmartFusion 2之規格模擬，其FPGA硬體使用率僅1.77%的LUT、FF及6.66%的DSP，加速後的影像增強迭代部分相較於原先僅使用ARM Cortex系列處理器迭代平均減少高達29秒的推理延遲。研究結果表明，本研究提出的模型相較先前的研究具更低的資源使用率、更快速的推理速度與更優異的遙測圖像強化品質。

    CubeSat has found extensive applications in various domains, significantly influencing society and people's lives. However, satellite-captured images often encounter low-light challenges due to the Earth's rotation and camera limitations, which impact the viewing experience and image quality. To address this issue, researchers have undertaken studies on low-light image enhancement techniques and proposed the Zero-DCE model based on zero-shot learning. Unlike traditional machine learning methods, this model does not rely on extensive satellite-specific data, making it applicable to satellite applications with limited availability of effective data.
    However, applying artificial intelligence to CubeSats poses challenges due to their limited hardware and software resources. To enable the deployment of the Zero-DCE model on CubeSats, this study proposes a novel approach to address the model's limitations, reducing its size while enhancing its performance in terms of inference speed. The proposed approach leverages quantization and transformation techniques to process image data, reducing image size through down-sampling and reducing the number of convolutional layers. Furthermore, depth-wise separable convolutions and linear approximation methods are introduced to replace exponential operations. Finally, the enhanced images are restored to their original size using bilinear interpolation.
    In addition, this study design novel loss functions to improve the original function's tendency to produce a color imbalance in filtered images after enhancement, aiming to enhance image quality. To better adapt to the hardware environment of CubeSats, the research also proposed a heterogeneous computing architecture consisting of MCU and FPGA. The FPGA is utilized for accelerated computation, while the MCU is responsible for system control.
    According to the evaluation results on multiple datasets, the proposed model in this study outperforms previous research in terms of signal-to-noise ratio, image dissimilarity, brightness, contrast, and structural similarity. Additionally, the new model exhibits advantages over the original model in terms of parameter count, memory utilization, and floating-point operations. The study employed Microchip SmartFusion 2 for simulation, with FPGA hardware utilization of only 1.77% in terms of LUT, FF, and 6.66% in terms of DSP blocks. Compared to the accelerated image enhancement iteration that used only ARM Cortex-M3 processors reduced the inference latency by 29 seconds. The data results demonstrate that the proposed model in this study exhibits lower resource utilization, faster inference speed, and superior remote sensing image enhancement quality compared to previous research.

    摘要 ......................................................... I Abstract ..................................................... II List of Figures .............................................. VI List of Tables ............................................... VIII Chapter 1 Introduction ....................................... 1 1.1 Motivation ............................................... 1 1.2 Contributions ............................................ 4 1.3 Organization ............................................. 6 Chapter 2 Related Work ....................................... 8 2.1 Potential Impact of Low-Light Images ..................... 8 2.2 Satellite Image Enhancement Under Low Light Conditions ... 9 2.3 Zero-Shot Learning Based Artificial Intelligence ......... 10 2.4 Neural Network Light Weighting Techniques ................ 14 2.5 Techniques for Accelerating in FPGAs ..................... 16 Chapter 3 Proposed System .................................... 19 3.1 System Architecture ...................................... 19 3.2 Zero-DCE QCS ............................................. 20 3.2.1 Normalization .......................................... 22 3.2.2 Down Scale ............................................. 23 3.2.3 Convolution Layer ...................................... 24 3.2.4 Activation ............................................. 29 3.2.5 Up Scale ............................................... 31 3.3 Model Loss Function Improve .............................. 33 3.4 Zero-DCE QCS Acceleration ................................ 36 3.4.1 Acceleration Circuit ................................... 38 3.4.2 Verification Framework ................................. 43 Chapter 4 Performance Analysis ............................... 45 4.1 System Environment and Parameter Settings ................ 45 4.2 Performance Evaluation Index ............................. 49 4.3 Performance Analysis ..................................... 52 4.4 Comparison of Different Studies .......................... 58 4.5 Summary .................................................. 66 Chapter 5 Conclusions and Future Works ....................... 67 5.1 Conclusions .............................................. 67 5.2 Future Works ............................................. 68 References ................................................... 71

    [1] X. Liu, C. Zhang, Y. Wang, K. Ding, T. Han, H. Liu, Y. Tian, B. Xu and M. Ju, "Low-Light Image Enhancement Based on Multi-Scale Network Fusion," IEEE Access, vol. 10, pp. 127853-127862, 2022.
    [2] KBV Research, Global Low Light Imaging Market Size, Share & Industry Trends Analysis Report, Retrieved from https://www.kbvresearch.com/low-light-imaging-market/, (Published on November 2022).
    [3] C. Li, C. Guo, L. Han, J. Jiang, M. M. Cheng, J. Gu and C. C. Loy, "Low-Light Image and Video Enhancement Using Deep Learning: A Survey," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, pp. 9396-9416, 2022.
    [4] C. Guo, C. Li, J. Guo, C. C. Loy, J. Hou, S. Kwong and R. Cong, "Zero-Reference Deep Curve Estimation for Low-Light Image Enhancement," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1777-1786, 2020.
    [5] Russo and G. Lax, "Using Artificial Intelligence for Space Challenges: A Survey, " Proceedings of the Multidisciplinary Digital Publishing Institute Applied Sciences, 2022.
    [6] Li, C. Guo and C. C. Loy, "Learning to Enhance Low-Light Image via Zero-Reference Deep Curve Estimation," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, pp. 4225-4238, 2022.
    [7] W. Mu, H. Liu, W. Chen and Y. Wang, "A More Effective Zero-DCE Variant: Zero-DCE Tiny," Multidisciplinary Digital Publishing Institute Electronics, vol. 11, 2022.
    [8] J. J. Chiu, Satellites are not what you think: Stop saying that the Starlink project is 6G!, Retrieved from https://www.storm.mg/lifestyle/2890880?page=2, (Published on July 2020).
    [9] S. Sakib, D. Faizllin, Y. Koga, M. Uetsuhara and S. Onishi, "In-Orbit FPGA Reprogramming Device for Small Satellites," Advances in Space Research, vol.71, pp. 4549-4556, 2023.
    [10] R. Pitonak, J. Mucha, L. Dobis, M. Javorka and M. Marusin, "CloudSatNet-1: FPGA-Based Hardware-Accelerated Quantized CNN for Satellite On-Board Cloud Coverage Classification," Multidisciplinary Digital Publishing Institute Remote Sensing, vol. 14, 2022
    [11] S. Akre, CubeSat Market Research Report: Information by Size, Subsystem, Application, End User, and Region — Global Forecast till 2027, Retrieved from https://www.marketresearchfuture.com/reports/cubesat-market-7523, (Published on March 2023).
    [12] Grand View Research, Inc., Nanosatellite And Microsatellite Market Size, Share & Trends Analysis Report By Mass, By Application, By End Use, By Region, And Segment Forecasts, 2022 – 2030, Retrieved from https://www.grandviewresearch.com/industry-analysis/nanosatellite-microsatellite-market, (Published on March 2023).
    [13] C. J. Fong, The Past, Present, and Future of Taiwan's CubeSat, Retrieved from https://www.scimonth.com.tw/archives/4659, (Published on October 2020).
    [14] O. Keiichi, Earth Low Orbit Environment Observation Satellite Ten-koh and Ten-kon 2, Retrieved from https://aerospacebiz.jaxa.jp/cubesatlv2022/5-6_detail_technology.html, (Published on January 2022)
    [15] W. Wang, X. Wu, X. Yuan and Z. Gao, "An Experiment-Based Review of Low-Light Image Enhancement Methods," IEEE Access, vol. 8, pp. 87884-87917, 2020.
    [16] W. Kim, "Low-Light Image Enhancement: A Comparative Review and Prospects," IEEE Access, vol. 10, pp. 84535-84557, 2022.
    [17] J. Guo, J. Ma, A. F. Garcia-Fernandez, Y. Zhang and H. Liang, "A Survey on Image Enhancement for Low-light Images, " Journal of Visual Communication and Image Representation, vol. 58, pp. 304-332, 2023
    [18] M. Ma, B. Guo and Y. Niu, "Real-Time GF-4 Satellite Image Enhancement in Low Light Conditions," Proceedings of the 5th China High Resolution Earth Observation Conference, vol. 552, 2019.
    [19] Rajathurai, Ablin, Sulochana, C. Helen and G. Prabin, "An Investigation in Satellite Images Based on Image Enhancement Techniques," European Journal of Remote Sensing, vol. 53, pp. 1-9, 2019.
    [20] Z. A. Ameen, "Satellite Image Enhancement Using an Ameliorated Balance Contrast Enhancement Technique," Traitement du Signal, vol. 37, pp. 245-254, 2020.
    [21] Y. Cao, X. Cheng, J. Mu, D. Li and F. Han, "Detection Method Based on Image Enhancement and an Improved Faster R-CNN for Failed Satellite Components," in IEEE Transactions on Instrumentation and Measurement, vol. 72, pp. 1-13, 2023.
    [22] A. Zhu, L. Zhang, Y. Shen, Y. Ma, S. Zhao and Y. Zhou, "Zeroshot Restoration of Underexposed Images via Robust Retinex Decomposition, " Proceedings of the IEEE International Conference on Multimedia and Expo, pp. 1–6, 2020
    [23] Z. Zhao, B. Xiong, L. Wang, Q. Ou, L. Yu and F. Kuang, "RetinexDIP: A Unified Deep Framework for Low-Light Image Enhancement," IEEE Transactions on Circuits and Systems for Video Technology, vol. 32 , pp. 1076-1088, 2022.
    [24] R. Liu, L. Ma, J. Zhang, X. Fan and Z. Luo, "Retinex-inspired Unrolling with Cooperative Prior Architecture Search for Low-light Image Enhancement," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10556-10565, 2021.
    [25] Z. Ll, N. Jia and H. Jin, "Night Fatigue Driving Detection Algorithm based on Lightweight Zero-DCE," Proceedings of the IEEE 7th International Conference on Smart Cloud, pp. 139-144, 2022.
    [26] S. Zheng and G. Gupta, "Semantic-Guided Zero-Shot Learning for Low-Light Image/Video Enhancement," Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision Workshops, pp.581-590, 2022.
    [27] K. Alex, S. Ilya and E. H. Geoffrey, "ImageNet Classification With Deep Convolutional Neural Networks," Communications of the ACM, vol. 60, pp. 84–90, 2017.
    [28] K. He, X. Zhang, S. Ren and J. Sun, "Deep Residual Learning for Image Recognition," Proceedings of the Computer Vision and Pattern Recognition, pp. 770-778, 2016.
    [29] A. G. Santos, C. O. de Souza, C. Zanchettin, D. Macêdo, A. L. I. Oliveira and T. Ludermir, "Reducing SqueezeNet Storage Size with Depthwise Separable Convolutions," Proceedings of the International Joint Conference on Neural Networks, pp. 1-6, 2018.
    [30] H. Qin, R. Gong, X. Liu, X. Bai, J. Song, and N. Sebe, "Binary Neural Networks: A Survey, " Pattern Recognition, vol. 105, pp. 107281, 2020.
    [31] X. Sui, Q. Lv, Y. Bai, B, Zhu. L. Zhi, Y. Yang and Z. Tan, "A Hardware-Friendly Low-Bit Power-of-Two Quantization Method for CNNs and Its FPGA Implementation," Multidisciplinary Digital Publishing Institute Intelligent Sensors, vol. 22, pp. 6618, 2022.
    [32] Y. H. Chen, T. Krishna, J. S. Emer and V. Sze, "Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep Convolutional Neural Networks," IEEE Journal of Solid-State Circuits, vol. 52, pp. 127-138, 2017.
    [33] Y. H. Chen, T. J. Yang, J. Emer and V. Sze, "Eyeriss v2: A Flexible Accelerator for Emerging Deep Neural Networks on Mobile Devices," IEEE Journal on Emerging and Selected Topics in Circuits and Systems, vol. 9 , pp. 292-308, 2019.
    [34] K. Y. Jiang, H. Y. Wang, C. B. Wu, Y. T. Hwang and C. P. Fan, "Quantized Lite Convolutional Neural Network Hardware Accelerator Design with FPGA for Face Direction Recognition," Proceedings of the Consumer Electronics, pp. 61-62, 2022.
    [35] L. R. Juracy, R. Garibotti and F. G. Moraes, "From CNN to DNN Hardware Accelerators: A Survey on Design, Exploration, Simulation, and Frameworks," Foundations and Trends in Electronic Design Automation, vol. 13, pp 270-344, 2023.
    [36] O. Ronneberger, P. Fischer and T. Brox, "U-Net: Convolutional Networks for Biomedical Image Segmentation," Medical Image Computing and Computer-Assisted Intervention, vol.9351, pp. 234-241, 2015
    [37] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov and L. C. Chen, "MobileNetV2: Inverted Residuals and Linear Bottlenecks," Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4510-4520, 2018.
    [38] C. Wei, W. Wang, W. Yang and J. Liu, "Deep Retinex Decomposition for Low-Light Enhancement, " Proceedings of the British Machine Vision Conference, 2018.
    [39] J. Xinyu, Z. Chuang, L. Minzhen, T. Wenqi and Z. Wenli, "LLVIP: A Visible-Infrared Paired Dataset for Low-Light Vision," Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3496-3504, 2021.
    [40] Y. P. Loh and C. S. Chan, "Getting to Know Low-Light Images with the Exclusively Dark Dataset," Computer Vision and Image Understanding, vol. 178, pp. 30-42, 2019.
    [41] J. Cornebise, I. Oršolić and F. Kalaitzis, "The WorldStrat Dataset: Open High-Resolution Satellite Imagery With Paired Multi-Temporal Low-Resolution," Proceedings of the NeurIPS Track Datasets and Benchmarks, 2022.

    QR CODE