橋梁建構材料大多以鋼筋混凝土和鋼材為主，後者因台灣潮濕多雨的氣候，易於腐蝕及材料彈性疲乏，將縮短鋼構橋梁之使用年限，故需要定期巡檢及養護。而橋梁背部面板位置較難以接近，傳統橋梁檢測方法為專業檢測人員透過橋檢車以目視來進行檢測判斷，該方式係憑藉個人的主觀經驗，缺乏現地實證依據，且巡檢過程耗時費力，若欲以深度學習技術建立橋梁即時自動檢測系統，不可或缺的環節為建構大量劣化圖像資料集，逐點標註影像中不規則形狀劣化區域的過程耗費龐大的時間及人力成本。經實地踏訪臺灣中部以北16座鋼構橋梁，以無人飛行載具蒐集橋背板主要劣化影像資料，並進行劣化態樣統計分析，發現鏽蝕為主要劣化形式，故希冀針對鋼構橋梁之背部面板鏽蝕劣化建構自動標註模組。本研究以電腦視覺深度學習技術Mask R-CNN作為模型骨架，經由不同影像資料集進行模型訓練，以十折交叉驗證法獲得模型評估指標，再將訓練集與驗證集合併重新學習，透過獨立測試集進行普適性預測能力評估。資料集於預處理過程中，裁切複雜背景漸少干擾，可使訓練模型具良好成效。分析成果顯示，為避免模型訓練過度適配，可根據標註需求選擇合適的擴增技術，透過旋轉技術擴增影像，使模型趨於穩定，或採用倍數擴增影像方法，使模型的平均精度有較佳表現，若採比例縮放技術，可使模型適配性較佳，另不建議採用改變顏色空間進行資料擴增。以橋梁檢測區域定位觀之，研發的自動標註預測模組皆能多區域的有效框列影像內多處不規則形狀的劣化鏽蝕範圍。本研發成果的具體貢獻為將經由標註工具以手動逐點標註多邊形劣化範圍，平均速度為3分鐘/張的傳統方式，提升為模組自動化產生劣化區域的預測標註框座標，平均速度為15秒/張，與以往手動標註速度相比增進12倍，大幅減少劣化區域需逐張圖像標註的時間耗費及人工成本。此模組可作為未來開發橋梁即時自動檢測系統之基石，改善劣化影像數據集建立之效率。

The majority of bridge construction materials consists of reinforced concrete and steel. Due to the humid and rainy climate in Taiwan, the latter is prone to erosion and elastic fatigue, which shortens the service life of steel bridges. Regular inspections and maintenance are therefore necessary. The underneath structure of bridges is difficult to access; conventional bridge inspections involve professional inspectors performing visual inspections using bridge inspection trucks. This approach relies on personal subjective experience, lacks any empirical foundation, and is time-consuming and laborious. To construct a real-time automatic bridge inspection system based on deep learning techniques, an essential step is to establish a large dataset of deterioration images. Labeling the irregularly-shaped deterioration regions point by point in the images is costly in both time and labor. We visited 16 steel bridges near north of Central Taiwan to collect images of deterioration on the underneath structure of steel bridges by unmanned aerial vehicles and performed a statistical analysis of their forms. The results indicated that rusting was the primary form of deterioration, so we aimed to develop an automatic labeling module for rust deteriorations on the underneath structure of steel bridges. We applied the computer vision-based deep learning model Mask R-CNN as the backbone and trained the model using different image datasets. We obtained model evaluation indicators using the ten-fold cross-validation method, combined the training and validation sets for retraining, and then conducted a generalizability assessment using an independent test set. During pre-processing, complex backgrounds were removed to reduce interference and improve model training effectiveness. The analysis results indicate that to prevent overfitting in model training, suitable augmentation techniques can be selected depending on the labeling needs. Using the rotation technique to augment the images can improve the stability of the model. The multiplication approach can also enhance the prediction power of the model. The scale technique to augment can raise the goodness of fit of the model Altering the color space to augment the data is not recommended. From the perspective of bridge inspection areas, the automatic labeling and prediction module developed in this study could effectively identify rust deteriorations in multiple small and large images. The specific contribution of this research and development achievement is to upgrade the manual labeling irregularly-shaped deterioration regions point by point with the mean speed of 3 min/image, to the automatic prediction and coordinate labeling module with a mean speed of 15 sec/image, which is 12 times as fast as manual labeling and greatly reduces the temporal and labor costs of deterioration labeling. The proposed module can serve as the foundation of real-time automatic inspection systems developed in the future and greatly increase the efficiency of image data compilation.

[1] N. Yau and Y. Chuang, "Analyzing Taiwan Bridge Management System for decision making in bridge maintenance: A big data approach," in 2015 10th International Joint Conference on Software Technologies (ICSOFT), 20-22 July 2015 2015, vol. 1, pp. 1-6. https://ieeexplore.ieee.org/document/7521105
[2] R. Santos, D. Ribeiro, P. Lopes, R. Cabral, and R. Calçada, "Detection of exposed steel rebars based on deep-learning techniques and unmanned aerial vehicles," Automation in Construction, vol. 139, p. 104324, 2022/07/01/ 2022, doi: https://doi.org/10.1016/j.autcon.2022.104324.
[3] B. Kim, N. Yuvaraj, H. W. Park, K. R. S. Preethaa, R. A. Pandian, and D.-E. Lee, "Investigation of steel frame damage based on computer vision and deep learning," Automation in Construction, vol. 132, p. 103941, 2021/12/01/ 2021, doi: https://doi.org/10.1016/j.autcon.2021.103941.
[4] N. Otsu, "A Threshold Selection Method from Gray-Level Histograms," IEEE Transactions on Systems, Man, and Cybernetics, vol. 9, no. 1, pp. 62-66, 1979, doi: 10.1109/TSMC.1979.4310076. https://ieeexplore.ieee.org/abstract/document/4310076
[5] L. Najman and M. Schmitt, "Watershed of a continuous function," Signal Processing, vol. 38, no. 1, pp. 99-112, 1994/07/01/ 1994, doi: https://doi.org/10.1016/0165-1684(94)90059-0.
[6] N. Dhanachandra, K. Manglem, and Y. J. Chanu, "Image Segmentation Using K -means Clustering Algorithm and Subtractive Clustering Algorithm," Procedia Computer Science, vol. 54, pp. 764-771, 2015/01/01/ 2015, doi: https://doi.org/10.1016/j.procs.2015.06.090.
[7] K. Simonyan and A. Zisserman, "Very Deep Convolutional Networks for Large-Scale Image Recognition," p. arXiv:1409.1556. [Online]. Available: https://ui.adsabs.harvard.edu/abs/2014arXiv1409.1556S
[8] K. He, X. Zhang, S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition," p. arXiv:1512.03385. [Online]. Available: https://ui.adsabs.harvard.edu/abs/2015arXiv151203385H
[9] J. Long, E. Shelhamer, and T. Darrell, "Fully convolutional networks for semantic segmentation," in 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 7-12 June 2015 2015, pp. 3431-3440, doi: 10.1109/CVPR.2015.7298965. [Online]. Available: https://ieeexplore.ieee.org/document/7298965
[10] O. Ronneberger, P. Fischer, and T. Brox, "U-Net: Convolutional Networks for Biomedical Image Segmentation," in Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015, Cham, N. Navab, J. Hornegger, W. M. Wells, and A. F. Frangi, Eds., 2015// 2015: Springer International Publishing, pp. 234-241. https://link.springer.com/chapter/10.1007/978-3-319-24574-4_28
[11] R. Girshick, "Fast R-CNN," in 2015 IEEE International Conference on Computer Vision (ICCV), Santiago, Chile, 7-13 Dec. 2015 2015, pp. 1440-1448, doi: 10.1109/ICCV.2015.169. [Online]. Available: https://ieeexplore.ieee.org/document/7410526
[12] K. He, G. Gkioxari, P. Dollár, and R. Girshick, "Mask R-CNN," in 2017 IEEE International Conference on Computer Vision (ICCV), 22-29 Oct. 2017 2017, pp. 2980-2988, doi: 10.1109/ICCV.2017.322. https://ieeexplore.ieee.org/document/8237584/keywords#keywords
[13] Z. Zhang, X. Yin, and Z. Yan, "Rapid data annotation for sand-like granular instance segmentation using mask-RCNN," Automation in Construction, vol. 133, p. 103994, 2022/01/01/ 2022, doi: https://doi.org/10.1016/j.autcon.2021.103994.
[14] F. Schindler and V. Steinhage, "Saving costs for video data annotation in wildlife monitoring," Ecological Informatics, vol. 65, p. 101418, 2021/11/01/ 2021, doi: https://doi.org/10.1016/j.ecoinf.2021.101418.
[15] A. Dutta and A. Zisserman, The VGG Image Annotator (VIA). 2019. https://arxiv.org/abs/1904.10699
[16] H. Kei Cheng, Y.-W. Tai, and C.-K. Tang, "Modular Interactive Video Object Segmentation: Interaction-to-Mask, Propagation and Difference-Aware Fusion," p. arXiv:2103.07941. [Online]. Available: https://ui.adsabs.harvard.edu/abs/2021arXiv210307941K
[17] X. Li, Y. Wang, and Y. Cai, "Automatic Annotation Algorithm of Medical Radiological Images using Convolutional Neural Network," Pattern Recognition Letters, vol. 152, pp. 158-165, 2021/12/01/ 2021, doi: https://doi.org/10.1016/j.patrec.2021.09.011.
[18] M. Andriluka, J. R. R. Uijlings, and V. Ferrari, "Fluid Annotation: A Human-Machine Collaboration Interface for Full Image Annotation," p. arXiv:1806.07527. [Online]. Available: https://ui.adsabs.harvard.edu/abs/2018arXiv180607527A
[19] D. Yang, X. Wang, H. Zhang, Z.-y. Yin, D. Su, and J. Xu, "A Mask R-CNN based particle identification for quantitative shape evaluation of granular materials," Powder Technology, vol. 392, pp. 296-305, 2021/11/01/ 2021, doi: https://doi.org/10.1016/j.powtec.2021.07.005.
[20] Y. Xu, D. Li, Q. Xie, Q. Wu, and J. Wang, "Automatic defect detection and segmentation of tunnel surface using modified Mask R-CNN," Measurement, vol. 178, p. 109316, 2021/06/01/ 2021, doi: https://doi.org/10.1016/j.measurement.2021.109316.
[21] H. Zhang et al., "Analyzing the pore structure of pervious concrete based on the deep learning framework of Mask R-CNN," Construction and Building Materials, vol. 318, p. 125987, 2022/02/07/ 2022, doi: https://doi.org/10.1016/j.conbuildmat.2021.125987.
[22] Y.-B. Lin et al., "The Artificial Intelligence of Things Sensing System of Real-Time Bridge Scour Monitoring for Early Warning during Floods," Sensors, vol. 21, p. 4942, 07/20 2021, doi: 10.3390/s21144942. https://pubmed.ncbi.nlm.nih.gov/34300679/
[23] Y. G. Lai, "A Two-Dimensional Depth-Averaged Sediment Transport Mobile-Bed Model with Polygonal Meshes," Water, vol. 12, no. 4, p. 1032, 2020. [Online]. Available: https://www.mdpi.com/2073-4441/12/4/1032.
[24] T.-Y. Lin et al., "Microsoft COCO: Common Objects in Context," p. arXiv:1405.0312. [Online]. Available: https://ui.adsabs.harvard.edu/abs/2014arXiv1405.0312L
[25] A. Paszke et al., "PyTorch: An Imperative Style, High-Performance Deep Learning Library," p. arXiv:1912.01703. [Online]. Available: https://ui.adsabs.harvard.edu/abs/2019arXiv191201703P
[26] A. Torralba, B. C. Russell, and J. Yuen, "LabelMe: Online Image Annotation and Applications," Proceedings of the IEEE, vol. 98, no. 8, pp. 1467-1484, 2010, doi: 10.1109/JPROC.2010.2050290. https://ieeexplore.ieee.org/abstract/document/5483185
[27] Á. Casado-García et al., "CLoDSA: a tool for augmentation in classification, localization, detection, semantic segmentation and instance segmentation tasks," BMC Bioinformatics, vol. 20, no. 1, p. 323, 2019/06/13 2019, doi: 10.1186/s12859-019-2931-1. https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-019-2931-1
[28] T.-Y. Lin, P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie, "Feature Pyramid Networks for Object Detection," p. arXiv:1612.03144. [Online]. Available: https://ui.adsabs.harvard.edu/abs/2016arXiv161203144L
[29] K. He, X. Zhang, S. Ren, and J. Sun, "Deep Residual Learning for Image Recognition," in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 27-30 June 2016 2016, pp. 770-778, doi: 10.1109/CVPR.2016.90. [Online]. Available: https://ieeexplore.ieee.org/document/7780459
[30] J. Dai, Y. Li, K. He, and J. Sun, "R-FCN: Object Detection via Region-based Fully Convolutional Networks," p. arXiv:1605.06409. [Online]. Available: https://ui.adsabs.harvard.edu/abs/2016arXiv160506409D
[31] J. Hosang, R. Benenson, P. Dollár, and B. Schiele, "What makes for effective detection proposals?," p. arXiv:1502.05082. [Online]. Available: https://ui.adsabs.harvard.edu/abs/2015arXiv150205082H
[32] D. Berrar, "Cross-Validation," in Encyclopedia of Bioinformatics and Computational Biology, S. Ranganathan, M. Gribskov, K. Nakai, and C. Schönbach Eds. Oxford: Academic Press, 2019, pp. 542-545. https://www.sciencedirect.com/science/article/pii/B978012809633820349X
[33] R. Kohavi, "A study of cross-validation and bootstrap for accuracy estimation and model selection," presented at the Proceedings of the 14th international joint conference on Artificial intelligence - Volume 2, pp. 1137–1143, Montreal, Quebec, Canada, 1995. https://dl.acm.org/doi/10.5555/1643031.1643047
[34] K. M. Ting, "Confusion Matrix," in Encyclopedia of Machine Learning, C. Sammut and G. I. Webb Eds. Boston, MA: Springer US, 2010, pp. 209-209. https://link.springer.com/referenceworkentry/10.1007/978-0-387-30164-8_157
[35] H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, and S. Savarese, "Generalized Intersection Over Union: A Metric and a Loss for Bounding Box Regression," in 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 15-20 June 2019 2019, pp. 658-666, doi: 10.1109/CVPR.2019.00075. [Online]. Available: https://ieeexplore.ieee.org/document/8953982
[36] J. Davis and M. Goadrich, "The Relationship Between Precision-Recall and ROC Curves," Proceedings of the 23rd international conference on Machine learning (ICML), vol. 148, pp. 233-240, 2006, doi: 10.1145/1143844.1143874.
[37] L. Yuan and Z. Qiu, "Mask-RCNN with spatial attention for pedestrian segmentation in cyber–physical systems," Computer Communications, vol. 180, pp. 109-114, 2021/12/01/ 2021, doi: https://doi.org/10.1016/j.comcom.2021.09.002.
[38] 中華民國交通部運輸研究所, "109年臺灣沿岸地區金屬材料腐蝕環境調查研究," 中華民國交通部運輸研究所, 2020 [Online]. Available: https://www.iot.gov.tw/dl-17132-9e3d2fc8feb64f42809a30ce67a7ab09.html
[39] 中華民國交通部運輸研究所, "2020年臺灣大氣腐蝕劣化因子調查研究資料年報," 中華民國交通部運輸研究所, 2021 [Online]. Available: https://www.iot.gov.tw/cp-78-206232-c7908-1.html
[40] 李家順, "鋼橋維護常見劣化缺失與成因探討," 中華民國交通部公路總局西濱北工處, 2021. https://www.anticorr.org.tw/down_1.files/2021/forum/%E6%9D%8E%E5%A50E%B6%E9%A0%86-49%E9%8B%BC%E6%A9%8B%E7%B6%AD%E8%AD%B7%E5%B8%B8%E8%A6%8B%E5%8A%A3%E5%8C%96%E7%BC%BA%E5%A4%B1%E8%88%87%E6%88%90%E5%9B%A0%E6%8E%A2%E8%A8%8E(110%E5%B9%B4%E8%AB%96%E5%A3%87%E8%AC%9B%E7%BE%A91101126).pdf
[41] C. Shorten and T. M. Khoshgoftaar, "A survey on Image Data Augmentation for Deep Learning," Journal of Big Data, vol. 6, no. 1, p. 60, 2019/07/06 2019, doi: 10.1186/s40537-019-0197-0. https://journalofbigdata.springeropen.com/articles/10.1186/s40537-019-0197-0
[42] M. Hamzah. "Auto-Annotate." GitHub. https://github.com/mdhmz1/Auto-Annotate.git