Structural health monitoring (SHM) on the bridge is important to know the usability of the bridges. However, conventional inspection is labor-intensive and expensive. This method is not suitable for post-earthquake inspections that require speed and consistency. Therefore, this research aims to develop an automated bridge inspection using STF-PointRend and EfficientNetB0. The STF-PointRend consists of two-part, namely symbiotic organism search as a hyper-parameter optimizer and PointRend as semantic segmentation. This model is used to recognize the component and the damage type which will be used to get the percentage of the damaged component. On the other hand, the EfficientNetB0 uses as the image classifier. The output of this model is used to get the damage level from each component. As a base to determine the safety of the bridge, this study uses the degree of earthquake resistance. This rating system is based on the DERU method but only considers the structural component. The result shows that STF-PointRend gets a good testing result with the mIoU of 82.67% and 71.42% for component and damage detection. Meanwhile, the EfficientNet got an average F1score of 0.85912 for the testing dataset. For further evaluation, this research uses two minor bridges that suffered catastrophic earthquakes from Palu Earthquake in 2018. The evaluation shows that both bridges need maintenance as soon as possible.

Alonso, I., Riazuelo, L., & Murillo, A. C. (2020). Mininet: An efficient semantic segmentation convnet for real-time robotic applications. IEEE Transactions on Robotics, 36(4), 1340-1347.
Barrile, V., Candela, G., Fotia, A., & Bernardo, E. (2019). UAV survey of bridges and viaduct: workflow and application. Paper presented at the International Conference on Computational Science and Its Applications.
Bibaeva, V. (2018). Using metaheuristics for hyper-parameter optimization of convolutional neural networks. Paper presented at the 2018 IEEE 28th International Workshop on Machine Learning for Signal Processing (MLSP).
Cha, Y. J., Choi, W., & Büyüköztürk, O. (2017). Deep learning‐based crack damage detection using convolutional neural networks. Computer‐Aided Civil and Infrastructure Engineering, 32(5), 361-378.
Chen, B., Gong, C., & Yang, J. (2018). Importance-aware semantic segmentation for autonomous vehicles. IEEE Transactions on Intelligent Transportation Systems, 20(1), 137-148.
Cheng, M.-Y., & Prayogo, D. (2014). Symbiotic organisms search: a new metaheuristic optimization algorithm. Computers & Structures, 139, 98-112.
Deng, W., Mou, Y., Kashiwa, T., Escalera, S., Nagai, K., Nakayama, K., . . . Prendinger, H. (2020). Vision based pixel-level bridge structural damage detection using a link ASPP network. Automation in Construction, 110, 102973.
Ezugwu, A. E., & Prayogo, D. (2019). Symbiotic organisms search algorithm: theory, recent advances and applications. Expert Systems with Applications, 119, 184-209.
Faes, L., Wagner, S. K., Fu, D. J., Liu, X., Korot, E., Ledsam, J. R., . . . Kern, C. (2019). Automated deep learning design for medical image classification by health-care professionals with no coding experience: a feasibility study. The Lancet Digital Health, 1(5), e232-e242.
Gattulli, V., & Chiaramonte, L. (2005). Condition assessment by visual inspection for a bridge management system. Computer‐Aided Civil and Infrastructure Engineering, 20(2), 95-107.
Hsien-Ke, L., Jallow, M., Nie-Jia, Y., Ming-Yi, J., Jyun-Hao, H., Cheng-Wei, S., & Po-Yuan, C. (2017). Comparison of Bridge Inspection Methodologies and Evaluation Criteria in Taiwan and Foreign Practices. Paper presented at the ISARC. Proceedings of the International Symposium on Automation and Robotics in Construction.
Hui, J. (2018). Understanding Feature Pyramid Networks for object detection (FPN). Retrieved from https://jonathan-hui.medium.com/understanding-feature-pyramid-networks-for-object-detection-fpn-45b227b9106c
Jordan, J. (2018). An overview of semantic image segmentation. Retrieved from https://www.jeremyjordan.me/semantic-segmentation/
Ke, T.-W., Hwang, J.-J., Liu, Z., & Yu, S. X. (2018). Adaptive affinity fields for semantic segmentation. Paper presented at the Proceedings of the European Conference on Computer Vision (ECCV).
Kirillov, A., Girshick, R., He, K., & Dollár, P. (2019). Panoptic feature pyramid networks. Paper presented at the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.
Kirillov, A., Wu, Y., He, K., & Girshick, R. (2020). Pointrend: Image segmentation as rendering. Paper presented at the Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.
Ko, J., & Ni, Y. Q. (2005). Technology developments in structural health monitoring of large-scale bridges. Engineering structures, 27(12), 1715-1725.
Liao, H.-K. (2008). Development of a bridge maintenance decision support module for Taiwan Bridge Management System. Paper presented at the Bridge Maintenance, Safety Management, Health Monitoring and Informatics-IABMAS'08: Proceedings of the Fourth International IABMAS Conference, Seoul, Korea, July 13-17 2008.
Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., & Belongie, S. (2017). Feature pyramid networks for object detection. Paper presented at the Proceedings of the IEEE conference on computer vision and pattern recognition.
M Rustowicz, R., Cheong, R., Wang, L., Ermon, S., Burke, M., & Lobell, D. (2019). Semantic segmentation of crop type in Africa: A novel dataset and analysis of deep learning methods. Paper presented at the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops.
Marr, B. (2019). 7 Amazing Examples Of Computer And Machine Vision In Practice. Retrieved from https://www.forbes.com/sites/bernardmarr/2019/04/08/7-amazing-examples-of-computer-and-machine-vision-in-practice/?sh=430c084b1018
Meese, N., & McMahon, C. (2012). Knowledge sharing for sustainable development in civil engineering: a systematic review. AI & society, 27(4), 437-449.
Mikołajczyk, A., & Grochowski, M. (2018). Data augmentation for improving deep learning in image classification problem. Paper presented at the 2018 international interdisciplinary PhD workshop (IIPhDW).
Montagnon, E., Cerny, M., Cadrin-Chênevert, A., Hamilton, V., Derennes, T., Ilinca, A., . . . Tang, A. (2020). Deep learning workflow in radiology: a primer. Insights into imaging, 11(1), 1-15.
Narazaki, Y., Hoskere, V., Yoshida, K., Spencer, B. F., & Fujino, Y. (2021). Synthetic environments for vision-based structural condition assessment of Japanese high-speed railway viaducts. Mechanical Systems and Signal Processing, 160, 107850.
Nieto, D., Brill, A., Kim, B., & Humensky, T. (2017). Exploring deep learning as an event classification method for the Cherenkov Telescope Array. arXiv preprint arXiv:1709.05889.
Pantaleón, M. J., Ramos, Ó. R., Ortega, G., Martínez, J. M., & Schanack, F. (2010). Dynamic analysis of a composite cable-stayed bridge: Escaleritas viaduct. Journal of Bridge Engineering, 15(6), 653-660.
Parmar, V., Bhatia, N., Negi, S., & Suri, M. (2020). Exploration of Optimized Semantic Segmentation Architectures for edge-Deployment on Drones. arXiv preprint arXiv:2007.02839.
Perez, L., & Wang, J. (2017). The effectiveness of data augmentation in image classification using deep learning. arXiv preprint arXiv:1712.04621.
R.O.C, M. o. T. a. C. (2020). 公路橋梁檢測及補強規範. In.
Raghu, M., & Schmidt, E. (2020). A survey of deep learning for scientific discovery. arXiv preprint arXiv:2003.11755.
Ros, G., Sellart, L., Materzynska, J., Vazquez, D., & Lopez, A. M. (2016). The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. Paper presented at the Proceedings of the IEEE conference on computer vision and pattern recognition.
Shorten, C., & Khoshgoftaar, T. M. (2019). A survey on image data augmentation for deep learning. Journal of Big Data, 6(1), 1-48.
Swersky, K., Snoek, J., & Adams, R. P. (2013). Multi-task bayesian optimization.
Tan, M., & Le, Q. (2019). Efficientnet: Rethinking model scaling for convolutional neural networks. Paper presented at the International Conference on Machine Learning.
Wada, K. (2016). Labelme: Image Polygonal Annotation with Python. Retrieved from https://github.com/wkentaro/labelme
Wang, L., Feng, M., Zhou, B., Xiang, B., & Mahadevan, S. (2015). Efficient hyper-parameter optimization for NLP applications. Paper presented at the Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing.
Wei, X., Yang, Z., Liu, Y., Wei, D., Jia, L., & Li, Y. (2019). Railway track fastener defect detection based on image processing and deep learning techniques: A comparative study. Engineering Applications of Artificial Intelligence, 80, 66-81.
WFPGeoNode. (2018). Earthquake in Sulawesi Tengah, Indonesia. 55km NNW of Palu. Magnitude: 6.1 Depth: 18.08. Retrieved from https://geonode.wfp.org/wfpdocs/earthquake-in-sulawesi-tengah-indonesia-55km-nnw-of-palu-magnitude-61-depth-1808/
Wu, Y., Kirillov, A., Massa, F., Lo, W.-Y., & Girshick, R. (2019). Detectron2. In.
Yau, N. J., & Liao, H. K. (2015). Indices for Fast Assessment of Bridge Condition in Taiwan. Paper presented at the Applied Mechanics and Materials.
Yeh, C.-H., Loh, C.-H., & Tsai, K.-C. (2006). Overview of Taiwan earthquake loss estimation system. Natural hazards, 37(1), 23-37.
Yoo, J.-H., Yoon, H.-i., Kim, H.-G., Yoon, H.-S., & Han, S.-S. (2019). Optimization of Hyper-parameter for CNN Model using Genetic Algorithm. Paper presented at the 2019 1st International Conference on Electrical, Control and Instrumentation Engineering (ICECIE).
Zhang, C., Chang, C.-C., & Jamshidi, M. (2018). Bridge damage detection using a single-stage detector and field inspection images. arXiv preprint arXiv:1812.10590.
Zhang, C., Chang, C. c., & Jamshidi, M. (2020). Concrete bridge surface damage detection using a single‐stage detector. Computer‐Aided Civil and Infrastructure Engineering, 35(4), 389-409.