The productivity of excavators is a major concern of managers because it
significantly influences the overall progress of a construction project, especially for
earthwork at the initial phase. Autonomous tracking of excavator performance in real-time
is essential to monitor/manage the earthwork. This study presents a vision-based
autonomous excavator action recognition and productivity measurement by integrating a
deep learning method, named You Only Watch Once (YOWO), to recognize excavators’
action. The recognized actions are used for the action time and average cycle time
calculation. The average cycle time is then used to calculate productivity. An excavator
action recognition dataset was developed to test the proposed method. The algorithm
recognized excavator actions with an F1 score of 87.6% and the mAP was 81.6%. The
outputs were then used for the proposed framework for excavator productivity
measurement. It was also found that by omitting the outliers/misclassifications (95%
confidence) to calculate average action times yielded better cycle time accuracy (99.7%)
compared to using mean (81.59%). The successful implementation of the proposed
framework in this study shows that autonomous productivity measurement in construction
is feasible and can be used to measure productivity cheaper, faster, and even continuously
in real-time.

The productivity of excavators is a major concern of managers because it
significantly influences the overall progress of a construction project, especially for
earthwork at the initial phase. Autonomous tracking of excavator performance in real-time
is essential to monitor/manage the earthwork. This study presents a vision-based
autonomous excavator action recognition and productivity measurement by integrating a
deep learning method, named You Only Watch Once (YOWO), to recognize excavators’
action. The recognized actions are used for the action time and average cycle time
calculation. The average cycle time is then used to calculate productivity. An excavator
action recognition dataset was developed to test the proposed method. The algorithm
recognized excavator actions with an F1 score of 87.6% and the mAP was 81.6%. The
outputs were then used for the proposed framework for excavator productivity
measurement. It was also found that by omitting the outliers/misclassifications (95%
confidence) to calculate average action times yielded better cycle time accuracy (99.7%)
compared to using mean (81.59%). The successful implementation of the proposed
framework in this study shows that autonomous productivity measurement in construction
is feasible and can be used to measure productivity cheaper, faster, and even continuously
in real-time.

Albawi, Saad, Tareq Abed Mohammed, and Saad Al-Zawi. 2018. “Understanding of a Convolutional Neural Network.” Proceedings of 2017 International Conference on Engineering and Technology, ICET 2017 2018-Janua: 1–6.
Aloysius, Neena, and M. Geetha. 2018. “A Review on Deep Convolutional Neural Networks.” Proceedings of the 2017 IEEE International Conference on Communication and Signal Processing, ICCSP 2017 2018-Janua: 588–92.
Bankvall, Lars, Lena E. Bygballe, Anna Dubois, and Marianne Jahre. 2010. “Interdependence in Supply Chains and Projects in Construction.” Supply Chain Management 15(5): 385–93.
Fang, Weili et al. 2018. “Automated Detection of Workers and Heavy Equipment on Construction Sites: A Convolutional Neural Network Approach.” Advanced Engineering Informatics 37(November 2017): 139–49. https://doi.org/10.1016/j.aei.2018.05.003.
Ji, Shuiwang, Wei Xu, Ming Yang, and Kai Yu. 2013. “3D Convolutional Neural Networks for Human Action Recognition.” IEEE Transactions on Pattern Analysis and Machine Intelligence 35(1): 221–31.
Khoshdeli, Mina, Richard Cong, and Bahram Parvin. 2017. “Detection of Nuclei in H&E Stained Sections Using Convolutional Neural Networks.” 2017 IEEE EMBS International Conference on Biomedical and Health Informatics, BHI 2017: 105–8.
Kim, Hongjo, Hyoungkwan Kim, Yong Won Hong, and Hyeran Byun. 2018. “Detecting Construction Equipment Using a Region-Based Fully Convolutional Network and Transfer Learning.” Journal of Computing in Civil Engineering 32(2): 1–15.
Kim, Jinwoo, and Seokho Chi. 2019. “Action Recognition of Earthmoving Excavators Based on Sequential Pattern Analysis of Visual Features and Operation Cycles.” Automation in Construction 104(December 2018): 255–64. https://doi.org/10.1016/j.autcon.2019.03.025.
Kim, Jinwoo, Seokho Chi, and Jongwon Seo. 2018. “Interaction Analysis for Vision-Based Activity Identification of Earthmoving Excavators and Dump Trucks.” Automation in Construction 87(May 2017): 297–308. https://doi.org/10.1016/j.autcon.2017.12.016.
Köpüklü, Okan, Xiangyu Wei, and Gerhard Rigoll. 2019. “You Only Watch Once: A Unified CNN Architecture for Real-Time Spatiotemporal Action Localization.” : 1–28. http://arxiv.org/abs/1911.06644.
LeCun, Yann, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998. “Gradient-Based Learning Applied to Document Recognition.” Proceedings of the IEEE 86(11): 2278–2323.
LeCun, Yann, and Corinna Cortes. 2010. “MNIST Handwritten Digit Database.” AT&T Labs [Online]. Available: http://yann. lecun. com/exdb/mnist. httpyann.lecun.comexdbmnist.
Ma, Ningning, Xiangyu Zhang, Hai Tao Zheng, and Jian Sun. 2018. “Shufflenet V2: Practical Guidelines for Efficient Cnn Architecture Design.” In Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), , 122–38.
Paszke, Adam et al. 2019. “PyTorch: An Imperative Style, High-Performance Deep Learning Library.” (NeurIPS). http://arxiv.org/abs/1912.01703.
Ren, Shaoqing, Kaiming He, Ross Girshick, and Jian Sun. 2017. “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks.” IEEE Transactions on Pattern Analysis and Machine Intelligence 39(6): 1137–49.
Roberts, Dominic, and Mani Golparvar-Fard. 2019. “End-to-End Vision-Based Detection, Tracking and Activity Analysis of Earthmoving Equipment Filmed at Ground Level.” Automation in Construction 105(August 2018): 102811. https://doi.org/10.1016/j.autcon.2019.04.006.
Sasaki, Yutaka. 2007. “The Truth of the F-Measure.” Teach Tutor mater: 1–5. http://www.cs.odu.edu/~mukka/cs795sum09dm/Lecturenotes/Day3/F-measure-YS-26Oct07.pdf.
Stone, R. 2002. The Civil Engineering Handbook The Civil Engineering Handbook. Second Edi. eds. W.F. Chen and J.Y. Richard Liew. CRC Press. https://www.taylorfrancis.com/books/9781420041217.
Sun, Ming, and Xianhai Meng. 2009. “Taxonomy for Change Causes and Effects in Construction Projects.” International Journal of Project Management 27(6): 560–72. http://dx.doi.org/10.1016/j.ijproman.2008.10.005.
Timmer, Marcel P, Robert Inklaar, and Mary O Mahony. 2011. “Productivity and Economic Growth in Europe: A Comparative Industry Perspective.” International Productivity Monitor 21: 3–23.
Tran, Van, and John Tookey. 2011. “Labour Productivity in the New Zealand Construction Industry: A Thorough Investigation.” Australasian Journal of Construction Economics and Building 11(1): 41–60.
Zhang, Xiangyu, Xinyu Zhou, Mengxiao Lin, and Jian Sun. 2018. “ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices.” Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition: 6848–56.
Zhi, Mao, Goh Bee Hua, Shou Qing Wang, and George Ofori. 2003. “Total Factor Productivity Growth Accounting in the Construction Industry of Singapore.” Construction Management and Economics 21(7): 707–18.