本研究旨在開發一個基於重複性動作影像技術應用於作業現場的例外動作辨識。本研究利用重複動作計次模型RepNet的預訓練（pre-trained）編碼器（Encoder）擷取影像特徵，再將其輸出特徵資料透過相似性比對，建立時間相似矩陣（Temporal Self-similarity Matrix, TSM）。再以TSM中各影像與接續影像之相似性，計算各影像與接續之影像相似性的亂度值（Entropy, 熵），以各個影像的熵依照順序建立成為一個具有時間序列特性的訊號資料。最後以LRR-RKMeans（Local Recurrence Rate - Robust K Means）演算法進行異質偵測，所找出的異質資料將被判定為例外動作所在影像順序。使用LRR-RKMeans演算法的目的在於無法窮舉所有可能出現在作業現場之規範外動作的情況下，以非監督式方法偵測出重複性標準動作以外的例外動作，解決例外動作資料蒐集的困難。本實驗以製造業產線上兩種常見的影像資料作為研究的主題：1）手部重複性動作畫面；2）全身性重複性動作畫面，並在兩種類中包含重複性標準動作以外的例外動作。研究結果發現若是為手部重複性動作畫面，選擇針對手部及工具的區域，預先框取感興趣區域（Region of Interest, ROI），並以本研究所提出熵訊號資料處理的異質偵測方法，即足以達成平均準確率為87.34%以及平均例外動作辨識率60.26%。

In this research, a video repetition counting method was applied to detect the exceptional motions. The objective is to detect exceptional motions which are not defined or cannot be listed exhaustively by using unsupervised method. Basically, the pre-trained encoder in RepNet, a video repetition counting model, was used to extract videos’ features. The per-frame features were compared with each other to form a Temporal Self-similarity Matrix (TSM). With similarity sequences in TSM, entropy was used to measure the uncertainty value of corresponding frame in a video. A time-series data based on entropy of each frame can be formulated. In order to detect anomaly in a time-series data, an anomaly detection method, Local Recurrence Rate - Robust K Means (LRR-RKMeans) was used. Discords subsequences detected by LRR-RKMeans are considered as exceptional motion with its corresponding frame sequence. In this research, two kinds of motions commonly in manufacturing line were studied; 1) repeated hand motions and 2) repeated body motions. Both kinds of motions contained the repeated standard operation process and the exceptional motions. The experimental result shows that the proposed method can achieve mean accuracy (87.34%), and mean recall of exceptional motion (60.26%) on the repeated hand motions with region of interest specified.

[1] F. Oleari, M. Magnani, D. Ronzoni, and L. Sabattini, "Industrial AGVs: Toward a pervasive diffusion in modern factory warehouses," in 2014 IEEE 10th International Conference on Intelligent Computer Communication and Processing (ICCP), 2014, pp. 233-238: IEEE.
[2] C. Liang et al., "Automated robot picking system for e-commerce fulfillment warehouse application," in The 14th IFToMM World Congress, 2015: IFToMM.
[3] J. Yang, Z. Shi, and Z. Wu, "Vision-based action recognition of construction workers using dense trajectories," Advanced Engineering Informatics, vol. 30, no. 3, pp. 327-336, 2016.
[4] S. Yan, Y. Xiong, and D. Lin, "Spatial temporal graph convolutional networks for skeleton-based action recognition," presented at the Thirty-second AAAI conference on artificial intelligence, New Orleans, USA, February 2-7, 2018.
[5] C.-L. Yang, W.-T. Li, and S.-C. Hsu, "Skeleton-based Hand Gesture Recognition for Assembly Line Operation," presented at the 2020 International Conference on Advanced Robotics and Intelligent Systems (ARIS), Taipei, Taiwan, August 19-21, 2020.
[6] C.-L. Yang, S.-C. Hsu, Y.-W. Hsu, and Y.-C. Kang, "Human Action Recognition on Exceptional Movement of Worker Operation," in Advances in Manufacturing, Production Management and Process Control, 2021, pp. 376-383: Springer International Publishing.
[7] R. Cutler and L. S. Davis, "Robust real-time periodic motion detection, analysis, and applications," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 781-796, 2000.
[8] T. F. Runia, C. G. Snoek, and A. W. Smeulders, "Real-world repetition estimation by div, grad and curl," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 9009-9017: IEEE.
[9] C. Sun, I. N. Junejo, M. Tappen, and H. Foroosh, "Exploring sparseness and self-similarity for action recognition," IEEE Transactions on Image Processing, vol. 24, no. 8, pp. 2488-2501, 2015.
[10] C. Panagiotakis, G. Karvounas, and A. Argyros, "Unsupervised detection of periodic segments in videos," in 2018 25th IEEE International Conference on Image Processing (ICIP), 2018, pp. 923-927: IEEE.
[11] C. Arteta, V. Lempitsky, and A. Zisserman, "Counting in the wild," in European conference on computer vision, 2016, pp. 483-498: Springer.
[12] D. Dwibedi, Y. Aytar, J. Tompson, P. Sermanet, and A. Zisserman, "Counting Out Time: Class Agnostic Video Repetition Counting in the Wild," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 10387-10396.
[13] C. Shannon, "A mathematical theory of communication," The Bell system technical journal vol. 27, no. 3, pp. 379-423, 1948.
[14] S. Z. Marvizadeh, "Entropy applications in industrial engineering," The University of Nebraska-Lincoln, 2013.
[15] R. Zhou, R. Cai, and G. Tong, "Applications of entropy in finance: A review," Entropy, vol. 15, no. 11, pp. 4909-4931, 2013.
[16] K. Xu, Z.-L. Zhang, and S. Bhattacharyya, "Profiling internet backbone traffic: behavior models and applications," ACM SIGCOMM Computer Communication Review vol. 35, no. 4, pp. 169-180, 2005.
[17] P. Bereziński, B. Jasiul, and M. Szpyrka, "An entropy-based network anomaly detection method," Entropy, vol. 17, no. 4, pp. 2367-2408, 2015.
[18] J. Garland, T. R. Jones, M. Neuder, V. Morris, J. W. White, and E. Bradley, "Anomaly detection in paleoclimate records using permutation entropy," Entropy, vol. 20, no. 12, p. 931, 2018.
[19] W. Kay et al., "The kinetics human action video dataset," arXiv preprint arXiv:.06950, 2017.
[20] K. Soomro, A. R. Zamir, and M. Shah, "UCF101: A dataset of 101 human actions classes from videos in the wild," arXiv preprint arXiv:1212.0402, 2012.
[21] A. Mishra, V. K. Verma, M. S. K. Reddy, S. Arulkumar, P. Rai, and A. Mittal, "A generative approach to zero-shot and few-shot action recognition," in 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), 2018, pp. 372-380: IEEE.
[22] Y. Qi, T. Liu, and Y. Fu, "Anomalous Action Recognition Research for Few-shot Learning," in 2020 IEEE 4th Information Technology, Networking, Electronic and Automation Control Conference (ITNEC), 2020, vol. 1, pp. 1306-1310: IEEE.
[23] D. Xu, E. Ricci, Y. Yan, J. Song, and N. Sebe, "Learning deep representations of appearance and motion for anomalous event detection," arXiv preprint arXiv:.01553, 2015.
[24] R. T. Ionescu, S. Smeureanu, M. Popescu, and B. Alexe, "Detecting abnormal events in video using narrowed motion clusters," arXiv preprint arXiv:.05030, 2018.
[25] S. Amraee, A. Vafaei, K. Jamshidi, and P. Adibi, "Abnormal event detection in crowded scenes using one-class SVM," Signal, Image and Video Processing, vol. 12, no. 6, pp. 1115-1123, 2018.
[26] S. Bouindour, M. M. Hittawe, S. Mahfouz, and H. Snoussi, "Abnormal event detection using convolutional neural networks and 1-class SVM classifier," in 8th International Conference on Imaging for Crime Detection and Prevention (ICDP 2017), 2017, pp. 1-6: IET.
[27] Y. Yuan, J. Fang, and Q. Wang, "Online anomaly detection in crowd scenes via structure analysis," IEEE transactions on cybernetics, vol. 45, no. 3, pp. 548-561, 2014.
[28] B. Liu, S. Yeung, E. Chou, D.-A. Huang, L. Fei-Fei, and J. C. Niebles, "Temporal modular networks for retrieving complex compositional activities in videos," in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 552-568: Springer.
[29] R. Morais, V. Le, T. Tran, B. Saha, M. Mansour, and S. Venkatesh, "Learning regularity in skeleton trajectories for anomaly detection in videos," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 11996-12004: IEEE.
[30] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778: IEEE.
[31] H. Sutrisno and C.-L. Yang, "Discovering defective products based on multivariate sensors data using local recurrence rate and robust k-means clustering," presented at the International Conference on Production Research (ICPR 2021), (virtual conference), Taichung, Taiwan, July 18-21, 2021.
[32] J. Lei, T. Jiang, K. Wu, H. Du, G. Zhu, and Z. Wang, "Robust K-means algorithm with automatically splitting and merging clusters and its applications for surveillance data," Multimedia Tools and Applications, vol. 75, no. 19, pp. 12043-12059, 2016.
[33] H. Sakoe and S. Chiba, "Dynamic programming algorithm optimization for spoken word recognition," IEEE transactions on acoustics, speech, signal processing, vol. 26, no. 1, pp. 43-49, 1978.
[34] M. Abadi et al., "Tensorflow: A system for large-scale machine learning," in 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), 2016, pp. 265-283.