This study is to establish a simple and effective supplement system for rehabilitation by using a Kinect RGB-D camera to monitor patient motions in a real time manner and using the Unity3D animation system to create an augmented reality human-machine interaction system to help the rehabilitation process. In the motion detection, a depth calibration process is built to correct the depth value owing to the existence of a pitch angle of the Kinect used. Through the depth calibration, the Kinect RGB-D camera can be flexibly set up at various locations. Also, in the system, because of possibly sheltering by the rehabilitation equipment or complex rehabilitation environment, joint re-location approaches are considered to solve the problem of joint misalignment or instability. In this approach, several physics and spatial principles are used to re-locate the joints positions. They include the contour of the body, the relationship between body parts, the correlation of depth values, the calculation of the spatial angle, the slope calculation in space, and the length conversion. Those approaches are selected to properly relocate wrong joint positions found in the actual system. Finally, the Kalman filter is employed to deal with possible noise carried in the joints obtained from the Kinect skeleton package. In the Unity3D user interface, it is to have a good visualization system for the proposed automated motion detection rehabilitation system and also to reduce the workload of medical personnel by creating a website for physical information of patients so that doctors can easily manage and analyze patient information. It can be experienced that the proposed approach can effectively be used to establish an automated rehabilitation system.

This study is to establish a simple and effective supplement system for rehabilitation by using a Kinect RGB-D camera to monitor patient motions in a real time manner and using the Unity3D animation system to create an augmented reality human-machine interaction system to help the rehabilitation process. In the motion detection, a depth calibration process is built to correct the depth value owing to the existence of a pitch angle of the Kinect used. Through the depth calibration, the Kinect RGB-D camera can be flexibly set up at various locations. Also, in the system, because of possibly sheltering by the rehabilitation equipment or complex rehabilitation environment, joint re-location approaches are considered to solve the problem of joint misalignment or instability. In this approach, several physics and spatial principles are used to re-locate the joints positions. They include the contour of the body, the relationship between body parts, the correlation of depth values, the calculation of the spatial angle, the slope calculation in space, and the length conversion. Those approaches are selected to properly relocate wrong joint positions found in the actual system. Finally, the Kalman filter is employed to deal with possible noise carried in the joints obtained from the Kinect skeleton package. In the Unity3D user interface, it is to have a good visualization system for the proposed automated motion detection rehabilitation system and also to reduce the workload of medical personnel by creating a website for physical information of patients so that doctors can easily manage and analyze patient information. It can be experienced that the proposed approach can effectively be used to establish an automated rehabilitation system.

[1] A. Ota et al., “Differential effects of power rehabilitation on physical performance and higher-level functional capacity among community-dwelling older adults with a slight degree of frailty,” Journal of Epidemiology, vol. 17, no. 2, pp. 61-67, 2007.
[2] L. P. Yu, Effect of Produce Outcome Worthwhile for the Elderly Rehabilitation (POWER) training on physical and ADL performance among older adults, Master’s Thesis of Department of Physical Therapy and Assistive Technology, National Yang-Ming University, Taipei, 2009.
[3] J. Pineau et al., “Automatic detection and classification of unsafe events during power wheelchair use,” Translational Engineering in Health and Medicine, IEEE Journal of, vol. 2, pp. 1–9, 2014.
[4] Y. X. Zhi et al., “Automatic detection of compensation during robotic stroke rehabilitation therapy,” IEEE Journal of Translational Engineering in Health and Medicine, vol. 6, pp. 1-7, 2018.
[5] Y. Su et al., "A upper limb rehabilitation system with motion intention detection," 2017 2nd International Conference on Advanced Robotics and Mechatronics (ICARM), Hefei, pp. 510-516, 2017.
[6] C. C. Sun, Y. H. Wang and M. H. Sheu, "Fast motion object detection algorithm using complementary depth image on an RGB-D camera," IEEE Sensors Journal, vol. 17, no. 17, pp. 5728-5734, 2017.
[7] P. Y. Chen et al., Lower limb power rehabilitation (LLPR) using interactive video game for improvement of balance function in older people, Department of Physical Medicine and Rehabilitation, Taipei Veterans General Hospital, pp. 677-682, 2012.
[8] Kinect for Windows Sensor Components and Specifications: http://msdn.microsoft.com/en-us/library/jj131033.aspx
[9] Kinect for Windows SDK v2 basic introduced:
https://kheresy.wordpress.com/2014/12/29/kinect-for-windows-sdk-v2-basic/
[10] Unity3D system requirements:
https://unity3d.com/unity/system-requirements
[11] Ultrasonic Sensor:
http://education.rec.ri.cmu.edu/content/electronics/boe/ultrasonic_sensor/1.html
[12] SHARP INFRARED RANGER COMPARISON:
https://acroname.com/articles/sharp-infrared-ranger-comparison
[13] VL53L0X laser range :
http://www.st.com/content/ccc/resource/technical/document/datasheet/group3/b2/1e/33/77/c6/92/47/6b/DM00279086/files/DM00279086.pdf/jcr:content/translations/en.DM00279086.pdf
[14] Time of Flight camera :
http://en.wikipedia.org/wiki/Time-of-flight_camera
[15] MEDIATEK labs, Linklt Smart 7688 :
https://labs.mediatek.com/en/platform/linkit-smart-7688
[16] S. Y. Chen, Development of Kinect-based rehabilitation system in Parkinson’s disease, Master’s Thesis of Department of Mechanical Engineering, National Cheng Kung University, 2017.
[17] M. Sivan et al., “Home-based computer assisted arm rehabilitation (HCAAR) robotic device for upper limb exercise after stroke: results of a feasibility study in home setting,” Journal of NeuroEngineering and Rehabilitation, vol. 11, pp.164, 2014.
[18] G. J. Wu, Feasibility Study of Unity and Kinect-based Upper Limb Rehabilitation and Evaluation System for Stroke Survivors, Master’s Thesis of Department of Communication Engineering, Chung Yuan Christian University, 2017.
[19] M. S. H. Aung et al., "The automatic detection of chronic pain-related expression: requirements, challenges and the multimodal emopain dataset," IEEE Transactions on Affective Computing, vol. 7, no. 4, pp. 435-451, 2016.
[20] M. Muñoz-Organero et al., "Identification of walking strategies of people with osteoarthritis of the knee using insole pressure sensors," IEEE Sensors Journal, vol. 17, no. 12, pp. 3909-3920, 2017.
[21] M. A. Brodie et al., “New methods to monitor stair ascents using a wearable pendant device reveal how behavior, fear, and frailty influence falls in octogenarians,” IEEE Transactins in Biomedicak Engineering, vol. 62, no.11 , pp.2595-2601, 2015.
[22] K. J. Li, Design of Rehabilitation Monitoring System Based on Sensor Devices, Master’s Thesis of Department of Electrical Engineering, National Taipei University of Technology, Taipei, 2017.
[23] Y. Y. Liu, An Effective Approach to Tracking Rehabilitation After Knee Replacement, Master’s Thesis of Department of Electrical Engineering, National Taipei University of Technology, Taipei, 2016.
[24] L. Xia, C. C. Chen and J. K. Aggarwal, "Human detection using depth information by Kinect," CVPR 2011 WORKSHOPS, Colorado Springs, CO, pp. 15-22, 2011.
[25] S. Monir, S. Rubya and H. S. Ferdous, "Rotation and scale invariant posture recognition using Microsoft Kinect skeletal tracking feature," 2012 12th International Conference on Intelligent Systems Design and Applications (ISDA), Kochi, pp. 404-409, 2012.
[26] XAMPP :
https://www.apachefriends.org/about.html
[27] A. A. Girgis and T. L. Daniel Hwang, "Optimal Estimation Of Voltage Phasors And Frequency Deviation Using Linear And Non-Linear Kalman Filtering: Theory And Limitations," IEEE Transactions on Power Apparatus and Systems, vol. PAS-103, no. 10, pp. 2943-2951, 1984.
[28] D. H. Dini, D. P. Mandic and S. J. Julier, "A Widely Linear Complex Unscented Kalman Filter," IEEE Signal Processing Letters, vol. 18, no. 11, pp. 623-626, Nov. 2011.
[29] F. Alonge et al., "Descriptor-Type Kalman Filter and TLS EXIN Speed Estimate for Sensorless Control of a Linear Induction Motor," IEEE Transactions on Industry Applications, vol. 50, no. 6, pp. 3754-3766, 2014.
[30] C. C. Liang, Kinect Based Motion and Breath Monitoring for Frailty Syndrome Rehabilitation, Master’s Thesis of Department of Electrical Engineering, National Taiwan University of Science and Technology, Taipei ,2017
[31] T. Zhang and W. Chen, "LMD based features for the automatic seizure detection of EEG signals using SVM," IEEE Transactions on Neural Systems and Rehabilitation Engineering, vol. 25, no. 8, pp. 1100-1108, 2017.