Action Recognition From Thermal Videos Using Joint and Skeleton Information

Abstract

Although various studies based on thermal images have been conducted, few studies have focused on the simultaneous extraction of joints and skeleton information of an object from a thermal image, and performed human action recognition using this information. Unlike in the case of visible light images, performing joint detection and skeleton generation on thermal images often leads to the complete disappearance of spatial information such as joints. In this case, it is extremely difficult to extract joints information from the object. Moreover, the accuracy of action recognition is significantly reduced owing to this issue. Therefore, a new method to extract joints and skeleton information is proposed in this study to address these issues. In the proposed method, an original 1-channel thermal image was converted into a 3-channel thermal image and then the images were combined to improve the extraction performance. A generative adversarial network (GAN) was used in the proposed method for extracting joints and skeleton information. In addition, research to recognize various human actions was conducted using the joints and skeleton information extracted by this method. The proposed human action recognition is performed by combining a convolutional neural network (CNN) and long short-term memory (LSTM). As a result of the experiments using self-collected and open data, it was found that the method proposed in this study shows good performance compared to other state-of-the-art methods.