Motion recognition for Chinese isolated word sign language based on deep learning method

Abstract

As society continues to develop, the number of deaf and hard-of-hearing individuals has been increasing. As a primary mode of communication, sign language plays a vital role in facilitating daily interactions for these individuals, making sign language recognition increasingly important. In this context, artificial intelligence and deep learning have introduced new opportunities and challenges in the field of sign language recognition. Building on existing research, this thesis provides a summary and analysis of commonly used recognition algorithms and neural network models, focusing on isolated words. This thesis identifies and analyzes the issues and challenges associated with sign language video recognition.

Given that sign language videos involve vast amounts of data, their processing is resource-intensive. To address this, this thesis proposes a fusion of Residual Networks (ResNet) and Long Short-Term Memory networks (LSTM) tailored for practical considerations. This approach includes detailed preprocessing of sign language videos, model pre-training, feature extraction, and classification, demonstrating strong recognition accuracy on the Chinese Sign Language and the Argentine Sign Language (LSA64) datasets. By recognizing the spatiotemporal characteristics of video data, this thesis further proposes a fusion of R(2+1)D and LSTM networks. This thesis also discusses the advantages and disadvantages of the R(2+1)D networks. It details the feature extraction process for sign language videos, with LSTM networks playing a key role in extracting long-sequence features. The experiments on the CSL and the LSA64 datasets reveal high recognition accuracies up to 96.21% and 99.69%, respectively.