Designing a sign language intelligent game-based learning framework with kinect

Abstract

Sign language is a visual language for communication among the deaf people around the world in everyday life. Sign language, however, is not a universal language but different sign languages have been used in many countries, for instance, there are several sign languages in the UK, the USA and Thailand. A sign language comprises two types; signs and finger-spellings. Signs are gestures of continuous hand motion as well as hand configurations, on the other hand, fingerspellings are a mixture of continuous hand configurations. Nevertheless, there is a lack of expertise in a specification of standard sign languages. In this paper, we present a framework of intelligent game-based learning using Kinect that aims at training sign language for the hearing-impaired students or deaf students. Game-based learning is not merely a constructive game for students to play but it provides activities to learn that can guide students to achieve an end goal. Besides, the gamebased learning also supports the sign language teachers for sign language improvement to their students. Kinect is a composite device of an IR projector and an IR camera. Kinect delivers three outputs as IR image, RGB image and Depth image. The knowledge engineering method is used to capture knowledge from sign language experts. The idea is to capture an action of sign language, then analyse and give them immediate feedback and suggestion based on knowledge transfer from American Sign Language (ASL) experts. Additionally, the game-based learning is to facilitate positive student experience in order to maximize the impact of educational game.