A smart dialogue box for the hearing impaired

Abstract

Conversations between a person with severe hearing impairment and a hearing person is difficult because people who know sign language are rare. Therefore, to facilitate the conversation, we design and develop a portable dialogue box that can be used without an interpreter. The system detects sign language of a deaf person by capturing hand images with a camera. The images are pre-processed and the histograms of oriented gradients (HOG) are extracted as image features. A multi-class support vector machine uses the HOGs as inputs to classify the alphabets. The vocal reply from a hearing person is received through a microphone and interpreted via Google cloud speech API. Both parts are processed by a microcomputer to produce messages of the conversation. The dialogue is displayed on the graphical user interface. The system can also record the conversation logs into the database.