Abstract

Finding an effective artificial intelligence real-time sign language recognition translator is still a barrier even with recent advancements in computer vision and machine learning more so for British sign language users as the bulk of the research has been done on American and Indian sign language. There have been methods proposed to tackle the challenges of building an effective SLR translator using data gloves, arm sensors and computer vision. Every other method aside from the computer vision method is intrusive, computationally intensive and can be inconvenient which makes them unscalable at least for now.

This work aims to present a vision-based realistic approach to bridging the communication gap between the public and the deaf-dumb community of British sign language users using a real-time BSL to English language translator capable of recognising both static and dynamic BSL gestures.

A web app was built using the Streamlit framework which hosts the deep learning systems trained to recognise the BSL signs. The system uses a webcam to feed live images to the mediapipe holistic model which detects the pose and hand landmarks. The landmarks are then extracted, pre-processed, and fed to the deep learning models for classification, with each model achieving over 90% accuracy on test data and the GRU model having 85% accuracy in a real-time test using just the hand landmarks to make a prediction.

Bridging the communication gap between the public and British sign language users with severe hearing loss using artificial intelligence