IT 15 - ASignal: An AI-Based Interactive Tool for ASL Beginners

One essential component of human interaction is communication. Unfortunately, verbal communication is a luxury that not all Canadians have. A great deal of the world's population is nonverbal, impeding their ability to exercise their rights to access basic services in the same ease as others. According to the Canadian Association of the Deaf, over 3.2 million Canadians are Hard of Hearing (HoH), and 357,000 are deaf. This got us thinking, can AI help bridge this communication gap?

Our capstone project, ASignal, is an AI-powered learning tool designed to support the next generation of American Sign Language (ASL) learners as we aim to reduce this ever so growing gap between non-ASL users and our fellow Canadians who are deaf and HoH. From the get-go, we understood that this project would necessitate meticulous detail and research. We explored Large Language Models because they’re incredible for translation. So, we thought, why not take that same transformer architecture and use it to translate ASL into English? From there, we started brainstorming how to bring our model to life.

While we liked the idea of integrating live recording with real-time feedback, we ultimately went with a more accessible approach. We developed a user-friendly system where learners can upload pre-recorded videos of ASL gestures which the model then interprets into readable text. While not live, this approach remains impactful as it offers a valuable and engaging learning experience.  More importantly it increases awareness and appreciation of ASL and the growing communities that rely on it. 

Individuals within the HoH community suffer daily from communication barriers due to the lack of widespread knowledge of ASL. These communication obstacles can lead to a lack of employment opportunities, poor health outcomes and social isolation. Our research revealed that many HoH individuals rely heavily on lip reading as a form of communication. However, during the height of the COVID-19 pandemic, when widespread mask mandates were thought to help most, it only created an unexpected and difficult barrier making it nearly impossible for the HoH and deaf community to understand people in public spaces. This not only disrupted communication but also had a negative impact on their mental and emotional wellbeing as they felt more disconnected from society than ever before. It served as a strong reminder of how inaccessible the world can be for those who rely on visual forms of language and why simple tools like ASignal are so important.

Our project uses machine learning to develop a real-time ASL-to-text translation. Our application will be primarily used by individuals who are in the process of studying the ASL language but hopefully also spark the interest of those all around. The application will require the user to upload a video of an individual performing ASL to our website, where our backend Flask API processes the video using MediaPipe to extract hand landmarks. Our model will then translate the ASL gesture and return a text translation of the ASL gesture.

Our primary data source consists of 2,647 videos from Dr. Bill Vicar’s YouTube channel. This dataset contains examples of individuals performing different ASL hand gestures. Through preprocessing and augmentation, we increased this to over 28,000 video samples. The videos are standardized to 173 frames, with 42 joint landmarks across three spatial dimensions. Data is stored as structured NumPy arrays to optimize model input.

The architecture combines CNN layers for spatial feature extraction and Transformer layers for temporal pattern recognition and contextual decoding. We also integrated a custom learning rate scheduler and early stopping callbacks to optimize training. Our model achieved a training accuracy of 94.46% and a validation accuracy of 98.75%, exceeding our performance benchmarks.

At the root of its core, ASignal isn’t just about recognizing signs but rather it’s about recognizing the people who use them. In a world designed primarily for the hearing, our project offers a stepping stone that unveils a more inclusive society.