A research prototype exploring communication between spoken English and Namibian Sign Language (NSL) through speech processing, computer vision and avatar-based visualisation.
Righteous Wasambo developed Sign Avatar as a final-year computer science project. It reached a desktop proof of concept and was evaluated under controlled conditions.
View the GitHub repositoryThe problem
A healthcare consultation depends on being able to ask questions, describe symptoms and understand the response. When a clinician and a patient who uses NSL cannot communicate directly and an interpreter is unavailable, even basic exchanges become difficult.
The research asked whether a prototype could support simple exchanges in both directions, and where the technology would fall short.
The approach
The system breaks each direction into stages so they can be tested separately. Both routes use glosses, short written labels for signs, as an intermediate representation. This made it easier to tell where a mistake started and how it affected the next stage.
Spoken English to sign visualisation
- Spoken English
- Speech transcription
- NSL glosses
- Landmark avatar
Signed video to English
- Signed video
- Landmark tracking and recognition
- Predicted glosses
- English text and optional speech
What was built
A Python desktop prototype brings the experiments into a tabbed PyQt interface. The forward path transcribes speech, maps English text to signs in a limited vocabulary, then plays a skeletal avatar using recorded movement landmarks.
The experimental reverse path tracks hand or body landmarks in video, predicts signs and tries to reconstruct English from those predictions. A separate integrated demo also tested reading the resulting text aloud.
Decisions and trade-offs
Balance speed against recognition accuracy
The experiments compared hand landmarks alone with hands plus body pose, and tested processing every frame against skipping frames. In the controlled tests, sampling fewer frames made processing faster. Adding pose cost more processing time and its benefit depended on the configuration.
Build the avatar from recorded movement
The visualisation replays representative landmark sequences and interpolates between signs. This made it possible to assemble demonstrations from a small vocabulary. How those transitions read to fluent NSL signers still needs evaluation.
Test what synthetic language data can support
With limited paired NSL and English data, the project generated training examples from text. The resulting English could be repetitive or incomplete, especially once recognition errors reached the translation stage.
What the research found
The evaluation covered held-out clips of isolated signs, gloss-to-text experiments and timing tests on a small set of audio and video inputs. It demonstrated that the modules could work together under controlled conditions. Recognising isolated signs is a different problem from finding where signs begin and end in continuous video.
The work remains a research prototype. Clinical deployment and evaluation with NSL signers and healthcare staff were outside the study. The next steps are to collect more authentic NSL data, improve continuous-sign segmentation and evaluate the prototype with signers under varied conditions.