Accessibility · Computer vision · On-device sign-language recognition
WASSEL
Egyptian Sign Language recognised in the browser, so deaf church members can make confession independently.
- signs recognised
- 58
- signs recognised
- source: public/sign-model/labels.json
- model ensemble (ONNX)
- 3
- model ensemble (ONNX)
- source: training/, public/sign-model/
- live-like top-1 (not signer-independent)
- 90.9%
- live-like top-1 (not signer-independent)
- source: training/runs/20261022-123633/report.md
- video frames uploaded
- 0
- video frames uploaded
- source: lib/recognition/* (client-side)

The problem
Deaf and hard-of-hearing members often need an interpreter for confession, one of the most private conversations there is. The interpreter is a third person in the room.
My role
- ML training pipeline, on-device inference and the real-time session app
What it is
The member signs to their own camera and the priest reads the recognised words as Arabic text. The priest’s reply plays back as a sequence of sign videos. Recognition runs entirely on the device: MediaPipe tracks the body and hands, and an ensemble of three ONNX models classifies the sign, so no video leaves the phone. Sessions are wiped when they end. The same engine powers a standalone sign translator and dictionary, with a “teach me” mode for personalising signs.
Key features
- Sign → Arabic text in real time, Arabic text → sign-video playback
- Runs in the browser: MediaPipe Holistic + onnxruntime-web, no video uploaded
- Live priest and member sessions over server-sent events (Postgres LISTEN/NOTIFY)
- Dictionary search in Arabic, Franco-Arabic and English
- “Teach me” mode: record a sign three times to personalise recognition
- Session data wiped at the end; anonymous ratings
Architecture
Device
- Camera
- MediaPipe Holistic
- Segmenter
- ONNX ensemble
App
- Next.js
- Sign → text
- Text → sign video
Realtime
- SSE stream
- Postgres LISTEN/NOTIFY
Training
- Mendeley ESL dataset (CC BY 4.0)
- PyTorch GRU
- Export to ONNX
AICustom-trained 3-model ensemble over pose and hand landmarks
AIClient-side inference: privacy by architecture
AIHonest evaluation: the training report flags that the model is not yet signer-independent
Stack
ML
App
Data
Engineering challenges
01
Privacy by architecture
Confession cannot be recorded, so recognition has to run where the camera is. Landmarks are extracted and classified in the browser, and only recognised words cross the network.
02
Reporting accuracy honestly
On live-like clips the ensemble reaches 90.9% top-1 over 58 signs, and 88.8% under a harsh new-signer stress test. The report says plainly that the test signers overlap the training signers, so the next step is recording more signers.
Screens
The app was run locally, and the ONNX model loaded in the browser. Camera recognition was not exercised during capture, so recognition screens come from the project’s own earlier presentation screenshots. The signer in those frames comes from the Mendeley Egyptian Sign Language dataset (CC BY 4.0).
Next project
ERP Business OS