arXiv:2511.09915v2 Announce Type: replace 
Abstract: Hearing-impaired individuals often face significant barriers in daily communication due to the inherent challenges of producing clear speech. To address this, we introduce the Omni-Model paradigm into assistive technology and present HI-TransPA, an instruction-driven audio-visual personal assistant. The model fuses indistinct speech with lip dynamics, enabling both translation and dialogue within a single multimodal framework. To address the distinctive pronunciation patterns of hearing-impaired speech and the limited adaptability of existing models, we develop a multimodal preprocessing and curation pipeline that detects facial landmarks, stabilizes the lip region, and quantitatively evaluates sample quality. These quality scores guide a curriculum learning strategy that first trains on clean, high-confidence samples and progressively incorporates harder cases to strengthen model robustness. Architecturally, we employs a novel unified 3D-Resampler to efficiently encode the lip dynamics, which is critical for accurate interpretation. Experiments on purpose-built HI-Dialogue dataset show that HI-TransPA achieves state-of-the-art performance in both literal accuracy and semantic fidelity. Our work establishes a foundation for applying Omni-Models to assistive communication technology, providing an end-to-end modeling framework and essential processing tools for future research.

HI-TransPA هو مساعد شخصي مبتكر مصمم لمساعدة الأفراد ذوي الإعاقة السمعية في التواصل. باستخدام نموذج أومني، يدمج الكلام غير الواضح مع ديناميات الشفاه لتسهيل الترجمة والحوار. يستخدم النظام خط أنابيب معالجة متعددة الوسائط لتحسين وضوح الكلام من خلال اكتشاف المعالم الوجهية وتثبيت حركات الشفاه، مما يعزز فعالية التواصل للمستخدمين.

HI-TransPA es un asistente personal innovador diseñado para ayudar a las personas con discapacidad auditiva en la comunicación. Utilizando el paradigma Omni-Model, integra el habla indistinta con la dinámica de los labios para facilitar la traducción y el diálogo. El sistema emplea un pipeline de preprocesamiento multimodal que mejora la claridad del habla al detectar puntos de referencia faciales y estabilizar los movimientos de los labios, mejorando así la efectividad de la comunicación para los usuarios.

HI-TransPA est un assistant personnel innovant conçu pour aider les personnes malentendantes dans leur communication. En utilisant le paradigme Omni-Model, il intègre la parole indistincte avec la dynamique des lèvres pour faciliter la traduction et le dialogue. Le système emploie un pipeline de prétraitement multimodal qui améliore la clarté de la parole en détectant les repères faciaux et en stabilisant les mouvements des lèvres, améliorant ainsi l'efficacité de la communication pour les utilisateurs.

HI-TransPA is an innovative personal assistant designed to aid hearing-impaired individuals in communication. Utilizing the Omni-Model paradigm, it integrates indistinct speech with lip dynamics to facilitate translation and dialogue. The system employs a multimodal preprocessing pipeline that enhances speech clarity by detecting facial landmarks and stabilizing lip movements, ultimately improving communication effectiveness for users.

HI-TransPA: Hearing Impairments Translation Personal Assistant

Was this article worth reading? Share it

Ready to build your own newsroom?