arXiv:2511.19907v2 Announce Type: replace 
Abstract: This paper employs a multimodal approach for continuous sign recognition by first using ML for detecting the start and end frames of signs in videos of American Sign Language (ASL) sentences, and then by recognizing the segmented signs. For improved robustness we use 3D skeletal features extracted from sign language videos to take into account the convergence of sign properties and their dynamics that tend to cluster at sign boundaries. Another focus of this paper is the incorporation of information from 3D handshape for boundary detection. To detect handshapes normally expected at the beginning and end of signs, we pretrain a handshape classifier for detection of 87 linguistically defined canonical handshape categories using a dataset that we created by integrating and normalizing several existing datasets. A multimodal fusion module is then used to unify the pretrained sign video segmentation framework and handshape classification models. Finally, the estimated boundaries are used for sign recognition, where the recognition model is trained on a large database containing both citation-form isolated signs and signs pre-segmented (based on manual annotations) from continuous signing-as such signs often differ a bit in certain respects. We evaluate our method on the ASLLRP corpus and demonstrate significant improvements over previous work.

تقدم دراسة جديدة نهجًا متعدد الوسائط للتعرف المستمر على لغة الإشارة، مع التركيز على لغة الإشارة الأمريكية (ASL). تستخدم الطريقة التعلم الآلي لاكتشاف بدايات ونهايات الإشارات في مقاطع الفيديو، مما يحسن عملية التعرف من خلال دمج ميزات هيكلية ثلاثية الأبعاد ومصنف أشكال اليد المدرب مسبقًا لتحديد فئات أشكال اليد الكانونية. يهدف هذا الإطار المبتكر إلى تحسين دقة وموثوقية أنظمة التعرف على لغة الإشارة.

Un nuevo estudio presenta un enfoque multimodal para el reconocimiento continuo de la lengua de señas, centrándose en la lengua de señas americana (ASL). El método utiliza aprendizaje automático para detectar los inicios y finales de los signos en videos, mejorando el proceso de reconocimiento al incorporar características esqueléticas en 3D y un clasificador de formas de mano preentrenado para identificar categorías de formas de mano canónicas. Este marco innovador tiene como objetivo mejorar la precisión y robustez de los sistemas de reconocimiento de la lengua de señas.

Une nouvelle étude présente une approche multimodale pour la reconnaissance continue de la langue des signes, en se concentrant sur la langue des signes américaine (ASL). La méthode utilise l'apprentissage automatique pour détecter les débuts et fins de signes dans les vidéos, améliorant le processus de reconnaissance en intégrant des caractéristiques squelettiques 3D et un classificateur de formes de main pré-entraîné pour identifier les catégories de formes de main canoniques. Ce cadre innovant vise à améliorer la précision et la robustesse des systèmes de reconnaissance de la langue des sig…

A new study introduces a multimodal approach for continuous sign language recognition, focusing on American Sign Language (ASL). The method utilizes machine learning to detect the start and end frames of signs in videos, enhancing the recognition process by incorporating 3D skeletal features and a pretrained handshape classifier for identifying canonical handshape categories. This innovative framework aims to improve the accuracy and robustness of sign language recognition systems.

MHB: Multimodal Handshape-aware Boundary Detection for Continuous Sign Language Recognition

arXiv:2512.08040v1 Announce Type: new 
Abstract: Our aim is to develop a unified model for sign language understanding, that performs sign language translation (SLT) and sign-subtitle alignment (SSA). Together, these two tasks enable the conversion of continuous signing videos into spoken language text and also the temporal alignment of signing with subtitles -- both essential for practical communication, large-scale corpus construction, and educational applications. To achieve this, our approach is built upon three components: (i) a lightweight visual backbone that captures manual and non-manual cues from human keypoints and lip-region images while preserving signer privacy; (ii) a Sliding Perceiver mapping network that aggregates consecutive visual features into word-level embeddings to bridge the vision-text gap; and (iii) a multi-task scalable training strategy that jointly optimises SLT and SSA, reinforcing both linguistic and temporal alignment. To promote cross-linguistic generalisation, we pretrain our model on large-scale sign-text corpora covering British Sign Language (BSL) and American Sign Language (ASL) from the BOBSL and YouTube-SL-25 datasets. With this multilingual pretraining and strong model design, we achieve state-of-the-art results on the challenging BOBSL (BSL) dataset for both SLT and SSA. Our model also demonstrates robust zero-shot generalisation and finetuned SLT performance on How2Sign (ASL), highlighting the potential of scalable translation across different sign languages.

تم تطوير نموذج موحد جديد لفهم لغة الإشارة، يركز على ترجمة لغة الإشارة (SLT) ومحاذاة العناوين الفرعية (SSA). يهدف هذا النموذج إلى تحويل مقاطع الفيديو المستمرة للإشارة إلى نص بلغة منطوقة ومحاذاة الإشارات مع العناوين الفرعية، مما يعزز التواصل العملي والتطبيقات التعليمية.

Se ha desarrollado un nuevo modelo unificado para la comprensión de la lengua de signos, centrado en la traducción de la lengua de signos (SLT) y la alineación de subtítulos (SSA). Este modelo tiene como objetivo convertir videos de signos continuos en texto en lengua hablada y alinear los signos con los subtítulos, mejorando así la comunicación práctica y las aplicaciones educativas.

Un nouveau modèle unifié pour la compréhension de la langue des signes a été développé, se concentrant sur la traduction de la langue des signes (SLT) et l'alignement des sous-titres (SSA). Ce modèle vise à convertir des vidéos de signes continus en texte de langue parlée et à aligner les signes avec les sous-titres, améliorant ainsi la communication pratique et les applications éducatives.

A new unified model for sign language understanding has been developed, focusing on sign language translation (SLT) and sign-subtitle alignment (SSA). This model aims to convert continuous signing videos into spoken language text and align signing with subtitles, enhancing practical communication and educational applications.

MHB: Multimodal Handshape-aware Boundary Detection for Continuous Sign Language Recognition

Was this article worth reading? Share it

LucidQuery AI

ShareSpeak

OpenL Translator