arXiv:2506.15881v2 Announce Type: replace 
Abstract: SHallow REcurrent Decoders (SHRED) are effective for system identification and forecasting from sparse sensor measurements. Such models are light-weight and computationally efficient, allowing them to be trained on consumer laptops. SHRED-based models rely on Recurrent Neural Networks (RNNs) and a simple Multi-Layer Perceptron (MLP) for the temporal encoding and spatial decoding respectively. Despite the relatively simple structure of SHRED, they are able to predict chaotic dynamical systems on different physical, spatial, and temporal scales directly from a sparse set of sensor measurements. In this work, we modify SHRED by leveraging transformers (T-SHRED) embedded with symbolic regression for the temporal encoding, circumventing auto-regressive long-term forecasting for physical data. This is achieved through a new sparse identification of nonlinear dynamics (SINDy) attention mechanism into T-SHRED to impose sparsity regularization on the latent space, which also allows for immediate symbolic interpretation. Symbolic regression improves model interpretability by learning and regularizing the dynamics of the latent space during training. We analyze the performance of T-SHRED on three different dynamical systems ranging from low-data to high-data regimes.

تم تطوير T-SHRED، وهو نهج جديد يدمج الانحدار الرمزي مع فك التشفير المتكرر الضحل القائم على المحولات، لتحسين تحديد الأنظمة والتنبؤ من بيانات المستشعرات النادرة. تعدل هذه الطريقة نماذج SHRED التقليدية من خلال دمج آلية انتباه جديدة لتحديد الديناميات غير الخطية بشكل نادر، مما يسمح بتنبؤات فعالة للأنظمة الفوضوية عبر مقاييس مختلفة.

T-SHRED, un enfoque novedoso que integra la regresión simbólica con decodificadores recurrentes superficiales basados en transformadores, ha sido desarrollado para mejorar la identificación de sistemas y la previsión a partir de datos de sensores escasos. Este método modifica los modelos SHRED tradicionales al incorporar un nuevo mecanismo de atención para la identificación escasa de dinámicas no lineales, permitiendo predicciones efectivas de sistemas caóticos en diversas escalas.

T-SHRED, une nouvelle approche intégrant la régression symbolique avec des décodeurs récurrents peu profonds basés sur des transformateurs, a été développée pour améliorer l'identification des systèmes et les prévisions à partir de données de capteurs rares. Cette méthode modifie les modèles SHRED traditionnels en incorporant un nouveau mécanisme d'attention pour l'identification sparse des dynamiques non linéaires, permettant des prédictions efficaces des systèmes chaotiques à différentes échelles.

T-SHRED, a novel approach integrating symbolic regression with transformer-based shallow recurrent decoders, has been developed to enhance system identification and forecasting from sparse sensor data. This method modifies traditional SHRED models by incorporating a new attention mechanism for sparse identification of nonlinear dynamics, allowing for effective predictions of chaotic systems across various scales.

T-SHRED: Symbolic Regression for Regularization and Model Discovery with Transformer Shallow Recurrent Decoders

arXiv:2511.18146v1 Announce Type: new 
Abstract: This study introduce GeeSanBhava, a high-quality data set of Sinhala song comments extracted from YouTube manually tagged using Russells Valence-Arousal model by three independent human annotators. The human annotators achieve a substantial inter-annotator agreement (Fleiss kappa = 84.96%). The analysis revealed distinct emotional profiles for different songs, highlighting the importance of comment based emotion mapping. The study also addressed the challenges of comparing comment-based and song-based emotions, mitigating biases inherent in user-generated content. A number of Machine learning and deep learning models were pre-trained on a related large data set of Sinhala News comments in order to report the zero-shot result of our Sinhala YouTube comment data set. An optimized Multi-Layer Perceptron model, after extensive hyperparameter tuning, achieved a ROC-AUC score of 0.887. The model is a three-layer MLP with a configuration of 256, 128, and 64 neurons. This research contributes a valuable annotated dataset and provides insights for future work in Sinhala Natural Language Processing and music emotion recognition.

تقدم الدراسة GeeSanBhava، مجموعة بيانات شاملة لتعليقات الأغاني السنهالية المستخرجة من يوتيوب، والتي تم وضع علامات عليها بدقة باستخدام نموذج فالنس-أروزال الخاص برسل بواسطة ثلاثة مُعَلقين مستقلين، مما حقق توافقًا عاليًا بين المُعَلِقين بنسبة 84.96%. تسلط مجموعة البيانات هذه الضوء على الملفات العاطفية المرتبطة بأغاني مختلفة، مما يبرز أهمية رسم المشاعر بناءً على التعليقات.

El estudio presenta GeeSanBhava, un conjunto de datos integral de comentarios sobre canciones en cingalés extraídos de YouTube, que ha sido etiquetado meticulosamente utilizando el modelo de Valencia-Arousal de Russell por tres anotadores independientes, logrando un alto acuerdo inter-anotador del 84.96%. Este conjunto de datos resalta los perfiles emocionales asociados con diferentes canciones, enfatizando la importancia de la mapeo emocional basado en comentarios.

L'étude présente GeeSanBhava, un ensemble de données complet de commentaires sur des chansons sinhala extraits de YouTube, qui a été soigneusement étiqueté à l'aide du modèle de Valence-Arousal de Russell par trois annotateurs indépendants, atteignant un accord inter-annotateur élevé de 84,96 %. Cet ensemble de données met en évidence les profils émotionnels associés à différentes chansons, soulignant l'importance de la cartographie des émotions basée sur les commentaires.

The study introduces GeeSanBhava, a comprehensive dataset of Sinhala song comments sourced from YouTube, which has been meticulously tagged using Russell's Valence-Arousal model by three independent annotators, achieving a high inter-annotator agreement of 84.96%. This dataset highlights the emotional profiles associated with different songs, emphasizing the significance of comment-based emotion mapping.

T-SHRED: Symbolic Regression for Regularization and Model Discovery with Transformer Shallow Recurrent Decoders

Was this article worth reading? Share it

Shakker-ai

Attentive AI

Hypertune