arXiv:2505.18973v3 Announce Type: replace-cross 
Abstract: Selective state-space models excel at long-sequence modeling, but their capacity for language representation -- in complex hierarchical reasoning -- remains underexplored. Most large language models rely on \textit{flat} Euclidean embeddings, limiting their ability to capture latent hierarchies. To address this, we propose {\it Hierarchical Mamba (HiM)}, integrating efficient Mamba2 with hyperbolic geometry to learn hierarchy-aware language embeddings for deeper linguistic understanding. Mamba2-processed sequences are projected to the Poincar\'e ball or Lorentzian manifold with ``learnable'' curvature, optimized with a hyperbolic loss. Our HiM model facilitates the capture of relational distances across varying hierarchical levels, enabling effective long-range reasoning for tasks like mixed-hop prediction and multi-hop inference in hierarchical classification. Experimental results show both HiM variants effectively capture hierarchical relationships across four linguistic and medical datasets, surpassing Euclidean baselines, with HiM-Poincar\'e providing fine-grained distinctions with higher h-norms, while HiM-Lorentz offers more stable, compact, and hierarchy-preserving embeddings-favoring robustness. The source code is publicly available at https://github.com/BerryByte/HiM.

يمثل تقديم نموذج Hierarchical Mamba (HiM) تقدمًا كبيرًا في تمثيل اللغة، حيث يستخدم الهندسة الزائدية لتحسين فهم التفكير الهرمي المعقد في المهام اللغوية. يدمج هذا النموذج تقنية Mamba2، حيث يتم إسقاط التسلسلات على مساحات زائدة لتحسين تضمينات اللغة من أجل التفكير على المدى الطويل. تشير النتائج التجريبية إلى أن HiM يلتقط بفعالية العلاقات الهرمية، مما يحسن الأداء في مهام مثل الاستدلال متعدد القفزات والتصنيف الهرمي.

La introducción del modelo Hierarchical Mamba (HiM) marca un avance significativo en la representación del lenguaje, utilizando geometría hiperbólica para mejorar la comprensión del razonamiento jerárquico complejo en tareas lingüísticas. Este modelo integra la tecnología Mamba2, proyectando secuencias en espacios hiperbólicos para optimizar las incrustaciones lingüísticas para el razonamiento a largo plazo. Los resultados experimentales indican que HiM captura eficazmente las relaciones jerárquicas, mejorando el rendimiento en tareas como la inferencia de múltiples saltos y la clasificación j…

L'introduction du modèle Hierarchical Mamba (HiM) représente une avancée significative dans la représentation du langage, utilisant la géométrie hyperbolique pour améliorer la compréhension du raisonnement hiérarchique complexe dans les tâches linguistiques. Ce modèle intègre la technologie Mamba2, projetant des séquences sur des espaces hyperboliques pour optimiser les embeddings linguistiques pour le raisonnement à long terme. Les résultats expérimentaux indiquent que HiM capture efficacement les relations hiérarchiques, améliorant les performances dans des tâches telles que l'inférence mult…

The introduction of the Hierarchical Mamba (HiM) model marks a significant advancement in language representation, utilizing hyperbolic geometry to enhance the understanding of complex hierarchical reasoning in language tasks. This model integrates Mamba2 technology, projecting sequences onto hyperbolic spaces to optimize language embeddings for long-range reasoning. Experimental results indicate that HiM effectively captures hierarchical relationships, improving performance in tasks such as multi-hop inference and hierarchical classification.

Hierarchical Mamba Meets Hyperbolic Geometry: A New Paradigm for Structured Language Embeddings

arXiv:2510.07318v2 Announce Type: replace-cross 
Abstract: Long-sequence modeling faces a fundamental trade-off between the efficiency of compressive fixed-size memory in RNN-like models and the fidelity of lossless growing memory in attention-based Transformers. Inspired by the Multi-Store Model in cognitive science, we introduce a memory framework of artificial neural networks. Our method maintains a sliding window of the Transformer's KV cache as lossless short-term memory, while a learnable module termed Artificial Hippocampus Network (AHN) recurrently compresses out-of-window information into a fixed-size compact long-term memory. To validate this framework, we instantiate AHNs using modern RNN-like architectures, including Mamba2, DeltaNet, and GatedDeltaNet to augment open-weight LLMs. We also propose an efficient self-distillation training method where the base model's all parameters are frozen and only the parameters from AHNs are optimized. For inference, our method sets a default large sliding window size of 32k for attention, and AHNs activate only when the sequence length exceeds the 32k window, addressing the quadratic-complexity issue of attention that emerges at that scale. Extensive experiments on long-context benchmarks LV-Eval and InfiniteBench demonstrate that AHN-augmented models consistently outperform sliding window baselines and achieve performance comparable or even superior to full-attention models, while substantially reducing computational and memory requirements. For instance, augmenting the Qwen2.5-3B-Instruct with AHNs reduces inference FLOPs by 40.5% and memory cache by 74.0%, while improving its average score on LV-Eval (128k sequence length) from 4.41 to 5.88. Code is available at: https://github.com/ByteDance-Seed/AHN.

Artificial Hippocampus Networks for Efficient Long-Context Modeling

One More Thing in AI – Your Shortcut to AI Mastery

Hierarchical Mamba Meets Hyperbolic Geometry: A New Paradigm for Structured Language Embeddings

Was this article worth reading? Share it

One More Thing in AI

LucidQuery AI

Magicley AI

Supametas.AI

Https

Hypertune

Ready to build your own newsroom?