arXiv:2502.17129v2 Announce Type: replace 
Abstract: Long context is an important topic in Natural Language Processing (NLP), running through the development of NLP architectures, and offers immense opportunities for Large Language Models (LLMs), giving LLMs the lifelong learning potential akin to humans. Unfortunately, the pursuit of a long context is accompanied by numerous obstacles. Nevertheless, long context remains a core competitive advantage for LLMs. In the past two years, the context length of LLMs has achieved a breakthrough extension to millions of tokens. Moreover, research on long-context LLMs has expanded beyond length extrapolation to a comprehensive focus on architecture, infrastructure, training, and evaluation technologies.
  Inspired by the symphonic poem, Thus Spake Zarathustra, we draw an analogy between the journey of extending the context of LLM and the attempts of humans to transcend their mortality. In this survey, we will illustrate how LLM struggles between the tremendous need for a longer context and its equal need to accept the fact that it is ultimately finite. To achieve this, we give a global picture of the lifecycle of long-context LLMs from four perspectives: architecture, infrastructure, training, and evaluation, showcasing the full spectrum of long-context technologies. At the end of this survey, we will present 10 unanswered questions currently faced by long-context LLMs. We hope this survey can serve as a systematic introduction to research on long-context LLMs.
  Video: https://www.bilibili.com/video/BV11h9AYoEYj.
  Github: https://github.com/OpenMOSS/Thus-Spake-Long-Context-LLM.

تسلط دراسة حديثة الضوء على أهمية السياق الطويل في معالجة اللغة الطبيعية (NLP) وتأثيره على نماذج اللغة الكبيرة (LLMs). لقد تم توسيع طول السياق لنماذج LLMs مؤخرًا إلى ملايين الرموز، مما يقدم فرصًا وتحديات. هذه الخطوة مهمة لأنها تعزز قدرة النماذج على التعلم ومعالجة المعلومات بطريقة تشبه البشر، على الرغم من أنها تثير أيضًا تساؤلات حول حدودها.

Una reciente encuesta destaca la importancia del contexto largo en el procesamiento del lenguaje natural (NLP) y su impacto en los modelos de lenguaje de gran tamaño (LLMs). La longitud del contexto de los LLMs se ha ampliado recientemente a millones de tokens, presentando tanto oportunidades como desafíos. Este avance es crucial ya que mejora la capacidad de los modelos para aprender y procesar información de manera más similar a los humanos, aunque también plantea preguntas sobre sus limitaciones.

Une récente enquête souligne l'importance du long contexte dans le traitement du langage naturel (NLP) et son impact sur les modèles de langage de grande taille (LLMs). La longueur de contexte des LLMs a récemment été étendue à des millions de tokens, présentant à la fois des opportunités et des défis. Cette avancée est cruciale car elle améliore la capacité des modèles à apprendre et à traiter l'information de manière plus humaine, bien qu'elle soulève également des questions sur leurs limites.

A recent survey highlights the significance of long context in Natural Language Processing (NLP) and its impact on Large Language Models (LLMs). The context length of LLMs has recently expanded to millions of tokens, presenting both opportunities and challenges. This advancement is crucial as it enhances the models' ability to learn and process information more like humans, although it also raises questions about their limitations.

Thus Spake Long-Context Large Language Model

Was this article worth reading? Share it

LucidQuery AI

Airparser

Langtail

Langfuse

Supametas.AI

CodeSpaced

Ready to build your own newsroom?