arXiv:2602.01572v2 Announce Type: replace 
Abstract: Sentence representations are foundational to many Natural Language Processing (NLP) applications. While recent methods leverage Large Language Models (LLMs) to derive sentence representations, most rely on final-layer hidden states, which are optimized for next-token prediction and thus often fail to capture global, sentence-level semantics. This paper introduces a novel perspective, demonstrating that attention value vectors capture sentence semantics more effectively than hidden states. We propose Value Aggregation (VA), a simple method that pools token values across multiple layers and token indices. In a training-free setting, VA outperforms other LLM-based embeddings, even matches or surpasses the ensemble-based MetaEOL. Furthermore, we demonstrate that when paired with suitable prompts, the layer attention outputs can be interpreted as aligned weighted value vectors. Specifically, the attention scores of the last token function as the weights, while the output projection matrix ($W_O$) aligns these weighted value vectors with the common space of the LLM residual stream. This refined method, termed Aligned Weighted VA (AlignedWVA), achieves state-of-the-art performance among training-free LLM-based embeddings, outperforming the high-cost MetaEOL by a substantial margin. Finally, we highlight the potential of obtaining strong LLM embedding models through fine-tuning Value Aggregation.

أظهرت دراسة حديثة نُشرت على arXiv أن متجهات قيم الانتباه من نماذج اللغة الكبيرة (LLMs) تقدم تمثيلات جمل أفضل مقارنة بالحالات المخفية التقليدية، التي تم تحسينها بشكل أساسي لتوقع الرمز التالي. الطريقة المقترحة لتجميع القيم تجمع قيم الرموز عبر الطبقات والفهارس، مما يظهر أداءً محسنًا في التقاط الدلالات العالمية للجمل.

Un estudio reciente publicado en arXiv revela que los vectores de valores de atención de los Modelos de Lenguaje Grande (LLMs) proporcionan representaciones de oraciones superiores en comparación con los estados ocultos tradicionales, que están optimizados principalmente para la predicción del siguiente token. El método de Agregación de Valores propuesto agrupa los valores de los tokens a través de capas e índices, demostrando un rendimiento mejorado en la captura de la semántica global de las oraciones.

Une étude récente publiée sur arXiv révèle que les vecteurs de valeurs d'attention des grands modèles de langage (LLMs) fournissent de meilleures représentations de phrases par rapport aux états cachés traditionnels, qui sont principalement optimisés pour la prédiction du prochain jeton. La méthode de regroupement de valeurs proposée agrège les valeurs des jetons à travers les couches et les indices, montrant une performance améliorée dans la capture de la sémantique globale des phrases.

A recent study published on arXiv reveals that attention value vectors from Large Language Models (LLMs) provide superior sentence representations compared to traditional hidden states, which are primarily optimized for next-token prediction. The proposed Value Aggregation method pools token values across layers and indices, demonstrating enhanced performance in capturing global sentence semantics.