arXiv:2511.14531v1 Announce Type: new 
Abstract: With Retrieval Augmented Generation (RAG) becoming more and more prominent in generative AI solutions, there is an emerging need for systematically evaluating their effectiveness. We introduce the LiveRAG benchmark, a publicly available dataset of 895 synthetic questions and answers designed to support systematic evaluation of RAG-based Q&A systems. This synthetic benchmark is derived from the one used during the SIGIR'2025 LiveRAG Challenge, where competitors were evaluated under strict time constraints. It is augmented with information that was not made available to competitors during the Challenge, such as the ground-truth answers, together with their associated supporting claims which were used for evaluating competitors' answers. In addition, each question is associated with estimated difficulty and discriminability scores, derived from applying an Item Response Theory model to competitors' responses. Our analysis highlights the benchmark's questions diversity, the wide range of their difficulty levels, and their usefulness in differentiating between system capabilities. The LiveRAG benchmark will hopefully help the community advance RAG research, conduct systematic evaluation, and develop more robust Q&A systems.

يقدم معيار LiveRAG مجموعة بيانات عامة تحتوي على 895 سؤالًا وإجابةً صناعيًا مصممة لتقييم أنظمة الجيل المعزز بالاسترجاع (RAG). تم اشتقاق هذه المجموعة من البيانات من تحدي LiveRAG في SIGIR'2025، وتشتمل على إجابات صحيحة وادعاءات داعمة لم تكن متاحة للمنافسين خلال التحدي. يتم ربط كل سؤال بدرجات صعوبة وتمييز مستمدة من تطبيق نظرية استجابة العناصر، مما يعزز عملية تقييم أنظمة الأسئلة والأجوبة المعتمدة على RAG.

El benchmark LiveRAG presenta un conjunto de datos público de 895 preguntas y respuestas sintéticas diseñado para evaluar sistemas de Generación Aumentada por Recuperación (RAG). Este conjunto de datos, derivado del Desafío LiveRAG de SIGIR'2025, incluye respuestas verdaderas y afirmaciones de apoyo que no estaban disponibles para los competidores durante el desafío. Cada pregunta se asocia con puntajes de dificultad y discriminación basados en la Teoría de Respuesta a Ítems, mejorando así el proceso de evaluación de sistemas de preguntas y respuestas basados en RAG.

Le benchmark LiveRAG présente un ensemble de données publiques de 895 questions et réponses synthétiques destiné à évaluer les systèmes de génération augmentée par récupération (RAG). Cet ensemble de données, dérivé du défi LiveRAG de SIGIR'2025, comprend des réponses véridiques et des affirmations de soutien qui n'étaient pas accessibles aux concurrents pendant le défi. Chaque question est associée à des scores de difficulté et de discriminabilité basés sur la théorie de la réponse à l'item, améliorant ainsi le processus d'évaluation des systèmes de questions-réponses basés sur RAG.

The LiveRAG benchmark introduces a publicly available dataset of 895 synthetic questions and answers aimed at evaluating Retrieval Augmented Generation (RAG) systems. This dataset, derived from the SIGIR'2025 LiveRAG Challenge, includes ground-truth answers and supporting claims that were not accessible to competitors during the challenge. Each question is assigned difficulty and discriminability scores based on Item Response Theory, enhancing the evaluation process for RAG-based Q&A systems.

LiveRAG: A diverse Q&A dataset with varying difficulty level for RAG evaluation

arXiv:2511.16681v1 Announce Type: new 
Abstract: Retrieval-Augmented Generation (RAG) systems have become a dominant approach to augment large language models (LLMs) with external knowledge. However, existing vector database (VecDB) retrieval pipelines rely on flat or single-resolution indexing structures, which cannot adapt to the varying semantic granularity required by diverse user queries. This limitation leads to suboptimal trade-offs between retrieval speed and contextual relevance.
  To address this, we propose \textbf{Semantic Pyramid Indexing (SPI)}, a novel multi-resolution vector indexing framework that introduces query-adaptive resolution control for RAG in VecDBs. Unlike existing hierarchical methods that require offline tuning or separate model training, SPI constructs a semantic pyramid over document embeddings and dynamically selects the optimal resolution level per query through a lightweight classifier. This adaptive approach enables progressive retrieval from coarse-to-fine representations, significantly accelerating search while maintaining semantic coverage.
  We implement SPI as a plugin for both FAISS and Qdrant backends and evaluate it across multiple RAG tasks including MS MARCO, Natural Questions, and multimodal retrieval benchmarks. SPI achieves up to \textbf{5.7$\times$} retrieval speedup and \textbf{1.8$\times$} memory efficiency gain while improving end-to-end QA F1 scores by up to \textbf{2.5 points} compared to strong baselines. Our theoretical analysis provides guarantees on retrieval quality and latency bounds, while extensive ablation studies validate the contribution of each component. The framework's compatibility with existing VecDB infrastructures makes it readily deployable in production RAG systems. Code is availabe at \href{https://github.com/FastLM/SPI_VecDB}{https://github.com/FastLM/SPI\_VecDB}.

تم اقتراح إطار عمل جديد يسمى Semantic Pyramid Indexing (SPI) لتحسين أنظمة الجيل المعزز بالاسترجاع (RAG) من خلال السماح بفهرسة متجهات متعددة الدقة في قواعد بيانات المتجهات (VecDBs). تتناول هذه المقاربة المبتكرة قيود خطوط أنابيب الاسترجاع الحالية التي تعتمد على هياكل الفهرسة المسطحة، والتي تواجه صعوبات مع التباين الدلالي في استفسارات المستخدمين.

Se ha propuesto un nuevo marco llamado Semantic Pyramid Indexing (SPI) para mejorar los sistemas de Generación Aumentada por Recuperación (RAG) al permitir la indexación vectorial de múltiples resoluciones en bases de datos vectoriales (VecDBs). Este enfoque innovador aborda las limitaciones de los pipelines de recuperación existentes que dependen de estructuras de indexación planas, que tienen dificultades con la granularidad semántica variable en las consultas de los usuarios.

Un nouveau cadre appelé Semantic Pyramid Indexing (SPI) a été proposé pour améliorer les systèmes de génération augmentée par récupération (RAG) en permettant un indexage vectoriel multi-résolution dans les bases de données vectorielles (VecDBs). Cette approche innovante répond aux limitations des pipelines de récupération existants qui reposent sur des structures d'indexation plates, qui peinent à gérer la granularité sémantique variable dans les requêtes des utilisateurs.

A new framework called Semantic Pyramid Indexing (SPI) has been proposed to enhance Retrieval-Augmented Generation (RAG) systems by allowing for multi-resolution vector indexing in vector databases (VecDBs). This innovative approach addresses the limitations of existing retrieval pipelines that rely on flat indexing structures, which struggle with varying semantic granularity in user queries.

LiveRAG: A diverse Q&A dataset with varying difficulty level for RAG evaluation

Was this article worth reading? Share it

HubRE AI

Augmeta

Quail