arXiv:2510.26095v1 Announce Type: cross 
Abstract: Recommender systems are among the most impactful AI applications, interacting with billions of users every day, guiding them to relevant products, services, or information tailored to their preferences. However, the research and development of recommender systems are hindered by existing datasets that fail to capture realistic user behaviors and inconsistent evaluation settings that lead to ambiguous conclusions. This paper introduces the Open Recommendation Benchmark for Reproducible Research with HIdden Tests (ORBIT), a unified benchmark for consistent and realistic evaluation of recommendation models. ORBIT offers a standardized evaluation framework of public datasets with reproducible splits and transparent settings for its public leaderboard. Additionally, ORBIT introduces a new webpage recommendation task, ClueWeb-Reco, featuring web browsing sequences from 87 million public, high-quality webpages. ClueWeb-Reco is a synthetic dataset derived from real, user-consented, and privacy-guaranteed browsing data. It aligns with modern recommendation scenarios and is reserved as the hidden test part of our leaderboard to challenge recommendation models' generalization ability. ORBIT measures 12 representative recommendation models on its public benchmark and introduces a prompted LLM baseline on the ClueWeb-Reco hidden test. Our benchmark results reflect general improvements of recommender systems on the public datasets, with variable individual performances. The results on the hidden test reveal the limitations of existing approaches in large-scale webpage recommendation and highlight the potential for improvements with LLM integrations. ORBIT benchmark, leaderboard, and codebase are available at https://www.open-reco-bench.ai.

تقديم ORBIT، معيار التوصية المفتوح للبحث القابل للتكرار، يمثل تقدماً كبيراً في مجال أنظمة التوصية. تهدف هذه المبادرة إلى معالجة التحديات التي تطرحها مجموعات البيانات الحالية التي لا تعكس سلوكيات المستخدمين بدقة وعدم الاتساق في طرق التقييم. من خلال توفير إطار موحد، ستعزز ORBIT موثوقية نتائج البحث، مما يعود بالنفع على المطورين والمستخدمين على حد سواء. هذا أمر بالغ الأهمية حيث تلعب أنظمة التوصية دوراً حيوياً في توجيه مليارات المستخدمين نحو محتوى مخصص كل يوم.

La introducción de ORBIT, el Benchmark Abierto de Recomendación para la Investigación Reproducible, representa un avance significativo en el campo de los sistemas de recomendación. Esta iniciativa busca abordar los desafíos que presentan los conjuntos de datos actuales, que no reflejan con precisión los comportamientos de los usuarios y las inconsistencias en los métodos de evaluación. Al proporcionar un marco estandarizado, ORBIT mejorará la fiabilidad de los resultados de investigación, beneficiando tanto a desarrolladores como a usuarios. Esto es crucial, ya que los sistemas de recomendación juegan un papel vital al guiar a miles de millones de usuarios hacia contenido personalizado cada día.

L'introduction d'ORBIT, le Benchmark Ouvert de Recommandation pour la Recherche Reproductible, marque une avancée significative dans le domaine des systèmes de recommandation. Cette initiative vise à résoudre les défis posés par les ensembles de données actuels qui ne reflètent pas fidèlement les comportements des utilisateurs et les incohérences dans les méthodes d'évaluation. En fournissant un cadre standardisé, ORBIT améliorera la fiabilité des résultats de recherche, bénéficiant ainsi aux développeurs et aux utilisateurs. Cela est crucial car les systèmes de recommandation jouent un rôle vital en guidant des milliards d'utilisateurs vers un contenu personnalisé chaque jour.

The introduction of ORBIT, the Open Recommendation Benchmark for Reproducible Research, marks a significant advancement in the field of recommender systems. This initiative aims to address the challenges posed by current datasets that do not accurately reflect user behaviors and the inconsistencies in evaluation methods. By providing a standardized framework, ORBIT will enhance the reliability of research outcomes, ultimately benefiting developers and users alike. This is crucial as recommender systems play a vital role in guiding billions of users to personalized content every day.

ORBIT - Open Recommendation Benchmark for Reproducible Research with Hidden Tests

One More Thing in AI – Your Shortcut to AI Mastery

ORBIT - Open Recommendation Benchmark for Reproducible Research with Hidden Tests

Was this article worth reading? Share it

One More Thing in AI

Topic

Competitor Research

Octofy

Subtle

AIPortalX

Ready to build your own newsroom?