arXiv:2603.03194v2 Announce Type: replace 
Abstract: Current code-agent benchmarks primarily evaluate localized issue resolution within a single target repository, leaving under-tested many software engineering tasks that require external knowledge or broader repository-level changes. We introduce BeyondSWE, a 500-instance benchmark drawn from 246 real-world GitHub repositories to evaluate code agents beyond single-repository bug fixing. BeyondSWE covers four representative settings: cross-repository issue resolution, domain-specific issue resolution, dependency-driven migration, and document-to-repository generation, spanning both broader knowledge scope and broader resolution scope. Our evaluation shows that BeyondSWE remains far from saturated: the best OpenHands-based agent reaches 46.12 average score, while the strongest Codex harness with GPT-5.4 (xhigh) reaches 56.65 under a search-aware prompt. To study whether external information access closes this gap, we use SearchSWE as a controlled diagnostic baseline for search-augmented coding. Search access improves most models and substantially helps some tasks, but the gains remain limited and uneven, showing that current agents still struggle to convert retrieved information into precise, version-compatible, and locally actionable code changes. These results suggest that deep search for coding remains an open problem: progress requires agents that can reliably combine external evidence with repository-local reasoning and execution-based verification.

يمثل تقديم BeyondSWE تقدمًا كبيرًا في تقييم وكلاء الشيفرة خارج نطاقهم التقليدي في إصلاح الأخطاء في مستودع واحد. يتكون هذا المعيار الجديد من 500 حالة مستمدة من 246 مستودعًا حقيقيًا على GitHub، مع التركيز على مهام متنوعة مثل حل المشكلات عبر المستودعات وتوليد الوثائق من المستودعات.

La introducción de BeyondSWE marca un avance significativo en la evaluación de agentes de código más allá de su alcance tradicional de corrección de errores en un solo repositorio. Este nuevo benchmark consta de 500 instancias extraídas de 246 repositorios reales de GitHub, centrándose en tareas diversas como la resolución de problemas entre repositorios y la generación de documentos a partir de repositorios.

L'introduction de BeyondSWE représente une avancée significative dans l'évaluation des agents de code au-delà de leur portée traditionnelle de correction de bogues dans un seul dépôt. Ce nouveau benchmark se compose de 500 instances provenant de 246 dépôts GitHub réels, se concentrant sur des tâches diverses telles que la résolution de problèmes inter-dépôts et la génération de documents à partir de dépôts.

The introduction of BeyondSWE marks a significant advancement in evaluating code agents beyond their traditional scope of single-repository bug fixing. This new benchmark consists of 500 instances sourced from 246 real-world GitHub repositories, focusing on diverse tasks such as cross-repository issue resolution and document-to-repository generation.