arXiv:2605.26144v1 Announce Type: cross 
Abstract: We present VISTA (VIsual Spec-To-App Benchmark), a benchmark for evaluating the end-to-end web-app generation capabilities of LLM-based agents. Unlike prior code generation benchmarks that focus on algorithmic tasks, VISTA targets realistic UI-centric development, where agents must produce functional, visually coherent applications from underspecified inputs. We define five prompt-information conditions that vary along two axes, visual/structural fidelity and stack constraint: (1) text only with free stack choice, (2) text with reference screenshots under three specified stacks, (3) text with reference screenshots under free stack choice, (4) text with screenshots and pruned Figma structure under a single specified stack, and (5) text with screenshots and pruned Figma structure under free stack choice. To enable robust evaluation, each page in the benchmark is manually annotated with interactive UI components and around three visual anchor points, addressing the well-known limitations of script-based testing tools such as Playwright in open-ended code generation settings. Evaluation combines DOM-grounded reference matching, behavior-specific browser tests, and CLIP-based visual similarity, jointly measuring structural alignment, behavioral completeness, and overall visual fidelity. We use VISTA to assess four agent systems drawn from two model families and two harnesses, finding that visual fidelity and functional correctness are partially decoupled across both input conditions and agents, and that agent editing style varies sharply but is largely orthogonal to task quality. VISTA establishes a rigorous and reproducible foundation for advancing agent-based software engineering research.

تم تقديم VISTA، وهو معيار شامل لتقييم قدرات الوكلاء المعتمدين على LLM في إنشاء تطبيقات الويب من المواصفات المرئية. يركز هذا المعيار على تطوير واجهة المستخدم الواقعية، حيث يتعين على الوكلاء إنشاء تطبيقات وظيفية من مدخلات غير محددة عبر خمس حالات معلومات مختلفة.

VISTA, un benchmark de extremo a extremo para evaluar las capacidades de los agentes basados en LLM en la generación de aplicaciones web a partir de especificaciones visuales, ha sido introducido. Este benchmark se centra en el desarrollo centrado en la interfaz de usuario, exigiendo a los agentes que creen aplicaciones funcionales a partir de entradas no especificadas en cinco condiciones de información de prompt distintas.

VISTA, un benchmark de bout en bout pour évaluer les capacités des agents basés sur des LLM à générer des applications web à partir de spécifications visuelles, a été introduit. Ce benchmark se concentre sur le développement centré sur l'interface utilisateur, exigeant des agents qu'ils créent des applications fonctionnelles à partir d'entrées sous-spécifiées selon cinq conditions d'information de prompt distinctes.

VISTA, an end-to-end benchmark for evaluating the capabilities of LLM-based agents in generating web applications from visual specifications, has been introduced. This benchmark focuses on realistic UI-centric development, requiring agents to create functional applications from underspecified inputs across five distinct prompt-information conditions.

VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents

arXiv:2512.04123v4 Announce Type: replace-cross 
Abstract: LLM-based agents already operate in production across many industries, yet we lack an understanding of what technical methods make deployments successful. We present the first systematic study of Measuring Agents in Production, MAP, using first-hand data from agent developers. We conducted 20 case studies via in-depth interviews and surveyed 86 deployed systems practitioners across 26 domains. We investigate why organizations build agents, how they build them, how they evaluate them, and their top development challenges. Our study finds that production agents are built using simple, controllable approaches: 68% execute at most 10 steps before human intervention, 70% rely on prompting off-the-shelf models instead of weight tuning, and 74% depend primarily on human evaluation. Reliability (consistent correct behavior over time) remains the top development challenge, which practitioners currently address through systems-level design. MAP documents the current state of production agents, providing the research community with visibility into deployment realities and underexplored research avenues.

تكشف دراسة منهجية بعنوان 'قياس الوكلاء في الإنتاج' عن معلومات حول نشر الوكلاء المعتمدين على نماذج اللغة الكبيرة (LLM) عبر صناعات مختلفة، استنادًا إلى 20 دراسة حالة واستطلاعات رأي من 86 ممارسًا. تسلط الدراسة الضوء على أن معظم الوكلاء يعملون بعدد محدود من الخطوات قبل التدخل البشري ويعتمدون بشكل كبير على التقييم البشري، مما يشير إلى تفضيل الأساليب الأبسط والأكثر قابلية للتحكم.

Un estudio sistemático titulado 'Measuring Agents in Production' revela información sobre el despliegue de agentes basados en LLM en diversas industrias, basado en 20 estudios de caso y encuestas a 86 profesionales. La investigación destaca que la mayoría de los agentes operan con un número limitado de pasos antes de la intervención humana y dependen en gran medida de la evaluación humana, lo que indica una preferencia por enfoques más simples y controlables.

Une étude systématique intitulée 'Measuring Agents in Production' révèle des informations sur le déploiement des agents basés sur des LLM dans diverses industries, basée sur 20 études de cas et des enquêtes auprès de 86 praticiens. La recherche met en évidence que la plupart des agents fonctionnent avec un nombre limité d'étapes avant l'intervention humaine et s'appuient fortement sur l'évaluation humaine, indiquant une préférence pour des approches plus simples et contrôlables.

A systematic study titled 'Measuring Agents in Production' reveals insights into the deployment of LLM-based agents across various industries, based on 20 case studies and surveys from 86 practitioners. The research highlights that most agents operate with limited steps before human intervention and rely heavily on human evaluation, indicating a preference for simpler, controllable approaches.

VISTA: An End-to-End Benchmark for Visual Spec-to-Web-App Coding Agents

Was this article worth reading? Share it

Ready to build your own newsroom?