arXiv:2412.15701v5 Announce Type: replace-cross 
Abstract: While the advancement of large language models has spurred the development of AI agents to automate tasks, numerous use cases inherently require agents to collaborate with humans due to humans' latent preferences, domain expertise, or the need for control. To facilitate the study of human-agent collaboration, we introduce Collaborative Gym (Co-Gym), an open framework for developing and evaluating collaborative agents that engage in bidirectional communication with humans while interacting with task environments. We describe how the framework enables the implementation of new task environments and coordination between humans and agents through a flexible, non-turn-taking interaction paradigm, along with an evaluation suite that assesses both collaboration outcomes and processes. Our framework provides both a simulated condition with a reliable user simulator and a real-world condition with an interactive web application. Initial benchmark experiments across three representative tasks -- creating travel plans, writing related work sections, and analyzing tabular data -- demonstrate the benefits of human-agent collaboration: The best-performing collaborative agents consistently outperform their fully autonomous counterparts in task performance, achieving win rates of 86% in Travel Planning, 74% in Tabular Analysis, and 66% in Related Work when evaluated by real users. Despite these improvements, our evaluation reveals persistent limitations in current language models and agents, with communication and situational awareness failures observed in 65% and 40% of cases in the real condition, respectively. Released under the permissive MIT license, Co-Gym supports the addition of new task environments and can be used to develop collaborative agent applications, while its evaluation suite enables assessment and improvement of collaborative agents.

تم تقديم Collaborative Gym (Co-Gym) كإطار مفتوح مصمم لتعزيز وتقييم التعاون بين البشر والوكالات، خاصة في البيئات التي تتطلب التواصل الثنائي الاتجاه. يتيح هذا الإطار تطوير بيئات مهام جديدة ويسهل التنسيق بين البشر ووكلاء الذكاء الاصطناعي من خلال نموذج تفاعل مرن.

Se ha presentado Collaborative Gym (Co-Gym) como un marco abierto diseñado para mejorar y evaluar la colaboración entre humanos y agentes, especialmente en entornos que requieren comunicación bidireccional. Este marco permite el desarrollo de nuevos entornos de tareas y facilita la coordinación entre humanos y agentes de IA a través de un paradigma de interacción flexible.

Le Collaborative Gym (Co-Gym) a été introduit comme un cadre ouvert destiné à améliorer et évaluer la collaboration entre humains et agents, en particulier dans des environnements nécessitant une communication bidirectionnelle. Ce cadre permet le développement de nouveaux environnements de tâches et facilite la coordination entre les humains et les agents IA grâce à un paradigme d'interaction flexible.

Collaborative Gym (Co-Gym) has been introduced as an open framework designed to enhance and evaluate human-agent collaboration, particularly in environments requiring bidirectional communication. This framework allows for the development of new task environments and facilitates coordination between humans and AI agents through a flexible interaction paradigm.

Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration

A recent MIT report notes that the failure rate for generative AI projects in organizations is 95%, despite directing billions of dollars in investment.

يبرز تقرير حديث من معهد ماساتشوستس للتكنولوجيا معدل فشل مذهل بنسبة 95% لمشاريع الذكاء الاصطناعي التوليدية في المؤسسات، على الرغم من الاستثمارات الكبيرة التي تصل إلى مليارات الدولارات. هذه الإحصائية المثيرة للقلق تسلط الضوء على التحديات التي يواجهها القادة في دمج تقنيات الذكاء الاصطناعي بشكل فعال في عملياتهم.

Un informe reciente del MIT destaca una asombrosa tasa de fracaso del 95% para los proyectos de IA generativa en las organizaciones, a pesar de las inversiones significativas que ascienden a miles de millones de dólares. Esta estadística alarmante subraya los desafíos que enfrentan los líderes para integrar efectivamente las tecnologías de IA en sus operaciones.

Un rapport récent du MIT souligne un taux d'échec de 95 % pour les projets d'IA générative dans les organisations, malgré des investissements significatifs s'élevant à des milliards de dollars. Cette statistique alarmante met en évidence les défis auxquels les dirigeants sont confrontés pour intégrer efficacement les technologies d'IA dans leurs opérations.

A recent report from MIT highlights a staggering 95% failure rate for generative AI projects in organizations, despite significant investments amounting to billions of dollars. This alarming statistic underscores the challenges leaders face in effectively integrating AI technologies into their operations.

Dr. Curtis Rasmussen: The Specialist Reframing How Leaders Understand Human-Machine Collaboration

The tools are available to everyone. The subscription is company-wide. The training sessions have been held. And yet, in offices from Wall Street to Silicon Valley, a stark divide is opening between workers who have woven artificial intelligence into the fabric of their daily work and colleagues who have barely touched it.The gap is not small. According to a <a href="https://cdn.openai.com/pdf/7ef17d82-96bf-4dd1-9df2-228f7f377a29/the-state-of-enterprise-ai_2025-report.pdf">new report</a> from OpenAI analyzing usage patterns across its more than one million business customers, workers at the <a href="https://openai.com/index/the-state-of-enterprise-ai-2025-report/">95th percentile of AI adoption</a> are sending six times as many messages to ChatGPT as the median employee at the same companies. For specific tasks, the divide is even more dramatic: frontier workers send 17 times as many coding-related messages as their typical peers, and among data analysts, the heaviest users engage the data analysis tool 16 times more frequently than the median.This is not a story about access. It is a story about a new form of workplace stratification emerging in real time — one that may be reshaping who gets ahead, who falls behind, and what it means to be a skilled worker in the age of artificial intelligence.<h2>Everyone has the same tools, but not everyone is using them</h2>Perhaps the most striking finding in the <a href="https://cdn.openai.com/pdf/7ef17d82-96bf-4dd1-9df2-228f7f377a29/the-state-of-enterprise-ai_2025-report.pdf">OpenAI report</a> is how little access explains. <a href="https://chatgpt.com/business/enterprise?utm_source=google&amp;utm_medium=paidsearch_brand&amp;utm_campaign=GOOG_B_SEM_GBR_Core_ENT_BAU_ACQ_PER_BRD_ALL_NAMER_US_EN_080625&amp;utm_term=chatgpt%20enterprise&amp;utm_content=182507886919&amp;utm_ad=779434575256&amp;utm_match=b&amp;gad_source=1&amp;gad_campaignid=22855802308&amp;gbraid=0AAAAA-I0E5ew3OYocHAOSOiKw516uqob_&amp;gclid=Cj0KCQiArt_JBhCTARIsADQZaylE-8Q0a0VjiXwyRLvFkD9BIkgt4EpWCm6v7alzBtNKWKE5_7J_3SYaArnTEALw_wcB">ChatGPT Enterprise</a> is now deployed across more than 7 million workplace seats globally, a nine-fold increase from a year ago. The tools are the same for everyone. The capabilities are identical. And yet usage varies by orders of magnitude.Among monthly active users — people who have logged in at least once in the past 30 days — <a href="https://cdn.openai.com/pdf/7ef17d82-96bf-4dd1-9df2-228f7f377a29/the-state-of-enterprise-ai_2025-report.pdf">19 percent have never tried the data analysis feature</a>. Fourteen percent have never used reasoning capabilities. Twelve percent have never used search. These are not obscure features buried in submenus; they are core functionality that OpenAI highlights as transformative for knowledge work.The pattern inverts among daily users. <a href="https://cdn.openai.com/pdf/7ef17d82-96bf-4dd1-9df2-228f7f377a29/the-state-of-enterprise-ai_2025-report.pdf">Only 3 percent</a> of people who use ChatGPT every day have never tried data analysis; just 1 percent have skipped reasoning or search. The implication is clear: the divide is not between those who have access and those who don&#x27;t, but between those who have made AI a daily habit and those for whom it remains an occasional novelty.<h2>Employees who experiment more are saving dramatically more time</h2>The <a href="https://cdn.openai.com/pdf/7ef17d82-96bf-4dd1-9df2-228f7f377a29/the-state-of-enterprise-ai_2025-report.pdf">OpenAI report</a> suggests that AI productivity gains are not evenly distributed across all users but concentrated among those who use the technology most intensively. Workers who engage across approximately seven distinct task types — data analysis, coding, image generation, translation, writing, and others — report saving five times as much time as those who use only four. Employees who save more than 10 hours per week consume eight times more AI credits than those who report no time savings at all.This creates a compounding dynamic. Workers who experiment broadly discover more uses. More uses lead to greater productivity gains. Greater productivity gains presumably lead to better performance reviews, more interesting assignments, and faster advancement—which in turn provides more opportunity and incentive to deepen AI usage further.Seventy-five percent of surveyed workers report being able to complete tasks they previously could not perform, including programming support, spreadsheet automation, and technical troubleshooting. For workers who have embraced these capabilities, the boundaries of their roles are expanding. For those who have not, the boundaries may be contracting by comparison.<h2>The corporate AI paradox: $40 billion spent, 95 percent seeing no return</h2>The individual usage gap documented by OpenAI mirrors a broader pattern identified by a separate study from <a href="https://venturebeat.com/ai/mit-report-misunderstood-shadow-ai-economy-booms-while-headlines-cry-failure">MIT&#x27;s Project NANDA</a>. Despite $30 billion to $40 billion invested in generative AI initiatives, only 5 percent of organizations are seeing transformative returns. The researchers call this the &quot;<a href="https://www.searchyour.ai/en/genai-divide-state-ai-business-2025-mit-nanda">GenAI Divide</a>&quot; — a gap separating the few organizations that succeed in transforming processes with adaptive AI systems from the majority that remain stuck in pilots.The MIT report found <a href="https://www.legal.io/articles/5719519/MIT-Report-Finds-95-of-AI-Pilots-Fail-to-Deliver-ROI-Exposing-GenAI-Divide">limited disruption</a> across industries: only two of nine major sectors—technology and media—show material business transformation from generative AI use. Large firms lead in pilot volume but lag in successful deployment.The pattern is consistent across both studies. Organizations and individuals are buying the technology. They are launching pilots. They are attending training sessions. But somewhere between adoption and transformation, most are getting stuck.<h2>While official AI projects stall, a shadow economy is thriving</h2>The <a href="https://venturebeat.com/ai/mit-report-misunderstood-shadow-ai-economy-booms-while-headlines-cry-failure">MIT study</a> reveals a striking disconnect: while only 40 percent of companies have purchased official LLM subscriptions, employees in over 90 percent of companies regularly use personal AI tools for work. Nearly every respondent reported using LLMs in some form as part of their regular workflow.&quot;This &#x27;<a href="https://venturebeat.com/ai/mit-report-misunderstood-shadow-ai-economy-booms-while-headlines-cry-failure">shadow AI</a>&#x27; often delivers better ROI than formal initiatives and reveals what actually works for bridging the divide,&quot; MIT&#x27;s Project NANDA found.The shadow economy offers a clue to what&#x27;s happening at the individual level within organizations. Employees who take initiative — who sign up for personal subscriptions, who experiment on their own time, who figure out how to integrate AI into their workflows without waiting for IT approval — are pulling ahead of colleagues who wait for official guidance that may never come.These shadow systems, largely unsanctioned, often deliver better performance and faster adoption than corporate tools. Worker sentiment reveals a preference for flexible, responsive tools — precisely the kind of experimentation that separates OpenAI&#x27;s frontier workers from the median.<h2>The biggest gaps show up in technical work that used to require specialists</h2>The largest relative gaps between frontier and median workers appear in coding, writing, and analysis — precisely the task categories where AI capabilities have advanced most rapidly. Frontier workers are not just doing the same work faster; they appear to be doing different work entirely, expanding into technical domains that were previously inaccessible to them.Among <a href="https://chatgpt.com/business/enterprise?utm_source=google&amp;utm_medium=paidsearch_brand&amp;utm_campaign=GOOG_B_SEM_GBR_Core_ENT_BAU_ACQ_PER_BRD_ALL_NAMER_US_EN_080625&amp;utm_term=chatgpt%20enterprise&amp;utm_content=182507886919&amp;utm_ad=779434575253&amp;utm_match=b&amp;gad_source=1&amp;gad_campaignid=22855802308&amp;gbraid=0AAAAA-I0E5ew3OYocHAOSOiKw516uqob_&amp;gclid=Cj0KCQiArt_JBhCTARIsADQZaykhZ9zJ10fojuPYA8XhHxR0jVT-WbgS4kfA7IIaTohf7yAByZszI-AaAkeREALw_wcB">ChatGPT Enterprise</a> users outside of engineering, IT, and research, coding-related messages have grown 36 percent over the past six months. Someone in marketing or HR who learns to write scripts and automate workflows is becoming a categorically different employee than a peer who has not — even if they hold the same title and started with the same skills.The academic research on AI and productivity offers a complicated picture. Several studies cited in the OpenAI report find that AI has an &quot;<a href="https://cdn.openai.com/pdf/7ef17d82-96bf-4dd1-9df2-228f7f377a29/the-state-of-enterprise-ai_2025-report.pdf">equalizing effect</a>,&quot; disproportionately helping lower-performing workers close the gap with their higher-performing peers. But the equalizing effect may apply only within the population of workers who actually use AI regularly. A meaningful share of workers are not in that group at all. They remain light users or non-users, even as their more adventurous colleagues pull away.<h2>Companies are divided too, and the gap is widening by the month</h2>The divide is not only between individual workers. It exists between entire organizations.Frontier firms — those at the 95th percentile of adoption intensity — generate approximately twice as many AI messages per employee as the median enterprise. For messages routed through custom GPTs, purpose-built tools that automate specific workflows, the gap widens to seven-fold.These numbers suggest fundamentally different operating models. At median companies, AI may be a productivity tool that individual workers use at their discretion. At frontier firms, AI appears to be embedded in core infrastructure: standardized workflows, persistent custom tools, systematic integration with internal data systems.The <a href="https://cdn.openai.com/pdf/7ef17d82-96bf-4dd1-9df2-228f7f377a29/the-state-of-enterprise-ai_2025-report.pdf">OpenAI report</a> notes that roughly one in four enterprises still has not enabled connectors that give AI access to company data—a basic step that dramatically increases the technology&#x27;s utility. The MIT study found that companies that purchased AI tools from specialized vendors succeeded <a href="https://www.tekedia.com/the-genai-divide-mit-report-shows-why-most-business-ai-projects-are-stalling/">67 percent</a> of the time, while internal builds had only a one-in-three success rate. For many organizations, the AI era has technically arrived but has not yet begun in practice.<h2>The technology is no longer the problem — organizations are</h2>For executives, the data presents an uncomfortable challenge. The technology is no longer the constraint. OpenAI notes that it releases a new feature or capability roughly every three days; the models are advancing faster than most organizations can absorb. The bottleneck has shifted from what AI can do to whether organizations are structured to take advantage of it.&quot;The dividing line isn&#x27;t intelligence,&quot; the <a href="https://mlq.ai/media/quarterly_decks/v0.1_State_of_AI_in_Business_2025_Report.pdf">MIT authors write</a>. The problems with enterprise AI have to do with memory, adaptability, and learning capability. Problems stem less from regulations or model performance, and more from tools that fail to learn or adapt.Leading firms, according to the <a href="https://cdn.openai.com/pdf/7ef17d82-96bf-4dd1-9df2-228f7f377a29/the-state-of-enterprise-ai_2025-report.pdf">OpenAI report</a>, consistently invest in executive sponsorship, data readiness, workflow standardization, and deliberate change management. They build cultures where custom AI tools are created, shared, and refined across teams. They track performance and run evaluations. They make AI adoption a strategic priority rather than an individual choice.The rest are leaving it to chance — hoping that workers will discover the tools on their own, experiment on their own time, and somehow propagate best practices without infrastructure or incentive. The six-fold gap suggests this approach is not working.<h2>The window to catch up is closing faster than most companies realize</h2>With enterprise contracts locking in over the next 18 months, there&#x27;s a shrinking window for vendors and adopters to cross the divide.The GenAI Divide identified by the <a href="https://cdn.openai.com/pdf/7ef17d82-96bf-4dd1-9df2-228f7f377a29/the-state-of-enterprise-ai_2025-report.pdf">MIT report</a> is not going to last forever. But the organizations that figure out a way across it soonest will be the ones that define the next era of business.Both reports carry caveats. The OpenAI data comes from a company with an obvious interest in promoting AI adoption. The productivity figures are self-reported by customers already paying for the product. The MIT study, while independent, relies on interviews and surveys rather than direct measurement. The long-term effects of this technology on employment, wages, and workplace dynamics remain uncertain.But the core finding — that access alone does not produce adoption, and that adoption varies enormously even within organizations that have made identical tools available to all — is consistent with how previous technologies have diffused through the economy. Spreadsheets, email, and the internet all created similar divides before eventually becoming universal. The question is how long the current gap persists, who benefits during the transition, and what happens to workers who find themselves on the wrong side of it.For now, the divide is stark. Ninety percent of users said they prefer humans for &quot;mission-critical work,&quot; while AI has &quot;won the war for simple work.&quot; The workers who are pulling ahead are not doing so because they have access their colleagues lack. They are pulling ahead because they decided to use what everyone already has—and kept using it until they figured out what it could do.The 6x gap is not about technology. It is about behavior. And behavior, unlike software, cannot be deployed with a company-wide rollout.

يكشف تقرير حديث من OpenAI عن فجوة كبيرة في الإنتاجية، حيث يظهر أن مستخدمي الذكاء الاصطناعي المتقدمين يرسلون ستة أضعاف الرسائل إلى ChatGPT مقارنة بالموظف المتوسط في شركاتهم. هذه الفجوة أكثر وضوحًا في أدوار محددة، مثل البرمجة وتحليل البيانات، حيث يتفاعل المستخدمون الأكثر نشاطًا 17 مرة أكثر من نظرائهم.

Un informe reciente de OpenAI destaca una brecha significativa en la productividad, revelando que los usuarios avanzados de IA envían seis veces más mensajes a ChatGPT que el empleado medio en sus empresas. Esta disparidad es aún más pronunciada en roles específicos, como la programación y el análisis de datos, donde los usuarios más activos interactúan 17 veces más que sus compañeros.

Un rapport récent d'OpenAI met en lumière un écart de productivité significatif, révélant que les utilisateurs avancés de l'IA envoient six fois plus de messages à ChatGPT que l'employé médian dans leurs entreprises. Cette disparité est encore plus marquée dans des rôles spécifiques, tels que la programmation et l'analyse de données, où les meilleurs utilisateurs interagissent 17 fois plus que leurs pairs.

A recent report from OpenAI highlights a significant productivity gap, revealing that AI power users send six times more messages to ChatGPT than the median employee in their companies. This disparity is even more pronounced in specific roles, such as coding and data analysis, where top users engage 17 times more than their peers.

Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration

Was this article worth reading? Share it

Chattermate

Augmeta

Agentcloud