arXiv:2606.06622v3 Announce Type: replace 
Abstract: We introduce UnpredictaBench, an evaluation that tests the ability of large language models (LLMs) to capture true underlying distributions. As LLMs are increasingly used as substitutes for other entities (e.g., for humans in economic simulations), the tendency of many models to collapse towards a single plausible answer means a failure to capture the unpredictability of real systems. Recent work on improving output diversity is insufficient for this setting: simulation requires samples that are calibrated to a target distribution, not merely varied outputs. UnpredictaBench isolates a simplified but fundamental version of this problem: sampling outcomes from individual target distributions, including canonical statistical distributions, distributions induced by stochastic programs, and natural-language scenarios that describe random processes. We introduce 448 such problems together with KS@N, a general-purpose evaluation metric that quantifies how well a model outputs approximate black-box target distributions via the Kolmogorov-Smirnov statistical test. This is the rate at which we fail to reject model samples of size N against ground-truth samples, with larger N indicating greater difficulty. Tested across open and proprietary models, we find a large spread in distributional capabilities. For instance, when models generate samples of size 100 (KS@100, our standard metric), scores range from near 0 to over 20%. No model is able to achieve over 40% at KS@100, showing significant headroom in distributional sampling as a capability. Although adding reasoning can somewhat increase scores, we find no immediate solution for this issue. UnpredictaBench shows that even simple distributional simulation remains challenging, making it a necessary first step toward using LLMs as stand-ins for complex systems. Project website and resources are available at https://unpredictabenchmark.github.io/.

تم تقديم UnpredictaBench كمعيار يهدف إلى تقييم قدرات العشوائية التوزيعية لنماذج اللغة الكبيرة (LLMs). هذه التقييمات مهمة حيث يتم استخدام LLMs بشكل متزايد في المحاكاة التي تتطلب التقاط عدم القدرة على التنبؤ في الأنظمة الواقعية، بدلاً من الانجراف نحو إجابة واحدة محتملة. يتضمن المعيار 448 مشكلة تقيم قدرة النماذج على أخذ عينات من توزيعات مستهدفة متنوعة.

UnpredictaBench se ha introducido como un benchmark destinado a evaluar las capacidades de aleatoriedad distribucional de los grandes modelos de lenguaje (LLMs). Esta evaluación es crucial ya que los LLMs se utilizan cada vez más en simulaciones que requieren capturar la imprevisibilidad de los sistemas del mundo real, en lugar de converger hacia una única respuesta plausible. El benchmark incluye 448 problemas que evalúan la capacidad de los modelos para muestrear de diversas distribuciones objetivo.

UnpredictaBench a été introduit comme un benchmark visant à évaluer les capacités de randomisation distributionnelle des grands modèles de langage (LLMs). Cette évaluation est cruciale car les LLMs sont de plus en plus utilisés dans des simulations nécessitant la capture de l'imprévisibilité des systèmes réels, plutôt que de converger vers une seule réponse plausible. Le benchmark comprend 448 problèmes qui évaluent la capacité des modèles à échantillonner à partir de diverses distributions cibles.

UnpredictaBench has been introduced as a benchmark aimed at evaluating the distributional randomness capabilities of large language models (LLMs). This evaluation is crucial as LLMs are increasingly utilized in simulations that require capturing the unpredictability of real-world systems, rather than converging on a single plausible answer. The benchmark includes 448 problems that assess the models' ability to sample from various target distributions.