arXiv:2606.11409v1 Announce Type: new 
Abstract: Adversarial robustness evaluations of large language models (LLMs) typically report attack success rate (ASR) under fixed query budgets, implicitly treating all attacks as equally costly. In practice, the computational expense of different attack strategies can vary by orders of magnitude. Consequently, ASR at a fixed budget can obscure the true effort required to jailbreak a model, thereby making it hard to determine whether an attack's cost justifies its payoff to the attacker. We propose a compute-aware evaluation framework based on computational pressure, measured in cumulative floating-point operations (FLOPs), as a proxy for adversarial effort. We introduce risk-compute curves, which map compute budgets to attack risk, and derive two metrics that summarize the average pressure required for a given attack to succeed. Across ten models spanning three families and four different stages in language model training and alignment, evaluated with three attack strategies (gradient-based, iterative refinement, and template-based) on two jailbreak robustness benchmarks, we find: (1) alignment training has non-monotonic effects on compute-space robustness; (2) scaling model size reduces gradient-based attack effectiveness but has limited impact on cheaper template-based attacks; (3) gradient-based attacks optimized on a surrogate model can transfer to a separate target model, providing a way to reduce attacker costs; (4) compute cost varies by up to ${\approx}5{\times}$ across harm categories within a single model; and (5) safety-aligned RL increases aggregate cost while leaving some categories disproportionately accessible. We release our framework to enable compute-aware risk assessment and evaluation.

تم اقتراح إطار جديد لتقييم القوة المضادة للنماذج اللغوية الكبيرة (LLMs) يركز على التقييمات المدركة للحوسبة التي تأخذ في الاعتبار التكاليف الحاسوبية المتغيرة لاستراتيجيات الهجوم المختلفة. يقدم هذا النهج منحنيات المخاطر-الحوسبة لفهم العلاقة بين ميزانيات الحوسبة ومخاطر الهجوم بشكل أفضل.

Se ha propuesto un nuevo marco para evaluar la robustez adversarial de los grandes modelos de lenguaje (LLMs), centrándose en evaluaciones conscientes de los recursos que consideran los costos computacionales variables de diferentes estrategias de ataque. Este enfoque introduce curvas de riesgo-computación para comprender mejor la relación entre los presupuestos de computación y los riesgos de ataque.

Un nouveau cadre d'évaluation de la robustesse adversariale des grands modèles de langage (LLMs) a été proposé, se concentrant sur des évaluations conscientes des ressources informatiques qui prennent en compte les coûts computationnels variables des différentes stratégies d'attaque. Cette approche introduit des courbes risque-compute pour mieux comprendre la relation entre les budgets de calcul et les risques d'attaque.

A new framework for evaluating the adversarial robustness of large language models (LLMs) has been proposed, focusing on compute-aware evaluations that consider the varying computational costs of different attack strategies. This approach introduces risk-compute curves to better understand the relationship between compute budgets and attack risks.