arXiv:2606.10706v1 Announce Type: new 
Abstract: Resource constraints increasingly determine what can be trained, fine-tuned, and deployed in large language models (LLMs), yet efficiency is often studied through isolated techniques rather than as an interacting system of limits. This survey adopts a constraint-centric perspective and organizes recent progress around three coupled bottlenecks: data efficiency (what to train on), memory efficiency (how to fit training), and compute budget awareness (when and where to spend FLOPs). On the data axis, we review selection and pruning methods that maximize learning per token, ranging from scalable proxy signals based on learning dynamics to gradient- and influence-based scoring, as well as difficulty-aware and curriculum-style strategies. We highlight emerging evidence that different notions of good data dominate in different regimes, implying that optimal subsets depend on the task objective and resource budget rather than being universal. On the systems side, we show that GPU memory, not raw compute, is often the dominant bottleneck in fine-tuning, and that effective scaling requires jointly reducing weight storage, optimizer states, and activation memory rather than optimizing any single component in isolation. Beyond memory, we frame training and inference as compute-governed processes in which optimization, data selection, and decoding must explicitly account for finite FLOP budgets. We review evidence for compute-optimal allocation and stopping rules, where computation should be halted or reallocated once marginal performance gains fall below a budget-dependent threshold. Together, these results unify compute-aware data selection, scaling laws, and adaptive inference under a common principle of resource-conditioned decision-making.

تسلط دراسة حديثة نشرت على arXiv الضوء على الأهمية المتزايدة للقيود على الموارد في تدريب نماذج اللغة الكبيرة (LLMs)، مع التركيز على ثلاثة مجالات رئيسية: كفاءة البيانات، وكفاءة الذاكرة، والوعي بميزانية الحوسبة. تؤكد الدراسة على أن الكفاءة يجب أن تُعتبر نظامًا مترابطًا بدلاً من تقنيات معزولة.

Una reciente encuesta publicada en arXiv destaca la creciente importancia de las limitaciones de recursos en el entrenamiento de modelos de lenguaje de gran tamaño (LLMs), centrándose en tres áreas clave: eficiencia de datos, eficiencia de memoria y conciencia del presupuesto de cómputo. La encuesta enfatiza que la eficiencia debe verse como un sistema interconectado en lugar de técnicas aisladas.

Une récente enquête publiée sur arXiv met en lumière l'importance croissante des contraintes de ressources dans l'entraînement des modèles de langage de grande taille (LLMs), en se concentrant sur trois domaines clés : l'efficacité des données, l'efficacité de la mémoire et la prise en compte du budget de calcul. L'enquête souligne que l'efficacité doit être considérée comme un système interconnecté plutôt que comme des techniques isolées.

A recent survey published on arXiv highlights the increasing importance of resource constraints in training large language models (LLMs), focusing on three key areas: data efficiency, memory efficiency, and compute budget awareness. The survey emphasizes that efficiency should be viewed as an interconnected system rather than isolated techniques.