arXiv:2510.25977v3 Announce Type: replace 
Abstract: AI accelerators, customized to AI workloads, provide cost-effective and high-performance solutions for training and inference. Trainium, an AI accelerator recently developed by Amazon Web Services (AWS), provides an attractive option for LLM training and inference through its heterogeneous architecture. However, leveraging Trainium architecture for high performance can be challenging because of its systolic array architecture and special requirement on data layout. In this paper, we design high-performance matrix multiplication (matmul), a critical compute kernel, for LLM inference on Trainium. We introduce a series of techniques customized to Trainium based on kernel fusion and novel caching strategies to reduce data movement across the software-managed memory hierarchy, maximize SRAM bandwidth, and avoid expensive matrix transpose. Evaluating with nine datasets and four recent LLMs, we show that our system largely outperforms the state-of-the-art matmul implemented by AWS on Trainium: at the level of matmul kernel, it achieves an average 1.35x speedup (up to 2.22x), which translates to an average 1.66x speedup (up to 2.49x) for end-to-end LLM inference.

قدمت خدمات أمازون ويب (AWS) ترينيوم، وهو مسرع ذكاء اصطناعي مصمم لضرب المصفوفات عالية الأداء في استنتاج نماذج اللغة الكبيرة (LLM). تُظهر دراسة حديثة أن تقنية جديدة لضرب المصفوفات، مصممة خصيصًا لترينيوم، تحقق تحسينات كبيرة في الأداء، مع زيادة متوسطة قدرها 1.35x لنواة الماتمول و1.66x لاستنتاج LLM من البداية إلى النهاية. هذه الخطوة مهمة لتحسين أعباء العمل في الذكاء الاصطناعي وتقليل التكاليف في التدريب والاستنتاج.

Amazon Web Services (AWS) ha presentado Trainium, un acelerador de IA diseñado para la multiplicación de matrices de alto rendimiento en la inferencia de modelos de lenguaje de gran tamaño (LLM). Un estudio reciente demuestra que una nueva técnica de multiplicación de matrices, adaptada a Trainium, logra mejoras significativas en el rendimiento, con un aumento promedio de 1.35x para el núcleo matmul y 1.66x para la inferencia LLM de extremo a extremo. Este avance es crucial para optimizar las cargas de trabajo de IA y reducir costos en el entrenamiento e inferencia de IA.

Amazon Web Services (AWS) a lancé Trainium, un accélérateur d'IA conçu pour la multiplication de matrices à haute performance dans l'inférence de modèles de langage de grande taille (LLM). Une étude récente montre qu'une nouvelle technique de multiplication de matrices, adaptée à Trainium, atteint des améliorations de performance significatives, avec un gain moyen de 1,35x pour le noyau matmul et de 1,66x pour l'inférence LLM de bout en bout. Cette avancée est cruciale pour optimiser les charges de travail d'IA et réduire les coûts dans la formation et l'inférence de l'IA.

Amazon Web Services (AWS) has introduced Trainium, an AI accelerator designed for high-performance matrix multiplication in large language model (LLM) inference. A recent study demonstrates that a new matrix multiplication technique, tailored for Trainium, achieves significant performance improvements, with an average speedup of 1.35x for the matmul kernel and 1.66x for end-to-end LLM inference. This advancement is crucial for optimizing AI workloads and reducing costs in AI training and inference.

NeuronMM: High-Performance Matrix Multiplication for LLM Inference on AWS Trainium

Was this article worth reading? Share it

Ready to build your own newsroom?