arXiv:2410.12982v2 Announce Type: replace 
Abstract: While transformers have been at the core of most recent advancements in sequence generative models, their computational cost remains quadratic in sequence length. Several subquadratic architectures have been proposed to address this computational issue. Some of them, including long convolution sequence models (LCSMs), such as Hyena, address this issue at training time but remain quadratic during inference. We propose a method for speeding up LCSMs' exact inference to quasilinear $O(L\log^2L)$ time, identify the key properties that make this possible, and propose a general framework that exploits these. Our approach, inspired by previous work on relaxed polynomial interpolation, is based on a tiling which helps decrease memory movement and share computation. It has the added benefit of allowing for almost complete parallelization across layers of the position-mixing part of the architecture. Empirically, we provide a proof of concept implementation for Hyena, which gets up to $7.8\times$ end-to-end improvement over standard inference by improving $110\times$ within the position-mixing part.

تم اقتراح طريقة جديدة لتحسين سرعة الاستدلال لنماذج تسلسل الالتفاف الطويلة (LCSMs)، وخاصة Hyena، مما يحقق تعقيد زمني شبه خطي قدره O(L log^2 L). تعالج هذه الخطوة التكلفة الحسابية التربيعية المرتبطة سابقًا بالمحولين أثناء الاستدلال، مما يحسن الكفاءة بشكل كبير حتى 7.8 مرات بشكل عام و110 مرات في مكونات محددة.

Se ha propuesto un nuevo método para mejorar la velocidad de inferencia de los modelos de secuencia de convolución larga (LCSMs), específicamente Hyena, alcanzando una complejidad temporal cuasilineal de O(L log^2 L). Este avance aborda el costo computacional cuadrático previamente asociado a los transformadores durante la inferencia, mejorando significativamente la eficiencia hasta en 7.8 veces en general y 110 veces en componentes específicos.

Une nouvelle méthode a été proposée pour améliorer la vitesse d'inférence des modèles de séquence à convolution longue (LCSMs), en particulier Hyena, atteignant une complexité temporelle quasi-linéaire de O(L log^2 L). Cette avancée répond au coût computationnel quadratique précédemment associé aux transformateurs lors de l'inférence, améliorant considérablement l'efficacité jusqu'à 7,8 fois au total et 110 fois dans des composants spécifiques.

A new method has been proposed to enhance the inference speed of long convolution sequence models (LCSMs), specifically Hyena, achieving a quasilinear time complexity of O(L log^2 L). This advancement addresses the previously quadratic computational cost associated with transformers during inference, significantly improving efficiency by up to 7.8 times overall and 110 times in specific components.

Flash Inference: Near Linear Time Inference for Long Convolution Sequence Models and Beyond

Was this article worth reading? Share it

LucidQuery AI

Magicley AI

Humanize AI

The Visualizer

FETCH HIVE

FastML

Ready to build your own newsroom?