arXiv:2412.16326v2 Announce Type: replace-cross 
Abstract: Current image generation methods are based on a two-stage training approach. In stage 1, an auto-encoder is trained to compress an image into a latent space; in stage 2, a generative model is trained to learn a distribution over that latent space. This reveals a fundamental trade-off, do we compress more aggressively to make the latent distribution easier for the stage 2 model to learn even if it makes reconstruction worse? We study this problem in the context of discrete, auto-regressive image generation. Through the lens of scaling laws, we show that smaller stage 2 models can benefit from more compressed stage 1 latents even if reconstruction performance worsens, demonstrating that generation modeling capacity plays a role in this trade-off. Diving deeper, we rigorously study the connection between compute scaling and the stage 1 rate-distortion trade-off. Next, we introduce Causally Regularized Tokenization (CRT), which uses knowledge of the stage 2 generation modeling procedure to embed useful inductive biases in stage 1 latents. This regularization improves stage 2 generation performance better by making the tokens easier to model without affecting the stage 1 compression rate and marginally affecting distortion: we are able to improve compute efficiency 2-3$\times$ over baseline. Finally, we use CRT with further optimizations to the visual tokenizer setup to result in a generative pipeline that matches LlamaGen-3B generation performance (2.18 FID) with half the tokens per image (256 vs. 576) and a fourth the total model parameters (775M vs. 3.1B) while using the same architecture and inference procedure.

دراسة حديثة نُشرت على arXiv تستكشف التوازن بين الضغط والتوليد في ترميز الصور، خاصة في سياق توليد الصور التلقائي التكراري. تكشف الأبحاث أن الضغط الأكثر عدوانية في المرحلة الأولى من معالجة الصور يمكن أن يفيد النماذج التوليدية الأصغر، حتى لو أدى ذلك إلى جودة إعادة بناء أقل.

Un estudio reciente publicado en arXiv explora la compensación entre compresión y generación en la tokenización visual, particularmente en el contexto de la generación de imágenes auto-regresivas discretas. La investigación revela que una compresión más agresiva en la primera etapa del procesamiento de imágenes puede beneficiar a modelos generativos más pequeños, incluso si conduce a una calidad de reconstrucción inferior.

Une étude récente publiée sur arXiv explore le compromis entre compression et génération dans la tokenisation visuelle, en particulier dans le contexte de la génération d'images auto-régressives discrètes. La recherche révèle qu'une compression plus agressive dans la première étape du traitement d'image peut bénéficier à de plus petits modèles génératifs, même si cela entraîne une qualité de reconstruction inférieure.

A recent study published on arXiv explores the trade-off between compression and generation in visual tokenization, particularly in the context of discrete, auto-regressive image generation. The research reveals that more aggressive compression in the first stage of image processing can benefit smaller generative models, even if it leads to poorer reconstruction quality.

When Worse is Better: Navigating the compression-generation tradeoff in visual tokenization

Was this article worth reading? Share it

SwapAnything.io

Airparser

4o Image Gen

Bulk Image Generation AI

ImgUpscaler AI

Republiclabs.ai

Ready to build your own newsroom?