arXiv:2507.15724v2 Announce Type: replace 
Abstract: Enabling image generation models to be spatially controlled is an important area of research, empowering users to better generate images according to their own fine-grained specifications via e.g. edge maps, poses. Although this task has seen impressive improvements in recent times, a focus on rapidly producing stronger models has come at the cost of detailed and fair scientific comparison. Differing training data, model architectures and generation paradigms make it difficult to disentangle the factors contributing to performance. Meanwhile, the motivations and nuances of certain approaches become lost in the literature. In this work, we aim to provide clear takeaways across generation paradigms for practitioners wishing to develop transformer-based systems for spatially-controlled generation, clarifying the literature and addressing knowledge gaps. We perform controlled experiments on ImageNet across diffusion-based/flow-based and autoregressive (AR) models. First, we establish control token prefilling as a simple, general and performant baseline approach for transformers. We then investigate previously underexplored sampling time enhancements, showing that extending classifier-free guidance to control, as well as softmax truncation, have a strong impact on control-generation consistency. Finally, we re-clarify the motivation of adapter-based approaches, demonstrating that they mitigate "forgetting" and maintain generation quality when trained on limited downstream data, but underperform full training in terms of generation-control consistency.

تناقش هذه المقالة التقدم في توليد الصور التي يمكن التحكم فيها مكانيًا باستخدام المحولات. وتبرز أهمية تمكين المستخدمين من إنشاء صور وفقًا لمتطلبات محددة، مثل خرائط الحواف والأوضاع. على الرغم من وجود تحسينات كبيرة في هذا المجال، إلا أن التركيز على تطوير نماذج أقوى قد طغى أحيانًا على الحاجة إلى مقارنات علمية شاملة.

Este artículo discute los avances en la generación de imágenes controladas espacialmente utilizando transformadores. Destaca la importancia de permitir a los usuarios crear imágenes según requisitos específicos, como mapas de bordes y poses. Aunque ha habido mejoras significativas en este campo, el enfoque en desarrollar modelos más fuertes a veces ha eclipsado la necesidad de comparaciones científicas exhaustivas.

Cet article aborde les avancées dans la génération d'images contrôlées spatialement à l'aide de transformateurs. Il souligne l'importance de permettre aux utilisateurs de créer des images en fonction de besoins spécifiques, tels que des cartes de contours et des poses. Bien qu'il y ait eu des améliorations significatives dans ce domaine, l'accent mis sur le développement de modèles plus puissants a parfois éclipsé la nécessité de comparaisons scientifiques approfondies.

This article discusses advancements in spatially-controlled image generation using transformers. It highlights the importance of allowing users to create images based on specific requirements, such as edge maps and poses. While there have been significant improvements in this field, the focus on developing stronger models has sometimes overshadowed the need for thorough scientific comparisons.

A Practical Investigation of Spatially-Controlled Image Generation with Transformers

One More Thing in AI – Your Shortcut to AI Mastery

A Practical Investigation of Spatially-Controlled Image Generation with Transformers

Was this article worth reading? Share it

One More Thing in AI

Attentive AI

GPTHumanizer

Republiclabs.ai

4o Image Gen

PicPicAI

Ready to build your own newsroom?