GalaxyDiT: Efficient Video Generation with Guidance Alignment and Adaptive Proxy in Diffusion Transformers

A recent study has highlighted the importance of motion representation in human motion generation using diffusion models, specifically focusing on the motion diffusion model (MDM) and its prediction objectives. The research evaluates various motion representations and their performance, aiming to enhance understanding of latent data distributions in generative models.

A new method called Soft Q-based Diffusion Finetuning (SQDF) has been proposed to enhance the alignment of diffusion models with downstream objectives, addressing issues of reward over-optimization that lead to unnatural samples. This method incorporates a reparameterized policy gradient of a differentiable soft Q-function estimation, along with innovations like a discount factor for credit assignment and off-policy replay buffers.

A new framework called Measurement-Aware Consistency Sampling (MACS) has been introduced to enhance the efficiency of diffusion models in solving inverse imaging problems. This approach utilizes a measurement-consistency mechanism to regulate stochasticity, ensuring fidelity to observed data while maintaining computational efficiency. Comprehensive experiments on datasets like Fashion-MNIST and LSUN Bedroom show significant improvements in both perceptual and pixel-level quality.

A new framework has been introduced that reinterprets Maximum Entropy Reinforcement Learning (MaxEntRL) as a diffusion model-based sampling problem, focusing on minimizing the reverse Kullback-Leibler divergence between the diffusion policy and the optimal policy distribution. This leads to the development of diffusion-based variants of existing algorithms like Soft Actor-Critic, Proximal Policy Optimization, and Wasserstein Policy Optimization, termed DiffSAC, DiffPPO, and DiffWPO.

Recent advancements in diffusion models have led to the development of a phase-aware strategy that accelerates image generation by applying different speedups to various stages of the process. This approach utilizes lightweight LoRA adapters, named Slow-LoRA and Fast-LoRA, to enhance efficiency without extensive retraining of models.