Fine-tune video and image models at scale with NVIDIA NeMo Automodel and 🤗 Diffusers
Special thanks to Sayak Paul from Hugging Face for their contributions to the integration work and for co-authoring this blog. Diffusion models power some of the most exciting open-source releases of the last two years - such as FLUX.1-dev for text-to-image and Wan 2.1 and HunyuanVideo for text-to-video. The 🤗 Diffusers library has become the de facto home for these models, giving researchers and builders a single, consistent interface for inference, adaptation, and pipeline composition.
Key Takeaways
- In addition, training and fine-tuning diffusion models are also on the rise, requiring utilities that offer memory-efficient sharding, latent caching, multiresolution bucketing, and configurations that scale gracefully from one GPU to hundreds.
To cater to these technical demands, we offer the NVIDIA NeMo Automodel open-source library.
- Supported diffusion models What this collaboration unlocks A look at the fine-tuning workflow 1.
- Point at any Diffusers model ID on the Hub and start training.
NeMo Automodel uses Diffusers model classes (e.g. ) for loading and Diffusers pipelines ( ) for generation.
- AutoModel currently supports flow-matching models only.
Under the hood, it uses flow matching as the training objective, with latent-space training (via pre-encoded VAE outputs) and multiresolution bucketed dataloading to accelerate throughput.
- Your fine-tuned checkpoint loads directly into a for inference, or back to the Hub for sharing.
In addition, training and fine-tuning diffusion models are also on the rise, requiring utilities that offer memory-efficient sharding, latent caching, multiresolution bucketing, and configurations that scale gracefully from one GPU to hundreds. To cater to these technical demands, we offer the NVIDIA NeMo Automodel open-source library. Today, we're highlighting the collaboration between NVIDIA and Hugging Face that brings production-grade, distributed diffusion training to any Diffusers-format model on the Hugging Face Hub - with no checkpoint conversion and no model rewrites for any new model.
The integration is documented in the Diffusers training guide and is fully open source under Apache 2.0. Table of contents What is NeMo Automodel? Supported diffusion models What this collaboration unlocks A look at the fine-tuning workflow 1.
Launch training with the existing FLUX YAML 3. Generate from the fine-tuned checkpoint 4. Performance Other Finetuned/LoRA examples Try it today Coming next: Pythonic recipe APIs Resources What is NeMo Automodel?
For more details please read the original article at Hugging Face.
Continue Learning
Comments
Comments appear only after moderation. Your email identifies your submission to the moderator and is never displayed here.
No approved comments yet.