Quick Overview
Presented by Katja Sirazitdinova from NVIDIA and Ivan Nardini from Google Cloud, this video is an educational technical tutorial on scaling machine learning models in JAX across multi-GPU setups. It builds on previous single-GPU tutorials to explain distributed training mechanisms using modern JAX and Flax NNX libraries.
Key Points
- 1.Scaling JAX workloads across multiple GPUs focuses on array placement and sharding declarations rather than rewriting training routines with low-level communication code.
- 2.Data parallelism in JAX operates by replicating model parameters across devices, splitting data batches into shards, and performing automatic gradient averaging across GPUs.
- 3.Four core concepts structure multi-GPU placement in JAX: Mesh, PartitionSpec, NamedSharding, and jax.device_put.
- 4.JAX supports both declarative automatic sharding and explicit low-level control through shard_map combined with jax.lax.pmean.
- 5.A Tiny Transformer language model built with Flax NNX demonstrates multi-GPU training, checkpointing with Orbax, and length-padded JIT generation.
Summary
Katja Sirazitdinova and Ivan Nardini present a technical tutorial on scaling machine learning models to multi-GPU systems using JAX and Flax NNX. In JAX, transitioning from a single GPU to a multi-GPU environment does not require writing explicit communication or networking logic. Instead, the framework relies on specifying array placement across devices while keeping standard training loops intact.
Data parallelism serves as the primary scaling pattern. In this setup, model parameters are fully replicated across all available GPUs, while the training dataset is sharded so each device processes a separate slice of the global batch. During the forward and backward passes, each GPU calculates local gradients, which are then combined and averaged using an all-reduce operation before optimizer updates are applied in sync.
Four fundamental concepts govern array placement in JAX. Mesh defines the hardware device topology and names the device grid. PartitionSpec, abbreviated as P, describes whether array dimensions are split along mesh axes or left unreplicated. NamedSharding combines the mesh and the partition specification into a concrete sharding strategy. Finally, jax.device_put transfers arrays onto devices according to that strategy. Developers can verify that data batches are sharded and weights are replicated using jax.debug.visualize_array_sharding.
Scaling performance is evaluated using weak scaling on an MLP workload with a per-GPU batch size of 1024. A single GPU achieves 22,442 examples per second at 45.63 milliseconds per step, while two GPUs achieve 41,136 examples per second at 49.79 milliseconds per step, yielding a 1.83x throughput ratio. Small workloads can experience communication overhead, meaning scaling throughput gains are not entirely free. While high-level declarative sharding is recommended for experimentation, JAX also offers shard_map and collective primitives like jax.lax.pmean for fine-grained, explicit per-device execution.
These principles are applied to training a Tiny Transformer model on the Tiny Shakespeare dataset using byte-level tokens. The architecture uses a model dimension of 256, 4 attention heads, a feed-forward dimension of 1024, and 4 transformer blocks configured with pre-layer normalization and causal self-attention. Using Flax NNX, the model and optimizer states are extracted with nnx.state, placed on the mesh with jax.device_put, and updated with nnx.update. Loss is computed using Optax softmax cross-entropy, and perplexity is monitored. Weights are serialized with Orbax checkpointing, verified against fresh models, and deployed for autoregressive generation using fixed-length padding to avoid repeated JIT recompilation.
Core concepts of JAX multi-GPU sharding
Running workloads on multiple GPUs in JAX relies on declaring array placement rather than writing custom communication scripts. Four primary abstractions manage this workflow: Mesh identifies the device grid, PartitionSpec defines how array dimensions are partitioned or replicated, NamedSharding binds the mesh to the partition specification, and jax.device_put applies the sharding configuration to concrete arrays.
Automatic placement versus shard_map
When arrays are configured with NamedSharding, JAX automatically handles parallel execution, including gradient all-reduce communication during optimization. Developers can inspect placement using jax.debug.visualize_array_sharding. For scenarios requiring explicit per-device execution and manual collective communication, JAX provides the shard_map primitive alongside collective operations such as jax.lax.pmean.
Training a Tiny Transformer with Flax NNX
The demonstrated language model applies multi-GPU sharding to a four-block Tiny Transformer trained on byte-level tokens from the Tiny Shakespeare dataset. Built with the Flax NNX API, the training workflow replicates model and optimizer states across the GPU mesh, shards training batches, computes cross-entropy loss with Optax, and saves checkpoints using Orbax.
Efficient inference and text generation
After restoring trained weights with Orbax, the model generates text conditioned on prompts. To maintain high throughput during autoregressive token generation, input sequences are padded to a fixed maximum sequence length, preventing repeated compilation overhead in the JIT-compiled forward path.
The Bottom Line
The video establishes how JAX uses declarative sharding abstractions like Mesh, PartitionSpec, and NamedSharding to scale deep learning models across multiple GPUs with minimal modifications to core training loops. It proves this methodology by scaling an MLP benchmark and building an end-to-end Tiny Transformer using Flax NNX and Orbax checkpointing. While declarative data parallelism is shown to be effective, the presenters note that communication overhead impacts scaling efficiency and leave advanced model parallelism techniques for future exploration.
FAQ
What is multi-GPU scaling in JAX and how does data parallelism work in practice?
Multi-GPU scaling in JAX enables models to train across multiple accelerators by distributing array placement across a device mesh. In data parallelism, model parameters are replicated across all GPUs while training batches are sharded, allowing local gradients to be computed concurrently and averaged across devices using an all-reduce operation.
What are the four core concepts used for array placement and sharding in JAX?
The four core concepts are Mesh, which names the device grid; PartitionSpec, which specifies how array dimensions are split or replicated; NamedSharding, which merges the device mesh with the partition specification; and jax.device_put, which applies the placement plan to concrete arrays.
How can developers verify whether array sharding and parameter replication in JAX are configured correctly?
Developers can use jax.debug.visualize_array_sharding to inspect arrays visually and confirm that data batches are properly partitioned across devices while model weights are replicated.
When should machine learning engineers use shard_map instead of declarative automatic sharding in JAX?
Declarative automatic sharding is recommended for rapid experimentation, whereas shard_map and collective functions like jax.lax.pmean are suited for debugging and scenarios requiring explicit, fine-grained control over individual device shards.
Why is input padding to a fixed maximum length used during Tiny Transformer text generation in JAX?
Inputs are padded to a fixed maximum sequence length so that the JIT-compiled forward generation path avoids recompiling the execution graph for every new token length.
Worth watching for
Machine learning engineers and JAX developers seeking to scale deep learning training workflows across multi-GPU hardware architectures using modern Flax NNX and JAX sharding APIs.
- jax
- gpus
- flax
- transformers
- distributed-training