Profiling in PyTorch (Part 2): From nn.Linear to a Fused MLP
Hugging Face published the second part of its PyTorch profiling series, showing how to read GPU performance data and speed up a common AI building block by packaging its math into fewer kernels. The standard version fired 5 separate GPU kernels per forward pass, while compiling the same model with torch.compile cut that to 4 by fusing the small pointwise steps into one kernel. The fused path skipped roughly 50 MB of intermediate memory traffic per pass and ran in 89.4 microseconds, slightly faster than a hand-tuned alternative at 92.8 microseconds.
Key Takeaways
- We're on a journey to advance and democratize artificial intelligence through open source and open science.
We also discussed several other topics that came our way - the CPU dispatch chain, launch overhead, the difference between an overhead-bound and a compute-bound regime, and some internals of torch.compile .
- Like before, it helps to open them in a separate tab and walk through the code as you read.
- From matmul-add to Linear nn.Linear is a module wrapper around the same matrix multiplication and addition we already profiled in Part 1 .
The only difference is that it owns its weight and bias as parameters and exposes a forward method that PyTorch users have grown familiar with.
- This is why we see three Profile Steps in the CPU and GPU lanes.
Figure 2: The transpose CPU row If we zoom into the profiler trace, as we do in Figure 2, we notice an aten::t (transpose) op before the aten::addmm (multiplication and addition) op.
- Figure 3: No aten::add in the profile of a linear layer There is no aten::add (the bias addition) in the dispatch chain of the linear layer, as seen in Figure 3.
We're on a journey to advance and democratize artificial intelligence through open source and open science. We also discussed several other topics that came our way - the CPU dispatch chain, launch overhead, the difference between an overhead-bound and a compute-bound regime, and some internals of torch.compile . In the second iteration (this blog post), we climb one rung up the ladder.
We replace the hand-written matmul-add pair with an nn.Linear (with bias=True ). This is the building block every deep learning model uses. We then stack three of them (specific to our example), with an activation in between, to form a Multilayer Perceptron (MLP) block.
The scripts for this blog post live here: 02_linear.py , 03_simple_mlp.py , and 03_kernels_mlp.py . Like before, it helps to open them in a separate tab and walk through the code as you read. We use an NVIDIA A100-SXM4-80GB GPU to run the scripts.
For more details please read the original article at Hugging Face.
Continue Learning
Comments
Comments appear only after moderation. Your email identifies your submission to the moderator and is never displayed here.
No approved comments yet.