Profiling in PyTorch (Part 3): Attention is all you profile
We're on a journey to advance and democratize artificial intelligence through open source and open science. profiler Profiling in PyTorch (Part 2): From nn. Linear to a Fused MLP Profiling in PyTorch (Part 3): Attention is all you profile (current) The series "Profiling in PyTorch" is meant to make you comfortable reading profiler traces and tables.
Key Takeaways
- In Part 1 we profiled basic math operations like addition and multiplication.
We saw how the profiler table uncovers hotspots, and how the profiler trace shows the order in which an algorithm runs over time.
- While being infamous for its quadratic-time complexity, many clever tricks exist to mitigate that issue and make it fast.
- One could also run the scripts with the Hugging Face Jobs pipeline .
Naive attention Attention works with Queries ( ), Keys ( ), and Values ( ).
- Inside (our annotated forward call) we can see exactly the operations we guessed.
The matmul is an old friend by now, and the new operations are easy to spot: : the scaling : the causal masking : the softmax kernel Now let's unfold the GPU lane and see which kernels were actually launched.
- The memory copy is the odd one out, so where does this come from?
In Part 1 we profiled basic math operations like addition and multiplication. We saw how the profiler table uncovers hotspots, and how the profiler trace shows the order in which an algorithm runs over time. In Part 2 we wrapped that addition and multiplication into a torch linear layer.
We then stacked several linear layers on top of each other (a multilayer perceptron) and profiled that. Along the way we also profiled fused and hand-tuned kernels. From the perspective of the Transformer architecture, the next logical step for us to profile is yet another fundamental algorithm, attention.
While being infamous for its quadratic-time complexity, many clever tricks exist to mitigate that issue and make it fast. Our goal here is not to cover every trick in detail. Instead, we want to see how each one looks different under the profiler.
For more details please read the original article at Hugging Face.
Continue Learning
Comments
Sign in to join the conversation