AI Creators
StatQuest with Josh Starmer
The GELU, SiLU and SwiGLU activation functions, clearly explained!!!
Sep 8, 202624 min watch
Quick Overview
This educational video by Josh Starmer from StatQuest provides an illustrated walkthrough of activation functions in modern neural networks. The tutorial traces the historical progression from sigmoid and ReLU activations to newer functions such as GELU, SiLU, and SwiGLU, explaining their mathematical derivations and practical roles in deep learning architectures.
Key Points
- 1.The GELU (Gaussian Error Linear Unit) and SiLU (Sigmoid Linear Unit) activation functions provide smooth, differentiable alternatives to the ReLU activation function.
- 2.Unlike the standard ReLU function which outputs zero for all negative values, GELU and SiLU create a subtle dip between negative two and zero.
- 3.The GELU activation function can be mathematically derived from the weighted average of applying input-dependent dropout with a Gaussian cumulative distribution function.
- 4.The SiLU activation function is similarly derived using a standard sigmoid function to determine the input-dependent dropout probability.
- 5.SwiGLU enhances the Swish/SiLU approach by incorporating extra trainable weights and a gated linear unit multiplication, offering greater shape flexibility.
- 6.Major artificial intelligence architectures employ these functions, with Google often using GELU and Meta frequently adopting SwiGLU.
Summary
Neural network architectures historically relied on the sigmoid activation function, which produced values bounded between 0 and 1. While intuitively matching biological neuron firing states, the sigmoid function suffered from vanishing gradients as networks grew deeper. Because the derivative of a sigmoid curve approaches zero for values far from zero, gradient descent took excessively small steps during backpropagation, making deep models nearly impossible to train and frequently resulting in underfitting.
Around 2010, researchers widely adopted the Rectified Linear Unit, or ReLU. With a derivative of one for all positive values and zero for negative values, ReLU dramatically sped up gradient descent and enabled deep network training, eventually powering milestones such as the 2017 Transformer. Despite this success, large networks using ReLU tended to create jagged, piecewise linear predictions that overfitted training data. While techniques like dropout helped by randomly removing nodes during training to average multiple smaller subnetworks, standard dropout acted independently of whether input signals were large or small.
To address this limitation, researchers developed input-dependent dropout, where the probability of dropping a node varies based on the incoming value. By computing the weighted average of keeping versus dropping an input across all values, a continuous mathematical function is formed. When using the cumulative distribution function of a standard Gaussian curve to calculate the probability of avoiding dropout, this weighted average yields the Gaussian Error Linear Unit, or GELU. When substituting the Gaussian curve with the standard sigmoid function, the resulting weighted average yields the Sigmoid Linear Unit, or SiLU, which is also commonly known as Swish.
The Swish Gated Linear Unit, or SwiGLU, builds further upon this concept by introducing trainable weight parameters into the input and multiplying the resulting Swish output with a gated linear bypass term. This structure allows the network to learn and customize the shape of its activation functions during training. While original research papers offer no formal theoretical proof for its exact advantage, the added mathematical flexibility allows smaller models to achieve superior performance without overfitting. Consequently, GELU and SwiGLU have become standard components in modern artificial intelligence, with GELU commonly favored in Google models and SwiGLU widely adopted in Meta architectures.
Limitations of Early Activation Functions
Before 2010, neural networks commonly relied on sigmoid activation functions to model biological neuron firing. As networks grew deeper, sigmoid derivatives approached zero for extreme input values, causing severe vanishing gradient problems that prevented large models from training effectively and led to underfitting. The introduction of ReLU solved this training bottleneck with constant derivatives of one for positive values, enabling deep architectures like the 2017 Transformer. However, deep networks built with ReLU often created sharp, jagged fits that overfitted training data.
Deriving GELU and SiLU from Input-Dependent Dropout
Traditional dropout prevents overfitting by randomly disabling activation functions during training, effectively averaging many smaller sub-networks. However, standard dropout applies independently of the input magnitude. By making dropout probability dependent on the input value, an equivalent single activation function emerges. When a standard Gaussian cumulative distribution function determines the retention probability, the resulting weighted average produces GELU. When the standard sigmoid curve determines the retention probability, the resulting weighted average produces SiLU, also referred to as Swish.
Constructing the SwiGLU Activation Function
SwiGLU, or Swish Gated Linear Unit, expands on SiLU by adding trainable weight parameters and multiplying the resulting output with a linear bypass term. By allowing the neural network to learn both weights during optimization, the activation function dynamically adapts its shape to fit the training data. While authors note there is no definitive theoretical proof for why this architecture outperforms alternatives, its increased parameter flexibility and tendency to yield smaller, more expressive networks make it widely used in modern language models, particularly those developed by Meta.
The Bottom Line
The video establishes that modern activation functions like GELU, SiLU, and SwiGLU evolved directly to balance the underfitting of sigmoid curves and the overfitting tendencies of ReLU. By framing continuous activation functions as weighted averages of input-dependent dropout, it provides an intuitive bridge between regularisation techniques and modern architecture design. While the empirical success of SwiGLU is demonstrated across advanced models, the exact mathematical explanation for its superiority remains an open question in active artificial intelligence research.
FAQ
What is the GELU activation function and what does GELU stand for?
GELU stands for Gaussian Error Linear Unit. It is a smooth, differentiable activation function derived from taking the weighted average of input-dependent dropout using the cumulative distribution function of a standard Gaussian distribution.
Why did the ReLU activation function replace the sigmoid activation function around 2010?
Sigmoid activation functions caused severe vanishing gradient issues in deep networks because their derivatives approached zero for large positive or negative inputs, making optimization extremely slow. ReLU solved this because its derivative remains a constant one for all positive inputs, facilitating fast backpropagation in deep architectures.
What is the primary difference between the SiLU and GELU activation functions?
GELU determines the probability of avoiding dropout using the cumulative distribution function of a standard Gaussian distribution, whereas SiLU (Sigmoid Linear Unit) determines that probability using a standard sigmoid curve. Both produce very similar smooth curves with a slight dip between negative two and zero.
How does the SwiGLU activation function differ from the standard SiLU activation function?
SwiGLU incorporates additional trainable weight parameters into the input calculation and multiplies the resulting Swish output by a linear bypass term, creating a gated structure that allows the network to dynamically adapt the function's curve shape during training.
Which major artificial intelligence organizations commonly use the GELU and SwiGLU activation functions?
According to the video, Google commonly favors the GELU activation function in its artificial intelligence models, while Meta frequently uses the SwiGLU activation function.
Worth watching for
Machine learning engineers, data scientists, and students looking to understand the mathematical derivation, history, and structural mechanics behind modern activation functions used in large language models.
- activation-functions
- gelu
- silu
- swiglu
- neural-networks
- deep-learning