DiScoFormer: One transformer for density and score, across distributions
Pinning down that distribution means estimating two quantities: the distribution's density and, more useful as dimensionality grows, its score. The density is the smooth version of a histogram-high where points cluster and low where they're scarce. The score-the gradient of the log-density-points in the direction the density rises fastest: move a point along the score and it heads toward a more probable region.
Key Takeaways
- Diffusion-based generative models (the technology behind AI image generators like Stable Diffusion and DALL-E) start from random noise and repeatedly follow the score, turning that noise into a realistic image.
The same score drives Bayesian sampling and the particle simulations used to model systems such as plasma.
- Alternatively, neural score-matching models trained to predict the score stay accurate even in high dimensions, but each needs to learn the distribution and must be retrained from scratch for another.
We introduce a new solution called the DiScoFormer (Density and Score Transformer) -one model that, given a set of data points, estimates both the density and the score of the distribution in a single forward pass without retraining.
- We leverage this by having a shared backbone with two output heads, one for the density and one for the score.
- Kernel density estimation has a single bandwidth-how far each point's influence reaches, fixed in advance and applied identically everywhere.
Attention is a strict generalization of it: we analytically show that a single attention head's weights are nearly a Gaussian kernel over the data, so one cross-attention block can already reproduce KDE's density and score.
- Firstly, GMMs are universal density approximators-with enough components they match essentially any smooth distribution to arbitrarily small error.
Diffusion-based generative models (the technology behind AI image generators like Stable Diffusion and DALL-E) start from random noise and repeatedly follow the score, turning that noise into a realistic image. The same score drives Bayesian sampling and the particle simulations used to model systems such as plasma. Extracting the density and score from a finite sample is challenging, and today's tools force a trade-off between generalizability and accuracy.
One classical approach, kernel density estimation (KDE), computes the density at any location from the data points around it: the closer and more numerous they are, the higher the density. It needs no training and applies to any distribution, but its accuracy falls off sharply as dimensionality grows. Alternatively, neural score-matching models trained to predict the score stay accurate even in high dimensions, but each needs to learn the distribution and must be retrained from scratch for another.
We introduce a new solution called the DiScoFormer (Density and Score Transformer) -one model that, given a set of data points, estimates both the density and the score of the distribution in a single forward pass without retraining. Training a transformer for density and score estimation DiScoFormer maps an entire sample to the density and score of the distribution behind it using stacked layers of transformer blocks. The model utilizes cross-attention, which allows it to evaluate density and score at any point-not just where you have data.
For more details please read the original article at Hugging Face.
Continue Learning
Comments
Comments appear only after moderation. Your email identifies your submission to the moderator and is never displayed here.
No approved comments yet.