What exactly does word2vec learn?
Researchers from Berkeley's BAIR lab present a quantitative theory of how word2vec learns word representations. They prove that in realistic regimes the learning problem reduces to unweighted least-squares matrix factorization, and they solve the gradient flow dynamics in closed form so that the final representations are given by PCA. When trained from small initialization, word2vec learns one concept at a time in discrete steps, each incrementing the rank of the embedding matrix.
Key Takeaways
- What exactly does word2vec learn, and how?
Answering this question amounts to understanding representation learning in a minimal yet interesting language modeling task.
- When trained from small initialization, word2vec learns in discrete, sequential steps.
Left: rank-incrementing learning steps in the weight matrix, each decreasing the loss.
- This so-called linear representation hypothesis has recently garnered a lot of attention since LLMs exhibit this behavior as well , enabling semantic inspection of internal representations and providing for novel model steering techniques .
In word2vec , it is precisely these linear directions that enable the learned embeddings to complete analogies (e.g., "man : woman :: king : queen") via embedding vector addition.
- Then (under some mild approximations) the embeddings collectively learn one "concept" (i.e., orthogonal linear subspace) at a time in a sequence of discrete learning steps.
It's like when diving head-first into learning a new branch of math.
- Our theory allows us to compute each of these features a priori in closed form - they are simply the eigenvectors of a particular target matrix which is defined solely in terms of measurable corpus statistics and algorithmic hyperparameters.
What exactly does word2vec learn, and how? Answering this question amounts to understanding representation learning in a minimal yet interesting language modeling task. Despite the fact that word2vec is a well-known precursor to modern language models, for many years, researchers lacked a quantitative and predictive theory describing its learning process.
In our new paper , we finally provide such a theory. We prove that there are realistic, practical regimes in which the learning problem reduces to unweighted least-squares matrix factorization . We solve the gradient flow dynamics in closed form; the final learned representations are simply given by PCA.
Right: three time slices of the latent embedding space showing how embedding vectors expand into subspaces of increasing dimension at each learning step, continuing until model capacity is saturated. Before elaborating on this result, let's motivate the problem. word2vec is a well-known algorithm for learning dense vector representations of words.
For more details please read the original article at Berkeley BAIR.
Continue Learning
Comments
Comments appear only after moderation. Your email identifies your submission to the moderator and is never displayed here.
No approved comments yet.