Gradient-based Planning for World Models at Longer Horizons
Berkeley BAIR researchers introduce GRASP, a gradient-based planner for learned dynamics models known as world models. GRASP aims to make long-horizon planning practical by lifting the trajectory into virtual states for parallel optimization, adding stochasticity to the state iterates for exploration, and reshaping gradients so actions get clean signals. The work addresses why planning with modern world models becomes fragile over long horizons.
Key Takeaways
- Large, learned world models are becoming increasingly capable.
They can predict long sequences of future observations in high-dimensional visual spaces and generalize across tasks in ways that were difficult to imagine a few years ago.
- But having a powerful predictive model is not the same as being able to use it effectively for control/learning/planning.
In practice, long-horizon planning with modern world models remains fragile: optimization becomes ill-conditioned, non-greedy structure creates bad local minima, and high-dimensional latent spaces introduce subtle failure modes.
- This blog post discusses work done with Mike Rabbat, Aditi Krishnapriyan, Yann LeCun, and Amir Bar (* denotes equal advisorship), where we propose GRASP.
- Suppose you take actions $a_t \in \mathcal{A}$ and observe states $s_t \in \mathcal{S}$ (images, latent vectors, proprioception).
A world model is a learned model that, given the current state and a sequence of future actions, predicts what will happen next.
- The key point is that a world model gives you a differentiable simulator ; you can roll it forward under hypothetical action sequences and backpropagate through the predictions.
Large, learned world models are becoming increasingly capable. They can predict long sequences of future observations in high-dimensional visual spaces and generalize across tasks in ways that were difficult to imagine a few years ago. As these models scale, they start to look less like task-specific predictors and more like general-purpose simulators.
But having a powerful predictive model is not the same as being able to use it effectively for control/learning/planning. In practice, long-horizon planning with modern world models remains fragile: optimization becomes ill-conditioned, non-greedy structure creates bad local minima, and high-dimensional latent spaces introduce subtle failure modes. In this blog post, I describe the problems that motivated this project and our approach to address them: why planning with modern world models can be surprisingly fragile, why long horizons are the real stress test, and what we changed to make gradient-based planning much more robust.
This blog post discusses work done with Mike Rabbat, Aditi Krishnapriyan, Yann LeCun, and Amir Bar (* denotes equal advisorship), where we propose GRASP. These days, the term "world model" is quite overloaded, and depending on the context can either mean an explicit dynamics model or some implicit, reliable internal state that a generative model relies on (e.g. when an LLM generates chess moves, whether there is some internal representation of the board). We give our loose working definition below.
For more details please read the original article at Berkeley BAIR.
Continue Learning
Comments
Comments appear only after moderation. Your email identifies your submission to the moderator and is never displayed here.
No approved comments yet.