NVIDIA Cosmos-H-Dreams: Bringing Real-Time Generative Simulation to Surgical Robotics
But evaluating and training these systems remains difficult. Physical robotic platforms are expensive to operate, experiments are slow to reproduce, and failures can damage instruments or biological material. Conventional simulators provide a safer alternative, but surgical scenes are exceptionally difficult to model: deformable tissue, fine instrument interactions, specular surfaces, sutures, needles, smoke, and occlusions all matter.
Key Takeaways
- World foundation models offer a different path.
Instead of manually authoring every object and physical interaction, they learn visual dynamics directly from synchronized video and robot kinematics.
- Today, we are introducing the next step: Cosmos-H-Dreams , a real-time, action-conditioned generative simulator for surgical robotics.
Cosmos-H-Dreams distills the capabilities of Cosmos-H-Surgical-Simulator into a causal, few-step student model and serves it through FlashDreams , NVIDIA's accelerated streaming-inference library.
- Given a surgical context frame and a future robot trajectory, it generates video showing the likely visual consequences of those actions.
This makes it useful for offline policy evaluation and synthetic data generation.
- The released model receives an initial RGB frame and a live stream of robot kinematics, then produces the next chunk of frames before continuing with the following action block.
We have also demonstrated the versatility of Cosmos-H-Dreams by collaborating with CMR Surgical and Cambridge Consultants to integrate it with the Versius surgeon controller, enabling real-time operation on the Versius platform.
- For the released dVRK tabletop model, the dual-arm dVRK action content, consisting of relative end-effector translation, rotation, and gripper state, is mapped into this common representation.
World foundation models offer a different path. Instead of manually authoring every object and physical interaction, they learn visual dynamics directly from synchronized video and robot kinematics. NVIDIA's Cosmos-H-Surgical-Simulator demonstrated this approach by generating future surgical video from an initial scene and a sequence of robot actions.
It enabled faster-than-physical evaluation and synthetic data generation across the Open-H-Embodiment ecosystem. Today, we are introducing the next step: Cosmos-H-Dreams , a real-time, action-conditioned generative simulator for surgical robotics. Cosmos-H-Dreams distills the capabilities of Cosmos-H-Surgical-Simulator into a causal, few-step student model and serves it through FlashDreams , NVIDIA's accelerated streaming-inference library.
Running on a single NVIDIA RTX PRO 6000 GPU, the result is an interactive environment that a person or a learned policy can control in a closed loop. From Surgical World Model to Interactive Simulator Cosmos-H-Surgical-Simulator is an action-conditioned world foundation model built on NVIDIA Cosmos-Predict2. 5-2B and post-trained on the Open-H-Embodiment dataset.
For more details please read the original article at Hugging Face.
Continue Learning
Comments
Sign in to join the conversation