How to train generalist robots with NVIDIA's research workflows and foundation models - The Robot Report

NVIDIA researchers are advancing scalable robot training by leveraging generative AI, world foundation models (WFMs), and synthetic data generation workflows to overcome the traditional challenges of collecting and labeling large datasets for each new robotic task or environment. Central to this effort is the use of WFMs like NVIDIA Cosmos, which are trained on millions of hours of real-world data to predict future states and generate video sequences from single images. This capability enables rapid, high-fidelity synthetic data generation, significantly accelerating robot learning and reducing development time from months to hours. Key components of NVIDIA’s approach include DreamGen, a synthetic data pipeline that creates diverse and realistic robot trajectory data with minimal human input, and GR00T models that facilitate generalist skill learning across varied tasks and embodiments. The DreamGen pipeline involves four main steps: post-training a world foundation model (e.g., Cosmos-Predict2) on a small set of real demonstrations, generating synthetic photorealistic robot videos from image and language prompts, extracting pseudo-actions