NVIDIA releases four Cosmos 3 open-weight world simulation models for robotics and autonomous driving

NVIDIA expanded its Cosmos 3 platform with four specialized post-trained checkpoints derived from the 16B-parameter Cosmos 3 Nano base model. The release targets two domains: autonomous driving and robotics. For driving, Cosmos3-Nano-Transfer-Auto generates synchronized multi-view video and LiDAR from scenario controls, while Cosmos3-Nano-Sim-Auto provides a causally distilled streaming variant for real-time interaction. Cosmos3-Nano-Sim-Transfer converts physics simulator outputs—depth, segmentation, edges, and blur—into photorealistic RGB video. For robotics, Cosmos3-Nano-Sim-Bimanual predicts visual consequences from two-arm robot actions, supporting both YAM and AgiBot embodiments with joystick or policy control.

All four models ship as downloadable weights under the OpenMDW1.1 license, which permits commercial and non-commercial use. NVIDIA accompanies the release with inference notebooks, technical documentation, and explicit post-training recipes so researchers can adapt Transfer-Auto to custom sensor rigs and distill their own streaming variants. The models require no NVIDIA-specific runtime for basic inference, though training and distillation assume CUDA-accessible hardware.

Model architecture and simulation pipeline design

• Bidirectional offline generation vs. causal streaming: Transfer-Auto operates bidirectionally for high-quality offline synthetic data generation across up to 11 camera views including fisheyes, with optional joint RGB-LiDAR output and per-camera World Scenario Map controls; Sim-Auto is a causally distilled variant that generates successive small video chunks in few denoising steps using rolling history for closed-loop streaming simulation.
• Multi-sensor control interfaces: Transfer-Auto and Sim-Auto accept aligned HD-map range-map controls for LiDAR generation alongside per-camera WSM controls that specify scene structure, traffic motion, and environmental conditions including weather, lighting, and time of day.
• Physics-decoupled rendering pipeline: Sim-Transfer consumes spatial controls (depth, segmentation, edges, blur) computed by an external physics simulator rather than raw physics state, keeping ground-truth dynamics in the simulator while the model handles appearance generation guided by text prompts.
• Bimanual action schema: Sim-Bimanual uses a 64-dimensional control vector (59 active values padded) with defined coordinate frames, rotations, gripper calibration, validity masks, and control domains, requiring T+1 observations for T transitions to maintain temporal alignment during training and inference.
• OpenMDW1.1 licensing and adaptation path: The weights are released under OpenMDW1.1 permitting commercial use; NVIDIA documents a clear adaptation workflow starting from Transfer-Auto post-training on domain-specific data, followed by causal training and few-step distillation to produce custom Sim-Auto variants.

Why it matters

These checkpoints give the open-source robotics and autonomous driving research communities a shared, commercially permissible foundation for world model development without requiring proprietary simulation stacks. The explicit post-training and distillation recipes reduce vendor lock-in by enabling adaptation to custom sensor configurations and embodiments, though practical training still depends on CUDA-accessible compute. For Linux-based robot operating systems and simulation frameworks like ROS 2, Isaac Sim, and CARLA, the models provide drop-in generative backends that can replace or augment traditional rendering pipelines.