TeraSim-World: Worldwide Safety-Critical Data Synthesis for End-to-End Autonomous Driving

Jiawei Wang1†, Haowei Sun2†, Xintao Yan3, Shuo Feng4, Jun Gao1,5, Henry X. Liu1*
1University of Michigan, 2SaferDrive AI, 3The University of Hong Kong, 4Tsinghua University, 5NVIDIA
arXiv Video Code

Starting from a global coordinate, TeraSim-World automatically synthesizes diverse and geographically grounded safety-critical data for E2E autonomous driving at anywhere in the world.

Abstract

Safe and scalable deployment of end-to-end (E2E) autonomous driving requires extensive and diverse data, particularly safety-critical events. Existing data are mostly generated from simulators with a significant sim-to-real gap or collected from on-road testing that is costly and unsafe.

This paper presents TeraSim-World, an automated pipeline that synthesizes realistic and geographically diverse safety-critical data for E2E autonomous driving at anywhere in the world. Starting from an arbitrary location, TeraSim-World retrieves real-world maps and traffic demand from geospatial data sources. Then, it simulates agent behaviors from naturalistic driving datasets, and orchestrates diverse adversities to create corner cases. Inspired by street views of the same location, it achieves photorealistic, geographically grounded sensor rendering via the frontier video generation model Cosmos-Drive.

By bridging agent and sensor simulations, TeraSim-World provides a scalable and critical data synthesis framework for training and evaluation of E2E autonomous driving systems.

Anywhere in the World

We showcase some representative safety-critical events from TeraSim-World. More synthesized data will be released.

Pedestrian Crossing

From Test Facility to the Real World

Test facilities like Mcity provide a controlled environment, but its miniature setup cannot fully reflect the real-world complexity. With TeraSim-World, existing facilities can now generate more realistic and diverse scenarios.

Related Links

We also provide resources and related work for TeraSim-World.

  • TeraSim presents the capability of uncovering unknown unsafe events in a Mcity Carla environment.
  • Cosmos-Drive-Dreams provides video generation pipeline from NVIDIA.
  • HDMap video rendering from trajectories and maps are built on top of the original Cosmos-Drive Toolkits.

We will continue to release supplementary materials and more synthesized data as they become available.

BibTeX

@article{wang2025terasim-world,
  author    = {Wang, Jiawei and Sun, Haowei and Yan, Xintao and Feng, Shuo and Gao, Jun and Liu, Henry},
  title     = {TeraSim-World: Worldwide Safety-Critical Data Synthesis for End-to-End Autonomous Driving},
  journal   = {arXiv preprint arXiv:2509.13164},
  year      = {2025},
}

Based on the template from this site

Web Analytics