Imagine-2-Drive: High-Fidelity World Modeling in CARLA for Autonomous Vehicles

Anant Garg1, K. Madhav Krishna1,
1International Institute of Information Technology - Hyderabad

Looking for MS/PhD positions for FY 2025

Paper arXiv Video Code (coming soon)

VISTA Future Frames Denoising Results

Denoising of future frames predictions using VISTA. Frames are predicted according to the input trajectory shown as color coded. Unlike single-step models, VISTA predicts future states simultaneously, eliminating compounding errors.

Teaser Image

Using front camera RGB image as the sole input modality, Imagine-2-Drive provides a framework to combine VISTAPlan, VISTA based World Model with DPA, a multi-modal diffusion based policy actor. Given a trajectory output by DPA, shown in Red and the corresponding predicted future observations from VISTAPlan, the DDPO tries to find an optimal policy by maximizing the cumulative sum of rewards from future states. The proposed architecture is shown along with the gradient flow for joint end-to-end training.

Abstract

In autonomous driving with image based state space, accurate prediction of future events and modeling diverse behavioral modes are essential for safety and effective decision- making. World model-based Reinforcement Learning (WMRL) approaches offers a promising solution by simulating future states from current state and actions. However, utility of world models is often limited by typical RL policies being limited to deterministic or single gaussian distribution. By failing to capture the full spectrum of possible actions, reduces their adaptability in complex, dynamic environments. In this work, we introduce Imagine-2-Drive, a framework that consists of two components, VISTAPlan, a high-fidelity world model for accurate future prediction and Diffusion Policy Actor (DPA), a diffusion based policy to model multi-modal behaviors for trajectory prediction. We use VISTAPlan to simulate and eval- uate trajectories from DPA and use Denoising Diffusion Policy Optimization (DDPO) to train DPA to maximize the cumulative sum of rewards over the trajectories. We analyze the benefits of each component and the framework as a whole in CARLA with standard driving metrics. As a consequence of our twin novelties- VISTAPlan and DPA, we significantly outperform the state of the art (SOTA) world models on standard driving metrics by 15% and 20% on Route Completion and Success Rate respectively.

Video

VISTA Future Frames Prediction Results

Future obervation predictions from the VISTAPlan World Model, conditioned on the input trajectory and current observations. Demonstrates the VISTAPlan’s ability to accurately predict future observations based on the provided context, highlighting its robust trajectory prediction capabilities.

Long Run Demo

Multi-Modal Demo

BibTeX

@misc{garg2024imagine2drivehighfidelityworldmodeling,
        title={Imagine-2-Drive: High-Fidelity World Modeling in CARLA for Autonomous Vehicles}, 
        author={Anant Garg and K Madhava Krishna},
        year={2024},
        eprint={2411.10171},
        archivePrefix={arXiv},
        primaryClass={cs.RO},
        url={https://arxiv.org/abs/2411.10171},
  }