LeGo-Drive: Language-enhanced Goal-oriented Closed-Loop End-to-End Autonomous Driving

Pranjal Paul*1, Anant Garg*1, Tushar Chaudhary1, Arun Singh2, K. Madhav Krishna1,
1International Institute of Information Technology - Hyderabad
2University of Tartu, Estonia
Paper arXiv Video
Teaser Image

The proposed method, LeGo-Drive estimates a goal location queried with a navigation instruction- “Park near the bus stop on the front-left” on a single front-facing camera image and coupled it with a differentiable optimizer-based planner that jointly optimizes the trajectory and the goal location. (Left) The proposed architecture is shown along with the gradient flow for joint end-to-end training. (Right-Top) Goal improvement from initial estimation in Green to improved location in Red. (Right-Bottom) Trajectory output in Red leading to the improved goal location, compared with the trajectory generated by baseline in Green.

Abstract

Existing Vision-Language Models (VLMs) produce long-term trajectory waypoints or directly control actions based on their perception input and language prompt. However, these VLMs are not explicitly aware of the constraints imposed by the scene or kinematics of the vehicle. As a result, the generated trajectories or control inputs are likely to be unsafe and/or infeasible. In this paper, we introduce LeGo-Drive † , which aims to address these issues. Our key idea is to use the VLM to just predict a goal location based on the given language command and perception input, which is then fed to a downstream differentiable trajectory optimizer with learnable components. We train the VLM and the trajectory optimizer in an end-to-end fashion using a loss function that captures the ego-vehicle’s ability to reach the predicted goal while satisfying safety and kinematic constraints. The gradients during the back-propagation flow through the optimization layer and make the VLM aware of the planner’s capabilities, making more feasible goal predictions. We compare our end-to-end approach with a decoupled framework where the planner is just used at the inference time to drive to the VLM-predicted goal location and report a goal reaching Success Rate of 81%. We demonstrate the versatility of LeGo-Drive † across various driving scenarios and navigation commands, highlighting its potential for practical deployment in autonomous vehicles.

Video

Park-In

Park-Out

Right Turn

Cross Left Turn

Visual Question Answering

Given the front-view RGB image, we employ GPT-4V [29] to determine the best course of action from a range of potential driving maneuvers. GPT-4V accurately identifies an obstruction ahead and recommends switching to the left lane to continue moving forward. With this recommended command, our pipeline is able to predict a collision-free goal point and an optimized trajectory.

Compound Command

Compound commands such as "Turn right and stop by the food stall on the right" is broken into atomic commands: Turn right and Stop by the food stall on the right. These atomic commands are then executed sequentially.