Taxi Demo
Seaquest Demo

Evaluating Video In-Context Learning

for Multimodal Agents in Interactive Environments

Paper Code Case Data

Human demonstration

Taxi human demonstration: initial state
›
Taxi human demonstration: middle state
›
Taxi human demonstration: final state
→

Agent decisions

Taxi agent: initial state
›
Taxi agent: passenger picked up
›
Taxi agent: successful dropoff
0 Tasks | 0 Environments | 0 Models
Tetris Demo
Highway Demo

Benchmark Overview

Benchmark Overview

Explore Evaluation Tasks

Human demonstrations and recorded agent trajectories across every benchmark environment.

Watch the model play Taxi

Benchmark Results

Model Overall Navigation Control Strategy
Human 94.2 96.8 93.4 92.5
GPT-4o 68.3 72.1 65.8 67.0
Claude Sonnet 3.5 65.7 70.3 62.4 64.5
Gemini 1.5 Pro 63.2 68.9 59.8 61.0

Trajectory Analysis

Trajectory Analysis