Evaluating Video In-Context Learning
for Multimodal Agents in Interactive Environments
Human demonstrations and recorded agent trajectories across every benchmark environment.
No matching games.
| Model | Overall S / P% |
Maze S / P% |
Shooter S / P% |
Control S / P% |
Dodge S / P% |
Collect S / P% |
Sport S / P% |
Single S / P% |
Multiple S / P% |
Static S / P% |
Dynamic S / P% |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Closed-source Models | |||||||||||
| Seed-2.1-Pro | 54.4 / 58 | 64.4 / 70 | 43.3 / 49 | 88.4 / 73 | 47.1 / 51 | 52.3 / 60 | 54.3 / 80 | 68.9 / 70 | 52.7 / 57 | 74.3 / 70 | 47.0 / 54 |
| Gemini-3.6-Flash | 49.5 / 53 | 61.0 / 70 | 37.9 / 39 | 88.1 / 80 | 39.9 / 44 | 42.2 / 63 | 29.7 / 50 | 67.1 / 70 | 47.3 / 52 | 77.2 / 75 | 39.2 / 45 |
| Gemini-3.1-Pro | 48.0 / 53 | 49.1 / 54 | 44.9 / 46 | 65.5 / 53 | 42.0 / 48 | 40.3 / 45 | 34.3 / 70 | 61.5 / 70 | 46.4 / 51 | 67.7 / 69 | 40.8 / 47 |
| GPT-5.6-sol | 46.7 / 51 | 52.0 / 53 | 37.6 / 43 | 82.7 / 87 | 38.1 / 43 | 41.5 / 43 | 35.0 / 60 | 82.5 / 85 | 42.4 / 47 | 68.3 / 67 | 38.8 / 46 |
| Gemini-3-Flash | 39.6 / 42 | 36.9 / 37 | 37.6 / 39 | 64.5 / 67 | 34.7 / 36 | 31.1 / 40 | 34.3 / 60 | 52.2 / 50 | 38.1 / 41 | 54.7 / 53 | 34.0 / 38 |
| Seed-2.0-Lite | 37.0 / 41 | 39.0 / 51 | 26.5 / 24 | 86.3 / 80 | 31.4 / 36 | 36.2 / 48 | 32.0 / 60 | 54.2 / 60 | 34.9 / 39 | 50.6 / 49 | 31.9 / 38 |
| Open-weight Models | |||||||||||
| DeepSeek-V4.1-Flash | 34.8 / 35 | 34.1 / 36 | 28.7 / 26 | 71.6 / 67 | 32.4 / 33 | 26.0 / 30 | 32.3 / 50 | 46.1 / 45 | 33.5 / 34 | 51.0 / 50 | 28.8 / 30 |
| Kimi-K3 | 32.5 / 39 | 32.3 / 40 | 26.4 / 34 | 70.1 / 73 | 27.2 / 34 | 25.4 / 30 | 26.3 / 40 | 51.8 / 55 | 30.2 / 37 | 46.7 / 46 | 27.3 / 36 |
| Qwen3.5-397B-A17B | 32.5 / 31 | 28.2 / 33 | 30.3 / 27 | 73.6 / 67 | 28.1 / 27 | 25.4 / 30 | 25.7 / 40 | 44.2 / 45 | 31.1 / 30 | 45.0 / 42 | 27.9 / 27 |
| DeepSeek-V4F-V | 25.2 / 30 | 21.0 / 29 | 26.0 / 34 | 47.1 / 53 | 24.1 / 30 | 19.3 / 30 | 21.7 / 30 | 26.6 / 15 | 25.0 / 32 | 30.1 / 26 | 23.4 / 32 |
| gemma-4-31B-it | 21.9 / 25 | 20.4 / 28 | 17.7 / 29 | 32.8 / 27 | 19.7 / 25 | 14.4 / 18 | 13.7 / 10 | 40.1 / 35 | 19.7 / 24 | 29.8 / 25 | 18.9 / 25 |
| MiMo-v2.5 | 20.4 / 23 | 20.0 / 21 | 18.3 / 26 | 34.6 / 27 | 21.4 / 27 | 13.1 / 28 | 16.0 / 20 | 25.6 / 15 | 19.8 / 24 | 22.2 / 14 | 19.8 / 26 |
| Qwen2.5-VL-72B | 17.7 / 23 | 14.3 / 16 | 16.3 / 35 | 20.1 / 7 | 20.4 / 31 | 11.5 / 25 | 10.3 / 0 | 19.0 / 10 | 17.5 / 25 | 16.9 / 8 | 17.9 / 29 |
| MiMo-VL-7B | 17.4 / 14 | 13.9 / 15 | 15.8 / 10 | 19.7 / 13 | 19.9 / 16 | 13.3 / 10 | 12.7 / 10 | 20.6 / 20 | 17.0 / 13 | 17.1 / 14 | 17.5 / 14 |
| Qwen3.6-27B | 16.9 / 27 | 10.5 / 18 | 13.8 / 36 | 17.5 / 7 | 19.2 / 33 | 12.2 / 28 | 22.7 / 40 | 17.3 / 10 | 16.8 / 29 | 14.8 / 8 | 17.6 / 34 |
| GLM-4.5V | 16.6 / 21 | 10.9 / 15 | 13.1 / 27 | 28.7 / 27 | 18.1 / 26 | 11.9 / 15 | 12.0 / 10 | 17.2 / 5 | 16.5 / 23 | 16.6 / 10 | 16.6 / 25 |
| Qwen3.5-VL-27B | 16.4 / 24 | 13.3 / 15 | 12.4 / 35 | 22.4 / 13 | 17.8 / 32 | 12.2 / 20 | 20.0 / 40 | 18.5 / 0 | 16.2 / 27 | 14.2 / 4 | 17.3 / 32 |
| Qwen3-Omni-30B | 13.6 / 20 | 9.7 / 13 | 5.8 / 20 | 19.5 / 13 | 15.6 / 27 | 5.3 / 18 | 16.7 / 30 | 19.6 / 10 | 12.8 / 22 | 16.9 / 10 | 12.3 / 24 |
| Qwen3-VL-235B | 12.3 / 5 | 12.5 / 6 | 6.9 / 3 | 19.7 / 13 | 11.7 / 5 | 11.6 / 5 | 15.3 / 10 | 21.6 / 5 | 11.2 / 5 | 14.2 / 6 | 11.6 / 5 |
| Reference Baselines | |||||||||||
| Random | 14.0 / 14 | 7.9 / 10 | 19.1 / 23 | 16.8 / 13 | 15.5 / 15 | 15.0 / 15 | 19.4 / 30 | 5.6 / 2 | 15.0 / 15 | 8.4 / 5 | 16.1 / 17 |
| Human-score | 83.6 / 91 | 92.1 / 96 | 76.4 / 85 | 94.6 / 89 | 82.8 / 90 | 91.3 / 99 | 69.8 / 97 | 99.0 / 100 | 81.8 / 90 | 98.1 / 99 | 78.3 / 88 |
