Abstract
Autonomous personal mobility benefits greatly from the ability to predict executable spatial goals from language and a front-view image alone. In this study, we focus on absolute target position prediction from a language instruction and a front-view image, which requires referring expression disambiguation and absolute target localization without depth cues. To address this, we propose AiSCREAM, a language-conditioned target localization model based on cross-view vision-language reasoning. AiSCREAM constructs a cross-view geometric-semantic alignment between the front-view and an Aerial Semantic (AiS) view, an overhead-view image synthesized from the frontal image via a text-conditioned image-to-image generation model. By preserving appearance-level scene characteristics, the alignment can leverage language-conditioned semantic cues and enable more reliable absolute target localization from a single image. To evaluate AiSCREAM, we constructed DRAMATiST, a benchmark that introduces road and traffic diversity. Experimental results show that AiSCREAM achieved mean absolute position error of 4.03 m on the DRAMATiST, which outperformed baseline methods and human performance.
Method
AiSCREAM consists of two main modules: AiS cross-view Representation Alignment Module (AiRAM) and Multi Task Absolute Position Localizer (MTAPL).
AiRAM introduces a cross-view geometric-semantic alignment representation, constructed using semantic segmentations from both the frontal and AiS-view images as structural cues. Given a front camera image, it first generates an AiS-view image using a text-conditioned image-to-image generation model, then obtains semantic segmentations for both views using cross-view consistent semantic segmentation (CVSS). Bidirectional cross-attention fuses image and segmentation features within each view and aligns them across views.
MTAPL consists of a two-stage prediction pipeline. It first predicts an initial target position using three heads (frontal segmentation, AiS-view segmentation, and absolute target prediction), then refines this prediction using an MLLM Reshape Localizer that performs prediction-conditioned refinement using a visual prompt with an overlaid prediction and explicit numeric cues.
Framework of AiSCREAM. The model takes a front camera image and a navigation instruction, generates an AiS-view, constructs cross-view alignment via AiRAM, and predicts the target position via MTAPL with MLLM-based refinement.
Key Contributions
- Cross-view geometric-semantic alignment: A novel representation constructed from language-conditioned semantic segmentations of the frontal and Aerial Semantic views, enabling ego-centric spatial structure understanding.
- MLLM Reshape Localizer: Prediction-conditioned refinement using a visual prompt that overlays the initial prediction with explicit numeric cues, enabling correction relative to the initial reference.
- Height-based Cross-view (HC) loss: A loss function incorporating complementary cross-view supervision terms and obstacle region penalty for three prediction heads.
- DRAMATiST benchmark: A new benchmark of front-view images and navigation instructions that introduces road and traffic diversity beyond existing datasets.
Quantitative Results
Comparison between AiSCREAM and baseline methods on GRiN-Drive and DRAMATiST benchmarks. Best scores are in bold.
| Method | GRiN-Drive | DRAMATiST | ||||
|---|---|---|---|---|---|---|
| RMSE [m] ↓ | MAE [m] ↓ | mIoU [%] ↑ | RMSE [m] ↓ | MAE [m] ↓ | mIoU [%] ↑ | |
| Rufus et al. | 35.30 | 18.57 | 2.64 | 37.91 | 19.72 | 0.98 |
| GSVA | 25.58 | 21.83 | 2.53 | 22.15 | 18.15 | 0.84 |
| GENNAV | 10.51 | 6.35 | 7.25 | 8.56 | 6.45 | 2.83 |
| SAM3 | 74.22 | 44.67 | 0.49 | 33.85 | 16.37 | 0.71 |
| SimLingo | 25.64 | 21.79 | - | 22.19 | 18.84 | - |
| ORION | 30.27 | 28.34 | - | 30.67 | 28.22 | - |
| Alpamayo-R1 | 27.15 | 23.23 | - | 21.42 | 17.72 | - |
| Qwen3-VL | 27.05 | 23.00 | 0.42 | 18.25 | 13.79 | 0.76 |
| GPT-5.2 | 12.44 | 9.07 | 6.26 | 11.29 | 7.91 | 3.22 |
| AiSCREAM (Ours) | 4.72 | 3.95 | 15.09 | 5.00 | 4.03 | 6.16 |
| Human | 7.45 | 4.00 | 4.82 | 7.00 | 4.69 | 6.50 |
AiSCREAM achieved RMSE of 4.72 m on GRiN-Drive and 5.00 m on DRAMATiST, outperforming all baselines and even human performance.
Qualitative Results
Comparison of predictions by Ground Truth, GENNAV, GPT-5.2, and AiSCREAM. Red points indicate target positions and green regions indicate target distributions.
Example 1 (GRiN-Drive): "Drop me off next to that garbage can."
(a) Ground Truth
(b) GENNAV
(c) GPT-5.2
(d) AiSCREAM (Ours)
Example 2 (DRAMATiST): "Please wait by the pedestrian carrying backpack wearing brown pants walking on the left side of the road away from the ego car."
(a) Ground Truth
(b) GENNAV
(c) GPT-5.2
(d) AiSCREAM (Ours)
Example 3 (DRAMATiST): "Pull over next to the orange traffic cones on the right side of the road."
(a) Ground Truth
(b) GENNAV
(c) GPT-5.2
(d) AiSCREAM (Ours)
Additional Success Examples
Additional success examples of AiSCREAM on GRiN-Drive and DRAMATiST. Red points indicate target positions and green regions indicate target distributions.
GRiN-Drive
(i) "Get a bit closer to the black car in front."
(ii) "Park in front of the black car."
(iii) "Follow behind the silver van."
(iv) "Pull over next to this bike."
(v) "Move into the lane directly behind the black suv and follow him."
(vi) "Could you get closer to the police officer."
DRAMATiST
(vii) "Please pull over next to the blue sedan stopped in the ego lane."
(viii) "Stop around the pedestrian wearing white tshirt and walking on the left side of the road toward the ego car."
(ix) "Could you stop near the pedestrian wearing white top and black pants walking on the left side of the road towards the ego car?"
(x) "Could you stop next to the black van stopped on the left side of the road?"
Ablation Study
Module ablation. Each component of AiSCREAM contributes to the overall performance.
| Model | Condition | GRiN-Drive | DRAMATiST | ||||||
|---|---|---|---|---|---|---|---|---|---|
| RL | AiS | CVSS | RMSE [m] ↓ | MAE [m] ↓ | mIoU [%] ↑ | RMSE [m] ↓ | MAE [m] ↓ | mIoU [%] ↑ | |
| (i) Full model | ✓ | ✓ | ✓ | 4.72 | 3.95 | 11.89 | 5.00 | 4.03 | 6.16 |
| (ii) w/o RL | ✓ | ✓ | 4.95 | 4.18 | 10.48 | 5.19 | 4.29 | 5.90 | |
| (iii) w/o AiS | ✓ | 6.60 | 5.61 | 8.24 | 5.57 | 4.47 | 3.35 | ||
| (iv) w/o CVSS | ✓ | 5.80 | 4.83 | 9.50 | 5.34 | 4.31 | 3.43 | ||
RL: MLLM Reshape Localizer, AiS: Aerial Semantic view, CVSS: Cross-View Consistent Semantic Segmentation.
HC loss ablation. Both loss components contribute to cross-view consistency and error reduction.
| Model | HC loss | GRiN-Drive | DRAMATiST | |||||
|---|---|---|---|---|---|---|---|---|
| $\mathcal{L}_{\text{ais}}$ | $\mathcal{L}_{\text{obs}}$ | RMSE [m] ↓ | MAE [m] ↓ | mIoU [%] ↑ | RMSE [m] ↓ | MAE [m] ↓ | mIoU [%] ↑ | |
| (a) w/o RL | ✓ | ✓ | 4.95 | 4.18 | 10.48 | 5.19 | 4.29 | 5.90 |
| (b) w/o $\mathcal{L}_{\text{ais}}$ | ✓ | 5.17 | 4.26 | 9.86 | 5.24 | 4.35 | 5.30 | |
| (c) w/o $\mathcal{L}_{\text{obs}}$ | ✓ | 5.27 | 4.57 | 9.93 | 5.24 | 4.30 | 4.06 | |
$\mathcal{L}_{\text{ais}}$: AiS-view segmentation loss, $\mathcal{L}_{\text{obs}}$: Obstacle penalty loss. HC loss is ablated in the RL-disabled setting to directly assess impact on initial localization.
BibTeX
@inproceedings{katsumata2026aiscream,
title = {AiSCREAM: Absolute Target Localization with Language-Conditioned Cross-View Alignment for Autonomous Vehicles},
author = {Katsumata, Kei and Piao, Jun and Hosomi, Naoki and Yamada, Kentaro and Sugiura, Komei},
booktitle = {European Conference on Computer Vision (ECCV)},
year = {2026}
}