Interactive driving behaviors such as merging, crossing, and car-following are central to autonomous driving perception, prediction, and decision-making, yet automated systems still struggle in complex negotiation scenarios.
Download your copy for later!
With the rise of VLA models, there is growing demand for datasets that align trajectories, visual inputs, and language annotations to support interaction understanding and reasoning. However, existing public datasets remain dominated by routine driving behaviors, while safety-critical interactions are sparsely represented in a long-tail distribution.
Moreover, most resources lack explicit language annotations describing driver intentions and interaction causality, resulting in weak multimodal alignment.
Rather than building a new dataset from scratch, this work enhances five existing trajectory datasets through a scalable pipeline that extracts, physically quantifies, and semantically enriches egocentric interaction segments, yielding temporally aligned trajectory, visual, and language data for VLA training and evaluation.
Mining, Quantifying, and Enriching Driving Interactions
The IEDD construction pipeline proceeds through three sequential modules: trajectory preprocessing and scenario slicing, interaction quantification, and multimodal data synthesis.
Raw trajectories from five public datasets (Lyft Level 5, Waymo Open Motion, nuPlan, INTERACTION, and SIND) are first cleaned and standardized. All vehicle trajectories are resampled to a 0.1-second resolution, invalid points are removed, and heading angle fluctuations caused by low-speed driving or sensor noise are smoothed to ensure kinematic reliability.
The pipeline then employs a four-step cascade to mine interaction segments. A spatiotemporal intersection detector identifies candidate interaction moments using a double-pointer sliding window for efficiency. A two-stage classifier then categorizes each interaction. Finally, vehicles engaged in multiple simultaneous interactions are recursively merged into unified multi-agent groups, preserving the integrity of complex chain reactions.
Each extracted segment is then quantified through two complementary metric systems. The interaction intensity metric captures instantaneous conflict pressure by coupling three physical components: pose adjustment, risk variation, and an artificial potential field that models surrounding vehicles' risk distribution with heightened sensitivity to frontal threats.
Weights are adapted per interaction type, where merging emphasizes the potential field, crossing prioritizes risk variation, and head-on scenarios weight risk most heavily. The interaction efficiency metric evaluates traversal quality as the product of path consistency, time delay relative to free flow, and driving smoothness based on the standard deviation of acceleration.
The final module synthesizes multimodal instruction data. Continuous trajectories are discretized into behavioral atoms and temporally compressed into action chains. The interaction process is partitioned into three phases (approach, interaction, and outcome) using peak intensity as a time anchor.
Language annotations are generated through a rule-driven approach that maps intensity values to discrete linguistic modifiers, ensuring factual grounding without hallucination. Three text formats are produced, namely, global summaries, standardized action chains, and multi-turn question-answer pairs.
Visual inputs are rendered as bird's-eye-view (BEV) videos reconstructed from real trajectories in an egocentric coordinate system, chosen for both sensor-agnostic versatility and unoccluded global spatial awareness. BEV frames, linguistic descriptions, and trajectory timestamps are strictly synchronized at the frame level.
The resulting IEDD dataset contains 7.31 million interaction segments, of which 91% involve multi-agent interactions. IEDD-VQA further structures selected segments into a four-level evaluation framework spanning perception and recognition, behavior description, physical quantification, and counterfactual reasoning about alternative driver actions.
Evaluating Models Across Four Reasoning Levels
To validate IEDD-VQA, the authors established a hierarchical evaluation framework spanning four progressive levels: perception and identification (L1), action description quality (L2), quantitative and logical analysis (L3), and counterfactual reasoning (L4).
These are aggregated into a weighted integrated score, with L4 assigned double weight reflecting the paramount importance of safety-critical reasoning. An automated judge model (general language model (GLM)-4.7) scored all responses. Human validation on 100 samples confirmed strong alignment with expert judgments, yielding a Pearson correlation of 0.80 ± 0.03.
Ten mainstream vision-language models were evaluated zero-shot on 100 interaction scenarios using six uniformly sampled key frames per scenario. The open-source model Llama-4-Maverick achieved the highest overall score, surpassing closed-source alternatives including generative pre-trained transformer (GPT)-4o.
However, every model exhibited a severe bottleneck at L3, producing extremely large physical estimation errors, indicating that general-purpose models cannot reliably map visual features to numerical physical values without domain adaptation.
Introducing chain-of-thought prompting substantially improved Qwen2.5-VL-7B's L3 performance, reducing its estimation error dramatically. Fine-tuning this same model on the IEDD-VQA training set yielded a 78.7% overall improvement in L1–L3 score and reduced physical estimation error to near zero.
However, counterfactual reasoning performance collapsed almost entirely, revealing a trade-off between domain specialization and general reasoning capability. Post-fine-tuning, chain-of-thought prompting became counterproductive, suggesting the model had already internalized interaction reasoning logic.
Closing the Gap in Interaction Understanding
IEDD addresses critical gaps in autonomous driving research by providing 7.31 million interaction segments with physically grounded intensity and efficiency annotations. Validation reveals that, while general-purpose vision-language models struggle with physical quantification without domain adaptation, fine-tuning on IEDD-VQA yields substantial improvements in perception, description, and numerical estimation.
However, counterfactual reasoning degrades when excluded from training, highlighting a specialization-generalization trade-off. Several limitations remain, including BEV-only visual representation, potential domain bias from heterogeneous source datasets, and reliance on predefined rules. Future work will explore world-model-based generation to enrich trajectory reconstruction with appearance-level semantic details.
Journal Reference
Feng, H., et al. (2026). An interactive enhanced driving dataset for autonomous driving. Scientific Data. DOI:10.1038/s41597-026-07929-2. https://www.nature.com/articles/s41597-026-07929-2.
Disclaimer: The views expressed here are those of the author expressed in their private capacity and do not necessarily represent the views of AZoM.com Limited T/A AZoNetwork the owner and operator of this website. This disclaimer forms part of the Terms and conditions of use of this website.