LLMs Enhance Vision-and-Language Navigation

In a recent paper posted to the arXiv* server, researchers explored using language as a perceptual representation for vision-and-language navigation (VLN).

LLMs Enhance Vision-and-Language Navigation
Study: LLMs Enhance Vision-and-Language Navigation. Image Credit: BOY ANTHONY/Shutterstock.com

The team introduced a novel approach called language-based navigation (LangNav), which converts visual observations into natural language descriptions and fine-tunes a pre-trained language model to select actions based on instructions and text descriptions. Additionally, this approach demonstrated the capability to leverage large language models (LLMs) for generating synthetic data and transferring knowledge from simulated to real environments.


VLN is a challenging task that requires an agent to perceive and navigate a three-dimensional (3D) environment based on natural language instructions. A popular benchmark for VLN is the room-to-room (R2R) dataset, which consists of realistic navigation instructions in the Matterport3D environment.

The standard approach for VLN involves using pre-trained vision models to extract continuous visual features from panoramic images of the agent’s view and then combining them with language features from the instructions using a joint vision-language module. However, this method faces challenges in low-data regimes, where only a few annotated trajectories are available, as it requires learning a complex vision-language model from scratch. Additionally, it is not suitable for sim-to-real transfer, where the agent needs to adapt to different environments with varying visual characteristics.

About the Research

In this study, the authors proposed an alternative approach for vision-and-language navigation (VLN) that uses language as the perceptual representation space. They employed pre-existing vision models to capture textual descriptions of the egocentric panoramic view, such as image captions and detected objects of the agent. These descriptions were then input into a language model and fine-tuned to predict the next action based on instructions and previous actions.

The researchers argued that using language to represent the agent’s perceptual field offered several advantages. First, it enabled the use of large language models (LLMs) to generate synthetic trajectories from a few seed trajectories, thereby enhancing training data and improving the performance of smaller language models. Second, it facilitated sim-to-real transfer by employing language as a domain-invariant representation that abstracted low-level perceptual details in favor of high-level semantic concepts.

The method was tested on the R2R dataset through two case studies: synthetic data generation and sim-to-real transfer. The authors utilized the large language model Meta AI (LLaMA) as the core model for LangNav. LangNav’s performance was compared against several baselines, including the vision-based model recommendation Bert (RecBert) and the contemporary approach navigation generative pre-text transformer (NavGPT), which also employed language as a perceptual representation.

Research Findings

The outcomes revealed that in the synthetic data generation case study, LangNav was able to generate realistic and diverse trajectories from a prompted large language model GPT version 4 (GPT-4) using only 10 real trajectories from a single scene. Fine-tuning LLaMA on a mixture of synthetic and real trajectories resulted in better performance compared to RecBert, which was trained on the same real trajectories, highlighting the effectiveness of data augmentation with language models.

In the sim-to-real transfer case study, LangNav successfully transferred a policy trained in a simulated environment action learning from realistic environments and directives (ALFRED) to the real-world R2R environment. LangNav showed improved sim-to-real transfer performance compared to RecBert, which tended to overfit the simulated environment and performed poorly in the real environment. Additionally, LangNav demonstrated the capability for zero-shot transfer without any R2R data, a feat that RecBert was unable to achieve.

The authors showcased language’s potential for VLN, especially in low-data scenarios, suggesting it could serve as a potent tool leveraging LLMs for data generation and knowledge transfer. They further proposed language as a natural, intuitive interface for human-agent communication and collaboration, opening new avenues for applying language models to embodied tasks requiring perception and action.


In summary, the novel methodology proved effective for VLN and showed promise across diverse domains such as robotics, virtual reality, and augmented reality, where efficient and adaptable navigation was essential. For example, in assistive robotics, a language-based navigation system could enhance robots’ ability to comprehend and execute natural language instructions from users, thereby improving navigation and task performance in real-world scenarios.

Moving forward, the researchers acknowledged several limitations and challenges associated with their approach, including reliance on vision-to-text system quality, the complexity of accurately translating detailed visual information into language, and LangNav’s performance gap with state-of-the-art vision-based methods in data-rich environments. They proposed avenues for future research, including exploring advanced vision-to-text systems, integrating visual and language features for navigation tasks, and extending LangNav’s application to other embodied tasks.

Journal Reference

Pan, B., et, al. LangNav: Language as a Perceptual Representation for Navigation. arXiv, 2024, 2310, 7889. https://doi.org/10.48550/arXiv.2310.07889, https://arxiv.org/abs/2310.07889.

Disclaimer: The views expressed here are those of the author expressed in their private capacity and do not necessarily represent the views of AZoM.com Limited T/A AZoNetwork the owner and operator of this website. This disclaimer forms part of the Terms and conditions of use of this website.

Muhammad Osama

Written by

Muhammad Osama

Muhammad Osama is a full-time data analytics consultant and freelance technical writer based in Delhi, India. He specializes in transforming complex technical concepts into accessible content. He has a Bachelor of Technology in Mechanical Engineering with specialization in AI & Robotics from Galgotias University, India, and he has extensive experience in technical content writing, data science and analytics, and artificial intelligence.


Please use one of the following formats to cite this article in your essay, paper or report:

  • APA

    Osama, Muhammad. (2024, June 19). LLMs Enhance Vision-and-Language Navigation. AZoRobotics. Retrieved on July 21, 2024 from https://www.azorobotics.com/News.aspx?newsID=14996.

  • MLA

    Osama, Muhammad. "LLMs Enhance Vision-and-Language Navigation". AZoRobotics. 21 July 2024. <https://www.azorobotics.com/News.aspx?newsID=14996>.

  • Chicago

    Osama, Muhammad. "LLMs Enhance Vision-and-Language Navigation". AZoRobotics. https://www.azorobotics.com/News.aspx?newsID=14996. (accessed July 21, 2024).

  • Harvard

    Osama, Muhammad. 2024. LLMs Enhance Vision-and-Language Navigation. AZoRobotics, viewed 21 July 2024, https://www.azorobotics.com/News.aspx?newsID=14996.

Tell Us What You Think

Do you have a review, update or anything you would like to add to this news story?

Leave your feedback
Your comment type

While we only use edited and approved content for Azthena answers, it may on occasions provide incorrect responses. Please confirm any data provided with the related suppliers or authors. We do not provide medical advice, if you search for medical information you must always consult a medical professional before acting on any information provided.

Your questions, but not your email details will be shared with OpenAI and retained for 30 days in accordance with their privacy principles.

Please do not ask questions that use sensitive or confidential information.

Read the full Terms & Conditions.