NavLead is an AI-driven chatbot that gives navigation assistance in a 3D enviroment.

The chatbot was trained on real-world navigation assistance dialogs to give natural responses.

The 3D enviroments were reconstructed form real-world houses.

No downloads are neeeded; both the chatbot and 3D enviroments are on this site.

Demo instructions:

Choose a target household item

Navigate the house (using WASD keys)

Ask the chatbot for assistance

How does it work?

Each question is processed roughly as follows:

  • The 3D enviroment is displayed on your browser. Then you ask a question to the chatbot.
  • This pages sends the question along with the user's location in the 3D enviroment to an API server requesting the answer.
  • The API server validates and authentificates the request.
  • The server runs the NavLeadNet deep learning model. It outputs a textual answer.
  • Finally the answer is sent all the way back to the chatbot.
navlead process diagram

3D simulated enviroment

The enviroment used in the demo is a reconstruction of Frank Lloyd Wright's final design: the circular sun house.

Circular Sun House wikimedia photo

It was constructed using Blender and exported to Unity to run on your browser.

Circular Sun House CG model in Blender

The chatbot's UI was then added in Unity as well.

NavLeadNet

NavLeadNet is the navigation guidance deep learning model used here.

This model is a sequence-to-sequence architecture that takes the traveler's question and location (as input sequence) and, then, returns a textual answer (as output sequence).

navlead process diagram

It was trained using the CVDN (Cooperative Vision-and-Dialog Navigation) dataset. CVDN is a dataset of human-human dialogs situated in simulated, photorealistic home environments. The 3D enviroment in the demo was reconstructed based on one of those enviroments.

How does it work?

©   Frederick Roman 2022 &