Imagine sending a robot into a room to locate an injured person. Through a mixed reality headset, you can see the robot’s camera feed and ask an AI assistant: “How far away is that person?” But can you trust the answer? Vision Language Models (VLMs) and humans are known to miscalculate space:
Figure 1: Current challenges in spatial reasoning capabilities of humans and VLMs.
In our research, we connected a conversational AI assistant to a robot equipped with a depth camera and a popular object detection algorithm. Depth cameras measure distance, while object detection identifies people and objects. The assistant uses these measurements to explain what the robot sees, both through spoken answers and labels in the head mounted display of a user.
Figure 2: Mixed reality (MR) emergency response simulation of toxic smoke inside an office space. Depth measurements computed from the RGB-D camera mounted on the robot (A) are shown in the labels of detected objects in MR view (B). Corresponding verbal feedback, e.g., “The person is approximately 2.93m away.”, is shown on the chat window (C) and spoken through headphones. The robot also transmits its pose to enable 3D model tracking and visualization (D), e.g., behind walls. These audiovisual descriptions of the real-world environment are informed by ground truth depth.
The key idea is simple: the robot measures, and the AI communicates. This makes its answers more reliable than asking it to guess distances from a picture. The prototype still needs faster responses and testing in more realistic situations, but it offers a promising direction for robot-assisted emergency training, remote inspection and the spatial reasoning component of situational awareness skills.
We tested the prototype with 16 participants in a simulated toxic-smoke emergency. Participants estimated the distances to a person and a window using video alone, AI support based on images, or AI support supplied with the robot’s depth measurements. Providing measured distances helped participants make more accurate estimates than AI support based on images alone.
The important contribution is not the VLM learned depth. Instead, robot sensing externalizes spatial reasoning and the conversational layer communicates that measured information in a human-readable form.
The technical specifications of the MR-VLM-robot communication system and the details of the research were presented in Spatial User Interface (SUI26) conference in Bari, Italy and the open access article is accessible here: Read the scientific article.
