Imagine sending a robot into a room to locate an injured person. Through a mixed reality headset, you can see the robot’s camera feed and ask an AI assistant: “How far away is that person?” But can you trust the answer?
In our research, we connected a conversational AI assistant to a robot equipped with a depth camera. This camera measures distance, while object detection identifies people and objects. The assistant uses these measurements to explain what the robot sees, both through spoken answers and labels in the headset.
We tested the prototype with 16 participants in a simulated toxic-smoke emergency. Participants estimated the distances to a person and a window using video alone, AI support based on images, or AI support supplied with the robot’s depth measurements. Providing measured distances helped participants make more accurate estimates than AI support based on images alone.
Figure 1: Mixed reality (MR) emergency response simulation of toxic smoke inside an office space. Depth measurements computed from the RGB-D camera mounted on the robot (A) are shown in the labels of detected objects in MR view (B). Corresponding verbal feedback, e.g., “The person is approximately 2.93m away.”, is shown on the chat window (C) and spoken through headphones. The robot also transmits its pose to enable 3D model tracking and visualization (D), e.g., behind obstacles. These audiovisual descriptions of the real-world environment are informed by ground truth depth.
The key idea is simple: the robot measures, and the AI communicates. This makes its answers more reliable than asking it to guess distances from a picture. The prototype still needs faster responses and testing in more realistic situations, but it offers a promising direction for robot-assisted emergency training and remote inspection.
