Imagine a first responder training for an emergency by speaking with an AI assistant. The assistant needs to understand the situation and reply clearly. It also needs to protect sensitive information. How can we tell whether it is suitable for that work?

In our research, we developed EFR–GVA EvalKit, a set of checks for designing and evaluating AI voice assistants for emergency response. It brings practical questions together: Does the assistant respond well? Is it easy to use? Where is the data processed, and does the system meet relevant requirements?

We used the EvalKit in a virtual reality 3D training scenario where Chemical Biological Radiological Nuclear and Explosives responders practised sorting patients by urgency. We compared AI assistants running locally on consumer devices with a remote AI assistant, then studied the experience of 19 first responders. The findings showed that local assistants can meet responders’ expectations in this setting.

The project offers a practical way to assess these tools before bringing them into emergency training. The patient sorting exercise was its first test; the same checks can be adapted to other devices and tasks.

The results were presented in the EICS26 conference at the Computer Engineering and Informatics Department (CEID) in the campus of the University of Patras, Greece, highlighting the need to observe the unethical practices of big-tech companies and their involvement in human-rights violations:

Read the paper in Proceedings of the ACM on Human-Computer Interaction.