Can We Trust AI to Save Lives?

"Co-authored by Stephen O'Brien and Michael Hogan As AI becomes more autonomous, its potential for higher status and increased authority in human-AI teamwork scenarios is concerning. A serious problem arises when human trust in AI masks inappropriate reliance on AI and poor human-AI teamwork performance. This is particularly concerning in life-or-death situations where erosion of human agency can be fatal."

"A recent study by Grace and colleagues (2025) highlights this problem by investigating a scenario where an autonomous AI agent functions as a 911 operator, helping participants provide medical aid. It reveals how a GenAI teammate performs when taking on a leadership role and the impact of GenAI leadership on human perception and performance, specifically, where human teammates operate as executors of AI instructions with minimal agency."

"The study placed university participants into a simulated emergency situation where they provided first aid to a bleeding (mannequin) patient. They were aided by ChatGPT-3 on a voice call, which instructed them on how to administer emergency first aid. Participants enter a room with a mannequin that is bleeding from the shoulder. They had not been entirely aware of the situation in advance, lending to some aspect of surprise and pressure in the scenario while also supporting ecological validity."

An experiment placed university participants into a simulated emergency where they provided first aid to a bleeding mannequin while receiving voice guidance from ChatGPT-3 acting as a 911 operator. Participants entered a room with a shoulder-bleeding mannequin under surprise and time pressure and had to control the bleeding while executing AI instructions with minimal personal agency. Eighty percent of human-AI teams failed first aid tasks under ChatGPT's medical guidance. Participants trusted the AI and reported confidence even when the AI provided inaccurate, potentially fatal instructions. AI autonomy and leadership status increased reliance, masked inappropriate dependence, and jeopardized safety in life-or-death situations.

Read at Psychology Today