Who’s to blame if AI fails at work? Wharton study shows a split
The question of accountability is becoming more pressing as companies push increasingly powerful AI agents into the hands of employees.
Companies are giving artificial intelligence systems more autonomy to boost productivity and efficiency. But as bots take on more responsibility, executives, managers, and employees remain unclear on who should be held accountable when something goes wrong.
A new report published Tuesday by the Wharton School of the University of Pennsylvania found little agreement among managers. When asked who should be held responsible when a bot “makes a consequential error,” respondents were split roughly evenly between the employee deploying the AI agent, the employee’s manager, and the company’s risk team, with about a quarter choosing each. Roughly 10% blamed the vendor, while a similar share said they weren’t sure who should be held accountable.
That question of accountability is becoming even more pressing as companies including OpenAI, Alphabet Inc.’s Google, and Meta Platforms Inc. push increasingly powerful AI agents into the hands of employees. The bots are equipped with their own corporate identities and are able to independently work toward goals, in some cases over the course of several days.
While concerns of rogue AI agents have dominated headlines in recent months, the more immediate risk to companies may be far more ordinary and mundane. A minor prompt error or unclear instructions can quickly snowball into a larger crisis when agents have access to live systems such as email accounts or customer databases, said Leeron Walter, vice president at employee-monitoring software company Teramind.
Companies are already facing those kinds of mistakes. Walter pointed to one national U.S. retailer where an employee built an AI system to automate customer emails. A prompt error caused the bot to send incorrect refund and cancellation notifications to more than 18,000 customers in less than six hours, triggering thousands of customer-service inquiries. Walter said customer demand for employee-monitoring systems has ticked up in the wake of embarrassing mistakes like these.
At another U.S.-based consumer electronics company, Walter said a marketing employee used AI to automatically generate and publish product prices. A prompt error caused a flagship device to be listed at an 85% discount for three hours. More than 700 orders were placed before the mistake was caught, and the resulting customer-service fallout took weeks to resolve.
Mishaps like those, which can result in large losses and long-term reputational damage, are raising an uncomfortable question for workers and their managers: If employers are being encouraged to experiment with AI, who takes the blame when something goes wrong?
“Organizations are learning in real time,” said Jeremy Korst, a partner at the consulting firm GBK Collective and co-author of the Wharton study. “When I ask this question of a senior executive, the most frequent answer I get is that they hadn’t thought about it yet.”
