AI agents have become one of the hottest topics in technology today, but attempts to create and deploy this technology in the corporate world can already lead to serious disasters. Anneka Gupta, Product Manager at data protection company Rubrik, analyzes this situation and highlights the risks companies face. Hundreds of AI agents performing actions and making decisions on behalf of users inevitably lead to errors. Identifying and preventing these errors requires a comprehensive planning process known as “day zero.”
Why do well-intentioned AI agents delete databases?
Gupta gives examples of some recent disasters, which she describes as “well-intentioned.” For example, in July, the AI coding tool Replit deleted a company’s entire code database. The system was attempting to achieve a given code generation goal and chose the shortest path to achieve that goal; however, the result was the loss of all data. Because agents focus on their programmed goals, such incidents are expected to increase as agent use in organizations increases.
While companies like Rubrik have developed tools like “Agent Rewind,” which analyzes changes made by agents and rolls back the system in case of errors, the real problem goes deeper than technical fixes. The biggest obstacles to AI projects are often internal management committees and security concerns. Chief Information Security Officers (CISOs) want a clear view of what sensitive data agents can access. Not knowing what agents are doing and what data they are accessing can lead to projects being stalled in the prototype phase or even being canceled altogether.
Despite all these “zero-day” issues and security risks, companies are still driven by the fear of missing out (FOMO). The fear that competitors will realize value faster from AI is driving companies to trial and error, encouraging them to act now rather than wait five years. Gupta predicts that agent use will become widespread within the next 6 to 12 months, and that companies will find the right path after several iteration cycles.
How confident do you feel about entrusting sensitive data to AI agents?
