The next AI security incident may look like normal operation
Businesses are giving AI systems greater authority over everyday operations. They are being used in fraud controls, customer decision-making and applications that can retrieve information and act across connected systems. Deloitte’s 2026 State of AI in the Enterprise found that 74% of surveyed organisations expect to be using AI agents at least moderately by 2027, while only around one in five reported mature governance for agentic AI.
In 2025, researchers disclosed EchoLeak, a vulnerability in Microsoft 365 Copilot that allowed a specially crafted email to influence Copilot through indirect prompt injection and, under certain conditions, expose data the user was already authorised to access. The attack did not require the recipient to click a malicious link. Microsoft fixed the vulnerability and said it had no evidence of customers being affected in the wild. Microsoft has since discussed EchoLeak as an example of the security challenges created when AI systems interact with enterprise data and content.
The significance of EchoLeak lies in the way ordinary business information became part of the attack path. An email could be legitimate content for an employee while also carrying instructions capable of influencing the AI system processing it. Security teams therefore need to understand which sources can shape model behaviour, what information the system can reach and what can happen after it produces an output. Conventional controls around accounts, infrastructure and malicious code remain necessary, but they do not capture every way an AI-enabled workflow can be manipulated.
Serious harm can also occur without an attacker compromising the system. I saw this while assessing an AI-powered fraud detection system that flagged legitimate customer activity as potentially fraudulent. The technology was operational, yet an incorrect decision could lead to a rejected order, a cancelled transaction or unnecessary scrutiny of an account. Once that output entered a wider customer or operational workflow, the consequences could extend beyond the original model. The system did not need malware, stolen credentials or a compromised account to create a security and business problem.
How much authority an AI system has determines how far an error can travel. A recommendation that waits for human review presents a different exposure from an output that can alter records, approve a transaction or trigger another automated process. The Open Worldwide Application Security Project identifies “excessive agency” as a vulnerability where an AI system is given more functionality, permission or autonomy than it needs. Prompt injection, hallucinations or poor model performance can provide the initial trigger, while the system’s permissions determine the actions that follow. IBM’s 2026 study of 2,000 technology leaders gives some indication of that exposure. Respondents reported an average of 54 AI-agent incidents in the previous year, defined as unintended or potentially harmful events requiring human correction. Seventeen per cent were classed as high severity. Among those high-severity incidents, 37% involved data exposure or security breaches and 33% caused cascading system failures.
Inputs deserve the same scrutiny as permissions. Prompt injection can introduce malicious instructions through content the model is allowed to process, while poisoned or unreliable data can distort model behaviour before any downstream action occurs. NIST’s 2025 adversarial machine-learning taxonomy covers poisoning, evasion and privacy attacks across predictive AI, as well as misuse attacks affecting generative AI systems. Historical data can create problems without deliberate manipulation too. Incomplete records, outdated assumptions or unrepresentative patterns can produce unreliable decisions even when the model itself remains technically stable.
A security review of a consequential AI system therefore needs to follow the route from input to action. Teams need to know where information comes from, which sources the model treats as context, which systems it can access, what permissions it holds and which downstream processes depend on its outputs. A tool that drafts internal text carries a different level of exposure from an agent able to retrieve customer records, modify data or influence payment decisions. The controls should reflect those consequences.
Human approval is one safeguard, but it will not be appropriate for every use case. Some systems need restricted permissions. Others require independent validation, real-time monitoring or mandatory approval before a high-impact action proceeds. People affected by consequential decisions also need a practical route to challenge and correct an outcome before it moves deeper into a critical workflow. Without that mechanism, an incorrect decision can acquire authority simply because it has been produced and passed on automatically.
Identity controls, patching, access management and infrastructure security still protect the technology surrounding AI. Security teams now also need to understand what can influence the system, what the system can reach and what it is permitted to do when its output is wrong or manipulated.
Before an AI system receives access to sensitive information or influence over a critical process, those boundaries should be explicit. The next serious AI security event may arrive through a compromised system, a manipulated prompt or a model behaving exactly as designed. In each case, the scale of the damage will depend heavily on the authority the organisation gave the system before anything went wrong.
For more startup news, check out the other articles on the website, and subscribe to the magazine for free. Listen to The Cereal Entrepreneur podcast for more interviews with entrepreneurs and big-hitters in the startup ecosystem.
