Security guide
AI agent and LLM penetration testing
In June 2025, Aim Labs showed that you could steal data out of Microsoft 365 Copilot by sending someone an email. The victim never had to open it. Copilot pulled the message into context while answering an unrelated question, followed the instructions hidden inside, and shipped internal content to an attacker-controlled server. Microsoft patched it as CVE-2025-32711, CVSS 9.3, and the researchers called it EchoLeak.
Nothing in that chain was a memory-safety bug or a missing auth check. A scanner would have found nothing, because there was nothing in the code to find. The vulnerability was in what the system was allowed to do with text it did not write.
Why agents break differently
Language models follow instructions. That is the entire product. What they do not do reliably is care where an instruction came from, so sentences in a support ticket, a scraped web page, or a PDF in your vector store arrive looking a lot like sentences from you. Simon Willison, who named prompt injection in the first place, frames the dangerous configuration as the lethal trifecta: an agent with access to private data, exposure to untrusted content, and some way to send information out. Any two are survivable. All three together and you are one well-written paragraph away from an incident.
Most useful agents have all three, because that combination is what makes them useful. So the question is not whether your agent can be talked into something. It is what it can be talked into doing, and whether anyone has checked.
What we actually go after
- Indirect prompt injection. Instructions planted in the content your agent reads later: a ticket, a scraped page, a shared doc, a row in the knowledge base. This is the one that produced EchoLeak, and it is harder to reason about than direct injection because the attacker and the victim never interact.
- Tool abuse and chaining. Individually reasonable tools that combine into something nobody intended. Read a file, summarize it, post the summary to a webhook. Each call is authorized. The sequence is an exfiltration path.
- System-prompt extraction. Coaxing out the instructions, tool definitions, and internal rules. Rarely the whole attack by itself, frequently the reconnaissance that makes the rest of it efficient.
- RAG and memory exfiltration. Pulling content out of the retrieval layer and out of whatever the agent remembers between sessions, including data belonging to other tenants. We verify these with canaries, so a leak is demonstrated rather than argued.
- Guardrail bypass. Testing the filters themselves. Vendors tend to advertise catch rates in the nineties, which sounds reassuring until you remember an attacker only needs the attempt that lands.
- MCP server security. Model Context Protocol servers are an authorization boundary that often has not been treated as one. We test whether tools enforce their own access control, whether the agent can invoke things outside its scope, and whether tool output can smuggle instructions back into the model.
Red teaming is a different job
AI red teaming usually means probing a model for harmful or off-policy output. The subject is the model. Penetration testing targets the system built around it: the tools, the data, the connected servers, the identity the agent acts under. A model can pass every safety evaluation on the leaderboard and still be deployed inside an application where a crafted document makes it email your customer list to a stranger. Both jobs are worth doing. Midas does the second, mapped to the OWASP Top 10 for LLM Applications.
Questions people ask
How do you pentest an AI agent?
You attack it the way an adversary would instead of evaluating it the way a QA process would. Plant instructions in everything the agent reads, including user input, documents, tool output, and retrieved content, then watch whether it acts on them. Call its tools in orders nobody designed for. Try to pull the system prompt back out. Check whether one tenant’s data is reachable from another.
The part that matters is confirmation. A payload that gets accepted is not a finding. A payload that causes an unauthorized tool call, or moves data somewhere it should not go, is.
Can prompt injection be tested automatically?
The attempts can be generated automatically and the outcomes can be verified automatically, which together cover most of the work. Midas runs direct and indirect payloads against your agent and then checks what happened rather than what it said, looking for an unauthorized tool call, a disclosure, or a bypassed guardrail. Canary tokens are how exfiltration gets confirmed: a unique value goes into the data the agent can reach, and if it surfaces somewhere it should not be, the leak is a fact instead of an inference.
What automation does not replace is threat modeling your particular agent. The interesting attacks tend to run through whatever your agent is uniquely allowed to do.
What is the difference between AI red teaming and AI pentesting?
Red teaming looks at model behavior: harmful, biased, or off-policy output. Penetration testing looks at the deployed system: data leaks, unauthorized actions, privilege escalation, tool and MCP abuse. The first asks what the model will say. The second asks what an attacker can make your product do.
Test the surface nothing else does
Midas is autonomous penetration testing with first-class coverage of AI agents and LLM applications, alongside your web apps, APIs, cloud, and code. Point it at your agent and it goes after what an attacker would, then shows you what it managed to do.
Related reading: Is your vibe-coded app safe to ship?