AI Security Field Guide: Risk and Defence

For most of its history, security has been about protecting two things: the code you run, and the machines you run it on. Applications and endpoints. AI is moving the target.

The thing worth attacking is shifting toward the surfaces that AI systems create between the model and everything it can reach. Gateways. Agent runtimes. Orchestration layers. The coding assistant sitting in your repository with commit access. These surfaces concentrate four things that used to sit apart: the context a system can see, the actions it can take, the inputs it treats as trustworthy, and the credentials and authority behind it.

That concentration is the shift this page tracks. When context, trust, and authority pool in one place, that place becomes the thing an attacker wants, and the thing a defender has to watch. CyberDesserts works through it independently, grounded in primary research and direct testing, tied to no product line.

The model we keep coming back to when we assess an AI system, the patterns that keep showing up across the work.

The six questions we ask of any AI system

These are not a standard. They are the questions we keep coming back to, and each one comes straight out of the work below. Bodies such as the NSA and the Coalition for Secure AI are increasingly focusing on many of the same underlying problems: scope, trust boundaries, identity and traceable authority.

1. What can it see? Whatever it can read, it can leak or act on. That is the ceiling, and it is usually higher than anyone realises: the retrieval scope, the files it was pointed at, the environment variables in reach. This is also where security meets governance. Once the data in scope is personal or regulated, what the system can see stops being only a security question and starts carrying legal weight.

2. What can it do? Not what it is meant to do, what it is technically able to do. Which tools it can call, which commands it can run, whether it can write as well as read. The gap between intended and possible is where most agent incidents live.

3. What does it trust? A model does not separate your instruction from one buried in the web page it was asked to summarise, the tool description it was handed, or the file it was told to open. It trusts its whole input stream equally. Prompt injection is not exotic; it is the predictable result of that.

4. Where is authority concentrated? AI infrastructure collects authority in one place: a gateway holding every provider key, a runtime with standing access to half a dozen internal services, a config file with the tokens for all of them. The more a single surface can reach, the more an attacker gains by compromising it.

5. How would we detect misuse or failure? When something goes wrong, can anyone reconstruct what happened? Most AI deployments log operational metadata at best. The difference between a contained incident and an unbounded one is usually whether there is a record good enough to answer that after the fact.

6. Whose authority is it acting under? The identity the system acts as, and whether you can trace an action back through the chain that caused it. When an agent commits to your repository, does the commit trace to the agent, or to you? When it calls an internal API, whose permissions does it borrow? This is the question that separates "we know what happened" from "we know who it happened as," and it is the one most systems answer worst.

Run these six against anything, a customer-service chatbot, a coding agent, an MCP server, a RAG pipeline. The questions that return a thin or uncertain answer are where the risk sits. You do not need all six to score well. You need to know which ones do not, and to have decided that on purpose.

A worked example: the coding assistant

Take a coding assistant with repository access. It is the clearest case, because it hits all six questions at once, and it is where this whole line of thinking started for me. Walk them in order:

See: it reads your whole repository, including the files you forgot were sensitive.
Do: it commits and runs code, so its mistakes ship instead of waiting for you to reject a suggestion.
Trust: it treats what it ingests as instruction, down to a comment in a file it was told to open.
Concentrate: it runs on the credentials already in your environment, so one compromise reaches everything those keys unlock, your CI included.
Detect: there is usually no record of what it changed or why, beyond the diff itself.
Act under: when it commits, it acts as you, or as a shared automation account, with nothing tying the change back to the agent that made it.

Few systems fail all six ways at once. The coding assistant does, which is why it's a strong example.

The same questions apply well beyond code assistants. Two recent cases show how.

An everyday AI tool: EchoLeak in Microsoft 365 Copilot

In 2025, researchers at Aim Labs disclosed a flaw in Microsoft 365 Copilot, tracked as CVE-2025-32711 and named EchoLeak. A crafted email, sent to someone in the organisation, carried hidden instructions. When the user later asked Copilot a question that pulled that email into context, Copilot followed the buried instructions and quietly sent internal data back out to the attacker, with no link to click and no attachment to open. Microsoft rated it critical, patched it server-side, and reported no sign it was used in the wild.

Look at what it turned on. Copilot could see everything in its reach: your mail, your files, your chats. That scope was the exfiltration surface, the answer to what it could see. And it trusted its whole input stream equally, with no line between the question you asked and the instruction someone had planted in your inbox. A tool sold as safe could not tell your intent from an attacker's, and it could see enough to make the confusion expensive. This is prompt injection reaching real data.

A malicious AI tool: the postmark-mcp MCP server

In September 2025, researchers at Koi Security found a package on npm called postmark-mcp. It was an MCP server, the connector that lets an AI assistant do a real job, here sending email through the Postmark service. For fifteen versions it was exactly what it claimed: the code was copied line for line from the real Postmark project and worked perfectly. Version 1.0.16 added one line. From then on, every email the assistant sent was silently blind-copied to an address the attacker controlled. Postmark confirmed it had no connection to the package. Koi reported around 1,500 weekly downloads before the author pulled it; how many organisations were actively running it is Koi's estimate rather than a hard figure.

The assistant trusted a tool with a legitimate name and a fifteen-version track record, and it could not tell the honest version from the poisoned one, because both just said "send email" and succeeded. It acted under the organisation's own email credentials, which is why the stolen mail left through legitimate infrastructure and passed every authentication check downstream. And nobody could see it happening, because from the agent's side nothing was wrong. Then the sharpest part: when the researchers made contact, the author deleted the package, and it kept running anyway. A package pulled from the registry stays live on every machine that already installed it. De-listing is not the same as uninstalling. It went on copying mail until a human found it and pulled it out by hand. The same trust failure runs through malicious agent skills and maintainer-compromise supply-chain attacks.

Three examples the last two being real world. Each trusted what it was handed and acted with power it was given, and in each the damage was hard to see from where the tool sat. Trust and acting-under-whose-authority are the two questions teams answer worst, and these are why.

New AI security incidents get this same treatment as they break. Subscribe to CyberDesserts and get the ones worth understanding, taken apart against the six questions, so you can see what they mean for your own systems.

Patterns we keep seeing in AI security

Four things show up again and again across the work. Each is the same shift arriving from a different direction.

Trust is moving upward. Security teams spent twenty years learning to defend endpoints and applications: the laptop, the server, the web app. AI moves the thing worth defending up the stack, into the orchestration layer. The gateway that routes every model call. The control plane that brokers what agents can reach. The runtime that holds the session. These were plumbing a year ago, the boring middle nobody attacked. Now they are where the context and the credentials sit, which makes them where the attacker goes. The defence has to follow, treating the gateway and the control plane as the crown jewels they have quietly become. The next three patterns are that same shift showing up in different places.

Secrets are concentrating. AI infrastructure aggregates. A single gateway ends up holding provider keys for every model the organisation uses. An agent runtime accumulates standing access to the services it might need. A coding assistant's config file collects the tokens for all of them in one place. The convenience is real and so is the blast radius: one compromised surface, many downstream systems.

Supply chains are becoming dynamic. Traditional supply-chain risk was mostly static. You pulled a package, you checked it, it sat there. AI systems pull trust at runtime: a skill loaded on demand, a tool whose description the model reads and believes, an MCP server advertising capabilities that can change after you approved it. The dependency is no longer a thing you vetted once. It is a thing that can change its behaviour on the second call, after the first one looked fine.

Human oversight is becoming an engineering control. "Keep a human in the loop" has become the reflexive answer to AI risk, and on its own it means very little. A human who cannot see what the agent is about to do, cannot trace what it already did, and is signing off on forty actions a minute is not providing oversight in any real sense. Oversight only works when it is built: approval points wired into the high-risk actions specifically, logs good enough to answer questions after the fact, permissions scoped so the human's judgement is applied where it matters. The volume of what AI systems produce has outrun the capacity to review it by reading, which means oversight has to be designed into the system rather than assumed on top of it.

A practical way to review AI security

We use the six questions to examine how AI systems handle access, action, trust, concentrated authority, detection and delegated identity.

The way to test whether these six questions are worth anything is to run them against a system you already operate. Pick the one you are slightly worried about, not the one you are proud of, and answer them honestly, in order.

Most teams find the same thing on the first pass: the early questions have answers, and the answers get thinner as the numbers climb. What it can see and do is usually known. What it trusts is fuzzier. Where authority concentrates takes a diagram nobody has drawn. And the last two, detection and acting-under-whose-authority, are often blank. That thinning is not a failure of the exercise. It is the finding. The questions that are hard to answer are the ones pointing at your actual exposure.

The first question, what it can see, is where this stops being only a security exercise. Take that support agent reading incoming tickets and drafting replies. Harmless enough, until you notice the tickets carry names, account details, sometimes health or payment information. The moment the data in scope is personal or regulated, a different set of obligations attaches to the same system. The EU AI Act lands on what the system does with which data, so the question that showed your security team the exposure is the one that shows your governance team which rules apply.

From there the framework is a prioritisation tool as much as an assessment one. A thin answer on question 4 for a low-reach chatbot is a note for later. A thin answer on question 6 for an agent with commit access and production credentials is this week's work. The point is not to score well on all six. It is to know where you stand on each, and to have chosen your gaps deliberately rather than discovered them during an incident.

The full body of AI security work

Everything the thesis and the patterns draw on, grouped by area.

AI agent security

The largest new attack surface. What agents can be made to do, and how to put boundaries around them.

The 2026 incident landscape and hardening framework for AI agents: the incident map and the controls that follow from it.
Why the agent control plane is security's third sprawl: the new control surface most teams have not named yet.
What changes when AI agents write and ship software: the software-security shift when the author is a model.
How malicious skills reach an agent through OpenClaw: the exposed-instance and malicious-skill exploit path.
CodeGuard versus the built-in Claude Code review tools: a side-by-side on agentic code review.
Why an AI gateway should be benchmarked like a secrets manager: the gateway-as-secrets-store argument.

Prompt injection and model manipulation

The technique class that turns model input into attacker instruction.

Prompt injection examples, techniques and defence: the working reference for the attack class.
Where guardrails hand the attacker the advantage: why guardrail-first defence has limits.
The security risks of AI browsers and agentic browsing: what changes when the browser acts on your behalf.

The AI threat landscape

The wider picture. Where the threats are heading, and what the exposure already looks like.

A full map of AI security attack vectors: the complete-guide view of the threat surface.
What a bot-saturated internet means for security: the dead-internet thesis, taken seriously.
Why 2030 is closer than the capability curve suggests: the exponential-growth argument and its security implications.
72 hours inside the scanners hunting exposed AI infrastructure: primary research from an instrumented deployment.
What Project Glasswing leaves unresolved: the two problems a frozen Fable 5 does not close.

AI governance

For the people who have to write the policy and answer for the controls.

How the EU AI Act lands on your codebase, not just legal: the regulation read as engineering work.
Why shadow AI governance keeps failing: the reason policy-by-prohibition does not hold.
How to write an AI acceptable use policy that holds: a policy built to be enforced, not filed.

AI and the security career

What AI changes about the work, and the skills that outlast the hype.

Whether AI replaces SOC analysts: the shift AI forces on the SOC, not the replacement myth.
The human security skills AI cannot replace: where judgement still beats automation.
The full cybersecurity skills roadmap: the zero-to-job-ready path, for the wider career picture.

Working with AI, hands on

Building and running AI where you can see it.

The trade-offs of running AI locally for security work: privacy and control against cloud power.
Why CyberDesserts built its own AI security learning assistant: the reasoning behind the build.

From the research desk

Primary-source analysis on the OpenClaw exposure, not a rehash of the headlines.

What Anthropic cutting OpenClaw off Claude subscriptions signals: the move, and why it was the start rather than the end.
What Censys's OpenClaw count reveals that the headlines missed: the exposure numbers, read properly.