TL;DR: AI doesn’t know your security rules. It knows only what is written down, and follows only what is enforced. Below: four layers that help (documents, skills, workflow, audit), my own failures that shaped them, and a checklist for Monday at the end.
Treat AI like a new contributor
AI is a tool. Whether it attacks or defends is up to us. And we are the ones responsible for the security of the code, even when a model writes that code for us.
A leak I started myself
At work we have a Slack bot. It's called Janusz, and I build it mostly on my own, partly after hours, partly between other tasks. It used to be a simple bot for kudos and reminders. Today it has an LLM inside and can answer a question like "what happened on this channel last week?"
In late 2023 I made a small change: instead of Slack user IDs, the bot showed people's full names. Nobody wants to read U087U91LE4Y gave kudos to U0CD5AK8Z6W. An ID tells you nothing, a name tells you everything. Good change, people were happy.
Two years later, in October 2025, I added a kudos export, also with full names. Also a good change.
Then an LLM moved into the bot, and the same rule flipped completely.
In June 2026 I changed the speaker labels in the conversations the bot sends to the model: Slack IDs instead of names. From that moment the rule was clear: personal data doesn't go to the model or to the traces unless it has to.
A week later I found the tool the model uses to read channel history. It was still giving the model full names, and with them, the names ended up in the traces. I fixed it, and the fix already pointed to the new rule.
In September the same thing turned up in the kudos. In the commit message I wrote: "an admin trace pulled today showed exactly that."
Two days later, another place: data from the Slack API was carrying email addresses, phone numbers and Skype handles to the model. Today those fields are cut out before anything reaches the model.
I didn't write bad code. I wrote good code for a different reader. The code didn't change. The one reading it did.
And the second thing, which hurts more: the rule was written down. Before the later cases turned up, I already knew personal data shouldn't go to the model. I had it written down. And that wasn't enough.
That's what this post is about.
AI is a contributor without context
For years I've been on the Apache Struts PMC, and I help with OGNL. Open source projects like Struts have always worked with people they don't know. Someone I've never met sends a patch to code that runs in production at thousands of companies. I can't assume that person "knows how we do things here". That's why open source projects had to write down the knowledge that companies usually keep in people's heads: what we don't do, why, and what has already been reported and isn't a bug.
Companies handled this differently. Security rules lived in senior engineers' heads, in code review, in "we don't do it like that here". It worked, because there were few new people and they learned from the older ones.
AI changes that. It's a contributor without context, and in many teams today it writes more code than anyone else. It wasn't at the post-incident retro. It doesn't know why this parameter is validated twice. It knows only what you show it.
The best way to see this is a story that has nothing to do with AI.
The OGNL series: learning the same lesson, again and again
Struts is a Java web framework that has been developed for over twenty years. Its best-known class of vulnerabilities is tied to OGNL, an expression language the framework uses to connect parameters from an HTTP request to objects in your code. That's convenient, until a value from the request gets treated as an expression to run.
This isn't the story of one vulnerability. It's a decade of the same lesson.
Starting with the first security bulletins in 2008–2010, every fix added more forbidden classes, packages and parameter name patterns to a blocklist. And every time, someone found a way around it. In 2014 three bulletins came out in a row, all about the same thing: a request parameter could reach the class loader.
In 2017 came S2-045. The Content-Type header of a file upload request ended up in an error message, and that error message was evaluated as an OGNL expression. So an attacker could run their own code on the server just by sending a crafted header. This is the vulnerability behind the Equifax data breach. Not because there was no fix. There was one. Equifax hadn't deployed it.
Between 2018 and 2020 more bulletins followed. Same class of bug, different door. Among them, tag attributes that were evaluated twice.
You can't finish a blocklist
In the end we understood that a blocklist can never be finished. The model has to be turned around: instead of listing what is forbidden, require an explicit declaration of what is allowed. And do it in two ways at once.
Enforced in code:
- A length limit for OGNL expressions: 256 characters by default since 2022.
- An allowlist of classes instead of a blocklist: since version 6.4, on by default since 7.0.
- A request parameter reaches an action only through an explicitly marked setter: since version 6.4, required by default since 7.0. In 2026 the same rule was extended to JSON and REST request bodies.
Written down:
- "Only evaluate OGNL generated by the framework, never user input." This is in the security guide and in the instructions file for AI in the repository.
In code it looks like this:
The framework doesn't guess what is safe. The developer, or the AI, has to say it explicitly, and no declaration means "no".
Each of these rules is knowledge a contributor had only if they had lived through the previous bulletin. AI hasn't lived through any of them. It knows only what is written down, and it follows only what is enforced.
Hence my thesis: treat AI like a new open source contributor. It doesn't know your security rules, so they have to be written down, and ideally enforced.
And a second, less obvious point: today, security documents are read mostly by AI, not by people. Write them for a machine reader.
How I laid it out
It's easy to say "write it down and enforce it". It's harder to answer: what exactly do you write down, where do you put it, and how do you make sure the model doesn't skip it? There is no single safeguard that handles everything. Each one lets something through. So I think of it as several sieves stacked one under another: what falls through the first one has a chance to get caught by the next.
I have four of them:
- Documents: what the AI needs to know. The rules, the trust boundaries, the list of things that look like bugs but aren't.
- Skills: how the AI should act. A step-by-step procedure that uses those documents instead of making up its own knowledge.
- Workflow: so the AI can't skip a step. Because a written procedure only works when someone runs it.
- Audit: to catch what got through the first three. Especially in old code nobody looks at anymore.
I'll show the first two layers on Struts, because that's where they are most mature. The other two on Janusz, because that's where I painfully found out the first two aren't enough. Along the way there will be plenty of my own failures, and that's not an accident. Each of them improved one of the layers.
Layer 1: documents written for AI
In Struts we have three documents that form a chain: AGENTS.md points to SECURITY.md, and that one points to THREAT_MODEL.md.
SECURITY.md
SECURITY.md has been in the repository since 2019, but it only became the source of truth in May 2026. The most important part is the "Before Reporting" list: things that look like a vulnerability but aren't, each with an explanation why. For a reporter, it saves time. For a model reviewing the code for security, it's the only chance not to raise the same false alarm for the tenth time.
There are two more things in it that I like. The first is the rule "a fix is a disclosure": a public commit with a fix tells an attacker where to look, so it doesn't go out before it has gone through the private process. The second is a simple control question: does this report describe hardening, or a vulnerability? That one question settles most arguments during triage.
THREAT_MODEL.md
The threat model was created in 2026. As the PMC, we asked the ASF security team to prepare the first version. Then we reviewed it and took ownership of it.
Every statement in it has a tag that says where it comes from: documented (it follows from the documentation), maintainer (someone from the project confirmed it), or inferred (worked out from the code, not confirmed). A separate section collects open questions for the PMC. The reader, human or model, knows how far to trust each sentence.
Inside are the trust boundaries, and then two lists I consider the most valuable part: what the framework guarantees and what it doesn't. Plus a list of known non-findings and instructions on what to do with each type of report.
AGENTS.md
AGENTS.md is the shortest, and that's its strength.
A model that enters a repository doesn't read it like a human. It doesn't browse the project website, doesn't ask on the mailing list, doesn't remember a conversation from a year ago. It looks for files that look like instructions for it. AGENTS.md is that signpost: it doesn't repeat the rules, it says where the truth is and adds a few things the model must not do.
The most important sentence in that file is: "Never submit a pull request that fixes a suspected vulnerability."
To a human who knows the project, that's obvious. To a model, it's the opposite. A model has been taught that a helpful contributor finds a bug and sends a fix right away. It's fast, eager and means well. And that's exactly why you have to tell it plainly that in security, the best intentions, sent in public, mean disclosing a vulnerability before the fix is released. A new human contributor will usually ask first. A model will do what it thinks is helpful.
Documents for a machine
One of the reasons this chain was built says a lot about our times. One of the companies building AI models offered ASF projects a security review with its model, on one condition: the project had to prepare a threat model. The chain AGENTS.md → SECURITY.md → THREAT_MODEL.md had to be something the scanner could find.
The requirement came from the AI side. Not from an auditor, not from a reporter, not from a user of the framework. From the model that was going to read the code.
The conclusion is practical: if these documents are read mostly by AI, then using AI to write them makes sense. Just in the right order: AI prepares a first draft, people review it, and the project takes ownership of the document.
A failure that pushes you forward
After publishing it, we tested the threat model against the five most recent bulletins of the project. The question was simple: if these reports came in today, what would triage based on the document do with them? The answer, in the words of the commit message: "Three of the five would have been closed as non-findings." In other words, three out of five real vulnerabilities would have been dismissed as non-bugs.
The security document itself was wrong. We fixed it, and that's the whole lesson of this layer. Written rules are necessary, but they are a hypothesis, not a guarantee. You have to test them against real cases.
Layer 2: skills, and a subagent only runs them
A document says what is true. It doesn't say how to proceed step by step. For that, in Struts we use skills: written procedures the model loads when a task calls for them.
Triaging reports
The skill for triaging security reports was built test-first. First I ran the model without the skill, and, as I wrote at the time, "a baseline run produced contradictory, unverified claims". Only then did I write the procedure. At its centre is one rule, called the Iron Rule in the skill: no claim without pointing to a file and a line that were read in this session. Not "I remember that this class…", but "I read it here".
Security bulletins
The second skill is for writing bulletins. Its key idea is a "disclosure budget". Details from old bulletins were once used to build exploits, so a bulletin says as much as a user needs to assess the risk and upgrade, and nothing more. The second rule looks trivial until a model breaks it: never make up a CVE number. A CVE number comes from the process, not from the model.
My agents that knew better
From this layer I have one lesson I recommend to anyone building AI tooling in a team: skills work better than subagents. A subagent should run a defined skill, not carry its own knowledge about security.
I learned it from my own mistake. In 2025 I added security agents to the repository, each with its own prompt. In August 2026 I deleted them. They duplicated the skill and ignored both SECURITY.md and THREAT_MODEL.md. The commit that removed them says they would "produce security reports… without ever mentioning SECURITY.md or private disclosure." Knowledge baked into an agent drifts away from the document. A skill reads the document.
The easiest way to explain this is how I have it set up day to day. The skill is the engine, and SECURITY.md is the fuel that runs it.
A skill can live globally, in my agent's configuration directory, and work in every project I touch. It knows nothing about a specific project. It knows how to run a triage or a review. What counts as a vulnerability in a given project, and what doesn't, it reads from the SECURITY.md and the threat model that sits in the repository, next to the code. Same with security review: the rule "run the built-in security review before opening a PR" is in my global instructions file, but it says to first read the project's SECURITY.md, THREAT_MODEL.md and AGENTS.md. One engine, a different fuel in each repository.
My 2025 agents had it the other way round. Each one carried its own fuel in the tank: its own version of what is safe. When the project's documents changed, the agents knew nothing about it.
The second slip came with one of the recent reports. During triage, my agent prepared a draft reply to a reporter, although nobody asked it to. In the draft it promised we would request a CVE. The draft reached the reporter. Nothing terrible happened, but it's a good example of how a process improves: I added a gate for drafts and ran the subagents several times to check that it holds.
This layer has a built-in limit, though. A skill does what it's asked to do. Nothing guarantees anyone will ask.
Trade-offs: no global redaction of personal data, on purpose
Before I get to enforcement, I have to admit something. A security frame isn't only prohibitions. It's also decisions I'm not entirely comfortable with.
Back to Janusz. After the leaks from the start of this post, the simplest reaction was: let's cut out all personal data before anything goes to the model. One rule, one filter, peace of mind. It sounds responsible. It looks good on a slide.
Except then Janusz stops working.
A bot you ask "what did people discuss yesterday on the project channel?" has to read what people wrote. And people write about people. "Ania will fix it tomorrow", "ask Tomek", "when Kasia is back from holiday". Names are the main personal data in this system, and they don't sit in a single field you can cut out.
Two things are worth separating here. When someone tags a colleague on Slack with @, the message contains a user ID. An ID like that is easy to find and replace, because it has a fixed format. But when someone writes "ask Tomek", "Tomek" is plain text. No marker, no format, no API field. It's a word in a sentence, like any other. To cut it out, you'd have to guess which words in a message are names. And in Polish, a name changes its ending depending on how it's used in a sentence: Tomek, Tomka and Tomkowi are all the same name.
Cut them all out, and the bot will tell you about a meeting nobody attended and where nobody said anything.
So in September 2026 I decided differently and wrote it down in the project's SECURITY.md. We have a data processing agreement (DPA) and zero data retention (ZDR) with the model provider. The bot can't answer a question about a conversation without seeing its content. So the content goes to the model, names included.
In exchange, I minimise wherever I can:
- email, phone and so on are cut out before the data reaches the model; the bot doesn't need them for anything;
- users appear in traces and in the search index as opaque Slack IDs, not as names;
- data about someone's wellbeing never goes into the bot's memory;
- traces and conversation state are deleted after 30 days.
The most important part of this decision is one sentence at its end: "Revisit if the ZDR or DPA status changes."
That sentence is what separates a trade-off from an excuse. An excuse says "it is what it is". A trade-off says "it is like this because these conditions hold today, and if they stop holding, we're back at the table". A year from now someone (me, or a model reviewing this code) will read SECURITY.md and see not only the decision, but also the moment it stops applying.
Without that, the next reader sees a missing redaction and does one of two things. Either they "fix" it and break the product, or they copy the pattern somewhere the trade-off doesn't hold.
A trade-off without a revisit condition is an excuse that forgot where it came from.
Layer 3: a workflow the AI can't skip
The first two layers tell the AI what to do. The third one is supposed to make sure it can't skip it. I'll say it up front: I don't have a ready recipe here. I'm describing the problem and the possible solutions, because none of them works a hundred percent. And that's exactly the lesson of this layer.
Where things are today
In Janusz, the rule "run the security review before opening a PR" exists, as a sentence in the agent's instructions file. At the same time, the command that opens a PR is on the list of commands the agent can run without my approval. So the rule is written down, but nothing enforces it. Exactly the same problem I started with.
What Janusz does have is a scanner: the repository is connected to our company's Aikido instance.
Struts is no better here. It has no gate for AI either. It has allowlists of commands and scanners in CI (CodeQL, OWASP, Sonar).
The simplest answer and its holes
The simplest answer is two hooks in the agent tool. The first fires when the agent runs the security review and writes a marker: a plain file in the .git directory with the ID of the current commit. The second fires before the agent opens a PR and checks whether that file exists for this commit. No marker, blocked.
Sounds good. When I started analysing it, the list of things such a hook doesn't cover grew fast:
- The marker means "started", not "finished". The hook fires when the skill loads, not when the review ends.
- The agent can write the marker itself. It's just a file, and the agent has a terminal. One command is enough to create it without any review, and the hook can't tell that file from a real one. I accept this on purpose: this gate protects against a forgetful agent, not one that deliberately goes around it.
- A push to an already open PR skips the gate. It goes through git push, which the hook doesn't see, and that's exactly where most of the code lands after the first review.
- It works only locally and only in one tool. It won't stop a human, a direct API call or the web UI. And it proves the review happened, not that the findings were fixed.
On top of that come technical details: worktrees, and a review started by hand by a human instead of by the agent. Each one is another small hole in the same boat.
Every control needs its own threat model
In my opinion, this is the most important observation in the whole post. The list above is exactly the same thing as the "what we guarantee" and "what we don't guarantee" sections in the Struts threat model. Just applied to a tool instead of a framework.
A hook that "enforces review" without such a list gives a false sense of security. With the list, it's an honest tool: I know what it can do and what it can't.
Possible solutions, from the weakest to the strongest guarantee
A local hook. Cheap, catches forgetting, full of holes as above. Worth having, not worth calling a gate.
An orchestrator. Instead of asking the agent to remember the order of steps, you write the steps as code, and the agent doesn't decide what comes next. One example is Orca, an experimental open source project from VirtusLab under the Apache 2.0 license. VirtusLab also builds Visdom, an enterprise-grade orchestrator. The same pattern can be built with other tools.
And here, too, there's a trap worth showing, because it applies to every tool of this kind. The default reviewer selection looks like this:
In plain words: a cheap model picks which reviewers take part in each round, and it may decide the security reviewer isn't needed. On top of that, open review comments go into the PR description instead of blocking it. The workflow is deterministic, but inside it a model still decides whether a step makes sense. The fix is ReviewerSelector.allEveryRound (all reviewers in every round) and an explicit fail(...) when open comments remain.
A required check in CI on every push. It covers every contributor, human and AI, and every path, including a push to an open PR or a change made in the web UI. It's the only place code has to pass through, no matter who wrote it or how it got there.
The lesson from this layer has two parts. A deterministic workflow isn't enough if a model inside it still decides whether a step makes sense. And a client-side gate isn't enough if code can arrive another way.
Layer 4: a detective audit
All the previous layers look at changes. At new code, a new diff, a new PR. But the leak in Janusz's kudos sat in code written before the LLM moved in, when names were still a feature.
A review of the LLM changes wouldn't have caught it, because the code that turned IDs into names wasn't in the diff. It was older, proven code that did exactly what it was supposed to do. When the LLM moved in, nobody touched those lines. The world around them changed: at the end of the pipe, instead of a person reading a message on Slack, there was now a model and a tracing system that records everything.
This kind of security debt doesn't shout. It doesn't break tests, doesn't fail the build, doesn't show up on a dashboard. It sits quietly in code that works, and waits until someone happens to look in the right place.
For me, that accident was an MCP server for analysing traces.
To explain where it came from, I need to separate the two models that work on Janusz. The first one sits inside the bot. It's its brain: it reads messages, calls tools, answers people. Every step it takes is saved as a trace in a database. The second is the model I write Janusz's code with. It has no access to that database. It sees only the repository.
When the bot behaved oddly, I wanted to know why: which tool it called, what it got back, why it answered the way it did. So I built an MCP server through which the coding model can read traces from the database and analyse them. A tool for debugging the bot's behaviour, nothing more. I didn't think of it as a tool for finding personal data leaks. Why would I? The rule about personal data was written down.
And it was during one of those analyses that I saw names where there should have been IDs.
Observability built for debugging turned out to be a security tool. A nice surprise. But an accident isn't a control. This time I looked in the right place. Next time I may not.
So I have an item on my list that is still waiting to be built: a regular audit of what ends up in the traces, for example an automatic scan for personal data. Not instead of review, but next to it. Reviewing diffs catches new mistakes. An audit catches old ones that became mistakes later, in code nobody has touched in a long time because "it works".
The loop
We update the Struts threat model for three reasons: a report comes in, someone reviews it out of habit, new bulletins appear. In a company the equivalents are obvious: an incident, a retro, a new vulnerability in a dependency. I say "vulnerability", not "CVE", on purpose: not every vulnerability gets a CVE, especially in closed-source code.
Every failure in this post improved one of the layers:
- the threat model that would have closed three out of five real vulnerabilities was rebuilt;
- my agents with their own knowledge made way for a skill;
- the draft that promised a CVE gave us a gate for drafts;
- The leak of emails and phone numbers gave us contact fields cut out before the model sees them.
Security is a process, not a project with an end date. I once heard a sentence that captures it well: "all those kinds of docs are always draft". It was about a project's security documents: SECURITY.md, the threat model, the instructions for agents. They are always drafts. Not because someone didn't finish them, but because the world they describe keeps changing: a new report, a new feature, a new model inside the bot.
Let's go back to OGNL for a moment. Struts learned the same lesson for a decade, until it replaced a blocklist with a rule that is written down and enforced. The documents and skills I described are today where that blocklist was at the beginning. The next bulletin, the next leak or the next agent mistake will add another entry. "Always draft" isn't an excuse. It's a description of the same loop.
A checklist for Monday
- Write the rules for a machine reader. Create a SECURITY.md with a list of known non-findings, and add an AGENTS.md that points to it. Test it against your latest incidents: what would it have let through?
- Replace subagents that carry their own security knowledge with a skill based on those documents. A subagent can run the skill, but it shouldn't have its own version of the truth.
- Give every control its own threat model. Write down what it guarantees and what it doesn't. A local hook catches forgetting; the gate is a required check in CI.
- Write trade-offs down explicitly, with a revisit condition. "We don't do X because of Y; we come back to it when Z changes."
- Plan an audit of existing code and data. Check what actually reaches the model, the logs and the traces. Reviewing diffs doesn't see debt.
AI is a new contributor that writes most of the code and knows none of your history. It won't learn it in code review, one comment at a time. Write it down. Then make it impossible to skip.
Reviewed by: Bartłomiej Turos and Daniel Olszowski
