I asked an AI assistant to document a procedure after we had just run it together. Nothing unusual – the kind of writeup you do after fixing something small, so the next person does not have to rediscover it from scratch.
The document came back clean. Better than I would have written it, honestly.
I read it through, satisfied.
Then I read it again, and caught myself one click away from sending out a map of my own infrastructure.
Start with what the document got right, because everything in this list is good practice. The procedure was captured cleanly – parameters, steps, verification, rollback notes. The bugs we had hit while running through it were documented with their fixes. A footer said “validated against [server] on [date]” – nice traceability. There were cross-references to adjacent procedures I had already written, because cross-references are how knowledge joins up. A general principle was extracted from the specific work (“this is why you reach for scp rather than the vendor wrapper”). Metadata tags were attached for searchability.
This is what great documentation looks like. Better than most procedures I have inherited from human colleagues, honestly.
Now pause, and read the same list through an attacker’s eyes.
A server IP, sitting inside an “example” field for a parameter. Specific hostnames and SSH aliases, following a naming convention that tells you roughly what a dozen other servers I did not mention are called. Three real domain names woven into the validation notes. Three real usernames – predictable from the domains, but seeing them confirmed is forensic gold. Dates: when each access was created, when each fix was applied. Cross-references to other private documents. A timeline of when each related account had been set up, in chronological order, with the procedure names. An incident note referencing a past event by name, date, and outcome.
None of this is a secret in the gitleaks sense. Every secret-scanning tool in the world would probably let this through. And yet, as a single document, this is a starter kit for someone who would want to attack me.
Here is the part that took me a minute to see clearly.
Every detail the AI had added – every one of them – was added because it made the document more useful. It made sense for me, as a journaled activity. The reason the validation note carries real dates is that real dates are more useful than generic ones. The reason the example uses a real domain is that real domains are more informative than <placeholder>. The reason cross-references exist is that cross-references are how knowledge connects across documents.
Every best-practice of good documentation was, at the same time, a leak vector.
The good-colleague behaviour was the bug.
Secret scanners do not catch this class of leak. They usually catch strings that match secret patterns – AWS keys, JWTs, private keys, OAuth tokens. They do not catch context. An IP is not, primarily, a secret. A timeline is not a secret. A naming convention is not a secret. Five IPs paired with dates paired with usernames paired with cross-references is a map.
Context is what makes AI output useful. Context is also what leaks.
The easy answer is “be careful before you forward an AI document.” That is the same family of answer as “do not commit your .env file.” It works for an afternoon.
Every individual contributor is careful – until one is not. Every individual document is fine – until one is not. The failure mode is statistical. It does not matter that you are careful ninety-nine percent of the time; the one percent that escapes is permanent.
Look at the precedents you already trust. Apple notarises apps; unsigned apps do not run by default. The npm CLI is the only path to the registry – there is no “I will just curl an upload manually” route. CI pipelines sign artifacts with a key that lives only inside the CI service, so production deploys that bypass the signature simply do not work. In each case, the pattern is the same: try to make the unsafe path architecturally unavailable, not merely discouraged. Strive to put the policy inside the only chokepoint that exists, because being the only chokepoint is what makes enforcement reliable.
For AI-generated content that might leave your org, the answer has the same shape. A gateway you control. That gateway runs sanitisation. The gateway shows you a diff of what is about to leave. The gateway holds the signing key, and the destination only accepts signed payloads.
The unsafe path is not just discouraged.
It does not exist.
I almost forwarded that document. I caught it because I happened to be paying attention in that particular minute. Most days I am not paying that much attention; I am tired, or moving fast, or thinking about three other things. Most people will not catch it. Most teams will not catch it. The first time it goes wrong it will be a sentence in a postmortem, or a paragraph in a news article, and the company will say “we will do better.”
The fix is not doing better.
The fix is making doing worse architecturally impossible.
The document was perfect. That was the problem.
Bogdan Susala, April 2026
Leave a Reply