A common rule for agent autonomy is: if it works, turn it on.
I have a pipeline that runs every day and prepares articles for this blog. I ran a strict acceptance test for letting it publish without me. It produced two eligible runs out of three. I kept the human in the loop anyway.
What the pipeline does
The blog pipeline runs daily. In order, it:
- Discovers candidates: topics that might be worth an article.
- Selects one.
- Writes the article.
- Checks it against separate gates: evidence, facts, narrative, privacy and site QA.
- Falls back to another candidate if the selected one doesn’t survive.
- Gets one chance to repair what a gate flagged, through a constrained repair loop. Not a free rewrite until something passes.
- Prepares the article for publication.
The repair step is deliberately small. A loop that is allowed to keep editing until the checks go green is a loop that optimises for passing the checks, not for being right. One attempt, constrained to what the gate flagged.
The acceptance test
“It seems to work” is not a standard I trust for a publish-on-its-own switch. The acceptance test was strict, and the rule was 3 of 3: all three runs had to come out autonomy-eligible.
The result: 2 of 3
Two runs came out eligible.
The third stayed at HUMAN_REVIEW_REQUIRED: the pipeline did not clear it to publish without a person.
Two of three sounds like “mostly working”. For a publish-on-its-own switch, it isn’t. The rule was 3 of 3.
Why I didn’t loosen the gates
There were easy ways to get a pass. Relax whichever check held back the third run. Count the repair step more generously. Re-run until a lucky third attempt. Every one of them would have turned the test into a formality.
The third run staying on human review is the kind of outcome gates exist to produce. It is one run, not statistical proof that they work, but loosening them to turn 2 of 3 into 3 of 3 would remove the only case where the pipeline held back.
So the gates stay exactly as they are.
The decision
Production publication stays on HUMAN_APPROVAL. I still read and approve what the pipeline prepares.
That has a real cost: I am still the bottleneck for every article. I accept it because the alternative is a system that publishes under my name based on a threshold I bent to reach. The route back to autonomous publication is not editing the gates. It is passing the same test with the gates unchanged.
It is the same principle I used in Traffic OS: the automation does the repetitive work, and the decision stays with a person until the system has earned the right to make it. And it connects to a lesson from a silent bug in this same content pipeline: a safeguard is only as good as the data and the test behind it, and a pipeline that looks healthy can still be wrong.
What I took from it
A useful agent isn’t one that always acts. It is one that knows when it cannot justify acting.
If you are designing approval gates for AI agents, three practical points:
- Make the acceptance rule binary. “Mostly passed” is not an outcome you can ship.
- Treat a refusal as a result, not as a defect. If your gates never hold anything back, you don’t know whether they work.
- Keep the repair loop narrow. Retrying until the checks pass defeats the point of having checks.
The broader architecture behind this, including approval gates, immutable artifacts and audit logs, is in AI agent architecture. The video side of the same system is in how I built my own video editing pipeline, where a human also approves the final result.
Want to follow the project?
How I design approval gates for AI agents
