I built a system that publishes video automatically: the first real production failure

The context: what Traffic OS is

Traffic OS is my private, internal application for managing YouTube video publishing: it generates content candidates, runs them through review, applies a policy before publishing, and only then uploads the video. Every step with real consequences — publishing something, not just proposing it — goes through an explicit human approval. It isn’t a public product or anything I sell: it’s infrastructure I operate myself.

Before there was a real public canary, every candidate had only ever been uploaded in private-validation mode: a real upload to YouTube, but not visible, meant to confirm the pipeline works end to end without exposing anything yet. The first real public canary was different: the first time a candidate actually went out, visible and measurable.

What went wrong

The internal table that records “which video is the real one for this candidate” (the one both publishing and metrics collection use) identified each publication by candidate + platform only. There was no third dimension for the purpose of that publication — nothing distinguished “this is a private validation” from “this is the public canary.”

That wasn’t a problem as long as each candidate had, at most, one upload. But the moment a single candidate had two real uploads — the earlier private validation and the new public canary — the two competed for the same key. The public canary’s publishing path was also built to bypass the shared orchestrator that normally writes that table, so it left no new record at all. The old private-validation row stayed in place as if it were the “live” publication for that candidate.

The result, had I run metrics collection as-is: the system would have read the private-validation video’s statistics and attributed them to the public canary’s experiment. Correct data, wrong video.

I caught it because, when I wired metrics collection up against a real canary for the first time, the numbers didn't match what I expected for that video. Nobody had exercised that exact path with a real publication before, so the gap had been there since it was built, just never triggered.

How I fixed it

First, a manual data repair that same day: correcting the row with the real evidence available, without inventing anything. That fixed the symptom, not the cause.

The cause was fixed a few days later with a structural change: a migration that adds the publication’s purpose to the identity key, so two videos from the same candidate can never be confused again.

// Before
publish_key = (candidate_id, platform)

// After
publish_key = (candidate_id, platform, purpose)
// purpose ∈ { private_validation, public_canary, ... }

With the expanded key, each video gets its own row, and metrics collection resolves exactly the video that matches each purpose — never the other one. I also added a regression test whose own name describes the bug: metrics resolved for one purpose can never leak into a different purpose. If I ever touch that logic again, that test fails before the bug reaches production a second time.

The metrics themselves, to be clear, are read exclusively via the YouTube Data API’s videos.list — public statistics for the video itself (views, likes, comments), not the YouTube Analytics API. That didn’t change with this fix; what changed was which video they got attributed to.

What I learned

• The part of the system nobody has really exercised yet is where broken invariants hide. The code existed and looked fine, but it had never been tested against a real public publication until that point.
• An incomplete identifier is worse than no identifier. With candidate + platform, the system looked like it worked — until it stopped being true that the key identified exactly one thing.
• Bypassing the shared path for a “special” case (the public canary, one-shot) is exactly where the bookkeeping everything else takes for granted gets lost. The fast path didn’t reuse the point where the record got written.
• A test that names the bug, not just the symptom, is worth more than a generic one. The regression test literally states which invariant it protects, so if someone breaks it in the future, they’ll immediately understand why it existed.

How I’d build it now

If I were starting Traffic OS from scratch, every publication’s identity would carry purpose from day one — not something bolted on when the second real case shows up, but part of the original data model. And any “special” or “one-shot” path would go through the same recording point as everything else, instead of having its own shortcut. Shortcuts are exactly where the invariant the rest of the system assumes holds gets lost.

Why I’m writing this up

Publishing content in an automated way isn’t “calling an API.” It’s generation, review, policy, and publication as separate layers, each with its own way to fail — and the layer you trust the most (the one keeping track of what actually got published) is exactly where a silent failure does the most damage, because it doesn’t break anything visible: it just produces wrong data with all the confidence of correct data. This is the first real incident I hit taking Traffic OS to production, and that’s why I’m writing it up with real names and real purpose, without inflating or softening what happened.

Related: a different kind of refusal, from the content side of the same system, in my AI system passed 2 of 3 tests to publish alone.