A silent bug stalled my content pipeline for weeks: here's how I found it

Update, October 2026: this article describes the state of the pipeline on September 23, 2026. The last paragraph is stale; see the update at the end.

What I was trying to automate

I run a personal agent system I call Jarvis — private infrastructure I operate myself, not a product. One of its pieces automates part of my social-content pipeline: an agent I call Pepe proposes scripts for short-form video under three distinct “hats” (a technical series, growth content grounded in real analytics, and AI news), I record the raw footage, an editing agent I call Marcos edits it, and I confirm the final result from a dashboard before it publishes. Nothing publishes on its own: every step with real consequences goes through an explicit confirmation from me, the same principle I already applied in Traffic OS.

Each of Pepe’s three hats has one simple, deliberate rule: don’t propose a new script of that type if the previous one hasn’t been recorded yet. It’s a safeguard against piling up a backlog of scripts nobody will ever record. For that rule to work, Pepe needs to know, from real data, whether the previous script has actually been recorded.

The symptom: it just looked like there was no new content

During Pepe’s daily autonomous run, I started noticing the “growth” hat stopping cold: it would check the history, find its last script still marked “planned,” and propose nothing new. Day after day, the same stop. A couple of weeks later, the “AI news” hat — which until then had been generating a new proposal almost every day without issue — started stopping too, for the same reason. Eventually all three hats were stuck at once.

My first hypothesis was the obvious one: that Marcos had stopped receiving or processing videos, or that some cron feeding the analytics behind those proposals had gone down. And I wasn’t entirely off base — a different cron really had gone weeks without writing fresh data, a coincidence that at first seemed to confirm the wrong hypothesis. But once that cron started working again and the block stayed exactly the same, it was clear I was looking in the wrong place.

Why it was hard to catch

What’s insidious about this bug is that every component, looked at individually, was working exactly as designed:

• The crons ran without error.
• Pepe’s blocking rules were doing exactly what they were supposed to: refuse to propose new content without confirmation that the previous piece had been recorded.
• I really was recording and confirming videos normally the entire time.

The problem wasn’t that something was broken and failing visibly. It was that a piece of data the whole system trusted — “has the previous script been recorded yet?” — hadn’t been updated in weeks, and nobody had a reason to suspect it because nothing returned an error. A system can be operationally healthy and still have its state machine completely stuck.

The lifecycle: Marcos → content_calendar → Pepe

The link between “what Pepe proposes” and “what Marcos edits” runs through a table, content_calendar, with a status field that advances from planned to recorded (and later to published). When I upload raw footage linked to one of Pepe’s scripts, that editing job gets tied to its content_calendar row via a shared identifier. Marcos, when processing the job, reads that row to know which script and which editing notes correspond to that specific video.

That read link was built in an earlier change to the system, weeks before this bug showed up: Marcos already knew how to read the content_calendar row for a job. What never got built was the write-back half.

What was actually missing

When I confirmed a finished job from the dashboard — the point where I tell the system “this video is ready, accept it” — that endpoint marked the job as confirmed in its own table, but no code touched the linked content_calendar row. Its status stayed planned forever, even though the video was already recorded, edited, and confirmed by me.

So Pepe’s blocking rules were doing exactly what they were supposed to: reading whether the previous script had already been recorded. The problem is that read always came back “no,” even when the real answer had been “yes” for weeks.

// Before
PATCH /jobs/{id}/confirm
  → marks the job confirmed
  → content_calendar.status
    stays "planned"
    forever

// After
PATCH /jobs/{id}/confirm
  → marks the job confirmed
  → if a linked row exists
    and it's still "planned":
      status = "recorded"
    (never downgrades a row
    already "published"/"archived")

How I fixed it

The fix itself is small: the confirmation endpoint now also updates the linked content_calendar row, with two explicit guards — it only ever advances planned → recorded, it never rolls back a row that’s already published or archived, and it does nothing if the job has no linked row.

What wasn’t small was the regression suite: five cases verifying that confirming a job correctly advances the linked status, that a job with no link touches nothing, that an already-published row never gets downgraded, that the endpoint still requires the video to be downloaded before accepting confirmation, and that confirming the same job twice is idempotent and breaks nothing. I ran them against a disposable Postgres database, never against the real one — the same rule I use for any test that writes data.

How I repaired the historical state

The fix stops the bug from happening again, but it doesn’t correct the rows that had already gotten stuck while it was present. That took a one-time backfill, run directly against production inside a single transaction: an UPDATE that only touches content_calendar rows in planned status whose linked video job is genuinely finished and confirmed — computed with a JOIN against the real job data, not a hand-written list of IDs.

The result was six real rows that moved from planned to recorded. After the backfill I verified the full state of the table with a read-only query: the remaining entries in planned stayed there because they genuinely had no confirmed job behind them — real recording backlog, not a leftover from the bug.

What I learned about agent pipelines

The main lesson isn’t “watch out for database bugs” — it’s more specific than that: any read link between two parts of a pipeline needs its write half explicitly tested, not assumed. It’s easy to build the side that consumes a piece of data (Marcos reading the linked row) and assume that somewhere else in the system, someone took care of keeping it updated. Nobody had.

The second lesson is about the safeguards themselves: a blocking rule that does exactly what it was asked — don’t advance without confirmation — is only as good as the data it reads. If that data can go stale without anything noticing, the rule stops protecting and starts blocking for the wrong reasons, and it does so silently: there’s no error to catch, just a pipeline that stops advancing.

And the third is about diagnosis: the coincidence of an unrelated analytics cron being down at the same time nearly led me to “fix” the wrong cause. Before accepting the first explanation that fits the symptoms, it’s worth checking whether it actually predicts everything you’re observing — in this case, it still didn’t explain why the block continued exactly the same after that cron recovered.

How I’d avoid this kind of failure now

Two concrete changes I’m taking from this. First: whenever I build any one-directional link between two tables or two pipeline steps, I write the test for the write half in the same change as the read half, even if that write “will be added later” — that “later” is exactly the gap that sat here for weeks unnoticed. Second: for any automatic blocking rule that depends on state read from another table, I want an independent periodic check auditing whether that state ever actually moves, not just that the rule itself works — a permanent block on stale data and a correct block on real backlog look identical from the outside, and only an audit of the underlying data tells them apart.

There’s still one validation I can’t force from the code: the next time I confirm a real job linked to a planned entry, that will be the first true end-to-end case of the fix working in production with real data, not just the regression tests. Until then, this is verified by tests and by the backfill, but not yet by a real confirmation from Marcos.

Update, October 2026

The paragraph above is the state of things in September. It is kept as written.

Since then, the pipeline has taken a real long-form job, about 1.8 GB of raw video, to a finished 1080p video: the walkthrough is here. That is a real-data run of the pipeline as a whole. I have not verified the planned to recorded write-back on its own, so the specific validation I ended with is still open. And the pipeline still doesn’t publish alone: the blog side stays on human approval, on purpose.