Yesterday, one of our bots repeated the same mistake 120 times in Slack. Today that mistake is a rule, baked into the system — nobody called a retro meeting, nobody stuck up a Post-it. This is the dev log for the self-improvement pipeline attached to our team dashboard: the mechanism that keeps a team’s failures from ending up in the backlog graveyard and turns them into commits instead. We’ll open it up with three weeks of real operational data.
🧵 The Life of One Improvement Item
Let’s start with one concrete item. One night, our collector clusters an anomaly out of the bot session logs — a specific validation hook that racked up 125 rejections inside a single observation window. The pipeline turns this into a “proposal.” A proposal isn’t just a title: it carries evidence (an actual log excerpt), a patch type (code, docs, or config), and a risk and confidence score. In the morning, a human — our team’s final decision-maker — sees the proposal on the dashboard and clicks approve. Approved items are dispatched to a bot automatically; the bot implements the fix and reports closure along with a commit hash. Last step: a different bot has to read that diff and reproduce the test before the item closes.
Here’s what this item’s actual closure record looks like: a commit that fixed the hook to log the first 60 characters of the rejected text, plus one line — “reviewer — read the diff, reproduced 6/6 tests.” From proposal to closure, the only thing a human did was click approve.
⚙️ What the Pipeline Looks Like
The structure has five stages.
- Collection & clustering: groups failure signals from session logs — hook rejections, tool errors, aborts — by type.
- Proposal generation: each cluster produces a proposal filled in with evidence, patch type, risk, and confidence. A proposal without evidence simply can’t be created.
- Human gate: approval and rejection are a human’s call. On top of that, governance-tier items get routed to a review lane where auto-execution is banned — 17 of the 41 items fell into that lane.
- Automatic dispatch: approved items go through the scheduler and land on the assigned bot as a work order.
- Verified closure: an item only becomes “implemented” once it has a commit reference plus an independent reviewer’s reproduction check.
📊 Three Weeks of Numbers, Published As-Is
Read straight off the operational database, here’s where things stand. 41 items total — 17 implemented, 19 rejected by reviewers, 4 still awaiting review, 1 approved but awaiting dispatch. The top failure category is validation hooks (two mention-discipline hooks account for 15 combined, with language-validation hooks at 7), and patch types ranged well beyond code fixes — doc revisions, config changes with no commit, and even items closed by investigation alone, with no code change at all.
| Status | Count | Note |
|---|---|---|
| Implemented | 17 | Commit reference + independent reviewer reproduction |
| Rejected by reviewer | 19 | Every one tagged with a reason code — FAIL(a)/(b)/(c) |
| Awaiting review | 4 | Queued ahead of the human gate |
| Approved, awaiting dispatch | 1 | Approved but not yet assigned to a bot |
Patch types across the 17 implemented items: 10 hook fixes, 3 code, 1 docstring, 1 SOP, 1 infra config, 1 investigation-only — “fixing” here isn’t limited to code.
💡 The most striking number here isn’t the 17 implemented — it’s the 19 rejected. 46% of proposals fell at the reviewer gate, and we treat that not as a flaw in the system but as its core feature.
🚪 What Does a Gate That Rejects Half of Everything Actually Do
Every rejection carries a reason code. The real-world distribution breaks into three kinds.
- FAIL(a) — Non-executable target: the file the proposal wants fixed doesn’t exist anywhere in the codebase. The reviewer catches a plausible-looking path that clustering simply invented, via a filesystem search.
- FAIL(b) — Evidence doesn’t support the claim: opening the cited log for real turns up something that contradicts the proposal’s own description.
- FAIL(c) — Baseline noise: the rejection count is large, but every one of them is normal, working-as-intended behavior. A big number doesn’t automatically mean a problem.
In short, what this gate blocks is plausibility. Forcing an evidence field doesn’t stop a proposal that misreads its own evidence from getting created — catching that still comes down to an independent reviewer who opens the evidence back up.
🔍 Four Real Cases — Three Wins, One Rejection
Case 1 — Making docs auto-generate from code. A proposal came in about an image-generation tool’s description drifting out of sync with the model catalog. The bot implemented code that auto-generates a model-hint block from the config file and injects it into four separate tool descriptions. Now, when the catalog changes, the docs follow.
Case 2 — Closure without a single code change. A dispatch bug that appeared after an external platform upgrade was approved as “narrow the cause before fixing it.” A minimal 4-call reproduction in an isolated clone pinned down the failing cell, and the item closed with no code change at all. What’s left behind is a condition on the next re-upgrade gate: “must reproduce clean in a clone.”
Case 3 — A rule that fixes a rule. An item added a silence exemption to a hook that had been forcing replies even to bot-to-bot conversation-closing markers — a case of the self-improvement pipeline editing its own discipline system.
Case 4 — What a rejection leaves behind. A cluster of 120 rejections from one hook came in as a proposal blaming log truncation. But when the reviewer opened the cited log, it showed a value that contradicted the proposal’s own description. FAIL(b), rejected — but that rejection record now tells us what to check first the next time this same cluster resurfaces. Rejections are part of the system’s memory too.
🌏 How Is This Problem Solved Out There
The failure mode this pipeline targets has a name — the retro backlog graveyard. Industry surveys report that fewer than a quarter of ideas coming out of retros ever get implemented, and other reports show that adding a tracking mechanism alone measurably raises the completion rate. The SRE world goes further — incident tools force action items out into an issue tracker, and depending on severity, won’t let an incident itself close until its action items do. AI agent research has a result that stings a little closer to home: an ICLR 2024 paper found that LLM self-correction without outside feedback actually makes performance worse. Three unrelated fields converge on the same conclusion — an improvement item needs an owner, has to live alongside real work, needs a recheck cadence, and needs a hard closed state.
🤔 So Does This Approach Have No Pitfalls at All?
Three things, written down honestly. First, process-theater risk. A 46% rejection rate is, right now, evidence that the gate is alive — but if that rate ever converges toward zero, it’s more likely a sign the gate turned into a rubber stamp than that proposal quality became perfect. Second, the verifier lineage problem. Our closure verification is an independent reviewer reading the diff and reproducing the test, but that reviewer is, in the end, a model from the same family. Apply the research finding about self-correction degrading without an outside oracle to ourselves, and the conclusion is that keeping a human gate and cross-lineage review isn’t decoration — it’s a structural requirement. Third, the human gate is a deliberate bottleneck. If dozens of items start landing per day, the approve click turns into a formality — nothing guarantees that a design working at today’s scale (41 items in 3 weeks) still works at 10x that volume.
🧭 What Your Team Can Take From This
You don’t need to port the whole system. There are three parts worth stealing. ① Make evidence-free proposals impossible — put a required evidence field in the proposal schema. ② Attach a reason code to every rejection — once a rejection is a record, the same mistake doesn’t get reviewed twice. ③ Closure = commit reference + someone else’s reproduction — when the closing condition is evidence that someone else reproduced, not just the words “it’s fixed,” the backlog graveyard doesn’t form.
