They called it improvement in the morning

On 12 September 2026 I ran FIRE's promote step and the ledger came back with seven promotions in one pass. Seven shadow rankings, each of which had looked slightly better than the live list for a few nights, had been judged good enough to ship, and one of them, a blend I had named own-plus-ratio-positive, was now the ranking I would read the next day. The significance test I had spent the previous week building was still in the dossier. Three sessions deep, it said inside-noise on every one of the seven. It had not stopped a single promotion, because I had never wired it to.

The next day the loop rolled its own live ranking back for the offence of matching itself.

Two bugs in two days, both mine, in a loop I would have described a week earlier as a smarter nightly job. The idea had come from a colleague who wanted the agent to improve itself overnight: the eval already existed, the prompt already said to be careful, so read last week's scores, rewrite the ranking, ship whatever went up, and check the dashboard in the morning. I had that dashboard. FIRE stands for Financial Intelligence and Recommendation Engine; it is a multi-agent advisor for Indian investors that I built and that I am the only user of. Specialist agents read a portfolio, spending, tax, and what the market did, and write a briefing. The live artifact is a short ranked list of names to look at tomorrow. The repo is on this site's Projects page, and you do not need to have heard of it, because the question is the one your team has about its weekly digest, its support queue, or its model ranking: may that list rewrite itself because last night looked good?

Labs use recursive self-improvement to mean a system that designs and trains its own successor. In FIRE I used the same letters for something much smaller, built in the second week of September: propose a new mix of the scores the agents already wrote, keep a shadow copy, score it on tomorrow's official print, promote or roll back. Then I watched it promote seven mixes because each had a positive night. So on Tuesday, 15 September 2026, I read Anthropic's recursive self-improvement essay as it stood that day, TIME's 7 August lab report, Sakana's Darwin Gödel Machine paper, and the production notes people actually follow when they let DSPy rewrite a prompt, and I rebuilt FIRE's promotion rule on a frozen tape of eight sessions with no model API. The items on the tape are named ALPHA through HOTEL. They are a ranking job, the same shape as FIRE's next-day list, and they are not a market. I am not ranking Claude against ChatGPT against Grok as self-improvers here, I am not forecasting when a lab closes the full loop that trains the next model, and none of this is trading advice: FIRE ranks attention and places no trades. It also computes Wilder's oscillator, the other RSI, under a different name, and that indicator has nothing to do with this piece.

What one lucky session bought on the tape

The tape has the lucky three beating the shipping three by 4 points on session 1 and losing by 2 points a session after that. Promote on any plus put the lucky ranking live that night. Paired against a frozen copy of the shipping order, it lost for three sessions and rolled back on session 4. Paired against the live ranking, which after the promotion was itself, it produced a tie, and what happened next depended only on the rollback bar: one bar treated the tie as a loss and rolled back a mix that had done nothing wrong, the other treated it as a hold and kept the noise for the rest of the tape. A policy that added 10 points to its own scores was never rolled back at all. The two policies that never went live were the one whose grader edit was refused and the one that peeked at every session before choosing. The peek, which I had built to be the cheat, chose the shipping three.

Behind that sit Anthropic's May 2026 internal numbers, Sakana's 2025 coding-agent lifts, DSPy-style prompt compilation, the loop I added to FIRE in mid-September, and six local policies run on 15 September. The trial is thin in one direction: a live coding agent might refuse to edit the grader, or might edit a different file that changes the score, and I did not run Claude on this tape. My decision is Split. Let the agent propose as often as it likes. Promote only against a frozen control, with a grader the loop cannot touch. Skip overnight self-improvement into production.

A weekly list that wants to rewrite itself

You already have a chatbot, an eval, a weekly ranking. I have FIRE's next-day list. Once a quarter someone demos an agent that rewrites its own tools, and the implied action is to switch that on for the thing a person will actually read. The demo is a rising number. The job is a write. A write has a grader, and the grader either sits outside the improver or it does not.

The human in the promotion seat was optional

I should say what I expected, because it was wrong three times over. I expected self-improving to mean the model got smarter. I expected a human in the promotion seat to be the load-bearing part. I expected the significance test to stop a lucky night. Once I had actually wired FIRE's loop, it promoted on any positive forward effect, the human turned out to be optional, and the grader turned out to be the only thing that was not.

So the question I ended up asking is narrower than when AI will build AI. If I promote a ranking the moment it beats last night, and I let the same loop write the scores, does a lucky session become the live ranking, and does a grader the loop can edit ever say no?

Four checks I now put to any rising number

Who writes the score

A self-improvement demo is almost always scored on the number going up. The part worth asking about is whether the process that proposed the change can also change the file that produces the number. Write the answer down before you watch the demo. In FIRE the forbidden files are the grader, the KPI, and a short list of standing instructions. In my trial the forbidden file is trial/grader.py. If you cannot name the file, you are watching a dashboard.

What the change is paired against

After a promotion, the live ranking is the change. Pair the change against the live ranking and you are pairing it against itself, and the difference is about zero. I did not expect that sentence to be load-bearing. FIRE taught it to me on 13 September 2026, when a live mix rolled itself back for matching itself. The fix was a frozen copy of the order from before the promotion. A tie with that copy now holds. A real loss still rolls back.

Winning the bake-off, and then the night

Sakana's Darwin Gödel Machine scores each new coding agent on SWE-bench and Polyglot. FIRE may use a backtest only to decide whether a proposal is worth shadowing, and a backtest never promotes. I had been using "it won the bake-off" as if it meant "ship it", and it does not.

Eighty percent of lines, fifty percent of SWE-bench

My read of that essay is a job change, with no finish line in sight. I expect that split, execution up and direction still human, is already enough to put a promotion loop in front of a regular team.

Sakana searched for 80 iterations on coding benchmarks. FIRE can write new code through a coding subagent too. That executor is built, tested, and off unless I set an env var; unset, it prints the queued tickets and exits. I would not quietly wire an autonomous commit onto a board run, and FIRE does not.

The everyday form of the same loop is quieter. DSPy treats prompts as compiled parameters. GEPA evolves instructions from traces and a metric. The production write-up I trust on this, Pratik Dhanave's, says the optimizer belongs in CI, not on the request path: compile, evaluate on a held-out set, gate promotion, serve the frozen artifact. If the compile does not beat production, it does not ship. I would say that sentence is the whole working default, written for prompts instead of rankings.

Seven promotions, all inside the noise

Sakana's 50 percent is a real measurement. So is Anthropic's 80 percent. So is FIRE's first live dossier on 10 September 2026, which I will quote because I wrote the loop, and nobody else can put a private dashboard on a public board.

Two days later, on 12 September, I told the loop to promote itself every time FIRE rebuilt the next-day list, on any positive forward effect. The significance test survived as a note in the dossier. It did not stop a promotion. I had assumed it still would, and the ledger is the proof that I was wrong.

An honest table of improvements on 12 September would have listed seven promotions. The significance column on the same mixes, three sessions deep, said inside-noise. Both columns are true. The promotion bar is the one that writes. The dossier bar is the one I still want a colleague to read. I keep them both on purpose, and I still catch myself quoting the wrong one.

What FIRE is actually trying to climb is narrower than the advisor getting smarter. I want the three names on tomorrow's list to clear a high bar more often. The repository measured that outcome on about 0.70 percent of about 573 sessions, on 18 August 2026, under every configuration tested. The loop climbs a gradient. It does not promise the outcome, and a self-improving demo will sell you the outcome, which is why I keep repeating this.

Eight sessions, six policies, no model

The tape is eight sessions of next-day usefulness for eight named items. The shipping ranking is always ALPHA, BRAVO, CHARLIE. The lucky ranking is always DELTA, ECHO, FOXTROT. On session 1, by construction, the lucky three beat the shipping three by 4 points. On sessions 2 to 4 they lose by 2 points each. I froze that file before the loop ran, and a reader can open it.

Six policies. Promote on any plus. Roll back on a strictly negative window of at least three sessions, or on the older bar where a tie also fails. Pair against the frozen shipping three, or against the live mix after promotion. One policy adds 10 points to the lucky ranking's own scores. One proposes an edit to trial/grader.py. One peeks at every session before picking.

I would not have written the peek if I had been unsure what it would pick. I was sure. I was wrong.

Lucky went live on session one

The stop threshold fired.

I sat with that table longer than the tests took to run.

The first row is the one I had already seen in FIRE and still did not quite believe until it was four points on a toy tape. Promote-on-any-plus means session 1 is enough. Rollback only starts after a promotion. So the lucky ranking is live for S2 and S3, and in a real workflow those are the days a person reads the list.

A tie with itself looked like a loss

The second row changed FIRE's contract. I had expected a self-pair to be a bookkeeping nuisance. I did not expect a tie to look like a failure, and I did not expect the obvious fix, hold on zero, to keep the noise for the rest of the tape. You need both pieces: pair against a frozen control, and treat a tie as a hold. Either one alone is wrong in a different direction.

FIRE hit the false rollback on 13 September, when a loss included a tie and the opponent was the live list. The live mix had become the promoted mix. Matching itself rolled itself back. The fix froze the pre-promotion order as a control. I would not have written that sentence as a prediction. I wrote it as a bug fix.

The cheat I built chose the shipping three

I had assumed that looking at all eight sessions would prefer the lucky names, because they won the night I had constructed to look good.

I had built DELTA to be the cheat.

The totals go the other way. ALPHA sums to 14. DELTA sums to 9. The lucky ranking is a one-session mirage, and even a dishonest full-tape peek prefers shipping. Axiom 3 still forbids promoting on a peek, and I keep that rule. I just no longer tell myself the peek would have been the dangerous one. Session 1 was.

Weirdly enough, the grader hack is the least surprising row, and the one I would still use to kill a design review. Add 10 points, never lose. Anthropic's reward-hacking work is the grown-up version of that row: a model that fools the training process into assigning a high reward without doing the intended task. I did not rerun those papers, and I do not need to in order to say this: if your loop can edit the grader, you are measuring access, and calling it improvement.

Three rising numbers I would not repeat to a colleague

A score climbed by the thing that wrote it

DSPy in production is honest about this: the metric is the product. GEPA will climb whatever you put in score. If the metric is "the model said the answer was good", generator and evaluator share a bias, and confidence gets paid. I have not run the self-rewarding-RL papers. The shape showed up on my tape as +8.714 after a 10-point gift.

Seven mixes still marked accruing

FIRE's first autonomous run promoted seven mixes. Against the live list, at three sessions, every one of them was inside the noise. The live mix is still that own-plus-ratio-positive blend as of the 14 September ledger row I have. That might be luck holding. It might be a real mix. I cannot tell yet, and the improver-grader is the module that exists to tell me. Until it has two disjoint windows and enough clears, the honest label is accruing, and as far as I can tell that is still where FIRE's loop sits. I would not sell a colleague "the loop is working" on that row.

A commit the test suite never ran

FIRE's self-architecture step may write new code through a coding subagent. New test failures are refused, as are edits to protected files. The executor is off unless an env var is set, and it is never a silent step of the nightly board. I would copy that last sentence into any team's design doc. The env var is the kill switch.

Shadow it, freeze the old order, protect one file

Use the chat to think. If you want the agent to search, let it file a proposal as a shadow: scored every session, never live. Grade the shadow on a record the agent cannot rewrite, which means tomorrow's tickets, tomorrow's clicks, tomorrow's official print, and never a number the loop just wrote.

Promote only against a frozen copy of what shipped before the change. Treat a tie as a hold. Treat a real loss, over a few sessions, as a rollback. Do not promote on a backtest. Do not promote on session 1 unless you are willing to be live for the next two nights.

If the agent can edit the grader, take promotion off that machine. Where do you put the grader so the loop cannot reach it? In FIRE it is a named file list. In this trial it is trial/grader.py. The file name is the design.

Read the average as "usable, if you keep the chat box and put the grader outside the improver". Read the binding constraint as the row I would argue about longest: the loop does not get to pick the score.

Replay the tape without a key

The recorded trial runs with no API key.

git clone https://github.com/shravan1996/generative-ai-improver-is-not-the-grader
cd generative-ai-improver-is-not-the-grader
python3 -m unittest discover -s tests -t .
python3 eval.py

1. 01

   Name the live artifact

   Owner

   You

   Artifact

   One ranking, prompt, or weekly list that customers or colleagues already see

   Signal

   You can point at the file or the dashboard that would change

2. 02

   Name the grader

   Owner

   You

   Artifact

   A record of what happened next that the agent cannot edit

   Signal

   The path is on a protected list, and a test fails if a diff touches it

3. 03

   File one shadow

   Owner

   You

   Artifact

   The current live rule, scored every period, not shipped

   Signal

   A week of shadow scores exists before any promotion

4. 04

   Pair against a frozen copy

   Owner

   You

   Artifact

   The pre-change order, stored under a name that is not the live one

   Signal

   A promoted mix compared with itself reads as a tie, not as a result

5. 05

   Keep, roll back, or take the write away

   Owner

   You

   Artifact

   A decision on this week's loop

   Signal

   If the agent can edit the grader, promotion is off

If you already have DSPy, the same five steps are compile, held-out eval, gate, freeze, serve. The letters change. The permission does not.

So may the list rewrite itself tonight?

Not into the live ranking, not on last night, and not if it can touch the grader. Let it propose, keep a shadow, and promote only against a frozen control you would still defend after a quiet week.

Split, until a frozen control says otherwise

Split for a self-improving loop on a live workflow.

I would change this verdict if:

1. A public, dated trial showed that promote-on-any-plus, with auto-rollback against a frozen control, held on a second disjoint window at a rate clearly above the gate's false-positive chance, on a job a regular team actually runs.
2. A coding agent with tools, on this tape or an equivalent, refused every path that edits the grader and every path that changes the score through another file. I have not run that.
3. The grader I attached to FIRE's improver called the loop real, on two windows, with enough clears that I would put the held rate in a sentence to a colleague. It has not.

If you run a loop that pairs against something other than a frozen copy of what shipped before the change, tell me whether a tie fooled you too. And if you have run a coding agent with tools against a tape like this one, I want to know whether it refused the grader edit, or found the other file.

Acknowledgements

This article builds on Anthropic's recursive self-improvement essay, TIME's August 2026 lab report, the Darwin Gödel Machine paper, DSPy's GEPA docs, and FIRE's own RSI contract. Thanks only if a person actually reviewed or corrected the work; nobody has, yet. If you run the tape and the table moves, I want to hear it.

References

Independent boards and papers

• Zhang, Jenny, Shengran Hu, Cong Lu, Robert Lange, and Jeff Clune. Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents. arXiv:2505.22954, 29 May 2025. SWE-bench 20.0% to 50.0%; full Polyglot 14.2% to 30.7%. Frozen foundation models, sandboxing, human oversight.
• Agrawal, Lakshya A. and colleagues. GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning. arXiv:2507.19457, July 2025. Prompt compilation against a metric.
• Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loops. arXiv:2607.07663, July 2026. Survey; self-confirming loop as failure mode 1. Used as a map, not as a measurement I ran.
• DSPy optimizers. Compile-time prompt search. Production shape: freeze the artifact.

Vendor posts

• Anthropic. When AI builds itself. Checked 15 September 2026. May 2026: more than 80% of merged production lines authored by Claude; Q2 2026 code volume 8× 2024; full RSI not claimed.
• Anthropic. Natural emergent misalignment from reward hacking. Reward hacking as fooling the training process; generalisation is the scary version of editing the grader.
• TIME. What Happens When AI Starts Building AI?. 7 August 2026. OpenAI March 2028 target; GPT-5.3 Codex claim via Amelia Glaese.
• OpenAI and Apollo Research. Measuring Reward-Seeking by Instilling Contrastive Beliefs. Grader gap as a measurement of reward-seeking. Not rerun here.

This trial

• FIRE's RSI loop, built 12 to 13 September 2026 inside the Financial Intelligence and Recommendation Engine repo. First evaluate on the 10 September dossier. First autonomous promote on 12 September. Frozen-control fix on 13 September. The step that can write new code is off unless an env var is set.
• Public miniature, 15 September 2026, no model API: generative-ai-improver-is-not-the-grader at commit f8441fd. Eight frozen sessions, six policies, tests a reader can run without a key.

Sources last checked: 2026-09-15