Find out who wrote down the diff

The first run of my prototype scored 0.0% on every arm. Three arms, thirty tasks, nothing resolved, including the arm that by construction cannot fail: it applies a patch it generated seconds earlier from the very tree it is checking against. A result that impossible is a bug in the harness, not a finding. I had assumed the bug was in the patch appliers, and it still took me a while to stop staring at them and look at the parser instead. Splitting the patch text on newlines had left one empty string at the end. The parser counted it as a blank context line, so every hunk expected one more line than the file had. I had written a parser for the most standardised text format in software engineering, and I had got it wrong in a way that failed silently and completely.

Which is, give or take, the finding of the paper I was testing. Claw-SWE-Bench, a June paper from Mengyu Zheng and colleagues, holds a coding agent's model fixed and swaps the harness around it, then does the reverse. Swap the model and the benchmark score moves by 29.4 points. Swap the harness and it moves by 27.4. Same 350 tasks, same evaluator. One of those two numbers gets reported as progress in machine intelligence; the other usually goes undisclosed. And the largest single effect they found inside the harness sits nowhere near the layer I spend my afternoons on. It sits in the adapter, the plumbing that prepares a workspace and collects the finished patch. The system prompt and the CLAUDE.md or AGENTS.md context file, meanwhile, are the layer where the two most recent controlled studies find no measurable effect on correctness at all.

So the plumbing is what I rebuilt. Thirty tasks, the agent's file edits held byte-identical across every arm, and the only thing varying is how the patch gets collected. With the parser fixed, that one change moved the resolved rate from 30.0% to 80.0%. The 80.0% is what I keep coming back to. It is exactly the share of tasks the agent had got right in its workspace, and the good adapter reached it without going a single point past. I had expected that ceiling to be approximate, the way measured ceilings usually are. Exact surprised me more than the 50-point gap did, and once I understood why, it became the reason I would actually ship this.

None of what follows reproduces Claw-SWE-Bench, since there is no model, no container and no test suite anywhere in my prototype, and none of it will tell you which coding agent to buy. It also says nothing about whether context files help with anything other than correctness, correctness being the one thing anyone has actually measured them on.

Fifty points from one function, and no new capability

For the skimmer, four things:

1. Hold the model, prompt, budget and evaluator fixed and the harness becomes the experimental variable it always was. On Claw-SWE-Bench's protocol, harness choice moves Pass@1 nearly as far as model choice, and the adapter alone moves it further than either.
2. I ran one agent's identical workspace edits through three collection routes over 30 tasks in 6 synthetic repositories: 30.0%, 43.3% and 80.0% resolved. Workspace export hit the competence ceiling exactly, at 0.042 ms per patch.
3. The obvious fix, a forgiving patch applier, raised the score and quietened the failures. Rejections fell from 40.0% to 20.0% while wrong-file-with-no-error rose from 16.7% to 23.3%. I would not take that trade; it is probably the worst one on offer.
4. Ship the mechanical layer, pin the harness, and stop reading an undisclosed-harness score as model capability.

Vendor A at 78%, vendor B at 71%

Say you are choosing a coding agent for an internal repository. Vendor A reports 78% on a public suite, vendor B reports 71%. You pick A.

Whether that was a good choice depends on something neither number tells you: whether A's seven-point lead came from its model, its tool surface, its retry policy, its context budget, or the way its evaluation runner extracted a patch from the agent's final message. Those are different purchases. Only one of them travels to your repository, because you will be running the agent under your own harness and not theirs.

Two terms carry the rest of this. The harness is everything around the model: the prompt, the tools it can call, the step budget, the stopping rule, and the code that turns what the agent did into something an evaluator can score. The adapter is the slice of the harness each system supplies under Claw-SWE-Bench's protocol; everything else is shared.

Claw-SWE-Bench exists to make the vendor question answerable. It is a SWE-bench-style benchmark of 350 issue-resolution instances across 8 languages and 43 repositories, and the tasks are the least of it. What the authors actually contribute is an adapter protocol: a fixed prompt, runtime budget, workspace contract, patch extraction procedure and evaluator, so that different harnesses become comparable instead of merely comparable-looking.

The protocol is the contribution the authors would name. The thing worth reading twice is what they tripped over while building it.

The harness becomes a slot and the runner reads Git

Most harnesses ship prompt, scaffold, runtime and stopping policy together in one release. Claw-SWE-Bench splits them. Each harness supplies an adapter implementing five methods: create the agent, send the task, back up the session, delete the agent, and supply Docker arguments (Section 2.2). Everything else belongs to a shared orchestrator that starts the container, resets the repository, instantiates the prompt, collects the patch, writes the prediction and runs the evaluation identically for every harness (Section 2.3).

Split that way, you can hold the harness fixed and vary the model, or the reverse, and know which one moved the number. Without the split you are comparing two whole systems and calling it a model comparison.

Inside the shared orchestrator sits one decision that turns out to carry more weight than the rest of the protocol combined:

"candidate patches are collected from repository state rather than parsed from an agent's final message" (Section 2.2)

A general-purpose agent does not naturally satisfy SWE-bench's contract. It edits files; the benchmark wants a unified diff. You can close that gap two ways. Ask the agent to write the diff into its final response, or let the agent edit files and have the runner diff the repository afterwards.

The first route makes the agent describe its own change from memory. The second reads what it actually did. Only the second comes with a correctness guarantee, and it is a trivial one: a diff generated from the workspace reproduces the workspace.

The leak that cost Claude Opus eight points

One more piece of plumbing, and I want to name it because it cuts the other way. The Docker images for the seven non-Python languages exposed commits after base_commit, which means an agent could simply read the real fix with git log. The runner now strips reachable future commits before evaluation.

Eight points of a frontier model's reported score turned out to be a property of how the container was built. Same class of fact as the adapter result, pointed the other way, and worth holding onto before you read the rest of this as an argument that harness engineering is free performance. Plumbing takes points away as readily as it hands them back.

Same model, same tasks, 54.3 points apart

The apply-failure column is what persuaded me. The bare adapter does not produce worse engineering; it throws away engineering it already had. Sixty-nine percent of the time the agent's answer never reached the test suite at all. My read is that anyone who has watched a capable model mangle a diff header will find that plausible, and anyone who has not will assume I have made an error.

The two sweeps that follow are what let you put that number in proportion.

Read those two together and the harness spread turns out to depend on the model: half the model spread on the stronger backbone, nearly equal to it on the weaker one. A more capable model is more robust to a bad harness, then, though not immune to one. Which direction does that point in? Honestly, both. If you are an optimist, the harness effect shrinks as models improve. If you are not, it is still 12.5 points on the strongest backbone they tested, and nobody has shown where it bottoms out.

Is a 7.7-point lead worth 170 times the bill?

Not in any workflow I would run, and every workflow I would run has a human reviewing each change anyway. My read is that the reviewer absorbs the accuracy difference and nobody absorbs the cost difference.

Five harnesses, two models, one run each

Here the paper is at its weakest, and to its credit it says so first.

The claw sweep covers five harnesses and two models, which the authors call "sufficient to show that the harness is a first-order variable... but not sufficient to fully decompose harness × model interactions" (Section 7). Main experiments are single-run aggregates with no seeds. I would not read a few points between two adjacent systems as a stable ordering, and I am not sure the 12.5-versus-27.4 asymmetry I leaned on above survives a second seed either. The 19.1%-versus-73.4% result is a two-arm comparison, with no ablation over the adapter's parts, so it cannot tell you how much of the 54.3 points is patch extraction and how much is workspace preparation or the shared prompt.

That last gap is the one a weekend prototype can close, and it is the one I chose. But first the three papers that convinced me the gap was worth closing.

Three more studies point away from the prompt

Three recent results bound the claim from different directions, and the second one is the one I would put in front of a team that is about to spend a quarter on prompt libraries.

Put those beside the adapter result and the shape is consistent. Harness effects are large. They concentrate in the mechanical layer: tools, middleware, memory, patch collection. The prose layer, where a team's attention naturally goes because you can edit it in a text editor without deploying anything, has the weakest evidence of any effect. And harness coupling is now strong enough to be trained into model weights.

Why Claw-SWE-Bench and not the other five

I screened six 2026 candidates on harness and scaffold design against claim support, baseline strength, causal evidence, production transfer, and testability. One person picked those six out of a literature that is producing harness papers faster than anyone can read them, so treat the shortlist as a judgement and not a survey. I expect I missed at least one that belonged here.

Claw-SWE-Bench won on falsifiability. Its central claim can be tested by holding one thing fixed and varying one thing, which is more than most agent papers offer, and the specific claim I wanted to test, that patch collection is doing the work, can be isolated without a model, a container, or a dollar of spend.

Thirty tasks, nine ways to damage a diff

30 tasks over 6 synthetic Python repositories: billing, auth, catalog, scheduler, ingest, notify. Each task carries the agent's workspace edits, meaning the change it actually made to the files, and those edits are byte-identical in every arm. Six of the 30 (20%) carry a wrong intent, which is the part I would not skip if you are copying this design. In those, the agent did competent-looking work on the wrong thing: widening an unrelated suggestion limit when the task asked for a filter to be applied after the limit, not before it.

The diff the agent types into its final message starts life as a correct diff of its real edits, then gets damaged by one of nine injectors: a drifted hunk header, tab re-indentation, stale context, a wrong path root, CRLF, a fenced block with prose around it, a dropped hunk, a truncated generation, or nothing at all. None of the injectors touch the workspace. The whole comparison rests on that, and if it were not true the comparison would be worthless.

Three arms:

Why three arms and not two? Because real tools are forgiving. git apply searches for context at an offset and patch(1) has fuzz, so a strict applier on its own is a straw man. bare-fuzzy is the more conservative comparison, and it turned out to be the arm with the most interesting failure in it.

def workspace_patch(task: Task) -> str:
    """The diff the runner exports from repository state."""
    base, work = task.base(), task.workspace()
    return patchlib.clean(patchlib.generate(base, work))
 
 
def run_workspace(task: Task) -> RunResult:
    start = time.perf_counter()
    text = workspace_patch(task)
    result = patchlib.apply_strict(text, task.base())
    elapsed = (time.perf_counter() - start) * 1000.0
    return _score(task, "workspace", text, result, elapsed)

Two tests pin the workspace arm down. test_generated_patch_reproduces_the_work_tree_exactly asserts the round trip, and test_it_resolves_exactly_the_tasks_the_agent_got_right asserts that run_workspace(task).resolved == task.intent_correct for all 30 tasks. Both halves matter, so the arm cannot pass by resolving something the agent got wrong. The first of those tests exists because of the zero run I opened with; I would not have thought to write it if the first run had gone well.

git clone https://github.com/shravan1996/ship-the-paper-coding-agent-harness
cd ship-the-paper-coding-agent-harness
PYTHONPATH=. python3 -m unittest discover -s tests   # 24 tests
PYTHONPATH=. python3 eval.py

Python 3.9.6 on an arm64 Mac. No network, no model, no GPU, no third-party packages.

Two predictions went down on paper before the run. The workspace adapter wins by 30 to 50 points. The bare arms fail loudly, throwing errors you can count. My impression at the time was that the second one was the safe prediction of the two. The first held. On the second I was wrong, and being wrong about it is most of what I took away from this.

The workspace adapter reached 80.0% and stopped there

The workspace arm resolved exactly the 24 tasks whose intent was correct, and none of the 6 whose intent was wrong. Obvious once you say it, and it is the whole safety argument for making the change: fixing patch collection cannot manufacture a capability the agent lacks, only stop you throwing away one it has. If this arm had scored above the ceiling I would have gone looking for the leak, because at that point it is measuring something other than the agent. That was the one finding in the table I had on my list beforehand. The next two I did not.

Forgiveness bought 13.3 points and quieter failures

Making the applier forgiving is what any of us does after watching strict mode reject a stack of nearly-correct diffs, and it works: rejections fell from 40.0% to 20.0% and the score rose 13.3 points. Now look at what else moved. The tab_indent class went from a clean rejection to a silently wrong file, because the applier matched the context with whitespace collapsed and then wrote the agent's tab-indented lines into a space-indented Python file. Applied-but-wrong rose from 16.7% to 23.3%. One two-hunk task was left half-edited, which the strict arm cannot produce at all. You buy resolved tasks by converting some rejections into corruption, and the score does not tell you that you made the trade.

The finding that unsettled me came from the defects that were never loud in the first place, and I did not expect it, because the defects in question looked perfect. dropped_hunk and truncated_hunk applied cleanly in both bare arms and produced a file the agent did not write, every single time. Of course they did. A patch that is missing a hunk is perfectly well-formed; nothing about it looks wrong to a parser. Which means the apply-failure rate, the metric the paper reports at 69.1% and under 1.5%, is blind to both classes, and so is any dashboard you build out of it.

A flat line at 80.0% across every defect mix

How much of the headline depends on the mix of defects I chose? Enough to be worth stating separately. Recombining the measured clean and defective rates at other mixes:

The flat line is the finding. A bare adapter's score depends on how reliably a model can retype a diff from memory, and that skill has nothing to do with software engineering ability. It differs by model. It changes with every release, silently, and nobody publishes it. Where do real agents sit on this table? The paper's 69.1% apply-failure rate puts them somewhere near the right-hand end, which is an uncomfortable place for a leaderboard to be standing.

What thirty synthetic tasks cannot warn you about

A one-time gain that looks like a trend

You fix patch collection once and it never pays again. Ship the change, watch your internal score jump 40 points, and the temptation is to fund a harness team on the strength of the extrapolation. There is no second 40 points. AHE's automatic evolution needed ten iterations and roughly 32 hours of frontier-model rollouts to find 7.3 points on top of an already-competent seed harness, and that is what the curve looks like once the free win is gone. I would budget harness work the way you budget a defect class.

No manifest, no comparison

Leave the harness unversioned and every A/B you run afterwards is confounded. The 350 tasks were never really Claw-SWE-Bench's contribution; the fixed prompt, budget, workspace contract and extraction procedure are, and without them the two sweeps mean nothing. Internally that means a manifest attached to every run: adapter commit, tool surface, prompt hash, step budget, timeout, model id, cache policy. Ship a model upgrade in the same week as a tool change and you have learned nothing, and worse, you will not be able to work out which one to roll back.

I had spent real hours tuning a CLAUDE.md

I should own a miss here. Until I read these two studies I had assumed CLAUDE.md was a performance lever, and I had spent real hours tuning one. Two controlled studies now put the system prompt and the context file at or below zero for correctness, and neither result is close enough to the boundary for me to argue with it. Delete your AGENTS.md? No. Stop treating it as a performance lever and start treating it as what the evidence actually supports, which is a conventions document that shapes style and reduces review friction. The failure triage in the context-files study is the detail I keep coming back to: agents fail on implementation skill, feature design, pattern selection and exact wiring. A file of repository facts supplies none of those. Measure it against review time and it may well earn its place. Measure it against pass rate and it will keep losing.

The export collects the forgotten debug print too

Exporting the patch from repository state is strictly better at recovering the agent's intent, and that is exactly the problem. Everything the agent touched is now in the diff: the debug print it forgot, the dependency it pinned to make a test pass, the unrelated file it reformatted on the way past. The bare adapter lost work; this one collects all of it, so the control has to be scope, declared before the run and enforced after it: a file allowlist, a diff-size ceiling, and a hard block on changes to dependency manifests, lockfiles, CI configuration and anything holding a credential. Those are the changes where a plausible-looking diff does the most damage and gets the least reading.

Your own history leaks in more places than a container

Closing the future-commit leak cost Claude Opus 4.7 eight points. Your own evaluation set has the same hazard in a worse form, because an internal repository leaks in more places than a benchmark container does: the fix sits in a later commit, a linked ticket, a changelog, or the test somebody wrote alongside it. Cut the history at the base commit. Then re-run your baseline every time the harness changes, because an old score under a new harness cannot be compared with a new one, whatever the dashboard implies.

A 4.1 that rests on someone keeping the manifest

The scores below cover the mechanical harness layer: workspace-based patch collection, the pinned manifest, and the deterministic gates around them. I rebuilt that layer and the verdict covers it. Automated harness evolution and the prose layer score differently and are called out separately.

A 4.1 average on a Ship verdict looks generous until you notice how small the mechanism is, and the smallness is the argument. A cheap, deterministic, well-understood change with a measured effect the size of a model generation is a good trade at almost any confidence level. What I am less sure about is the row I scored lowest. Operations burden at 3 assumes a team will actually maintain the manifest, and I have never seen an organisation keep that kind of discipline for a year without someone owning it. So the binding constraint sits outside the mechanism: it is whether you can say, twelve months from now, what harness a number came from.

Export, then gates, then policy, then a person

Read it as one run. The manifest is written before the agent starts and travels with the result, so any number produced downstream can be attributed. The agent works on a throwaway branch in a sandbox with no credentials, which is what makes "collect everything it touched" a safe policy instead of a reckless one.

Run the export before the gates, the gates before the policy check, and the policy check before any human sees the diff. That ordering is the part I would defend hardest. A reviewer should never be the first check, because reviewers are good at judging whether a change is a good idea and terrible at noticing that a diff quietly touched the lockfile.

Alongside the run, two things never stop. A ledger records the manifest, the exported patch, every gate result and the approver for each run. A kill switch parks a repository when its gate failure rate climbs, on the theory that a rising failure rate means the harness changed or the repository did, and not that the tests got flaky. I am fairly confident about that theory and would want to see it survive a month of real traffic before betting an on-call rotation on it.

Start by tracing where your runner gets its diff

1. 01

   Find out how your runner collects patches

   Owner

   Platform

   Artifact

   A one-page trace of the path from the agent's last tool call to the diff your CI receives

   Signal

   You can name the function, and whether it reads the repository or the agent's message

2. 02

   Pin the harness

   Owner

   Platform

   Artifact

   A manifest per run: adapter commit, tool surface, prompt hash, step budget, timeout, model id, cache policy

   Signal

   Two runs a month apart can be compared, or provably cannot

3. 03

   Freeze an evaluation set from your own history

   Owner

   Applied science

   Artifact

   At least 100 resolved issues with deterministic test-based checks and history cut at the base commit

   Signal

   No task is passable by reading a later commit, verified by running an agent with the network off and the fix in reach

4. 04

   Run the adapter A/B

   Owner

   Applied science

   Artifact

   Paired runs of message-parsed and workspace-exported collection on one model

   Signal

   Apply-failure rate, resolved rate, and the count of well-formed patches that under-edit, reported separately

5. 05

   Add the scope gate before granting write access

   Owner

   Platform and security

   Artifact

   A file allowlist, a diff-size ceiling, and a hard block on manifests, lockfiles, CI config and credentials

   Signal

   Every out-of-scope change in a replayed month of runs is blocked before review

6. 06

   Shadow, then merge behind a named approver

   Owner

   Engineering leadership

   Artifact

   A decision record naming the repository, the permitted change classes, the approver and the rollback path

   Signal

   Gate failure rate and reviewer-minutes per merged change stay flat for four weeks

Before any of it goes live, four things have to be true. Apply-failure rate is under 2% on the frozen set, and every failure is reproducible from the ledger. Zero merged changes touch a file outside the declared scope. Two runs with the same manifest produce the same gate results, or the difference is explained. And reviewer-minutes per merged change have been measured before the rollout and after it.

Any of four things stops it. Any merge to a protected branch without a named approver and a ledger entry. Gate failure rate on one repository rising for three consecutive days. A model or adapter change landing without a manifest bump, making the following week's numbers unattributable. And well-formed patches that under-edit exceeding 2% of merged changes, which means the apply rate has stopped being a useful signal.

On review: every merge to a shared branch requires a named approver who has seen the exported patch, the gate results and the manifest. Review may be skipped only for changes confined to a declared scope with a passing test suite and no dependency, configuration or credential change, and only after four weeks of ledger evidence.

Ship the plumbing, wait on the prose

Ship workspace-based patch collection and a pinned harness manifest. Wait on the prose layer and on automated harness evolution.

The mechanical change is about as close to free as production changes get: one function in the runner, microseconds to run, no data and no model change, and both the paper and my prototype put its effect in the range of a model generation. What the prototype adds is the reason it is safe as well as effective. It recovered 100% of the agent's competence and 0% of the competence the agent did not have. An intervention shaped like that cannot inflate a capability claim, which is more than I can say for most things that move a benchmark number by 50 points.

The paper and my measurement agree on direction and split on emphasis. Its headline is that harness and model are non-separable. Mine says the practical consequence is narrower and more useful: most of the separable part is one decision about where the patch comes from, and the intervention everyone reaches for first has nothing to do with it. Where the two disagree, I trust the paper on how far harness effects reach across systems and the prototype on what the mechanism can and cannot manufacture, because those are the questions each one actually measured. On anything else, neither of us has the evidence.

What I would not do is read any of this as a capability story. Nothing here made an agent better at software engineering. It stopped a system throwing away work the agent had already done, and it made the resulting numbers attributable. Both are worth having. Neither is progress in the thing the leaderboards claim to be measuring, and I suspect a fair slice of the progress we have been reading about for the last two years is this, uncredited.

I would change this verdict if:

1. A multi-seed replication across more than two models shows the harness spread collapsing on current frontier backbones. The trend from 27.4 points on Qwen 3.6-flash to 12.5 on GLM 5.1 is the one number in the paper that points that way.
2. Workspace export on real repositories produces out-of-scope changes at a rate the scope gate cannot absorb, which would make the bare adapter's discarding behaviour an accidental safety property worth replacing deliberately.
3. A controlled study finds context files moving correctness by more than the 10-to-15-point bound the current equivalence testing places on them, on a task set larger than 17.

Rules I would keep, and the A/B I still want

The measurement I would trade this whole prototype for is the adapter A/B on a real repository: same model, same tasks, message-parsed collection against workspace export, with well-formed patches that under-edit counted separately from patches that fail to apply. If you have run it, or if your runner already exports from Git state and you think the gap is smaller than I am claiming, I would like to know. My whole case rests on a synthetic corpus and a defect mix I chose myself, and the parser I got wrong on the first run is a fair reminder of how much a single person's plumbing can hide.

Acknowledgements

This article builds on Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks by Mengyu Zheng, Kai Han, Boxun Li, Haiyang Xu, Yuchuan Tian, Wei He, Hang Zhou, Jianyuan Guo, Hailin Hu, Lin Ma, Chao Xu, Guohao Dai, Lixue Xia, Yunchao Wei, Yunhe Wang, and Yu Wang. The adapter protocol, the two sweeps and the leak-fix analysis are theirs; errors in my reading of them are mine.

The prototype is an original implementation. It does not use the authors' code, containers, tasks or data, and it reproduces none of their results. It is Python standard library only, under MIT. The six repositories and thirty tasks are synthetic and were written for this article.

Four further papers shaped the boundaries of the argument and are credited in the References below.

References

Primary research

• Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks. Mengyu Zheng et al., 2026 (arXiv:2606.12344, submitted 10 June 2026)

Supporting and contrary evidence

• Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories. Prakhar Khatri, 28 July 2026. The null result on the prose layer, with code and data released
• Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses. Jiahang Lin et al. The component ablation that localises gains to memory, tools and middleware
• DCAS: Decoupling CLI Agent Scaffolding to Internalize Planning across Scaffolds. Kishanthan Thangarajah, Boyuan Chen, Ahmed E. Hassan, 6 August 2026. Harness coupling surviving into fine-tuned weights
• When Does Restricting a Coding Agent to execute_code Help? A Regime × Agent-Design Ablation. Hong Yang, Qi Yu, Travis Desell, 12 July 2026. A tool-surface ablation where pass rates tie and cost does not
• Stop Comparing LLM Agents Without Disclosing the Harness. Yunbei Zhang et al. The disclosure standard this article argues for

Prototype

• Source, setup, tests, and evaluation
• Exact tested commit

Sources last checked: 2026-08-09