The harness decides what a coding agent looks capable of. The prompt layer does not

Swap the model under a coding agent and its benchmark score moves by 29.4 points. Swap the harness around the same model and it moves by 27.4.

Those two numbers come from the same paper, the same 350 tasks, and the same evaluator. One of them gets attributed to progress in machine intelligence. The other is usually described as an implementation detail, and is often not disclosed at all.

What surprised me is where inside the harness the effect lives. I assumed it would be spread thinly across a dozen small decisions. It isn't. The largest single effect in the paper comes from the adapter, which is the plumbing that prepares a workspace and collects the resulting patch. Meanwhile the layer we all actually spend our afternoons on, the system prompt and the CLAUDE.md or AGENTS.md context file, is the layer where the two most recent controlled studies find no measurable effect on correctness at all.

So I rebuilt the plumbing decision. On 30 tasks, with the agent's file edits held byte-identical across every arm, changing only how the patch was collected moved the resolved rate from 30.0% to 80.0%.

Three things this piece does not do, so you can stop reading now if you came for them. It does not reproduce Claw-SWE-Bench: there is no model, no container and no test suite anywhere in my prototype. It does not tell you which coding agent to buy. And it says nothing useful about whether context files help with anything other than correctness, which is the one place they have actually been measured.

The 80.0% is not a round number by luck. It is exactly the share of tasks the agent got right in its workspace, and the good adapter reached it without going one point past. I expected that ceiling to be approximate, the way most measured ceilings are. It was exact, which surprised me more than the 50-point gap did, and once I understood why it became the reason I would actually ship this.

The short version

• The idea: hold the model, prompt, budget and evaluator fixed, and treat the harness as the experimental variable it already is.
• Why it matters: on a fixed protocol, harness choice moves Pass@1 nearly as far as model choice, and the adapter alone moves it further than either.
• What I tested: three ways of getting one agent's identical workspace edits into a score, over 30 tasks in 6 synthetic repositories.
• What happened: 30.0%, 43.3% and 80.0% resolved. The workspace-export adapter hit the competence ceiling exactly, at 0.042 ms per patch.
• Where it breaks: making the patch applier forgiving raised the score and made the failures quieter, which is probably the worst trade available. Rejections fell from 40.0% to 20.0% while wrong-file-no-error rose from 16.7% to 23.3%.
• Decision: ship the mechanical layer, pin the harness, and stop reading undisclosed-harness scores as model capability.

Start with the problem

You are choosing a coding agent for an internal repository. Vendor A reports 78% on a public suite, vendor B reports 71%. You pick A.

The question that decides whether that was a good choice is not in either number. It is whether A's seven-point lead came from its model, its tool surface, its retry policy, its context budget, or from the way its evaluation runner extracted a patch from the agent's final message. Those are not the same purchase. Only one of them travels to your repository, because you will be running the agent under your own harness, not theirs.

Claw-SWE-Bench exists to make that question answerable. It is a SWE-bench-style benchmark of 350 issue-resolution instances across 8 languages and 43 repositories, and its contribution is not the tasks. It is the adapter protocol: a fixed prompt, runtime budget, workspace contract, patch extraction procedure and evaluator, so that different agent harnesses become comparable rather than merely comparable-looking.

The protocol is the contribution. But the thing worth reading twice is what the authors tripped over while building it.

How the method works

1. Make the harness a slot, not a bundle

Most harnesses ship prompt, scaffold, runtime and stopping policy together in one release. Claw-SWE-Bench splits them: each harness supplies an adapter implementing five methods: create the agent, send the task, back up the session, delete the agent, and supply Docker arguments (Section 2.2). Everything else belongs to a shared orchestrator that starts the container, resets the repository, instantiates the prompt, collects the patch, writes the prediction and runs the evaluation identically for every harness (Section 2.3).

Split that way, you can hold the harness fixed and vary the model, or the reverse, and know which one moved the number. Without the split you are comparing two whole systems and calling it a model comparison.

2. Collect the patch from the repository, not from the reply

Inside the shared orchestrator sits one decision that turns out to carry more weight than the rest of the protocol combined:

"candidate patches are collected from repository state rather than parsed from an agent's final message" (Section 2.2)

A general-purpose agent does not naturally satisfy SWE-bench's contract. It edits files; the benchmark wants a unified diff. There are two ways to close that gap. Ask the agent to write the diff into its final response, or let the agent edit files and have the runner diff the repository afterwards.

The first route makes the agent describe its own change from memory. The second reads what it actually did. Only the second has a correctness guarantee, and it is a trivial one: a diff generated from the workspace reproduces the workspace.

3. Remove the leak before measuring anything

One more piece of plumbing, and I want to name it because it cuts the other way. The Docker images for the seven non-Python languages exposed commits after base_commit, which means an agent could simply read the real fix with git log. The runner strips reachable future commits before evaluation.

Eight points of a frontier model's reported score turned out to be a property of how the container was built. Same class of fact as the adapter result, pointed the other way, and worth holding onto before you read the rest of this as an argument that harness engineering is free performance. Plumbing gives points back as easily as it takes them.

What the research actually shows

The strongest result

Same model. Same tasks. Same evaluator. 54.3 points.

The apply-failure column is what persuaded me, and it is a genuinely surprising number. The bare adapter is not producing worse engineering; it is throwing away engineering it already had. Sixty-nine percent of the time the agent's answer never reached the test suite at all. My read is that anyone who has watched a capable model mangle a diff header will find that plausible, and anyone who has not will assume I have made an error.

The two sweeps that follow are what let you put that number in proportion:

Read those two together and the harness spread turns out to depend on the model: half the model spread on the stronger backbone, nearly equal to it on the weaker one. A more capable model is more robust to a bad harness, then, but not immune to one. Which direction does that point in? Honestly, both. If you are optimistic, the harness effect shrinks as models improve. If you are not, it is still 12.5 points on the strongest backbone they tested, and nobody has shown where it bottoms out.

The cost result is the one a platform lead should read twice

Does the mechanism earn the credit?

Here the paper is at its weakest, and to its credit it says so first.

The claw sweep covers five harnesses and two models, which the authors call "sufficient to show that the harness is a first-order variable... but not sufficient to fully decompose harness × model interactions" (Section 7). Main experiments are single-run aggregates with no seeds. I would not read a few points between two adjacent systems as a stable ordering, and I am not sure the 12.5-versus-27.4 asymmetry I leaned on earlier survives a second seed either. The 19.1%-versus-73.4% result is a two-arm comparison, not an ablation over the adapter's parts, so it does not tell you how much of the 54.3 points is patch extraction and how much is workspace preparation or the shared prompt.

That last gap is the one a weekend prototype can close, so I built one.

Where the evidence points next

Three recent results bound the claim from different directions, and the second one is the one I would put in front of a team that is about to spend a quarter on prompt libraries.

Put those beside the adapter result and the shape is consistent. Harness effects are large. They are concentrated in the mechanical layer: tools, middleware, memory, patch collection. The prose layer, where a team's attention naturally goes because it is the layer you can edit in a text editor without deploying anything, is the layer with the weakest evidence of any effect. And harness coupling is now strong enough to be trained into model weights.

Why this paper earned the test

I screened six 2026 candidates on harness and scaffold design against claim support, baseline strength, causal evidence, production transfer, and testability. One person picked those six out of a literature that is producing harness papers faster than anyone can read them, so treat the shortlist as a judgement rather than a survey. I expect I missed at least one that belonged here.

Claw-SWE-Bench won on falsifiability. Its central claim can be tested by holding one thing fixed and varying one thing, which is more than most agent papers offer, and the specific claim I wanted to test, that patch collection is doing the work, can be isolated without a model, a container, or a dollar of spend.

Rebuilding the claim

Experimental setup

30 tasks over 6 synthetic Python repositories: billing, auth, catalog, scheduler, ingest, notify. Each task carries the agent's workspace edits, meaning the change it actually made to the files, and those edits are byte-identical in every arm. Six of the 30 (20%) carry a wrong intent, which is the part I would not skip if you are copying this design. In those, the agent did competent-looking work on the wrong thing: widening an unrelated suggestion limit when the task asked for a filter to be applied after the limit rather than before it.

The diff the agent types into its final message starts life as a correct diff of its real edits, then gets damaged by one of nine injectors: a drifted hunk header, tab re-indentation, stale context, a wrong path root, CRLF, a fenced block with prose around it, a dropped hunk, a truncated generation, or nothing at all. None of the injectors touch the workspace. That is the whole trick, and if it were not true the comparison would be worthless.

Three arms:

Why three arms and not two? Because real tools are forgiving. git apply searches for context at an offset and patch(1) has fuzz, so a strict applier is a straw man on its own. bare-fuzzy is the more conservative comparison, and it turned out to be the arm with the most interesting failure in it.

def workspace_patch(task: Task) -> str:
    """The diff the runner exports from repository state."""
    base, work = task.base(), task.workspace()
    return patchlib.clean(patchlib.generate(base, work))
 
 
def run_workspace(task: Task) -> RunResult:
    start = time.perf_counter()
    text = workspace_patch(task)
    result = patchlib.apply_strict(text, task.base())
    elapsed = (time.perf_counter() - start) * 1000.0
    return _score(task, "workspace", text, result, elapsed)

Correctness check: test_generated_patch_reproduces_the_work_tree_exactly asserts the round trip, and test_it_resolves_exactly_the_tasks_the_agent_got_right asserts that run_workspace(task).resolved == task.intent_correct for all 30 tasks. Both halves matter, so the arm cannot pass by resolving something the agent got wrong.

I would not have written that first test if the first run had gone well. It didn't. Every arm scored 0.0%, including the workspace arm, which by construction cannot fail: it applies a patch it generated seconds earlier from the very tree it is checking against. A result that impossible is a bug in the harness, not a finding, and it took me a while to stop looking at the appliers and start looking at the parser. Splitting the patch text on newlines left one empty string at the end, and the parser was counting it as a blank context line. Every hunk expected one more line than the file had.

Worth saying out loud. I was writing a parser for the most standardised text format in software engineering, and I got it wrong in a way that failed silently and completely. The article's thesis, arriving early and uninvited.

git clone https://github.com/shravan1996/ship-the-paper-coding-agent-harness
cd ship-the-paper-coding-agent-harness
PYTHONPATH=. python3 -m unittest discover -s tests   # 24 tests
PYTHONPATH=. python3 eval.py

Python 3.9.6 on an arm64 Mac. No network, no model, no GPU, no third-party packages.

What I expected

Two predictions, written down before the run. The workspace adapter wins by 30 to 50 points. The bare arms fail loudly, throwing errors you can count. My impression at the time was that the second one was the safe prediction of the two.

The first held. The second was wrong, and being wrong about it is most of what I took away from this.

What happened

Three things in that table changed how I would build this, and only the first one was on my list beforehand.

The good adapter adds nothing. The workspace arm resolved exactly the 24 tasks whose intent was correct, and none of the 6 whose intent was wrong. Obvious once you say it, and it is the whole safety argument for making the change: fixing patch collection cannot manufacture a capability the agent lacks, only stop you throwing away one it has. If this arm had scored above the ceiling I would have gone looking for the leak, because at that point it is measuring something other than the agent.

The obvious fix trades loud failures for quiet ones. Making the applier forgiving is what any of us does after watching strict mode reject a stack of nearly-correct diffs, and it works: rejections fell from 40.0% to 20.0% and the score rose 13.3 points. Look at what else moved. The tab_indent class went from a clean rejection to a silently wrong file, because the applier matched the context with whitespace collapsed and then wrote the agent's tab-indented lines into a space-indented Python file. Applied-but-wrong rose from 16.7% to 23.3%. One two-hunk task was left half-edited, which the strict arm cannot produce at all. You buy resolved tasks by converting some rejections into corruption, and the score does not tell you that you made the trade.

Some defects were never loud in the first place. This is the one that unsettled me. dropped_hunk and truncated_hunk applied cleanly in both bare arms and produced a file the agent did not write, every single time. Of course they did. A patch that is missing a hunk is perfectly well-formed; nothing about it looks wrong to a parser. Which means the apply-failure rate, the metric the paper reports at 69.1% and under 1.5%, is blind to both classes, and so is any dashboard you build out of it.

How much of this depends on my defect mix

Enough to be worth stating separately. Recombining the measured clean and defective rates at other mixes:

The flat line is the finding. A bare adapter's score depends on how reliably a model can retype a diff from memory, and that skill has nothing to do with software engineering ability. It differs by model. It changes with every release, silently, and nobody publishes it. Where do real agents sit on this table? The paper's 69.1% apply-failure rate puts them somewhere near the right-hand end, which is not a comfortable place for a leaderboard to be standing.

What breaks outside the experiment

A harness fix is a one-time gain that looks like a trend

You fix patch collection once and it never pays again. Ship the change, watch your internal score jump 40 points, and the temptation is to fund a harness team on the strength of the extrapolation. There is no second 40 points. AHE's automatic evolution needed ten iterations and roughly 32 hours of frontier-model rollouts to find 7.3 points on top of an already-competent seed harness, and that is what the curve looks like once the free win is gone. Budget harness work as a defect class, not a growth curve.

Pinning the harness is what makes anything else measurable

Leave the harness unversioned and every A/B you run afterwards is confounded. The 350 tasks are not really Claw-SWE-Bench's contribution; the fixed prompt, budget, workspace contract and extraction procedure are, and without them the two sweeps mean nothing. Internally that means a manifest attached to every run: adapter commit, tool surface, prompt hash, step budget, timeout, model id, cache policy. Ship a model upgrade in the same week as a tool change and you have learned nothing, and worse, you will not be able to work out which one to roll back.

The prose layer will keep absorbing effort anyway

I should own a miss here. Until I read these two studies I treated CLAUDE.md as a performance lever, and I had spent real hours tuning one. Two controlled studies now put the system prompt and the context file at or below zero for correctness, and neither result is close enough to the boundary for me to argue with it. Delete your AGENTS.md? No. Stop treating it as a performance lever and start treating it as what the evidence actually supports, which is a conventions document that shapes style and reduces review friction. The failure triage in the context-files study is the detail I keep coming back to: agents fail on implementation skill, feature design, pattern selection and exact wiring. A file of repository facts supplies none of those. Measure it against review time and it may well earn its place. Measure it against pass rate and it will keep losing.

Reading the workspace means trusting the workspace

Exporting the patch from repository state is strictly better at recovering the agent's intent, and that is exactly the problem. Everything the agent touched is now in the diff: the debug print it forgot, the dependency it pinned to make a test pass, the unrelated file it reformatted on the way past. The bare adapter lost work. This one collects all of it.

The control is scope, declared before the run and enforced after it: a file allowlist, a diff-size ceiling, and a hard block on changes to dependency manifests, lockfiles, CI configuration and anything holding a credential. Those are the changes where a plausible-looking diff does the most damage and gets the least reading.

Benchmark plumbing decides scores in both directions

Closing the future-commit leak cost Claude Opus 4.7 eight points. Your own evaluation set has the same hazard in a worse form, because an internal repository leaks in more places than a benchmark container does: the fix sits in a later commit, a linked ticket, a changelog, or the test somebody wrote alongside it. Cut the history at the base commit. Then re-run your baseline every time the harness changes, because an old score under a new harness is not a comparison, however much it looks like one.

The production scorecard

The scores below cover the mechanical harness layer: workspace-based patch collection, the pinned manifest, and the deterministic gates around them. That is what I rebuilt and what the verdict covers. Automated harness evolution and the prose layer score differently and are called out separately.

A 4.1 average on a Ship verdict looks generous until you notice how small the mechanism is, and the smallness is the argument rather than a caveat. A cheap, deterministic, well-understood change with a measured effect the size of a model generation is a good trade at almost any confidence level. What I am less sure about is the row I scored lowest. Operations burden at 3 assumes a team will actually maintain the manifest, and I have never seen an organisation keep that kind of discipline for a year without someone owning it. The binding constraint is not the mechanism. It is whether you can say, twelve months from now, what harness a number came from.

A production shape I would test

Read it as one run. The manifest is written before the agent starts and travels with the result, so any number produced downstream can be attributed. The agent works on a throwaway branch in a sandbox with no credentials, which is what makes "collect everything it touched" a safe policy rather than a reckless one.

Run the export before the gates, the gates before the policy check, and the policy check before any human sees the diff. That ordering is the part I would defend hardest. A reviewer should never be the first check, because reviewers are good at judging whether a change is a good idea and terrible at noticing that a diff quietly touched the lockfile.

Two things run continuously. A ledger records the manifest, the exported patch, every gate result and the approver for each run. A kill switch parks a repository when its gate failure rate climbs, on the theory that a rising failure rate means the harness changed or the repository did, and not that the tests got flaky. I am fairly confident about that theory and would want to see it survive a month of real traffic before betting an on-call rotation on it.

From prototype to a repository that matters

1. 01

   Find out how your runner collects patches

   Owner

   Platform

   Artifact

   A one-page trace of the path from the agent's last tool call to the diff your CI receives

   Signal

   You can name the function, and whether it reads the repository or the agent's message

2. 02

   Pin the harness

   Owner

   Platform

   Artifact

   A manifest per run: adapter commit, tool surface, prompt hash, step budget, timeout, model id, cache policy

   Signal

   Two runs a month apart can be compared, or provably cannot

3. 03

   Freeze an evaluation set from your own history

   Owner

   Applied science

   Artifact

   At least 100 resolved issues with deterministic test-based checks and history cut at the base commit

   Signal

   No task is passable by reading a later commit, verified by running an agent with the network off and the fix in reach

4. 04

   Run the adapter A/B

   Owner

   Applied science

   Artifact

   Paired runs of message-parsed and workspace-exported collection on one model

   Signal

   Apply-failure rate, resolved rate, and the count of well-formed patches that under-edit, reported separately

5. 05

   Add the scope gate before granting write access

   Owner

   Platform and security

   Artifact

   A file allowlist, a diff-size ceiling, and a hard block on manifests, lockfiles, CI config and credentials

   Signal

   Every out-of-scope change in a replayed month of runs is blocked before review

6. 06

   Shadow, then merge behind a named approver

   Owner

   Engineering leadership

   Artifact

   A decision record naming the repository, the permitted change classes, the approver and the rollback path

   Signal

   Gate failure rate and reviewer-minutes per merged change stay flat for four weeks

Ship gates

• Apply-failure rate under 2% on the frozen set, and every failure reproducible from the ledger.
• Zero merged changes touching a file outside the declared scope.
• Two runs with the same manifest produce the same gate results, or the difference is explained.
• Reviewer-minutes per merged change measured before the rollout and after it.

Kill criteria

• Any merge to a protected branch without a named approver and a ledger entry.
• Gate failure rate on one repository rises for three consecutive days.
• A model or adapter change lands without a manifest bump, making the following week's numbers unattributable.
• Well-formed patches that under-edit exceed 2% of merged changes, which means the apply rate has stopped being a useful signal.

Human review policy: every merge to a shared branch requires a named approver who has seen the exported patch, the gate results and the manifest. Review may be skipped only for changes confined to a declared scope with a passing test suite and no dependency, configuration or credential change, and only after four weeks of ledger evidence.

My verdict

Ship workspace-based patch collection and a pinned harness manifest. Wait on the prose layer and on automated harness evolution.

The mechanical change is about as close to free as production changes get: one function in the runner, microseconds to run, no data and no model change, and both the paper and my prototype put its effect in the range of a model generation. What the prototype adds is the reason it is safe as well as effective. It recovered 100% of the agent's competence and 0% of the competence the agent did not have. An intervention shaped like that cannot inflate a capability claim, which is more than I can say for most things that move a benchmark number by 50 points.

The paper and my measurement agree on direction and split on emphasis. Its headline is that harness and model are non-separable. Mine says the practical consequence is narrower and more useful: most of the separable part is one decision about where the patch comes from, and the intervention everyone reaches for first is not it. Where the two disagree, I trust the paper on how far harness effects reach across systems and the prototype on what the mechanism can and cannot manufacture, because those are the questions each one actually measured. On anything else, neither of us has the evidence.

What I would not do is read any of this as a capability story. Nothing here made an agent better at software engineering. It stopped a system throwing away work the agent had already done, and it made the resulting numbers attributable. Both are worth having. Neither is progress in the thing the leaderboards claim to be measuring, and I suspect a fair slice of the progress we have been reading about for the last two years is this, uncredited.

I would change this verdict if:

1. A multi-seed replication across more than two models shows the harness spread collapsing on current frontier backbones. The trend from 27.4 points on Qwen 3.6-flash to 12.5 on GLM 5.1 is the one number in the paper that points that way.
2. Workspace export on real repositories produces out-of-scope changes at a rate the scope gate cannot absorb, which would make the bare adapter's discarding behaviour an accidental safety property worth replacing deliberately.
3. A controlled study finds context files moving correctness by more than the 10-to-15-point bound the current equivalence testing places on them, on a task set larger than 17.

What to remember

The number I most want and do not have is the adapter A/B on a real repository: same model, same tasks, message-parsed collection against workspace export, with well-formed patches that under-edit counted separately from patches that fail to apply. If you have run it, or if your runner already exports from Git state and you think the gap is smaller than I am claiming, I would like to know. My whole case rests on a synthetic corpus and a defect mix I chose myself.

Acknowledgements

This article builds on Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks by Mengyu Zheng, Kai Han, Boxun Li, Haiyang Xu, Yuchuan Tian, Wei He, Hang Zhou, Jianyuan Guo, Hailin Hu, Lin Ma, Chao Xu, Guohao Dai, Lixue Xia, Yunchao Wei, Yunhe Wang, and Yu Wang. The adapter protocol, the two sweeps and the leak-fix analysis are theirs; errors in my reading of them are mine.

The prototype is an original implementation. It does not use the authors' code, containers, tasks or data, and it reproduces none of their results. It is Python standard library only, under MIT. The six repositories and thirty tasks are synthetic and were written for this article.

Four further papers shaped the boundaries of the argument and are credited in the References below.

References

Primary research

• Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks. Mengyu Zheng et al., 2026 (arXiv:2606.12344, submitted 10 June 2026)

Supporting and contrary evidence

• Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories. Prakhar Khatri, 28 July 2026. The null result on the prose layer, with code and data released
• Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses. Jiahang Lin et al. The component ablation that localises gains to memory, tools and middleware
• DCAS: Decoupling CLI Agent Scaffolding to Internalize Planning across Scaffolds. Kishanthan Thangarajah, Boyuan Chen, Ahmed E. Hassan, 6 August 2026. Harness coupling surviving into fine-tuned weights
• When Does Restricting a Coding Agent to execute_code Help? A Regime × Agent-Design Ablation. Hong Yang, Qi Yu, Travis Desell, 12 July 2026. A tool-surface ablation where pass rates tie and cost does not
• Stop Comparing LLM Agents Without Disclosing the Harness. Yunbei Zhang et al. The disclosure standard this article argues for

Prototype

• Source, setup, tests, and evaluation
• Exact tested commit

Sources last checked: 2026-08-09

Before you attribute a coding agent's score to its intelligence, find out who wrote down its diff.