Your company probably offers you the same model in three costumes. The chat window in the browser. The sidebar someone wired into the wiki. The new "agent" IT enabled last month that can open files. All three say they run the same frontier model, and a colleague deciding where to paste the July expense sheet asks you which one to use. The obvious answer is that it doesn't matter. Same model, same intelligence, same answer.
I used to think exactly that. I stopped after spending a week rebuilding a coding-agent result, where swapping the software around a fixed model moved its benchmark score almost as far as swapping the model itself. That finding lives in an earlier article on this site aimed at people who run evaluations for a living. This piece is the version for the rest of us, and it needed its own trial.
So I built one. A scripted model with fixed, documented competence, run byte-identically inside four wrappers, answering the same 24 questions about one 30-row expense sheet. The wrapper moved the share of correct answers from 25.0% to 91.7%. It also decided something I now think matters more than the score: whether the wrong answers announced themselves, or arrived sounding exactly like the right ones.
Three things this piece doesn't do. It doesn't rank Claude against ChatGPT against Gemini; no real model was called, which is the point and the main limitation at once. It doesn't cover coding agents in depth, though it borrows their research, because that's where anyone bothered to measure this. And it doesn't ask you to adopt the word "harness" at work. You only need the question that word stands for: what is the app around the model actually doing?
The short version
- The job: get trustworthy answers about your own files from an AI assistant, this week, without becoming a benchmark analyst.
- The trap: "powered by the same model" reads as "same results." The app decides what the model sees, what it can do, and whether you ever learn that something went wrong.
- What I compared: four wrappers around one deterministic scripted model, on 21 August 2026: a truncated paste, a full file in context, tools without error feedback, and tools with it.
- What I ran: 24 expense-sheet questions per wrapper. Scores of 25.0%, 50.0%, 50.0%, and 91.7%, with the model held byte-identical.
- Where it breaks: the two middle wrappers tied exactly, and could not be more different to live with. One failed loudly twelve times. The other was confidently wrong twelve times.
- Decision: split. Chat for what's in the paste. A harness with tools and visible errors for anything counted across a file. Fix the wrapper before upgrading the model.
Start with the job
It's the last week of the month and the expense sheet needs checking. What did we spend on travel? Is software still under budget? How many invoices are waiting on approval? You have thirty rows and an assistant, and you'd rather not build a pivot table for this.
The demo you saw made this look solved. And the demo wasn't lying, exactly. It was answering a different question than the one you're about to ask, because the assistant in the demo may have been allowed to see and do different things than the one on your screen.
The rest of this piece asks a narrower question than "which assistant is best." If the model is exactly the same, how much of the outcome do those four switches control?
How to read a "same model" claim
1. Ask what the assistant actually saw
A model can't use a row it was never shown. Chat apps truncate long pastes, summarise old turns, and retrieve fragments instead of whole files, mostly without telling you. The first question about any wrong answer is not "why is the model bad at this" but "was the relevant row in front of it at all." You can test this in one minute: ask the assistant to repeat the last line of what you pasted.
2. Ask what it did with its hands
"What did we spend on travel" over thirty rows is a database query wearing a chat costume. An assistant with a code runner or a query tool executes it. An assistant without one performs arithmetic in prose, which is a known weak spot of language models, and the result reads identically either way. Some chat apps run code for arithmetic and some don't, and the same app decides differently on different days. If the answer doesn't say what it did, assume prose.
3. Ask what happened on its last failed step
Real work fails midway. A query has the wrong date format, a file doesn't parse, a step times out. The harness decides whether the model hears about the failure and gets another turn, or whether the run just ends, or worst, whether the model shrugs and answers from memory. You'll almost never find this switch documented. You find it by watching what the assistant does the first time something breaks.
What the boards actually show
Nobody benchmarks office assistants this way yet, as far as I can tell. Coding agents are where the measurement exists, because their harnesses are public enough to swap, and the numbers are blunt.
Reported
Claw-SWE-Bench (June 2026) holds the evaluation protocol fixed and varies one thing at a time across 350 coding tasks. Swapping the model under one fixed harness moves the pass rate by 29.4 points. Swapping the harness around one fixed model moves it by up to 27.4 points. And a single harness decision, whether the agent's finished work is collected from its workspace or re-typed from its final message, moves one model from 19.1% to 73.4% on the same tasks.
Read that middle sentence again with your office assistant in mind. The wrapper is worth nearly as much as the model, on the one protocol where anyone has held everything else still.
Reported
Do Context Files Help Coding Agents? (July 2026) is the null result on the layer everyone actually tinkers with. Across 288 evaluated runs on two agents, instruction files like AGENTS.md produced no measurable effect on correctness. The layers that carried effects elsewhere in this literature were mechanical: tools, memory, and how work is collected.
Reported
DCAS (August 2026) reports that open models fine-tuned inside one harness degrade substantially when run inside a different one, while untrained base models don't show the divergence. The wrapper is now getting baked into model weights.
I'll admit the middle result stung, because I would have expected the opposite. I had assumed the prose layer was the lever available to me, and I'd spent real hours polishing instruction files on that assumption. Two controlled studies now put that lever at roughly zero for correctness, and the mechanical layers, the ones I'd dismissed as plumbing, are where the measured effects live.
Why an honest table can still mislead
Every number above is filed publicly under a model's name. "GLM 5.1 scores 73.4%." The harness that produced the score is named in the paper and nowhere else, and a position paper from May 2026 exists purely to beg vendors to disclose it. Until they do, a leaderboard entry is a lap time filed under the engine, and your assistant's vendor changed the chassis last Tuesday without a release note.
There's a second distortion, and my trial exists mostly because I wanted to see it up close. Accuracy tables have no column for how a failure behaves. Two systems can post the same score while one refuses visibly and the other fabricates confidently, and every table you've seen this year treats them as identical.
Inferred
A regular user comparing assistants has less information than a benchmark reader. The benchmark at least names its harness in the methods section. The app on your screen names its model and nothing else.
What I asked
I wanted the wrapper effect isolated so cleanly that nobody could blame the model, so I removed the model. In its place: a deterministic script with fixed, documented competence. It reads rows correctly, slips on arithmetic past five numbers (drops the last one), writes "July" where a tool wants "2026-07" until shown the error, misreads two loosely worded questions, and honestly says so when it can't see relevant data. Same object, same rules, every arm. The four wrappers around it differ only in what it sees, whether it has a query tool, and whether errors come back.
The question
If the model's behaviour is held byte-identical, how much does the app around it change the share of office questions answered correctly, and how does it change what the failures look like?
- Baseline
- All four harnesses land within a few tasks of each other, meaning the wrapper is cosmetic and the same-model intuition holds.
- Continue if
- A spread of 6 or more tasks out of 24 between the weakest and strongest harness, with the model unchanged.
- Stop if
- The harnesses reorder the model's competence, resolving a question the model actually got wrong, which would mean my wrappers add intelligence rather than transmit it.
The four arms, named for how you'd meet them at work: paste-chat sees only the first 12 of 30 rows, which is the paste that got cut off. attach-chat sees the whole sheet but works in prose, one turn. tools-mute has a query tool, and a tool error silently kills the run. tools-loop has the same tool, gets errors fed back, and may retry.
What I expected, written down before the run: a clean ladder. Each added capability buys points, so paste-chat worst, tools-loop best, and tools-mute comfortably ahead of attach-chat, because surely a real tool beats mental arithmetic even without retries.
What happened
Reproduced
Same model, four apps: 6, 12, 12, and 22 of 24 questions correct. paste-chat was silently wrong 12 times and visibly stuck 6 times. attach-chat was silently wrong on every one of its 12 misses. tools-mute never gave a wrong answer: all 12 of its misses were visible dead runs. tools-loop resolved 22 of 24, which is exactly the set of questions the scripted model read correctly, at 0 loud failures.
The ladder I predicted didn't survive contact. I did not expect the tie: attach-chat and tools-mute both landed on 12, and that tie is the most useful thing the trial produced. The tool arm wasn't better; it failed differently. Every question that mentioned "July" died inside tools-mute, because the model's first query wrote the month as a word, the tool wanted "2026-07", and the harness never showed the model the error message. The identical model inside tools-loop saw the error and fixed it on the next turn, every single time. The entire 41.7-point gap between those two arms is one switch: does the error message reach the model?
The quietest failure cost $118.40. The software budget question has a true answer of $1,568.38 against a $1,500 budget: over. attach-chat, summing six prices in prose, dropped the last row, a $118.40 storage charge, got $1,449.98, and answered "under budget" in a complete, fluent, plausible sentence. I designed the arithmetic slip myself and the verdict flip still surprised me when I tallied the arms. Nothing about that answer invites a second look. It's the kind of number that gets pasted into a slide.
And the ceiling behaved in a way I found genuinely reassuring. tools-loop got 22 right and the two it missed are the two questions the scripted model misreads by construction: "client entertainment," which it rounds to all meals, and "billable to Meridian," which it rounds to all travel. The best harness in the trial recovered every point of competence the model had and manufactured none it didn't. My stop threshold, a harness resolving a question the model got wrong, never fired.
Unknown
Whether real assistants sit where my arms sit. The scripted slips mimic documented model weaknesses, long mental arithmetic and format errors and scope misreading, but I chose their rates, so the per-cell grid travels and the totals don't. I also don't know how often commercial chat apps silently truncate a paste, and I'd honestly like to. My guess is more often than any of us assume, but that's a guess. Nobody publishes the number.
Where a regular user should not trust the number
An answer summed in prose over a whole file
attach-chat is the arm most office use looks like: attach the file, ask in chat, get a fluent paragraph. It was the arm where every miss was silent. If a total crosses more than a handful of rows and the assistant can't show it ran a query or code, treat the number as a draft. Ask it to list the rows it used. Counting the listed rows is a check you can do; re-adding its arithmetic isn't one you'll actually do.
A tie between two assistants
Two of my arms scored identically and I'd happily run one of them unattended and never the other. When you compare assistants on a handful of test questions, keep a third column beside right and wrong: what did it do when it missed? "Returned nothing, visibly" is a feature. My read is that failure loudness predicts operating cost better than a few points of accuracy, and no published table I've seen carries it.
A "same model" upgrade announcement
The week your assistant's badge changes from one model to a newer one, the wrapper usually changed too, and the wrapper isn't in the announcement. The coding-agent evidence says wrapper effects are the same order of magnitude as model effects. If quality shifts after an upgrade, in either direction, the model is only half the suspect list.
Any question you framed loosely
"Client entertainment" and "billable" sank every harness in my trial, including the best one, because the model settled on a definition and executed it flawlessly. No wrapper fixes the question it was handed. Define terms before the assistant defines them for you.
A working default
Chat over what you pasted
High confidence
paste-chat got right everything that sat fully inside its 12 visible rows. The paste is a contract: inside it, chat is fine; outside it, silence or confident fiction.
Whole-file totals in plain chat
High confidence
attach-chat's 12 misses were all confident. One dropped row flipped a budget verdict from over to under with no audible change in tone.
Tools without visible errors
Medium confidence
tools-mute never lied, which I value, but it discarded half the workload because one error message was never shown to the one party who could fix it.
Tools with error feedback
Medium confidence
tools-loop hit the model's competence ceiling exactly: 22 of 24, recovering everything the model knew and inventing nothing. In my trial this switch was worth 41.7 points.
Question framing
High confidence
The two loosely framed questions failed in all four arms. This dimension is yours, and no purchase improves it.
A weekly hand-check
High confidence
Every silent failure in this trial is caught by re-checking one answer against the sheet. Ten minutes, no tooling, and it tells you which regime your assistant is actually in.
- Average
- 3.2 / 5
- Binding constraint
- Silent failure. The arms that answered confidently while wrong are the ones I wouldn't leave alone with a budget question, whatever their score.
- Override applied
- None. Split is what a 3.2 with that constraint means in practice.
The average says the tooling mostly works. The binding constraint says the danger is concentrated in one cell: confident answers to whole-file questions in plain chat. Split accordingly, and spend your effort on framing and the weekly check rather than on the model picker.
How to try this yourself
The recorded trial replays without an API key, and the tests recompute every gold answer from the sheet.
git clone https://github.com/shravan1996/generative-ai-model-is-not-the-assistant
cd generative-ai-model-is-not-the-assistant
python3 -m unittest discover -s tests -t .
python3 eval.pyNo network, no GPU, no third-party package. Swapping in your own sheet and questions is a file edit.
- 01
Pick one real file and five questions you know the answers to
- Owner
- You
- Artifact
- A sheet and an answer key, written before any assistant sees them
- Signal
- If writing the key is hard, the assistant's answers were unverifiable all along
- 02
Ask the same five in each assistant you actually have access to
- Owner
- You
- Artifact
- A grid: right, wrong-but-confident, or visibly stuck
- Signal
- Any wrong-but-confident cell on a whole-file total
- 03
Ask each assistant to show the rows it used
- Owner
- You
- Artifact
- Its row list against yours
- Signal
- A missing row explains a wrong total faster than any model theory
- 04
Break one step on purpose
- Owner
- You
- Artifact
- A note on what it did: retried, stopped, or answered anyway
- Signal
- Answered-anyway is the behaviour to plan around
- 05
Set the default and a weekly re-check
- Owner
- You
- Artifact
- One line: which assistant for which job, and one hand-checked answer per week
- Signal
- A silent miss in the weekly check reopens this trial, not the model picker
Ship gates
- The answer key existed before the assistant ran
- Whole-file numbers come from an assistant that showed a query, code, or its rows
- You've seen each assistant fail once and know what it sounds like
Kill criteria
- A confident wrong number from a chat reached a decision
- An assistant answered from a truncated paste without saying so
- You're comparing assistants by model badge instead of by the grid you built
Should you switch assistants when a better model ships?
Not by default. The wrapper around the model moved my trial by more than most model upgrades claim to, so test the new model inside your current assistant on your five known-answer questions, and switch only if the grid improves.
My verdict
Split, for getting answers about your own files in August 2026.
Chat earns the jobs where everything relevant sits inside what you pasted: reading, rewriting, thinking out loud. Whole-file totals, counts, and lookups belong to a harness that can run a query or code and shows its failures, and if none of your assistants can do that visibly, the honest default is a pivot table and no model at all. Between two assistants on the same model, I'd now pick by failure behaviour before accuracy: the one that returns nothing beats the one that returns something plausible, because nothing gets a second look and plausible doesn't. I expect that split to hold whichever vendor's assistant you have open.
I'd also spend exactly none of this month's effort upgrading models. In my trial the whole 25.0-to-91.7 spread was wrapper. In the coding-agent literature the wrapper moved scores nearly as far as the model swap. The cheap experiment is always the wrapper first.
I would change this verdict if:
- Someone shows a current chat app reliably running code for whole-file arithmetic and saying so, at which point plain attach-and-ask becomes defensible for totals and my second scorecard row is wrong.
- A measured trial with real models finds the harness spread collapsing, on office tasks rather than coding tasks, with the harness actually disclosed.
- Vendors start publishing failure-loudness alongside accuracy, which would let you buy the property I currently make readers test by hand.
What to remember
The number I most want and don't have is the silent-truncation rate of the major chat apps: how often a long paste is quietly cut before the model sees it. If you've measured it, or if you run the five-question grid and your chat app beats my expectations on whole-file totals, write to me. This trial's totals are a design, and I'd rather replace them with your measurements than defend them.
Acknowledgements
This article builds on coding-agent harness research linked below, especially Claw-SWE-Bench, whose fixed-protocol design is the reason anyone can say "the harness moved the score" with a straight face. The framing owes a debt to my own earlier rebuild of that result, where the mechanism first surprised me.
The prototype is original, dependency-free Python. The scripted model, the sheet, and the questions were written for this article and share nothing with the cited papers' code.
References
Research
- Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks. Zheng et al., June 2026. The fixed-protocol model and harness sweeps quoted above.
- Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories. Khatri, July 2026. The null result on instruction files.
- DCAS: Decoupling CLI Agent Scaffolding to Internalize Planning across Scaffolds. Thangarajah, Chen, and Hassan, August 2026. Harness coupling surviving into fine-tuned weights.
- Stop Comparing LLM Agents Without Disclosing the Harness. Zhang et al., May 2026. The disclosure argument.
Earlier on this site
- The harness decides what a coding agent looks capable of. The research-engineer version, with the patch-collection rebuild.
This trial
- Source, sheet, questions, and tests
- Exact tested commit
- One scripted model, four harnesses, 24 questions, run 21 August 2026. Deterministic, no model API.
Sources last checked: 2026-08-21
Before you argue about which model is smarter, check who decided what it was allowed to see.