The mill in the dark

I almost published four honest pictures. Then I opened Qwen's file, hit render, and got a black frame.

For a stupid second I thought the GPU had died.

The GPU was fine, which made the black frame worse.

There was no camera in the scene. There were no lights. Qwen3.8 Max had built a mill in the dark, saved the file, and stopped. So I copied it, dropped in a camera and a sun, and rendered the copy. The score did not move. The picture you are about to see in Figure 4 is mine. I made it so this article would have a mill instead of a rectangle of nothing, and I still don't know if that was the right call. People remember pictures. They skip captions. If you catch yourself thinking the fourth mill is "a bit far away but basically fine," you've already been had. It's far away because I put the camera there. It's lit because I put a sun there. The model chose neither.

I used to treat a good picture as the proof. Open it, squint, keep the run. I've done that for years, on work that was less photogenic than a windmill, and I was wrong in a small way that turns out to be the whole job. A good picture can be a lie I made at 11pm because a black frame looks like a broken article. A passing score can belong to a mill that is a different building from the one you asked for. The only thing I trust now is a fresh Blender, opening the file the model actually saved, answering fourteen yes-or-no questions about what is in the scene, and then replaying the successful Python from empty to see whether the story still holds.

Thursday, 10 September 2026. Same instruction to GPT-6 Astra, Claude Opus 5, and Qwen3.8 Max: rebuild the windmill, from nothing, by writing Python that talks to Blender. No window. No person clicking. Fifty tries. Twenty minutes. A checker that never looks at the picture. One pictured run per model.

I am not ranking them. One run each, three different ways of calling the same tools, is a setup I was still shaking down, and it happened to spit out pictures. Opus ran on a subscription, so that dollar figure later is reconstructed: don't quote it as measured. I'm also not teaching you Blender. The Python is the product. The argument is what you're allowed to do with it once a model has typed it.

The mill we were supposed to be thinking about

This one is not a model. It's the human mill, rendered on the same Blender as everything else, and I don't score it. I look at it the way you look at a photograph of a house you're about to rebuild out of plywood.

The grass is a lie too, in a smaller way. The original texture never showed up, so the ground is just green. The sky is borrowed. I still catch myself treating this frame as "the real mill," which is funny, because even the real mill is missing two of its clothes.

The models were not asked to import those purchased trees. They were asked to build a mill from simple shapes, in an empty scene, and to survive a checker that looks for objects, materials, a camera, lights, and whether anyone parented the blades into orbit. A person will hold the output against this picture with their eyes. The machine will not. I had assumed, going in, that "pass" meant "recreated." I was already wrong and I hadn't started a run yet.

How you even do this without cheating

A new Blender file comes with a camera and a lamp, like a hotel room that already has a light. If you start there, a model can "succeed" by never touching lighting. A stray save can also overwrite the task, and you will not notice until a later run is graded against a file the model wrote. So every run starts from empty, on purpose. The emptiness is the test. Qwen's missing camera is only a missing camera because I refused to lend it one.

Then five tools: write code, look at the scene, look at one object, take a preview, save. The Python keeps running between steps, which feels generous until a step crashes after creating a material, and then every later "success" is standing on a ghost. The model writes, reads what came back, corrects. What Blender said stays in the tool result, not in the model's little speech. Hide the result in the wrong place and you're manufacturing a sample nobody can learn from.

Fifty calls. Twenty minutes. Both Qwen runs in this session died on the clock, which I mention because "the model ran out of ideas" and "the model ran out of minutes" are different stories and I only have the second one.

Scoring is a second Blender. Fresh process, the saved file, fourteen yes-or-no checks, all of them must pass or the run fails. After every change to those checks I run two sanity tests: our own solution has to pass all fourteen, and an empty scene has to pass none. If either test breaks, I've been labelling fiction. I know this because I have labelled fiction before. During setup, our own solution passed every check while the tree canopy and the windmill blades had been flung into space by a parenting bug. I sat there with a green 14 and a sky full of leaves, feeling like an idiot, and then I added checks for placement. Charm is still not on the list. It can't be. I don't know how to write a check for "this mill has charm."

An earlier version of this work came back as hand-made: someone thought a person had built the mill and called it generated. I built the rest of the pipeline so that sentence can't be honest. The model writes every line. A machine checks the result. What the model was told gets frozen at the start of the run and checked again at the end, so you cannot score it against a rule it was never given. None of that tells you whether you'd keep the mill. A person still has to look. I just don't let the person be the first look anymore.

Astra finished while I was still watching the logs

Fourteen calls. Zero errors. About seven and a half minutes. I had time to think the setup had short-circuited.

I expected a slightly worse version of Figure 1. A lattice mill with clumsier trees, maybe. Astra looked at the same instruction and built a Dutch cottage. Cream plaster, teal cap, four sails, a stone path, daisies in the grass, trees that look like they were borrowed from a mobile game. As a picture it's the most finished of the four. As a recreation it's a different building on a different continent.

I stared at the daisies for longer than I will admit.

They're good daisies. They're also a tell. The model wasn't copying the reference. The model was doing "windmill, outdoor, pretty," which is a prompt an image model would understand, and then it happened to also satisfy fourteen questions about the scene. I had been treating the human mill as a target image. The checks are a target state. A cottage with a camera, lights, a rotor, materials that are more than a flat colour, and objects that sit on the ground, will pass. A lattice mill whose blades have been parented into the sky will fail, even if a thumbnail still "looks like a windmill."

Weirdly enough, the pass looking this different is what changed my mind. I spent a minute hunting for a bug in the checks. There wasn't one. A check that compared pixels against Figure 1 would have failed Astra for being prettier than the spec, which is a sentence I did not expect to type.

Fifty-eight cents. If you came here for a winner, your eye will stop on this picture. Please don't. The interesting mill is the one that isn't a mill you can see.

Opus took the long way and unwrapped the brick

Twenty-eight calls, two errors, about sixteen minutes. I went and did something else. When I came back the mill had a surface.

I didn't predict that the only model to bother unwrapping would also be the only one whose cost I cannot print. Opus 5 ran on a subscription. There is no bill. I reconstructed the usage from the log and checked it against two measured runs. The input side lands within about eight percent. The output side undercounts by something like a sixth to a half, because the thinking never appears. The range later is mostly about whether the run got a discount for repeating itself. My guess is the truth sits near the lower end. Call it roughly a dollar. Don't quote it as measured. I'm already tired of that sentence and I still need it.

Look at the sails. That's wood grain wrapping a blade, which only happens if someone unwraps the mesh. Astra's mill is charming. Opus's mill is the one I'd zoom in on if I were trying to convince a skeptic that a language model had touched a 3D file and not just a picture. I still wouldn't ship the textures as "as good as the human mill." That mill used bought maps. Prompting will not grow a bought texture out of a cube. I keep saying this to myself because the temptation, looking at that brick, is to ask for one more pass. One more pass is how you get a pretty file you can no longer replay.

Then I made the lie

Qwen used 32 calls, raised 8 errors, and ran forty seconds past the limit. Fail. Nine of fourteen. It did build geometry. There's a mill, two round trees, a pine that I accidentally made poignant by putting the camera in the grass. I had assumed the interesting failure would be ugly: blades in the dirt, a floating canopy, plastic wood. A missing camera was not on my list. To my surprise, a missing camera is worse than ugly. Ugly you argue about. Dark you never see, so you never argue, so the run never enters the gallery, so the next person looking at "what the models did" is looking at a set with the failure already edited out.

Would Qwen have passed if we'd started from Blender's hotel-room file, lamp included? I didn't run that. I can tell you why I won't. A free camera would have given me a picture. It would also have scored a model for accepting a gift. I expect the missing camera is only a real failure because the start was empty on purpose.

The photograph is the failure you can see, once I cheat. The replay is the one you can't see in any of these pictures.

The cheap mill and the mill I cannot price

I priced from what the APIs actually billed. Nothing here is guessed from how long the run took or how many times someone called a tool. For Qwen, the vendor's own number matched ours to the cent, which I found disproportionately comforting, like a receipt matching a restaurant bill after a confusing meal.

A failed run that is cheaper than a passing one is a very ordinary fact, and the ordinary fact is the one that will get quoted if someone reads only the cost column. The cost column does not contain a camera. I don't know how to put a camera in a spreadsheet. I tried, rhetorically, and felt silly.

I would not have built the second check if Astra had gone first

The successful Python gets stitched into one script, replayed in a fresh Blender from empty, and checked again. Step by step, Qwen's log looks clean enough to keep. Two of its steps errored partway through, after creating a material and before the line that blew up. Later steps used that material and were recorded as successes. Stitch only the successful steps and the material is never created, so the script crashes on replay.

I sat with that for a while. The log said progress. The file had a mill in it, once I added a camera. The Python, the actual thing you'd train on if you shipped the successful steps, did not run. What I did not expect was the shape of the lie: not a step that failed out loud, a step that failed after doing work, then a later success standing on the work. If you train on that transcript, you're teaching the model a world that only existed because of a crash. The running Python remembered the material. The replay does not.

I would not have invented the replay check if the first messy run had been Astra. Astra's 14 calls all succeeded. Stitching them is boring.

Boring is the dream, and I mean that as a compliment.

Where do you look, then, if the picture is untrusted? You look at the saved file in a process that did not build it, and you look at the Python you would actually train on. My read of this session is that those two looks are the only ones I would defend to someone who has only seen Figure 4. Ugly is a reviewer problem. A mill that looks fine in a picture I took, whose log looks fine, and whose Python does not run, will enter a training pack if you select on screenshots. Then the next model will do the same half-finished work, and someone will write a nicer caption.

What I'd actually keep

A run is a candidate only if the machine can prove it is reproducible. A human then judges what the machine cannot, which is whether you'd be embarrassed to hang the mill on a wall.

The machine gate is simple, and all of it has to be true: every check passed, the replay matches, it was graded on what it was told, it said it was done, it saved the file, it stopped because it finished, and the task didn't drift underneath it. Of the five runs in this session, three clear that gate: both GPT runs and Opus 5. Neither Qwen run does. The pictures above are one run per model. The gate is counting the ones I didn't hang on the wall.

The person gate is a person, and I mean a person who is allowed to be rude. The checker will not score charm. A scene can be ugly and pass. A reviewer looks at the model's own picture, not at the one I made, and can reject on looks alone. My impression is that Astra survives the machine and still loses the person if the brief was "recreate the lattice mill," because Figure 2 is a cottage. I'd keep it for a pack whose brief was "a complete mill that a fresh Blender can open." Those are different jobs. The checks will not tell you which one you meant. Write the brief down before you start scoring, or the person gate becomes a vibe.

The Python is the product, so I keep it three ways, generated from the same steps, with the replay proving they agree: each step as it ran, the same code sitting inside the log you'd train on, and one stitched script that has to run from empty. Then the saved scene, the model's own pictures, why each check passed or failed, the replay, a frozen copy of what the model was told, the bill, and what Blender printed back, so a later model can learn recovery instead of only success.

For the pack as a whole I want a fingerprint of every file, the licences for the borrowed sky in one place, and the checker plus our own solution, so someone else can re-run scoring without trusting me. Our solution must pass all fourteen. Empty must pass none. If those two sentences ever stop being true, burn the pack.

The things I am not pretending

One pictured run per model. Five runs in the session. Nothing here is a leaderboard. If you put "Astra wins" in a slide I will know you didn't read the daisies paragraph.

Three different pipes. Instruction, tools, budget, checker: identical. How the model was called: not. "Opus takes more calls" has to travel with "Opus arrived through a different door."

I cannot pin a rerun. I already said this. I would most like to be wrong about it, because a pinned rerun is how you find out whether Astra's cottage was a mood. I don't have that rerun. I have a cottage.

The Opus cost is reconstructed. The textures trail the human mill because the human mill used bought maps. And the pictures, including Figure 4, still do more work than I'm comfortable with. I used them because you cannot write about a mill you cannot see. I labelled the one I faked. I do not know how to make a reader believe a caption more than a sky. If you remember one mill from this piece, I am afraid it will be the one I lit.

So do you keep the run?

Keep it if a fresh Blender can open the save, the fourteen checks pass, the successful steps replay from empty, and a person looking at the model's own picture wouldn't be embarrassed. Astra and Opus clear the machine side of that in this session. Whether they clear the person side depends on whether you asked for Figure 1 or for a mill that exists. I'd say recreate-the-lattice is a no for Astra, and a complete-mill brief is a yes.

I might be too fond of the daisies, and I'm allowed to be.

Don't keep it if you had to add a camera to see it, or if the stitched script crashes. Qwen fails both. Being cheaper is not a personality.

I would change my mind if a second task, on the same setup, showed replay matching while the saved file was still missing a camera. I don't expect that. I would also change my mind if a pinned Astra rerun produced the lattice mill, because then "pass" and "recreate" might actually be the same word, and I've spent this article telling you they aren't.

If you have a run that passed by looking at it, send me the replay. I want to know whether I learned this too hard from one windmill on one Thursday, and from a picture I still feel slightly guilty about hanging next to the honest three.

References

10 September 2026. Same empty scene, same fourteen checks, same time limit, three models. Astra: fourteen calls, no errors, fifty-eight cents, every check passed. Opus: twenty-eight calls, two errors, roughly a dollar that I did not measure, every check passed. Qwen: thirty-two calls, eight errors, fifty-one cents, nine of fourteen, and Figure 4 is a picture I lit. Five runs in the session. The machine gate let through both GPT runs and Opus. Neither Qwen run. Two measured bills, $1.09. The whole session, including the runs I threw away, about $2.85.