The teacher said no

At 15:20 on Saturday, 3 October, I started a job that asked Claude to write short reading notes for 1,608 old stock-exchange filings. Each note was meant to say where a filing keeps its numbers: which table, which column is this quarter and which is a year ago, what unit the figures are in. A small model running on my laptop would then learn to read by studying those notes.

Within forty seconds, five requests had come back with the same reply: "claude-sonnet-5-5 can't help with this." My job runner had been written to treat five failures in a row as a usage limit, so it printed "waiting 31 min for the teacher to come back" and settled in. It would have waited forever. The error linked to Anthropic's usage policy, which lists "model distillation" among the things that need Anthropic's permission first, and my prompt said in plain words that the notes existed to train a smaller model.

In the same quarter of an hour, the other half of the plan died too. Fine-tuning that small model on my Mac ran out of GPU memory before the first training step, and kept running out at every smaller setting I tried.

What came next took about twenty hours. By 11:07 the next morning, a fine-tuned copy of Qwen3.5-9B had passed a test I'd written down before reading any of its answers. On 200 filings that no step of the process had trained on, used for any choice, or even uploaded, it caught 55 of the 61 filings Claude calls buys, against 51 for the model it replaced, with half the false alarms. Its teacher was an open-weights Qwen model on a rented A100, and the cloud bill came to roughly $25–30 of free credit by my estimate. Claude's answers graded every exam along the way and trained nothing. I named the new model Argus, and since Sunday afternoon it has read every filing outside Claude's own categories in a pipeline I run called Golden Egg.

Two warnings before the details. Every number here measures agreement with Claude, which is a reader, not the market: I don't yet know whether Argus's buys make money, and none of this is investment advice. And it's one narrow job, filling in a fixed form from Indian exchange filings, so read it as a recipe for that kind of job rather than a verdict on Qwen. The code lives in my private project repository, wired into a Slack channel and a broker, so there's no public trial repo this time; the steps near the end are written so you can rebuild it around your own documents without mine.

By Sunday morning it was catching more buys

The fine-tuned model beat the live one on every measure I'd declared, on a test set it had never seen: 55 of Claude's 61 buys caught instead of 51, three false buys instead of six, the event type matched 88.0% of the time instead of 83.5%, and the amount matched 87.9% of the time instead of 72.7%. On the filings where exactly one of the two models agreed with Claude, Argus was that model 22 times and the live model 10 times, which a one-sided sign test puts at p = 0.025.

It also fixed the failure that sent me down this road. The day before, I'd tried to teach the same 9B model by pasting Claude's reading rules and worked examples into its prompt. That made it more careful and cost it three of eleven buys. On those same 150 filings, Argus caught ten of the eleven.

Three things didn't go the way I planned, and they're most of the story. Claude wouldn't be the teacher, so an open model was. The reasoning notes and the on-policy correction round I'd asked for never ran, because the evidence talked me out of both. And the teacher that won was only just good enough: it beat the untrained model by exactly the margin I'd set, and a student can't outread its teacher.

The free reader and the rationed one

Golden Egg reads every fresh filing on India's National Stock Exchange. The models in it never decide anything. They fill in a form: what kind of event the filing announces (an order win, quarterly results, an acquisition, a credit rating), the amount it states in its own unit and currency, the counterparty, which way profit moved, any adverse facts such as a penalty, and a few quotes copied word for word from the document. Code does the rest. It checks every quote really is in the text, converts the amount into crore, and runs a fixed rule (version 2.2.0) that turns the facts into a confidence score and a buy or an ignore. No model ever sees that rule.

Two models do the reading. Claude, as claude-sonnet-5-5, reads the handful of categories where most of the buys come from: results, board meeting outcomes, acquisitions, mergers, order wins, credit ratings and investor presentations. A 9-billion-parameter Qwen, qwen3.5:9b, runs in Ollama on my 24 GB Mac at temperature 0 with a fixed seed, and reads everything else. It's also the backup whenever Claude hits a usage limit, at which point everything lands on it.

Claude is better and rationed. Qwen is free and always on. So the question underneath all of this was whether the free reader could get closer to the rationed one. "I want qwen model to be as accurate as Claude" is how I put it to my coding agent on Saturday evening, and I meant it more literally than was sensible.

That agent, Claude Code, did most of the typing for everything below. It proposed each stage and its gate; I approved them, sometimes after a round of questions. Where a decision in this piece was mine, I've quoted what I actually said.

Telling it the rules only made it timid

The first attempt changed only the prompt. Overnight on 3 October, Claude read 3,702 old filings, every one either model had ever touched, and its answers became labels. Claude then wrote thirteen short reading rules from the older filings where Qwen had disagreed with it, and each taught read put those rules in the system prompt with three similar worked examples in front of the filing.

On the 150 newest filings, held out, the taught Qwen agreed with Claude on every field 70.0% of the time instead of 62.7%, and its false buys fell from seven to two. It also caught six of Claude's eleven buys where the plain model caught nine. I'd written down beforehand that losing buys meant the switch stayed off, so it stayed off. The prompts were 2.6 times longer, too, and the median read went from 17.5 seconds to 32.5.

My read was that the rules worked as brakes. If I wanted Qwen to read differently, rather than more timidly, I'd have to change its weights. At 14:38 I asked: "I want to do distillation of claude finance capabilities in qwen, can we start that".

So what kind of distillation was I actually doing?

At first, the simplest kind: copying the teacher's finished answers, because an API never shows you the probabilities behind them. Anything richer has to be bought with extra calls, extra permissions, or a teacher whose weights you can run yourself.

"Distillation" covers several quite different things, and I didn't have them straight until I asked "what kind of distillation are we doing? Is it appropriate one to get us to the results?" The answer I got back was refreshingly unsalesy: the simplest kind, the right first step, and probably not enough on its own. Here are the four kinds that mattered, and what happened to each.

The case for answer-only was that the job is narrow and the output is a fixed form, which is exactly where training on a teacher's outputs tends to work. The case against was specific. The misses that hurt looked like reasoning errors, such as reading the year-ago column as this quarter, and a finished answer doesn't show which column the teacher used. The buys were thin: a few hundred among thousands of routine filings. And the deepest objection was one nobody could answer that afternoon: copying Claude only helps if Claude's reads earn money, and nobody had checked.

I said "ok, run the local QLoRA first but also add rationale distillation and on-policy correction". Within the hour, the first of those had failed for lack of memory and the second for lack of permission.

The afternoon both plans died

The local plan was QLoRA: keep the 9B's weights frozen in 4-bit form and train small LoRA adapters on top, using Apple's MLX library. On paper it fit. The configuration trained adapters of rank 16 on sixteen layers with sequences up to 6,144 tokens, which is 21.6 million trainable parameters, 0.24% of the model. The first validation pass took 454 seconds and reported a loss of 0.800. Then:

RuntimeError: [METAL] Command buffer execution failed: Insufficient Memory

Four more probes followed, shrinking the run each time, down to four trained layers in one and 1,024 tokens in another. Every one died the same way, with most of the memory apparently still free: about 87% at the smallest setting, by the probe's own reading. My guess is the backward pass through Qwen3.5's hybrid layers, which mix ordinary attention with a newer recurrent kind called Gated DeltaNet, in the version of MLX I had. I never confirmed it, and a newer MLX might well train it fine.

Then came the refusal. My coding agent stopped the job and told me two things I needed to hear. It wouldn't reword the prompt to hide what the notes were for, since that would be working around a safety control. And the same rule reached further than the notes: my 3,702 labels had been collected through Golden Egg's ordinary reading prompt, but using them as training data was the same act as asking for training notes.

So I set a rule that shaped everything after it. Claude's answers became an answer key: they grade, they never train a model, and they never leave the Mac. Adopting it cost nothing but the plan, because no model had trained on a single Claude answer yet. The MLX run had never finished a step.

I needed two replacements: an answer key that no model wrote, and a GPU with enough memory. At 15:26 I replied "go with XBRL answer key", and asked whether any free cloud GPUs would do. The free notebook GPUs offered 16 GB, enough only for the 4-bit training Unsloth warns against. At 19:34 I told the agent I'd set up a Google Cloud account with $300 of credit.

It could read the table but wouldn't do the division

Listed companies in India also file their results in XBRL, a machine-readable format with tagged numbers. Code can read a quarter's profit straight from it, then take the year-ago quarter on the same basis from Screener, a public financial data site, and work out the profit direction and the change itself. No model touches those figures. The key ended up with 198 rows, and its own checks refused 19 of them. Of the results filings it covered, 68 had both a usable key and a stored text both models could read. On 11 of them the standalone and consolidated statements moved in opposite directions, so a read is graded against whichever basis it matches, and a sign flip only counts if it contradicts both.

Base Qwen, on those 68 filings, got the profit direction right 22 times out of 55 it could be graded on, flipped the sign 12 times, and came within one point of the true profit growth zero times out of 46.

Zero surprised me until I looked at the rows.

The model had mostly left the growth field empty. The schema asked it to read two numbers from a table, divide them, and write a percentage, and a 9B model at temperature 0 mostly declined to do the division. I had assumed the 9B couldn't read a results table. It could read the table. It wouldn't do the sum.

So, before any teacher read anything, the schema changed. Six new fields ask the model to copy figures instead: which statement it read, the unit, and profit and revenue for this quarter and the same quarter a year ago. Code computes the growth and the direction. The same untrained model, re-reading the same 68 filings with the new fields, got the direction right 32 times out of 55, flipped it 8 times, and landed within a point of profit growth 21 times out of 46.

I expect the copy schema to matter more in the September quarter, when a fresh batch of results filings arrives with XBRL keys, than it did in the ship test further down, which had no XBRL keys to grade it against. Its order mattered more than the change itself. Teacher labels teach whatever schema they're written in, so if the schema had changed after labelling, a day of teacher reads would have taught the old habit. The switch shipped behind a flag, off by default. I also said yes to a second decision the same evening: the new model would serve Golden Egg only, while the research agents elsewhere in the project kept the original 9B.

A GPU that turns itself off when I forget to

I'm not worried about cloud bills in the abstract. I'm worried about the GPU I forget about. My instructions at 20:00 were blunt: alerts for every $50; when the free credit ends, "stop the VMs"; when a job is done, stop "everything that uses the credits"; and before any of it, "cross-check if the entire workflow is correct and if it gives the desired results".

What got built was a budget that alerts every $50, wired to a small cloud function that stops every VM at ₹25,000 and detaches billing at ₹27,500, just under the credit. The machine was a Spot a2-ultragpu-1g, one A100 with 80 GB, in us-east4-c, because us-east5 had no Spot A100 free that night. Spot machines are much cheaper and can be taken away at any moment, so every job ran inside a wrapper that uploaded what it wrote and shut the machine down on success or failure. A watchdog shut it after 30 idle minutes, every VM carried a maximum run time, and the storage bucket deleted anything older than 60 days.

The cross-check produced a plan with a table at the top: each way the previous attempt had failed, beside the control meant to stop it happening again. The taught reader lost buys, so every gate now put buys caught, false buys and sign flips ahead of field agreement. Training text had been cut to 8,000 characters while the live lane reads up to 101,000, so training would run at live length. Claude had refused, so the teacher would be open-weights. The model computed percentages badly, so it would copy figures. MLX ran out of memory, so training would use full bf16 weights on the A100.

The boring checks caught two real bugs

None of the expensive stages could start until six cheap checks passed, four on the GPU machine and two on the Mac. Two of them found real bugs.

The template check sent the prompt exactly as the training code would format it to Ollama in raw mode, and compared the answers with Ollama's normal chat mode. All 20 matched, which meant the text I'd train on was the text the live lane sends. On the GPU, the base 9B served by vLLM parsed 16 of 16 answers, none cut off at the live limit of 700 tokens, with the longest prompt at 46,019 tokens. A first candidate teacher, Qwen3.5-27B, loaded in 299 seconds, shared the 9B's tokenizer exactly, and handed back the top 20 probabilities for every answer token, which is what on-policy training would need.

The memory check passed on its third run. An earlier run ran out of memory because the training script had left the model in evaluation mode, where Hugging Face quietly skips gradient checkpointing. Fixed, a LoRA step on a 46,385-token filing peaked at 50.1 GB and took 45 seconds. The other bug was a context window of 40,960 tokens in the vLLM setup, where the live lane allows 65,536; a table-heavy filing can run at 1.8 characters per token, so long filings would have been silently cut.

If you're doing this with Ollama, the export check is the one I'd underline. Ollama can import a fine-tuned model directly from Hugging Face's safetensors files, and I'd assumed that was the route. The imported model came back on Ollama's MLX engine, quantized to one of a short list of formats, and refused every request with "structured output is unavailable". Golden Egg decodes every read under a JSON schema, so that route was dead. The route that worked converts the weights with llama.cpp's own script (release b11146), quantizes to Q4_K_M, the same format as the live model, and imports that file with Qwen3.5's renderer and parser.

One honest footnote on that check. The plan first asked for identical answers on 18 of 20 filings. The version that ran asked for agreement with Claude within three points of the live model, on every field, across 60 filings. The converted base gave identical answers on 49 of 60, with the rest differing because two different quantizers round the same weights slightly differently, and it matched or beat the live file on every field. I believe the new bar was the right one. The bar still moved between the plan and the script, so I'm recording it here.

By then the cloud spend was about $3.

The biggest model didn't win

A distilled student can approach its teacher but can't pass it, so the teacher had to be better than the student on the answers that mattered, and by a margin I'd write down before reading anything. The gate used one number, S: Claude's buys caught, minus false buys. The winning teacher had to score at least five more than the base 9B, get the XBRL profit direction right at least as often, flip it no more often, and match Claude's event type at least as often. Otherwise the plan stopped and reported the gap.

The set held 317 filings: the 200 development filings, 50 of Claude's buys drawn from the training pool, and every XBRL-keyed filing, none of them from the test set. The readers were the base 9B in three settings (the table shows the live one; the other two scored 50), Qwen3.5-27B, Qwen3.5-35B-A3B, Qwen3.5-122B-A10B, and two newer 27B models, Qwen3.6 and Qwen3.8, which I added before anything was graded because they share the 9B's tokenizer. I left out Kimi K3, at 2.8 trillion parameters the largest open model I looked at: about 1.4 TB of 4-bit weights needs eight to sixteen top GPUs against my quota of one, and its tokenizer is different, which rules out token-level distillation.

The first variant came back with seven answers cut off at the token limit, and none of them were reading errors. Five had written "auditor_resignation" into the adverse-facts list over and over until they ran out of tokens. Two had written whitespace until they ran out. vLLM's JSON grammar allows unlimited whitespace, which Ollama's doesn't, and the schema's lists had no maximum length. Every variant was re-run with whitespace disabled and the lists bounded (six quotes, ten flags), with the live schema and every threshold unchanged.

The biggest model didn't win, and I didn't expect that. The 122B needed a retry with smaller memory settings to load at all, then caught the most buys of any thinking-off reader and made six false ones. I'm not sure it got a fair shot: it ran as a 4-bit copy, squeezed onto one GPU, and a less cramped setup might have scored differently. The winner was the 35B mixture-of-experts model, which only runs about 3 billion parameters for each token. That made it fast as well as good: later it read all 3,366 training filings in 26 minutes, where the Qwen3.8-27B took 2.2 hours.

It cleared the gate at exactly the minimum: S of 58 against 53 plus 5, direction 38 against 35, sign flips 2 against 6, and event agreement 221 against 220. The margin was one I'd set, and it held by nothing. I let it go on to the next stage, but the number stayed with me. This teacher still missed 15 of Claude's 75 buys, and nothing I did afterwards could teach the student those 15.

The bake-off also settled what to do about reasoning. Qwen's newer models can write out their reasoning before they answer, and I'd assumed reasoning would help most on exactly the filings that hurt, like results tables. Thinking ran at Qwen's recommended settings with a cap of 6,000 reasoning tokens, and then the answer was decoded greedily under the schema.

The reasoning hit the cap on 57–67% of filings, averaging around 5,500 tokens per read, far beyond anything the live lane on my Mac could afford. The 9B with thinking failed the eligibility bar outright, at 96.2% of answers parsed. The 35B with thinking matched Claude's event type more often and caught five more buys. What I didn't expect was the price: 23 false buys, against 2 without thinking. Somewhere in five thousand tokens of reasoning, the model seems to talk itself into the reading where something happened.

After five of fifteen variants, I chose "Thinking off only" for the rest. The teachers that hadn't read yet never thought out loud, and the reasoning notes I'd wanted for training never got written. I suspect a short, disciplined rationale could still help a 9B on results tables. I'm fairly sure this wasn't the way to get one.

A second opinion on every label

The teacher's labels would become the student's entire education, so its mistakes would be taught as faithfully as its insights. The filter was the part I spent longest on.

Both the winning teacher and Qwen3.8-27B, as a partner, read all 3,366 training filings. A label was kept only if the winner's answer parsed and wasn't cut off; its amount was anchored in the text; every copied profit and revenue figure appeared in the filing with all its decimals; only verified quotes stayed, and an answer with none was dropped. On a results filing the XBRL key covered, the key decided. Everywhere else, the partner had to agree on both the buy-or-ignore call and the event type. Claude's answers took no part in any of it.

A smoke test on 117 filings showed why the figure check had to be strict. It caught OCR noise such as "90379" where the filing said 903.79, figures that appear nowhere in the text, and one mis-copied decimal. It also showed that the live code's existing amount check was too forgiving: it had accepted 18.6 because "18 months" appeared in the filing. The training filter got a stricter version of its own.

The agreement filter earned its cost on the bake-off set, where Claude's answers could measure it without choosing anything. The winning teacher alone agreed with Claude 93.3% of the time on buy or ignore and 87.3% on the event type, across every filing. Where Qwen3.8-27B agreed with it, those became 95.3% and 95.3%, on 84.9% of filings. Adding a third teacher barely moved them. A handoff script refused to start training on fewer than 1,500 kept labels or below 90% measured agreement with Claude; it got 2,859 and 98.5%.

Then the set was balanced. Routine filings, such as AGM notices and compliance certificates, made up 1,731 of the kept labels, so they were capped at a quarter of the training set, and every buy was written twice. That gave 1,593 training examples, 418 of them buys, plus 143 for validation. About 1,300 routine labels sat out the first run.

One more thing broke before training could start. The GPU machine's disk had reached 86%, though I'd deleted every model folder I no longer needed. Hugging Face's download cache stores files once in a shared store and links them into each model's folder, so deleting the folder frees nothing. 140.6 GB of orphaned files, from the 122B and two of the 27Bs, came out with a proper cleanup, and later jobs clean up the right way.

Two hours of training, and why I threw half away

The training script is short enough to read in one sitting, and most of its lines are there because something went wrong in an earlier check. The weights stay in bf16, never 4-bit. LoRA adapters of rank 32, with alpha 64, go on every linear layer of the language model, covering attention, the Gated DeltaNet layers and the feed-forward blocks, while the vision tower and the output head stay frozen. That came to 86.6 million trainable parameters, under 1% of the model.

Each example is the live read exactly: the live system prompt and filing prompt, formatted by the chat template the template check proved, with thinking off, followed by the teacher's answer and the end-of-turn token. The loss covers only the answer. The script also computes the output probabilities only at the answer's positions, because the longest filing in the set ran to 58,721 tokens, and the probabilities for every position of that one example would have needed about 29 GB on their own.

def target_loss(model, ids, start):
    """Mean cross-entropy over the target tokens; logits only for the positions that predict them."""
    hidden = model.model(input_ids=ids).last_hidden_state[:, start - 1:-1, :]
    logits = model.lm_head(hidden).float()
    return F.cross_entropy(logits.reshape(-1, logits.size(-1)), ids[:, start:].reshape(-1))

The rest is ordinary: one sequence at a time, sixteen per optimiser step, AdamW at a learning rate of 1e-4 with 20 warm-up steps and a cosine decay, gradients clipped at 1.0. Every 100 steps the adapter, optimiser and scheduler went to the bucket, so a Spot preemption would cost minutes, not the run. A filing that ran out of memory would be skipped and logged rather than ending the run. None did.

Two epochs came to 199 optimiser steps, about 35 seconds each, 1.93 hours in all, 14.7 million tokens, with memory peaking at 59.6 GB on the longest filing. Training loss fell from 0.391 to about 0.054 by the end of the first epoch and 0.027 by the end of the second. Validation loss told a different story: 0.0576 at step 50, 0.0531 at step 100, 0.0531 at step 150, 0.0526 at the end. The second epoch taught the model its training set and almost nothing else.

Each epoch's merged model then read the 200 development filings on the cloud GPU, with live settings, and the Mac graded them. Only 22 of those were Claude buys, so S was noisy, and event and amount agreement across all 200 counted too.

I expected the student to land somewhere between the base model and the teacher, and it surprised me more than anything else in the run. It matched the teacher on S after one epoch, and beat it on event type, 91.5% against 87.5%, and on amounts, 89.7% against 82.1%. Against the base model, it was better on 19 filings and worse on 4, a one-sided p of 0.0013. The teacher, in the same comparison, managed 20 better and 12 worse, p = 0.108.

How does a student beat its teacher? My guess, and I haven't tested it, is the filter. The student never learned from the teacher's full output. It learned from the 85% where a second teacher agreed or XBRL confirmed, so it studied a cleaner version of the teacher than the teacher is.

The second epoch doubled the false buys, from three to six. Every buy was written into the training set twice, so two epochs showed each one four times, and the likeliest reading is that the model learned buys are more common than they are. Epoch 1 it was. The validation loss, flat across that whole stretch, never hinted at the problem.

The clever parts I never ran

At 19:40 on Saturday I'd written "also I want both on-policy and reasoning distillation too". Neither happened, and I'm the one who called both off.

Reasoning went first: thinking made the teachers over-buy, and it ran far past what the live lane could spend per read. The on-policy round, in which the student reads, the teacher scores each of the student's tokens, and the student trains on its own mistakes, was built and memory-tested: a step combining the usual loss with a divergence from the teacher's top 20 probabilities fit at 46,385 tokens. On Sunday morning the agent laid out the case, and I chose "Skip GKD, run the ship test".

The reasoning was simple once the development grade was in. On-policy training pulls the student toward the teacher's judgement, and this student already matched the teacher on buys and beat it on events and amounts. Another round would spend money teaching it to be more like a model it had already caught. The next correction round should learn from something stronger than this teacher, and the September-quarter results season, starting around 10 October, will produce exactly that: hundreds of fresh results filings with an XBRL key.

I went into this thinking the clever parts, reasoning and on-policy rounds, were where the gain would come from. I no longer think that, at least for this job. Plain answer copying from a filtered teacher did all of it.

Ollama wouldn't open the file

The epoch-1 adapter was merged into the full bf16 model on the GPU machine's CPU, converted with the same llama.cpp release the export check had used, and quantized to Q4_K_M. The file was 5,629,108,672 bytes, with matching checksums on the cloud machine and the Mac. Ollama refused to load it:

tensor 'blk.32.attn_norm.weight' not found

Qwen3.5 carries one extra block for multi-token prediction, a side layer that can draft several tokens ahead. Its 15 tensors sit apart from the 32 main blocks. Hugging Face's library doesn't load that block, so the merged checkpoint was saved without it, while its configuration still declared it. The converter wrote a model header promising 33 blocks and delivered tensors for 32.

The fix copies the 15 missing tensors, byte for byte, from the converted base model, which was made by the same release with the same quantization, and refuses to write anything unless the result holds exactly the base file's 442 tensor names, shapes and types. The extra block was never trained, and at temperature 0 it can't change an answer: normal decoding doesn't use it, and when llama.cpp does use it to draft tokens, the main model checks every draft.

Then I compared the new file with the converted base, tensor by tensor, to see what had changed. Every matrix LoRA had trained was different, as it should be. A set of 24 small normalisation weights differed by up to 0.37%, because Qwen ships them in 32-bit and the merge had loaded them in bf16. The embedding and output tables differed by single quantization steps on 15–16% of rows, because the cloud machine's quantizer and my Mac's round slightly differently; their correlation with the base was 0.999994 and 1.000000. Everything else was identical. With the graft, the file grew to 5,780,090,176 bytes and loaded on Ollama's qwen35 engine with structured output working. Its Modelfile, minus my local path, is short:

FROM argus-v1-q4km.gguf
RENDERER qwen3.5
PARSER qwen3.5
PARAMETER presence_penalty 1.5
PARAMETER temperature 1
PARAMETER top_k 20
PARAMETER top_p 0.95

The temperature there doesn't matter in practice. Golden Egg sends temperature 0 and seed 7 with every request, which override it.

Sitting an exam it had never seen

The ship gate went into the plan at 06:55 on Sunday, before anything read the test set. The 200 test filings were the newest 150, filed between the evening of 1 October and the night of 2 October, plus 50 of Claude's buys drawn from older filings so the buy measures had some power. That made 61 Claude buys in all. No stage had trained on them or chosen anything with them, and they'd never been uploaded.

Both models read them on my Mac, in Ollama, exactly as the live lane reads: the copy schema as the output format, temperature 0, seed 7, a 65,536-token window, 700 answer tokens, thinking off, presence penalty 1.5. The live model took about 55 minutes, and Argus about 75.

It shipped, on paper. The switch was still mine, and the agent's report was careful to say so, along with two side effects of flipping it naively that would have reached lanes the test never covered. That turned into the integration work further down. The whole cloud run, estimated from VM hours rather than read off the bill, came to roughly $25–30 of the $300, plus about $1 a day for the stopped machine's disk until I delete it.

What changed, filing by filing

A table tells you a model got better. Only the rows tell you how, so I went through every buy the two models disagreed on.

Argus caught five of Claude's buys that the live model missed. Jubilant Ingrevia's board approved a new subsidiary to buy a 40% stake in another company, which Claude sized at ₹10 crore; Argus recorded the amount and the live model left it empty, so the rule had nothing to size. Transport Corporation of India's board proposed a ₹150 crore buyback, in a filing headlined as a new subsidiary in China; the live model filed it under "other". Electrosteel Castings disclosed a promoter transaction; the live model left the event type empty. Lemon Tree Hotels opened a 65-room hotel in Darjeeling. And Gillette India's results are the cleanest example of what the training changed. The live model copied revenue as 6,855,578,182 and labelled the unit lakh, which is the number multiplied out and then still called lakh. Argus copied 68,555 and 63,946, in the filing's own unit, and code computed the rest.

Argus lost one buy the live model had caught. Thomas Cook's subsidiary Sterling opened a 36-room hotel in Agra. The live model recorded 36 as capacity added, which the rule reads as an expansion; Argus left capacity empty, so the opening had no size. Claude had called it the start of commercial production. One missing number, and a buy became an ignore.

Then there's Pranav Constructions, the small Mumbai builder whose investor presentation started all of this. Claude read the deck, found revenue up 15.7% and profit up 45.5%, and called it a buy. The live model decided the deck was about two redevelopment projects. Argus copied every figure correctly, profit of ₹144 million against ₹99 million and revenue of ₹1,645 million against ₹1,422 million, wrote the growth as 45.5% and the direction as up, and summarised the deck as a results presentation. Then it filed the whole thing as routine compliance.

The facts are all in its answer; the field the rule reads is wrong. I don't know why, and I'd like to. My guess is the training mix, where routine filings outnumber investor presentations many times over, but it might just as well be something about how decks are laid out that I haven't spotted.

Its three false buys were Ambika Cotton, which every version of Qwen I've run calls a buy, and two filings from Ganesh Consumer Products. Ganesh sold its Hyderabad unit for ₹32.5 crore. Argus's own summary says "sold". Its event type says acquisition. The live model made the same mistake on one of the two filings and got the other right.

What about the buys the prompt-taught model had lost the day before? Argus read Aurobindo Pharma's US acquisition as USD 15.247 million, which code converted to ₹129.6 crore, a buy, where the prompt-taught model had written "units" and turned it into almost nothing. It called MPI Manipal's results and both of Pranav's formal results filings buys at 8, matching Claude. And it recorded Leap India's investment as the single ₹4.03 crore Claude recorded, instead of adding two amounts together into a false buy.

Was it the training, or the new form?

Almost all of it was the training. Argus differs from the live model in two ways at once: its weights, and the copy schema it was trained on. To separate them, I'd had the live model read the test set a third time with the old schema. That run stopped at 170 of the 200 filings, and I only graded it on Monday morning, for this article, with the same code.

On those 170 filings, which hold 38 of Claude's buys, the live model on the old schema scored an S of 25, and on the copy schema 26. Argus scored 33. Against the old-schema run, Argus was better on 22 filings and worse on 9, a one-sided p of 0.015. The copy schema against the old one was 6 against 4, p = 0.38, which is nothing. Nearly all of the gain against Claude came from training. The copy schema was built for profit direction against XBRL, which this test set can't grade, so the result doesn't make the schema useless; it means the schema didn't carry the ship test.

The earlier prompt experiment's 150 filings sit inside the test set, so all four approaches can be lined up on them. Plain Qwen caught 9 of 11 buys with 7 false; Qwen with thirteen rules and three examples in its prompt caught 6 with 2 false; plain Qwen on the copy schema caught 9 with 6 false; Argus caught 10 with 3 false, and got 30 of 36 amounts right against 24 for the prompt-taught model. Against the prompt-taught model, Argus was better on 20 filings and worse on 12, a p of 0.108. That isn't significant on 150 filings with eleven buys, and I don't want the cleanness of 10 against 6 to sound like more than it is.

Why I'd discount my own result a little

Most of the reasons to discount the headline are visible from the table, and I'd rather list them than have you find them.

Everything is agreement with Claude. When Claude is wrong, a model that disagrees with it scores as wrong too, and the Claude lane doesn't take a seed, so even Claude doesn't repeat itself exactly. Whether any of these buys made money is a separate question this test doesn't ask.

The test set was split by filing, not by company. When I checked on Monday, 162 of the 200 test filings, and 45 of the 61 buys, came from companies that also had at least one other filing in the training pool. A different filing from the same company isn't a leaked answer, but it's a familiar letterhead. I suspect it helped most on companies that file the same kind of notice every quarter, and I'm not sure by how much.

The test set was built for power, which makes it unlike the live stream. Fifty of the 200 were Claude buys added on purpose, so buys made up 30% of the test set, against about 10% of everything Claude labelled. If you run a model like this on live traffic, a false buy costs you more, relative to a caught one, than this table makes it look.

The sign test rests on 32 filings where the two models disagreed. p = 0.025 cleared the bar of 0.10 written into the gate, which is looser than the 0.05 the prompt test used the day before; it would have cleared 0.05 too, but not by much. Both models are deterministic at temperature 0 with a fixed seed, so re-reading wouldn't add noise. I expect a fresh set of 200 to narrow the gap, probably without closing it.

And the ceiling is real. The open teacher missed 15 of Claude's 75 buys on the bake-off set. Argus can't learn what its teacher doesn't know. Claude still reads the high-value categories live, and nothing in this run argues for changing that.

Calling it Argus, and planning for the day it breaks

At 11:15 on Sunday I asked the agent to "rename the new model as argus and integrate it reliably and safely into golden egg". Most of what followed was about failure, not success.

The lane now asks Ollama for argus:v1, with a plain argus as an alias for the same weights, and the old working name was removed only after confirming the weights matched the ship test. The schema follows the model: Argus always reads with the copy schema it was trained on, whatever the flag says, while Claude and the other lanes keep the old one. The exact build is pinned by its Ollama digest, 77082c1e437caad5, and at start-up the lane runs Argus only if that exact build is installed. Otherwise it falls back to the original qwen3.5:9b with the old schema, and the start-up line says why in capitals.

If Argus disappears mid-run, Ollama's "not found" makes the lane re-read that one filing with the fallback model, in the fallback's own schema, and the row records which model read it. The start-up check now requires only the fallback, so a missing Argus degrades the pipeline instead of stopping it. A future training run becomes argus:v2, with its own ship test and pinned digest; v1 is never overwritten.

Then a check that I'd now run after any change at all. npm run golden-egg:argus-check makes the same start-up decision the loop makes and reads ten test filings through the real production code; the facts must equal the ship-tested answers. On Sunday: 10 of 10 identical, including the Aurobindo acquisition and a Knack Packaging order win, both buys. Pointing the lane at a model that doesn't exist, argus:v9, produced the fallback and the warning, as designed.

One failure had nothing to do with the model. The 5.8 GB file was copied to the cloud bucket for safekeeping, and the upload printed a throughput line, so it looked fine. My organisation forces a fresh Google sign-in every few hours, the session had expired partway through, and the file never arrived. It was re-uploaded after I signed in again, and its checksum matched the local copy, 5,780,090,176 bytes either side, in a folder exempt from the bucket's 60-day deletion rule, next to the 346 MB LoRA adapter.

At 14:50 I said "yes, start the golden egg loop with argus". The first cycle, catching up on twelve hours, fetched 177 filings, found 9 fresh, excluded 3 as routine, sent 1 to Claude and 5 to Argus, with no fallbacks and one buy delivered to Slack in paper mode, which places no order. That evening my Mac's fan wouldn't stop. A backlog job was re-reading 478 older filings with Argus, which held 7.6 GB of memory and 163% of a CPU while it worked. I stopped it at 200. The other lanes still use the original 9B, so on a 24 GB Mac Ollama sometimes swaps between two 9B models, about four seconds each time.

Golden Egg can now turn a buy into a small limit order when its executor is set to live, which is the reason I cared so much that Argus answers the same way twice.

If I had to distill another reader tomorrow

If I had to do this again for a different document, here's what I'd keep. None of the settings are magic; each one is there because something in this run broke without it.

You don't need my code. You need a reading job with a fixed form, a small model you run yourself, a frontier model whose answers you're allowed to grade with, and some patience about order.

1. 01

   Build an answer key you're allowed to use

   Owner

   You

   Artifact

   Stored frontier answers on your own machine, plus any key code can compute from structured data

   Signal

   Every document can be graded by code, and no key answer appears in any file you will train on.

2. 02

   Fix the schema before any labels

   Owner

   You

   Artifact

   Fields that copy figures from the text, with code doing the arithmetic

   Signal

   The untrained small model already scores better against the key, before any training.

3. 03

   Prove the round trip on the base model

   Owner

   You

   Artifact

   The export route, the chat template, and the live decoding settings, tested on the untrained model

   Signal

   The converted base gives the same facts as the model you serve today, and your training prompt equals your live prompt.

4. 04

   Bake off open teachers against a written margin

   Owner

   You

   Artifact

   A gate file committed before the first teacher read

   Signal

   One teacher beats the base by the margin on your rare yes-calls, or you stop and report the gap.

5. 05

   Filter labels with a second teacher

   Owner

   You

   Artifact

   Kept and dropped counts, by reason

   Signal

   Measured agreement with your answer key is high, and the key chose nothing.

6. 06

   Train one epoch and grade it on held-out documents

   Owner

   You

   Artifact

   One adapter per epoch and a graded development set

   Signal

   False positives do not rise from one epoch to the next.

7. 07

   Ship only through a gate written before the test

   Owner

   You

   Artifact

   A test set never uploaded, read in your real serving setup

   Signal

   Every declared check passes. Then pin the build and keep the old model as the fallback.

The step I'd fight for is the fourth. Everything after it is only as good as the teacher, and the bake-off is the one place you find out cheaply.

I'd do it again, and keep Claude marking the exams

Use, for a narrow reading job: fine-tune the small model on an open teacher's filtered answers, and let the frontier model grade it. On this job, that took a free local model from catching 51 of Claude's 61 buys to 55, with half the false alarms, for roughly the cost of a dinner out, where a prompt-only approach the day before had traded buys for caution. Keep the frontier model reading the categories where a miss costs the most, because the student's ceiling is its teacher's.

I'd change this verdict if:

1. On the September-quarter results, graded against XBRL on filings no stage has seen, Argus gets profit direction wrong more often than the untrained 9B on the same copy schema.
2. On the next 200 filings, from companies with no filing in the training pool, Argus no longer passes the same five-check ship gate against the original model.
3. Over the first two weeks of shadow reads, Argus agrees with Claude on buy or ignore less often than the 92.0% the original model managed on the test set.

The thing I trust least in all of this is the filter. Two teachers agreeing is a much better signal than one, and it's still two models from the same family that can be wrong the same way. My impression is that Ganesh Consumer selling a factory, with Argus writing "sold" in its summary and "acquisition" in its event type, is exactly the kind of error two Qwen models could share and a filter like mine would wave through. Version 2 will train on the September quarter with XBRL as the extra judge, which helps for results and does nothing for an asset sale.

So the question I'm carrying into v2 is a narrow one, and I don't have an answer to it: what catches a confident mistake that two teachers share, without asking the model I'm not allowed to learn from? Ganesh's two filings stay in the test set, unchanged, as the first check for whatever idea works. Send me yours. Every number above comes from a file I can reopen in a minute, so if one looks off, say which one and I'll check it.

Acknowledgements

This builds on the Qwen team's open-weights models, which made an open teacher possible at all, and on llama.cpp, vLLM, Hugging Face Transformers and PEFT, Ollama, and Unsloth's public fine-tuning notes. Claude Code did most of the building and, unprompted, ruled out rewording a prompt to get past a refusal, which turned out to be the most useful thing it did all weekend. Errors in the pipeline and in this reading of it are mine.

References

Models and tools

• Qwen3.5-9B on Hugging Face. The student, trained in bf16 and served as Q4_K_M.
• Qwen3.5-35B-A3B-FP8 on Hugging Face. The selected teacher.
• Qwen3.8-27B-FP8 on Hugging Face. The partner teacher used to filter labels.
• Qwen3.5 on Ollama. The live qwen3.5:9b model Argus replaces in Golden Egg's reading lane.
• llama.cpp release b11146. The conversion and quantization route for both the base and Argus.
• vLLM. The server for teacher and development reads on the cloud GPU.
• Hugging Face PEFT. The LoRA implementation used for training and merging.
• MLX LM. The Apple-silicon training library behind the failed local attempt.
• Unsloth, Qwen3.5 fine-tuning guide. The advice against 4-bit QLoRA on Qwen3.5, and the 22 GB figure for a bf16 LoRA of the 9B.
• Google Cloud Spot VMs. The preemptible A100 the whole run used.
• Anthropic Usage Policy. The policy the refusal linked to.
• NSE corporate announcements. The filings Golden Egg reads.
• Screener. The year-ago quarters behind the XBRL answer key.

Background

• Distilling the Knowledge in a Neural Network. Hinton, Vinyals and Dean, 2015. Training a small model on a large model's probabilities.
• Sequence-Level Knowledge Distillation. Kim and Rush, 2016. Training on the teacher's finished outputs, the kind Argus used.
• Distilling Step-by-Step!. Hsieh et al., 2023. Training on the teacher's rationales as well as its answers.
• On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. Agarwal et al., 2024. The GKD method behind the skipped round.
• LoRA: Low-Rank Adaptation of Large Language Models. Hu et al., 2021. The adapters Argus was trained with.
• QLoRA: Efficient Finetuning of Quantized LLMs. Dettmers et al., 2023. The 4-bit method the local attempt used.

This run

• Golden Egg distillation, 3 to 4 October 2026, private FIRE repository: the plan and its declared gates, the teacher bake-off (317 filings), the label filter (2,859 of 3,366 kept), LoRA training on one A100 80 GB (199 steps), the export, and the ship test (200 unseen filings, 61 Claude buys).
• My own grading on 5 October 2026, with the project's grading code: the per-filing comparison of Argus and the live model, the 170-filing comparison of the old and copy schemas, and the company overlap between the test set and the training pool.

Sources last checked: 2026-10-05