Seven queues it had never seen

On Friday, 23 October, OpenAI switches off every fine-tuned copy of gpt-4.1-nano. The suggested replacement, gpt-5.6-luna, isn't on OpenAI's list of models you can fine-tune, and on 6 January 2027 the company stops taking new fine-tuning jobs from anyone. A fine-tuned model keeps answering only until its base model retires.

Suppose your team at a bank or an insurer did the sensible thing last year and fine-tuned a small hosted model on its own tickets, through the company's cloud contract. That model now has a retirement date nobody on the team chose. Claude was never the way around it: Anthropic's API doesn't offer fine-tuning, and the one Claude model you could fine-tune on Amazon Bedrock, Claude 3 Haiku, reached end of life in its main regions on 10 September.

What's left, for a team that wants a model trained on its own work, is an open-weights model whose files it keeps. So the question I wanted answered is the one underneath: should a team inside a big company fine-tune a small model at all, and if it does, what is it signing up to own? I ran the textbook case on my laptop, twice: on the day a team starts, and ninety days later, after the business has changed its mind. The job is routing a bank's customer messages into queues, using BANKING77, a public set of 13,083 short messages labelled with 77 intents. On day one there are 70 queues. Claude Haiku 5.5 and Claude Sonnet 5.5 got the queue names and 2,800 messages they'd never seen, and later a written line on each queue. Qwen3.5-0.8B, an open model small enough to run on a phone, was fine-tuned on 8, 32 or every labelled message per queue and routed the same 2,800. On day ninety, the bank adds seven queues.

A few things this piece won't do. It won't rank small models; I tried one, chosen because it's current, Apache-licensed and tiny. It doesn't cover agents, retrieval or chat assistants, only the narrow, repetitive sorting jobs where fine-tuning has always made most sense. Two sections lean on banking rules and the EU AI Act, and I'm reading those as an engineer who has to ship something, not as a lawyer. And BANKING77 is clean, short and English; your tickets aren't.

Training beat the names, then one line caught up

Trained on 32 examples per queue, Qwen3.5-0.8B routed 86.9% of 2,800 unseen messages to the right queue, against 81.5% for Claude Sonnet 5.5 and 76.0% for Claude Haiku 5.5, both given the list of queue names. That looked like a clear win for training, until I gave Claude one written line on what belongs in each queue. Sonnet then reached 88.3%, a little ahead of the trained model, and Haiku 82.9%. Only the model trained on every label, at 93.1%, stayed clearly in front.

When the bank added seven queues, Claude handled them as soon as I added seven names to its prompt. The trained model sent all but six of those customers to the wrong queue, and its own confidence flagged only about half of them. Retraining it with eight labelled examples of each new queue got 61% of them right, short of the 70% I'd set and of both Claude models.

So my answer for a team in a big company is still Split, in a different order from the one I started with. Describe your queues to the frontier model you already pay for, first. Fine-tune a small open model only when you can export hundreds of past tickets per queue, keep the frontier model reading beside it for new queues, and name the person who retrains before you launch. Cost decides little: at list price, Sonnet with the queue lines routes 10,000 messages for about $8.72, and Haiku with the names alone for 38 cents.

Every closed ticket is already a label

Message routing is the job fine-tuning was made for. The list of answers is fixed and short, and the volume is high, often millions of messages a year at a large bank. Above all, the training data already exists. Every ticket an agent closed carries the queue it ended up in, and every ticket moved from one queue to another is a correction someone made by hand. A team doesn't have to commission labels. It has to export them.

The person I have in mind runs that queue: an operations lead, or the engineer who keeps the routing rules working. They already have an enterprise chatbot. Training something of their own is a different decision from choosing one, because a trained model is a thing you then have to look after.

Six years of small-beats-big claims, with the dates on

BANKING77 has been public since 2020, and several "small model beats big model" claims have been made on it.

Read with the dates on, those numbers say more about what each side was given than about any model's name: GPT-4 moved nine points on its choice of examples, and BERT ten on its number of labels. Who wrote the training answers matters too, and that changed my mind about something I wrote ten days ago. In the Argus piece I said a student trained on a teacher's answers can approach that teacher but not pass it. Preply's 0.8B learned from GPT-5.4's annotations and then beat prompted GPT-5.4. As far as I can tell from the paper, the training answers had been reviewed, re-annotated and filtered by a rule-based checker first. A student can pass its teacher's first drafts once someone has thrown the bad ones away, which is more or less what a bank's closed tickets already are.

None of these studies asks what happens after training day, because a paper trains once and reports a number. A team trains once and then lives with the model while the business renames products, splits a queue in two and launches a card.

Where a company can still fine-tune, and keep what it trained

The options narrowed this year, and the details matter to a team picking something small.

A hosted fine-tune lives as long as its base model, on someone else's calendar. And the licence on an "open" model matters more in a big company. Qwen3.5 and Gemma 4 are Apache 2.0. Meta's Llama licence makes companies with more than 700 million monthly users ask Meta first, and wants any fine-tuned model you make available to carry "Llama" at the start of its name. Liquid AI's LFM Open License doesn't cover commercial use at any company with $10 million or more in annual revenue, which rules out nearly every company this article is about.

For a bank I'd want weights the team can store, a licence legal has already read, and a model small enough that training it takes an afternoon rather than a procurement request. That's how I ended up with a 0.8B.

Seventy queues on day one, seventy-seven on day ninety

I wrote the plan down and committed it before scoring any model; two plumbing checks touched 60 messages first, and the pre-registration says which.

The seven new queues were drawn at random with a fixed seed before anything ran: card linking, contactless not working, country support, disposable card limits, exchange charge, fiat currency support, and getting a virtual card. Several sit right next to an existing queue, which is what new queues tend to do. "Exchange charge" arrives in a world that already has "exchange rate".

The small model learns with LoRA, short for low-rank adaptation: instead of changing the model's 800 million numbers, it trains an add-on of about 11 million that fits in a 43-megabyte file, with settings fixed in advance. Claude got the queue names and 20 messages per call; Haiku ran at low effort, the setting you'd pick for high-volume sorting, and Sonnet at its default. BANKING77 is old enough that both Claude and Qwen might have seen it in training. I can't rule that out, and my guess is that it flatters both sides about equally.

A tiny model that wanted 35 gigabytes

The first training run ran out of GPU memory before its first step, on a model that takes 1.7 gigabytes on disk; one batch of 16 short messages was asking for 35. Eighteen of Qwen3.5's 24 layers use a newer kind of attention called Gated DeltaNet, and Apple's MLX library runs them with a fast routine that can't compute gradients, so training falls back to a slow version that keeps everything for the backward pass. Recomputing instead of storing, and four messages at a time, brought it to about 5 gigabytes at a third of a second per message. The full set of 9,024 labels took two hours, part of it on a laptop battery that had quietly run down to 1%.

The other early mistake was mine, and a team will make its own version of it. My first scoring run marked three right answers out of 20 as wrong: BANKING77 spells one queue with a capital letter, Refund_not_showing_up, and my scorer lowercased every reply. Your queue list will have its own capital R, whether a renamed queue, a trailing space, or two spellings of the same team.

It caught up with Sonnet in an afternoon

I didn't expect the 8-label adapter to come close; my plan assumed the interesting number would be how many labels it took to catch Haiku. Instead, 560 labelled messages, an afternoon's work by hand, and seventeen minutes on a laptop put a model you could run on a phone ahead of Haiku and level with Sonnet, 1.1 points behind and inside the noise. At 32 per queue, a few days of labelling from nothing, it was 11 points ahead of Haiku and 5.4 ahead of Sonnet, well past the line I'd written down for "a reason to fine-tune".

One queue explains a lot of it, and it surprised me more than the totals. BANKING77 has a queue called get_physical_card. Read the name and you'd send it messages about ordering a card. The bank's labellers used it for PIN questions: "Is the PIN delivered separately?", "I cannot seem to find my PIN." Haiku sent 34 of its 40 test messages to change_pin and got none right; Sonnet got none right either. The 8-label model, having seen eight real examples of what the queue holds, got 33.

My guess is that every team's queue list has a get_physical_card in it: a name that made sense when someone created the queue and has since come to mean something else. A frontier model reading the list sees the name; a model trained on closed tickets sees what the team does with it. When I first wrote this I guessed that a sentence about each queue in the prompt would win back a good part of that gap. I hadn't tested it, so I did, and it's further down this section.

Two things went the other way. With only eight examples per queue, the small model sometimes invented a queue: 128 replies, 4.6%, named something that doesn't exist, mostly near-misses such as top_up_by_card for the real topping_up_by_card. They count as wrong. At 32 per queue that fell to 1.6%, and letting the model write only names from the list would stop it outright. And I had assumed the bill would be the case for fine-tuning. It isn't: at list price, Haiku cost 38 cents per 10,000 messages in my runs and Sonnet $4.87, so a million messages a year is roughly $38 or $487. The case is accuracy, and keeping customer messages inside the building.

Claude's numbers needed a fix of mine before they were fair, and the cause caught me out. Five of Sonnet's 140 replies printed a list of queues, then "Wait, I need to correct that", then a second list, and five of Haiku's had the wrong number of names. My first parser marked all of them wrong. Reading the last complete list, and asking again when a reply came up short, gave both Claude models about two and a half points back; the change is logged in the pre-registration, and it only makes the small model's job harder.

Then I gave Claude a line on each queue

The first version of this article ended with a condition: if a frontier model given a one-line description of each queue, or a few examples, came within 2 points of the 32-label model, the cheaper thing to maintain would be a prompt rather than a model. On 16 October I wrote that test down, committed it, and ran it.

Claude Opus 5.5 read the same 2,240 labelled training messages the 32-label model learned from and wrote one line for each queue, at most 20 words, for 50 cents. For get_physical_card it wrote "Where to find the PIN for a new card, or when and how it arrives." Separately, I tried three random training messages under each queue in place of the lines.

So my guess was wrong in the direction that matters. For Sonnet, a sentence per queue won back the whole gap and a little more, which surprised me; I'd credited the trained model with learning what each queue really means, and it turns out most of that was simply being told. Two things survive. The model trained on every label still beat the best prompt by 4.8 points, and the lines cost something: Sonnet with them came to $8.72 per 10,000 messages at list price, about $870 a million, where the small model costs whatever your own hardware does. Haiku, the one you'd pick on price, didn't get there with lines or examples.

Then the bank added seven queues

On day ninety I added seven lines to Claude's queue list and sent all 3,080 test messages again, 280 of them meant for the new queues. Sonnet routed 86.8% of the new-queue messages correctly and Haiku 75.4%. Both slipped on the old queues, Sonnet by 2.0 points and Haiku by 4.2, because a longer list puts more near neighbours side by side.

The trained model has no list to edit. The best one, trained on every label, still got 93.1% of the old-queue messages right, and 6 of the 280 new ones. I didn't expect any. It had made up the name contactless_not_working from card_not_working and virtual_card_not_working, and that happened to be the real name of a new queue. Almost everything else went to an old queue that sounded close: disposable card limits to get_disposable_virtual_card, questions about which currencies the bank supports to exchange_via_app.

Averaged over all 3,080 messages, the unretrained model still scored 84.9%, above Sonnet's 80.2%. That average is the trap. Every customer asking about the seven things the bank had just added landed in the wrong queue, and a dashboard showing 85% wouldn't say so.

Does the model at least know it's guessing? I'd written down that its confidence would count as a usable alarm at an AUROC of 0.80, where 0.5 is a coin toss and 1.0 is perfect. It scored 0.81, past the line by a hair, and I'm not sure I'd trust it. A threshold that sends one ordinary message in ten to a person caught 149 of the 280 new-queue messages and let the rest through. Sixty-nine of the 280 came out with the model at least 90% sure, and every one of those answers was wrong. It's a smoke alarm that goes off for half the fires.

Then I did what the team would do: added eight labelled messages for each new queue, 56 in all, and retrained from the base model on everything. It took a little over two hours on the laptop. The old queues didn't suffer, 93.2% against 93.1%. The new ones reached 60.7%, short of the 70% I'd written down and well short of what Claude managed from the names alone. Eight examples were plenty for contactless_not_working and exchange_charge, 36 of 40 right each. They were nowhere near enough for fiat_currency_support, which went to the old currency queues 24 times out of 40, or for getting_virtual_card, which split almost evenly with the old disposable-card queue. My read is that this is the real cost: a new queue has no history on the day it opens, so the 32 closed tickets that made the small model good take weeks of traffic to pile up, and someone has to come back and retrain each time.

The cheapest fix I found wasn't a better alarm, and it wasn't in my pre-registration, so read it as a lead rather than a result. Let Claude read every message with the current list, and use its answer only when it names a queue the small model was never trained on. With Sonnet watching, day ninety went to 90.6% overall and 86.8% on the new queues; with Haiku watching, 89.5%. The price was 67 old-queue messages pulled into a new queue by mistake, and a Claude call for every message, which at Haiku's rate is still about 38 cents per 10,000.

If you keep the model, you keep the paperwork

A model you trained and keep is yours in every sense a regulator cares about, and the rules moved this year too.

Is a fine-tuned 0.8B that only ever writes one of 77 queue names generative? I don't know which side of SR 26-2's footnote a supervisor would put it on. I suspect a bank's model-risk team would inventory and validate it either way, and I'd want that settled before the first message is routed.

The EU question is easier for training. By the usual rule of six operations per parameter per token, my largest adapter took about 5 × 10¹⁵, some seven orders of magnitude below a third of 10²³, so a LoRA on your own tickets doesn't make you Qwen's provider. What the Act asks of a routing system follows from its use, and I'm not the person to tell you where yours lands.

Who retrains it when the queues change?

Someone you can name before launch, with a retrain they've already rehearsed and can finish inside a day; if you can't name them, don't fine-tune. In my trial Claude took on seven new queues the moment I edited its list, while the small model needed new labels and two hours of training, and still got only 61% of the new queues right.

Describe the queues first, then train the stable ones

Here's what I'd set up for a contact-centre team this month, and most of it is plumbing rather than machine learning. Start with a line on what belongs in each queue, drafted from closed tickets by a model or a person and checked by whoever runs the queue, and give the list to the frontier model you already use. Score it on last month's tickets. If you can export hundreds of tickets per queue, train the small model on them and let it own the stable queues, with its answers checked against the live list. The frontier model, lines and all, keeps reading every message and wins whenever it names a queue the small model doesn't know. The least confident few percent go to the frontier model or a person; on day one, handing the bottom tenth of the 32-label model's answers to Haiku lifted it from 86.9% to 88.2%, and handing over more made it worse.

Rerun my numbers before you trust them

Everything above is in the trial repository: the splits, the pre-registration with its one deviation and its follow-up, the queue lines Opus wrote, every prediction, and every raw Claude reply with its token counts. The checks run with plain Python and no key.

git clone https://github.com/shravan1996/generative-ai-fine-tuning-for-teams
cd generative-ai-fine-tuning-for-teams
python3 -m unittest discover -s tests -t .
python3 analyze.py

Retraining the adapters needs an Apple-silicon Mac and a few hours; the README has the commands, and the three things that cost me time. For your own queues, this is the test I'd run before anyone approves a project:

1. 01

   Export a quarter of closed tickets with the queue each one ended in

   Owner

   You

   Artifact

   A table of message, final queue and date

   Signal

   You know how many examples each queue has, and which queues have fewer than 32.

2. 02

   Score the model you already pay for, with a written line on what belongs in each queue

   Owner

   You

   Artifact

   Accuracy on the most recent month, queue by queue, and the list of lines

   Signal

   You can name the queues it still gets wrong and say why.

3. 03

   Train a LoRA adapter on the older months and score the same recent month

   Owner

   You

   Artifact

   One adapter file and a paired comparison on identical messages

   Signal

   It beats the described prompt by more than 2 points on a few thousand messages, or you stop here and keep the prompt.

4. 04

   Rehearse the retrain before launch

   Owner

   Whoever will own it

   Artifact

   A script that rebuilds the adapter from a fresh export and fails if any queue in the routing table has no training examples

   Signal

   Someone other than its author runs it end to end in under a day.

5. 05

   Launch with a way out for unsure and unfamiliar messages

   Owner

   You

   Artifact

   A confidence threshold, a fallback route, and a weekly sample of routed tickets

   Signal

   No queue in the routing table lacks training examples, and the weekly sample stays within 3 points of launch accuracy.

Split, but describe the queues before you train

Split, for routing and other narrow sorting jobs at a big company, in that order: describe, then train. Give the frontier model you already pay for a written line on each queue, and measure it. Fine-tune a small open model when you can export hundreds of past tickets per queue, where it was still almost five points better than the best prompt, or when messages can't leave your systems. Keep the frontier model reading beside it with the current lines, and don't launch until someone owns the retrain.

The first version of this verdict said fine-tune first and listed a described prompt catching a few days of labels as the first thing that would change it. That happened within a day, and this is the changed verdict. I'd change it again if:

1. On your tickets or on BANKING77, a frontier model with written queue lines comes within 2 points of a model trained on every label. Then there's no accuracy reason left to fine-tune, only price and privacy.
2. A retrain with 8 labelled messages per new queue reaches 70% on the new queues without costing the old ones 2 points, on BANKING77 or on your tickets. Mine reached 60.7%. If a handful of examples is enough, the frontier model's job beside the small one shrinks to the first week of each new queue.
3. On your own tickets, a model trained on your full history doesn't beat the described prompt by 2 points on a month it hasn't seen. BANKING77 is cleaner than your queue, and my margins may not survive the move.

The test I most want to see is the one I couldn't run: real tickets, a real reorganisation, and a real retrain on a bank's own hardware. If you have fine-tuned a router and then lived through a queue change, tell me how long the misrouting lasted and who noticed first. And if you've written queue descriptions by hand, tell me whether yours did better than a model's; mine were drafted by Opus from 2,240 labelled messages, which is its own kind of labelling.

Acknowledgements

This builds on PolyAI's BANKING77, released under CC BY 4.0, on the Qwen team's open weights, and on Apple's MLX. Claude Code did most of the building, including the parser fix after Sonnet started correcting itself mid-reply. Errors in the trial and in this reading of it are mine.

References

This trial

• generative-ai-fine-tuning-for-teams on GitHub. Code, splits, pre-registration, every prediction and every raw Claude reply.
• BANKING77 on Hugging Face. The dataset, CC BY 4.0, 10,003 training and 3,080 test messages over 77 intents.
• Qwen3.5-0.8B on Hugging Face. The small model, Apache 2.0; the card lists task-specific fine-tuning among its intended uses.
• MLX LM. The training and inference library, version 0.32.0.
• Claude pricing. List prices for Claude Haiku 5.5 and Claude Sonnet 5.5, and the Batch API discount.

Research

• Casanueva et al., Efficient Intent Detection with Dual Sentence Encoders, 2020. The paper that released BANKING77; Table 3 holds the fine-tuned BERT results.
• Loukas et al., Breaking the Bank with ChatGPT, 2023. GPT-3.5 and GPT-4 few-shot results on BANKING77, Table 3.
• Ganesh, Dozier and Seals, Selecting Open-Weight Language Models for Zero-Shot Intent Classification, July 2026. 41 open models given the label list; BANKING77 in Table 4.
• Celikik, Peleteiro Ramallo and Morales, Grammar Concept Annotation at Scale, October 2026. Preply's deployed fine-tuned Qwen3.5-0.8B against prompted GPT-5.4 and GPT-5.6 Sol.
• Hu et al., LoRA: Low-Rank Adaptation of Large Language Models, 2021. The training method.

Platforms and licences

• OpenAI deprecations. The fine-tuning wind-down dates and the 23 October shutdown of fine-tuned gpt-4.1-nano and o4-mini.
• OpenAI model optimization guide. The models that can still be fine-tuned.
• Anthropic glossary, Fine-tuning. "The Claude API does not currently offer fine-tuning."
• Amazon Bedrock, fine-tuning and model lifecycle. Claude 3 Haiku as the only fine-tunable Claude model, its legacy and end-of-life dates, and no new fine-tuning jobs on legacy models.
• Microsoft Foundry, fine-tuning overview. Supported models and methods, updated 8 October 2026.
• Google Vertex AI, open model tuning. Gemma 4 by LoRA, Qwen3.5 9B by full fine-tuning.
• Gemma 4 licence, Llama 3.2 licence, and LFM Open License v1.0.

Regulation

• Federal Reserve, OCC and FDIC, SR 26-2, Revised Guidance on Model Risk Management, 17 April 2026, with the guidance itself; footnote 3 sets generative and agentic AI outside its scope.
• Bank of England PRA, SS1/23, Model risk management principles for banks.
• European Commission, Guidelines on the scope of obligations for providers of general-purpose AI models. The one-third compute criterion for when a modifier becomes a provider, paragraphs 59 to 61.

Sources last checked: 15 October 2026