A colleague wanted the launch email "in the GPT by Friday." The brief was already in Slack: $49 a seat from 1 September, fourteen-day trial, existing free-plan teams. Also in the same note, because Slack notes are how companies actually write: do not quote the $29 grandfather rate, ignore the competitor's 50% off, the 48-hour flash sale did not ship, and the 200-seat figure is an internal target. The implied application was a Custom GPT with those paragraphs pasted into the instructions. Someone would type "write the email" and send whatever came back.

I used to treat that as an application. I no longer do. I expected a schema to save me. Valid JSON still carried $29 in an extras bag. The surprise that changed the week was smaller and ruder: the brief's own date, 18 August 2026, showed up in the outbound text as if it were a customer deadline.

I wrapped that one Northwind brief four ways on 18 August 2026: a fluent chat wrapper that copies every number the notes mentioned, an open JSON object with the required keys plus an extras bag, an allowlist that fills only the fields I would defend, and the same dirty JSON run through a claim gate. No model API. A regex reader pulled money, percents, hours, seats, days, and dates from whichever text was about to be sent. The brief, the gold list, and the arithmetic live in the prototype repository.

Three things this piece doesn't do. It doesn't rank Claude against ChatGPT against Grok for copywriting. It doesn't run a frontier model on the brief, so every claim about what a chatbot would say is labelled. And the piece is not a legal or brand-safety policy, even though a leaked grandfather price is exactly the sort of thing those policies exist to catch.

The short version

  • The job: produce text a customer or a colleague will see, without shipping a number that was only in the notes as a warning.
  • The trap: a good prompt, a Custom GPT, and valid JSON all look like an application. They are wrappers. Wrappers copy.
  • What I compared: chat passthrough, open JSON, allowlist fill, and open JSON behind a claim gate, on 18 August 2026.
  • What I ran: one messy launch brief and an extractive gate. Chat and open JSON both leaked $29, 50%, 48 hours, 200 seats, and 18 August. Allowlist fill was sendable. The gate blocked the dirty JSON.
  • Where it breaks: adjectives, tone, and a model that invents a number the brief never contained. The gate I wrote catches numbers and dates. It won't catch "hundreds of teams."
  • Decision: split. Chat to think. Allowlist plus gate to send. Skip the Custom GPT as the whole product.

Start with the job

You already have a chatbot. You already write emails, support macros, one-pagers, in-app copy. Once a quarter someone demos a draft that sounds like the company, and the implied action is to make that the way the team works: paste the brief, share the GPT, send the output.

The demo is a conversation. The job is an artifact. An artifact has claims. Claims have numbers. Numbers either came from a list you would defend, or they came from a Slack thread that also contained a warning.

Four labelled checks to apply before sending a generated draft: required claims present, forbidden numbers absent, remaining numbers on the allowlist, and whether you would put your name on the send.
Fig. 1 The four checks I used on every draft in this piece. I added the fourth after the trial, because a blocked send is a default and a pretty JSON object is not.

The rest of this piece is my attempt to answer a narrower question than "which writing model." If I wrap a messy launch brief in fluent prose, or in valid JSON, does the customer email stay inside the facts I would defend, or do forbidden numbers from the notes still ship?

How to read a demo

1. Ask which numbers can leave

A writing demo is almost always scored on tone. Tone is the easy part. The hard part is the set of numbers that are allowed to appear in the outbound text. Write that set down before you watch the demo. $49, 14, 1 September 2026. Then watch what else shows up. In my brief the "else" was sitting in the same paragraph as the gold facts, labelled as a warning. Warnings are still tokens.

2. Do not confuse a schema with a source

OpenAI's Structured Outputs documentation is honest about the job it actually does: the model will emit JSON that matches the schema you supplied. Missing keys and illegal enums are the failure mode it exists to prevent. Factual grounding is not on that list. Anthropic's structured-output docs, as of this writing, say the same thing in a different API shape: valid JSON matching the schema. I had been using "structured" in conversation as if it meant "grounded." It doesn't. A schema can require an other_numbers field. It can also forbid extra keys and still let $29 occupy price_usd if your extractor put it there.

3. Completeness is not safety

The chat wrapper in this trial included the gold facts. $49 was in the email. The 14-day trial was in the email. 1 September was in the email. The draft was still unsendable, because $29, 50%, 48 hours, 200 seats, and 18 August were in there too. I used to score drafts on whether the required claims were present. Presence is the wrong passing grade for anything a customer will read. Leakage is the grade.

What the boards actually show

Reported

Vectara's hallucination leaderboard, last updated 11 May 2026, scores how often a model introduces unsupported statements when summarising a short document, using HHEM. On that snapshot, gpt-5.4-nano-2026-03-17 sat at a 3.1% hallucination rate, anthropic/claude-opus-4-5-20251101 at 10.9%, and xai-org/grok-4-fast-reasoning at 20.2%. The board is summarisation of the supplied document, temperature 0 where the API allowed it. The board is not an email job, and it doesn't include Grok 4.6, GPT-5.6 Sol, Claude Opus 5, or Claude Fable 5.

Quote that table as "which model to write customer email" and you have already left the job. A 3% summarisation model will still emit $29 if $29 is in the prompt. The board measures unfaithfulness under a summarisation instruction. Your send button measures whether a number you wouldn't defend can leave.

Reported

RAGTruth (Niu, Wu, Zhu, Xu, Shum, Zhong, Song, and Zhang, ACL 2024) is a corpus of nearly 18,000 RAG responses with word-level hallucination labels, across question answering, data-to-text writing from JSON, and news summarisation. The authors found data-to-text the most hallucination-prone of the three. In their suppression experiment, GPT-4-0613 had a 9.3% response-level hallucination rate on the slice they report. GPT-4-turbo, asked to detect hallucinations given the reference, reached an average F1 of 63.4%. Fine-tuning Llama-2-13B on their training split reached 78.7% F1.

Two facts in that paragraph pulled in different directions for me. Data-to-text is the closest public analogue to "turn this JSON into an email," and it was the worst of the three RAGTruth tasks. Asking GPT-4 to catch the bad spans, with the reference sitting right there, still missed a lot of them. I had assumed "the model will notice" was a plan. On this evidence, treat it as a hope with a published F1.

Reported

OpenAI's Structured Outputs guide, as checked on 18 August 2026, distinguishes JSON mode (valid JSON, any shape) from Structured Outputs (valid JSON that matches a supplied schema). The documented guarantee is schema adherence: required keys present, illegal enum values blocked. The guide doesn't claim the values are faithful to a source document.

I'm not going to pretend a 2024 RAGTruth number is a 2026 Grok 4.6 number. I'm going to say the paste-the-brief demo still invites you to skip the only check both results make expensive: look at the numbers in the outbound text, against a list you wrote down first.

Why an honest table can still mislead

Vectara's cells can be right and still point at the wrong default: the task is summarisation, and the missing columns are the models you actually type into. RAGTruth can be right and still point at the wrong default if you read "GPT-4 still hallucinates 9.3% of the time" as "don't use a writing model." Even with retrieved context, even on JSON-to-text, models still introduce unsupported spans. Asking another model to find those spans is not a substitute for a list of numbers that are allowed to appear.

Vectara says summarisation. RAGTruth says RAG responses. The misleading move is ours: we watch a demo, we remember a hallucination percentage, and we ship a GPT.

Inferred

A regular user who pastes the Slack brief into a Custom GPT because the demo sounded like the company has not listed the numbers that must not appear, and has not checked the draft against that list.

What I asked

I wanted a test a regular reader would recognise as a launch note, not as a benchmark.

The question

If I wrap a messy launch brief in fluent prose, or in valid JSON, does the customer email stay inside the facts I would defend, or do forbidden numbers from the notes still ship?

Baseline
Chat passthrough is sendable: required claims present, and $29, 50%, 48 hours, 200 seats, and 18 August absent.
Continue if
Chat passthrough and open JSON are both sendable on that same rule.
Stop if
Chat passthrough leaks a forbidden number, or open JSON is schema-valid and still leaks one.

Gold, fixed in data/gold.json before I wrapped anything:

  • Price $49 per seat per month
  • Start 1 September 2026
  • 14-day trial
  • Audience: existing free-plan teams
  • CTA: start the trial

Forbidden if they appear in the email: $29, 50%, 48-hour, 200 seats, 18 August.

Four applications: chat passthrough, open JSON with an other_numbers bag, allowlist fill, and the dirty JSON behind a claim gate. The reader is extractive. The reader is not Claude. If a forbidden number is in the outbound text, a better model doesn't get a vote. The wrapper already voted.

What I expected: the chat wrapper would leak, because it is a dump. I also expected JSON to be the grown-up version. I walked in with that view, and I was wrong about JSON.

What happened

Table of the 18 August 2026 trial. Chat passthrough and open JSON both leaked $29, 50 percent, 48 hours, 200 seats, and 18 August. Allowlist fill was sendable. Gating the dirty JSON blocked the send.
Fig. 2 My run, not a product bake-off. A win was required claims present and no forbidden number in the outbound text. I did not get one from fluency or from a schema with extras.

Reproduced

Chat passthrough was unsendable. The email contained $49, $29, 50%, 48-hour, 200 seats, 14-day, 18 August 2026, and 1 September 2026. Forbidden set: 18, 29, 48, 50, 200. Open JSON was schema-valid, with required keys correct, and other_numbers equal to [18, 29, 48, 50, 200, 20260818]. The rendered email still contained $29, 50%, 48-hour, 200 seats, and 18 August 2026. Allowlist fill was sendable: $49, 14-day trial, 1 September 2026, existing free-plan teams, start the trial. Open JSON behind the gate was blocked.

The stop threshold fired twice, on the same brief. Fluency leaked. JSON leaked.

The first surprise is the one that killed my working default. I had assumed a schema was how grown-ups stop the $29. Open JSON in this trial is an extras bag on a fixed schema, which is a choice I made, and I should say so. Vendor structured outputs can set additionalProperties: false and refuse extra keys. They still won't look at $49 and ask whether it is the price you meant. My extras bag is a loud version of a quieter failure: a required price_usd field whose extractor read the wrong dollar amount. Schema-valid, unsendable.

I almost made the gate miss it. The first render of other_numbers wrote 29, 48, 50 as bare integers. The checker looks for labelled money, percents, hours. A draft that said "29" would have walked past a gate that was hunting "$29". I labelled the extras so the gate could see them. A checker that cannot read its own output is not a checker.

The second surprise was the date. I put "Northwind launch notes, 18 August 2026" at the top of the brief as metadata, the way every real note starts. The chat wrapper treated it as a date in the notes, because it is a date in the notes. 18 August 2026 left the building next to 1 September 2026. I wouldn't have predicted I needed to forbid the timestamp of the file itself.

Reproduced

Allowlist fill used only labelled gold fields: price 49, trial days 14, start date 2026-09-01, audience, CTA. The gate found no forbidden number and no extra. The same gate, pointed at the open JSON email, refused to send.

The third surprise is the one I should have seen coming and did not. Completeness was never the problem. The dirty drafts were complete. They were complete the way a paste-all folder is complete: everything you needed, plus everything you needed to keep in the thread.

Unknown

I'm not sure whether Claude, ChatGPT, or Grok, given the same brief and "only use the facts you would put in a customer email," would drop $29. I didn't run those. A live model might drop it. It might also invent a number this harness won't show. Vectara and RAGTruth both say models still introduce unsupported spans with a source in front of them. My wrapper doesn't settle a product comparison. It settles what the wrapper itself will do.

Where a regular user should not trust the number

Fluency is not a source

A draft that sounds like the company will include the numbers that were in the prompt, because those numbers are how it sounds informed. "Do not quote $29" is still the token $29. I wouldn't send on tone. I would send on the allowlist.

Schema-valid is not grounded

JSON mode means it parses. Structured outputs mean the keys match. Neither means price_usd is the price you meant, and neither means an extras field is empty. Did you check the extras, or did you assume empty because the keys looked right? Extra keys belong in the schema. Values belong in a second program. I had collapsed those jobs into the word "structured." I don't anymore.

A second model is not a gate

RAGTruth's GPT-4-turbo detector, with the reference available, put up a 63.4% F1. Those points are a published number on a different task, and I shouldn't pretend they are my email job. They are enough to stop me treating "ask the model if this is faithful" as the application. Ask the list. The list doesn't need to be a model.

The gate will not catch "hundreds"

My checker looks at money, percents, hours, seats, days, and dates. It won't catch "limited time," "exclusive," or "hundreds of teams." Those are real leaks. They need a different check, or a person. I'd rather name that hole than let the 5/5 on numbers pretend to be a 5/5 on language.

A working default

Four-row decision table. If you are thinking, use the chat. If a customer will see the text, allowlist the fields then gate. If you want a shared Custom GPT, share the gate. If a week of sends shows a number you would not sign, revert to a human-written template.
Fig. 3 A starting point for this week, with the kill switch in the last line. The chat box is allowed to be messy. The send button is not.

Chat passthrough

1/51 out of 5

High confidence

Leaked $29, 50%, 48-hour, 200 seats, and 18 August, while also including the gold facts. Completeness with leakage.

Open JSON

2/52 out of 5

High confidence

Required keys were correct. other_numbers still carried 18, 29, 48, 50, 200, and the 18 August date code. Schema-valid, unsendable.

Allowlist fill

5/55 out of 5

High confidence

Only labelled gold fields. The gate found no forbidden number and no extra. The draft I would actually send.

Dirty JSON, then gate

5/55 out of 5

High confidence

The gate refused. A blocked send is the correct application behaviour when extras are dirty.

Custom GPT instructions

2/52 out of 5

Low confidence

I didn't run one. Instructions are sampled. I wouldn't start there for a number that lives on a list I can check in code.

Checking numbers against the brief

5/55 out of 5

High confidence

Every leak in this trial is visible if you read the outbound text against the forbidden list. That check is available on Monday without a new tool.

Average
3.3 / 5
Binding constraint
Forbidden numbers in the notes. Allowlist fill was sendable. Chat and open JSON were not, and the gate only helped when it was allowed to refuse.
Override applied
None. Split is already the conservative reading of a 3.3 average.

The average says the landscape is usable if you allowlist. The binding constraint says don't let the notes pick the number the customer sees. My read: chat is for thinking, and the send path is a list plus a refusal.

How to try this yourself

You can replay the recorded trial without an API key. You still need your own brief if you want to choose a default.

bash
git clone https://github.com/shravan1996/generative-ai-chat-is-not-the-application
cd generative-ai-chat-is-not-the-application
python3 -m unittest discover -s tests -t .
python3 eval.py

No network, no GPU, no third-party package. The full implementation is in the prototype repository.

  1. 01

    Name one artifact you sent last week that started in a chatbot

    Owner
    You
    Artifact
    The email, macro, or one-pager, and the brief it came from
    Signal
    If you cannot name the brief, you were chatting, not applying
  2. 02

    List every number in the outbound text

    Owner
    You
    Artifact
    A column of numbers, with the one-line source for each
    Signal
    A number with no source is the actual problem
  3. 03

    Write the allowlist and the forbidden list before the next draft

    Owner
    You
    Artifact
    Two short lists a colleague could use without you
    Signal
    If the forbidden list is empty, you have not read the notes
  4. 04

    Run the next draft through those lists, or through this repo with your brief swapped in

    Owner
    You
    Artifact
    Sendable, or a refusal
    Signal
    A fluent draft with a leaked number is a skip
  5. 05

    Keep chat for thinking, or take it off the send path

    Owner
    You
    Artifact
    A one-line default and a date to look again
    Signal
    If a week of sends shows a warning from the notes, the prompt doesn't get a vote

Ship gates

  • Required claims are written down before anyone opens the chatbot
  • Forbidden numbers are written down from the same notes
  • A draft cannot be sent if a forbidden number is in it

Kill criteria

  • A week of real sends contains a number that was only in the notes as a warning
  • The team is scoring drafts on tone and skipping the list
  • You needed a second model to tell you whether $49 was the price

Should you put a Custom GPT in front of customers?

Not as the application. A shared GPT is a shared prompt, and a shared prompt will still copy $29 if $29 is in the file. Put the allowlist and the gate in front of the send button, and let the GPT draft behind them if you want.

My verdict

Split for turning a chatbot into customer-facing text in August 2026.

As far as I can tell, the prompt is not the application. The application is the list of numbers that are allowed to leave, plus the refusal when something else shows up. I would start a colleague on allowlisted fields and a gate. I would still use the chat box to think. I wouldn't ship a Custom GPT as the product, and I wouldn't treat valid JSON as grounded JSON. I expect Claude, ChatGPT, and Grok can all draft behind that gate. The logo is not the decision. The gate is.

I would change this verdict if:

  1. Your live Custom GPT, on three real briefs that contain a labelled warning number, never puts that number in the outbound draft, and you can show the transcripts.
  2. A schema with extra keys forbidden, on a brief whose labelled price is unambiguous, still needs no value check because the extractor cannot see the $29. I have not seen that extractor.
  3. You have a tone or format job that few-shot prompting cannot hold, and the numbers are already gated. Fine-tune that. Leave the price list in the gate.

What to remember

If you ran your own three briefs with a labelled warning still in the notes and the outbound draft stayed clean, write to me. I would rather correct the default than defend the GPT.

Acknowledgements

This article builds on public boards and vendor docs linked in References. Vectara's HHEM leaderboard and RAGTruth are why I refused to treat "the model will notice" as a plan.

References

Independent boards

  • Vectara Hallucination Leaderboard. HHEM summarisation faithfulness; snapshot last updated 11 May 2026. Rates cited above are from that README table.
  • RAGTruth. Cheng Niu, Yuanhao Wu, Juno Zhu, Siliang Xu, Kashun Shum, Randy Zhong, Juntong Song, and Tong Zhang. ACL 2024. Nearly 18,000 labelled RAG responses; data-to-text the most hallucination-prone task; GPT-4-turbo detection F1 63.4%.
  • Benchmarking LLM Faithfulness in RAG with Evolving Leaderboards. Tamber et al., EMNLP 2025 Industry. Cited as the later Vectara faithfulness effort, not scored in my trial.

Vendor posts

This trial

Sources last checked: 2026-08-18

The chat can think. The gate decides what leaves.