Ninety percent sure, in English

The first thing I looked for in TypeSafe's docs was a calibration plot. It's a boring chart: take every answer the model said it was 70% sure of, and check whether it was right about 70% of the time. Then do the same at 80%, at 90%, and so on. If a company is selling you honest probabilities, I figured that chart would be on the front page.

I couldn't find it. Not in the launch post, not in the docs, not on the evals site.

That's where this piece comes from. Jev came out on 15 September, and it's an odd thing to call an AI model. It doesn't write. You send it some text, say a support ticket, along with a question and a fixed list of answers, like "approve" or "deny". It sends back a probability for each answer, usually in a few hundred milliseconds. No explanation, no paragraph. Just something like approve, 0.93.

TypeSafe says a new training method makes those numbers trustworthy. They call it RLCD, short for Reinforcement Learning for Calibrated Decisions. And the number matters, because the docs suggest that anything at 0.9 or above can be acted on automatically, without a person looking at it.

So that's my question. If Jev says it's 90% sure, can you let it make the call?

I haven't run Jev myself; this piece doesn't use an API key. What I did was read the four independent studies that came out in its first two weeks, and then build a small toy version of the idea behind RLCD to see where it holds and where it slips. I'm not ranking Jev against ChatGPT or Claude, which makes little sense for a model that can't write a sentence. And if you've seen "RLCD" before, that was a 2023 alignment method from Berkeley, Meta, and UCLA researchers. Same letters, unrelated idea.

Cheap, fast, and honest on average

Here's where I landed, before the details.

Jev is very cheap and very fast. Every study agrees on that, and the gap isn't small: tens to hundreds of times cheaper than calling a chatbot. On accuracy it's roughly as good as a mid-tier chatbot for routing and simple sorting, and clearly worse on hard calls like legal or medical judgement.

Its probabilities do look honest on average. The catch is "on average". In my toy test, a model trained the way RLCD is described gave confidence scores you could safely act on, right up until the tickets coming in looked different from the ones it trained on. Then the same 90% quietly meant something closer to 85%.

I believe the right move is to shift the easy sorting work over now, and to hold off on letting the score make decisions alone until you've checked it on your own data.

What you actually get back from Jev

You send Jev two things. The first is the input, which TypeSafe calls the "state": a string, a JSON object, or a list of text. The second is one or more questions. A question can be yes-or-no, a choice from up to 255 options, or a score on a short scale. For each one you get back the pick and a probability for every option.

There's also a "confidence" field, but it isn't a separate opinion. It's just a measure of how lopsided the probabilities are. If all the options are equally likely, confidence is 0. If one option has all of it, confidence is 1.

Jev also has no "I don't know" answer. If you want one, you add "none of these" to your list yourself.

The docs give a rule of thumb for the confidence number: act automatically at 0.9 and above, send it to a person below 0.5, and add a check in between. The very next line says the right thresholds "depend on your domain and the performance of the model for your use case." I skimmed past that line the first time. By the end of my test I thought it was the most important sentence in the docs.

It's worth knowing who's behind this. TypeSafe's CEO, Diogo Almeida, spent about four years at OpenAI and was one of the lead authors on InstructGPT, the paper that made RLHF the standard way to train chatbots. The company raised $40 million, led by DCVC. Early users sound happy. Vercel told TechCrunch it moved a safety classifier to Jev and got answers 5 to 18 times faster "with greater accuracy". That's a customer quote, though, not something I can check.

The training method TypeSafe hasn't shown anyone

The easiest way to see what RLCD is supposed to do is to compare it with the two methods it's positioned against.

Each of the older methods has a blind spot about confidence. A reward that only checks correctness never looks at how sure the model said it was, so that number never gets trained at all. A reward based on human preference has the opposite problem: people tend to prefer answers that sound sure of themselves. A 2024 paper found the reward models used in RLHF lean toward confident-sounding answers whether or not they're right. TypeSafe's homepage names overconfidence as one of the things RLCD fixes.

That's about all TypeSafe has said. The method has a name, a goal, and, according to press interviews, training data that is entirely synthetic. There's no paper, no reward formula, and no calibration plot.

My guess at what's underneath is something called a proper scoring rule, like the Brier score. Under one of those, a model earns the most on average by reporting exactly what it believes. Overstating or understating its confidence both cost it. An MIT group published a method along these lines in 2025, adding a Brier penalty to a plain right-or-wrong reward. That's the version I copied for my toy. To be clear, though, TypeSafe hasn't confirmed any of this.

I'll admit I was a bit surprised that the training method is both the headline and the secret. I get why a startup would keep it close. It does mean every claim about RLCD itself rests on trust, and only Jev's measured behaviour can be checked.

Can you let it act on its own when it says 90%?

Not on the strength of calibration alone. A score can be honest on average and still fail to tell you which answers are the wrong ones, and it can be honest on someone else's data and off on yours.

I used to think "well calibrated" meant "knows when it's wrong". I was wrong about that. Picture a model that says 85% about every single ticket and is right on 85% of them. It's perfectly calibrated, and it's no help at all in deciding which tickets a person should look at. A medical study of Jev is what made this click for me, and my toy later turned it into a number.

What four papers found in two weeks

Researchers moved fast on this one. None of the four studies below has a TypeSafe author. Here are the headlines, with the setting each one used.

The Bonn study is the one I'd read first, and it gave me my first real surprise. Averaged across all 37 datasets at once, Jev's calibration error was 0.028, much better than the open Qwen model's 0.063. Averaged one dataset at a time, the two were basically tied: 0.074 against 0.075.

Same model, same data, and the numbers tell two different stories. The difference is in how you average. Lump everything together and the big, easy datasets dominate. Look dataset by dataset and you see how it does on a typical task. Your workload is one task, so the second number is the one that applies to you.

And TypeSafe's own evals? Across four workflows the company wrote itself, Jev scores 67.8% at $0.0004 a case, roughly level with mid-tier chatbots and behind the best. Before quoting that, look at how "accuracy" is defined. It means agreement with the average answer of GPT-6 Astra and Fable 5.1. Agreeing with two strong models is useful. It still isn't the same as being right, and calibration is about being right.

A review of the first 28 Jev papers summed it up fairly: the clearest gains are "in latency and cost, while accuracy gaps remain on harder tasks."

I built a tiny refund desk to find out

None of those papers could answer my actual question, so I set up a toy.

Imagine a team handling refund requests. This is a made-up example, but a common kind of job. They'd like the model to approve or deny tickets on its own when it's confident, and pass the rest to a person. A wrong automatic approval costs money. A ticket sent to a person costs a few minutes.

In my version, every ticket has a single clue, a number from 0 to 9 for how much it reads like a valid refund. Each ticket is also in English or Portuguese, and the model can't see which. In Portuguese the clue is weaker, so the same wording is a less reliable sign. Nine in ten training tickets are English. Because I wrote the world down, I know the true odds for every ticket. That means I can measure the model exactly rather than estimate.

The model is about as simple as they come: a lookup table that, for each clue, picks approve or deny and states a confidence. I trained it three times on the same tickets, changing only the reward.

• Correct only: a point for a right answer. It never checks the confidence. (The RLVR shape.)
• Sound sure: a point for being right, plus a bonus for sounding confident. It exaggerates the RLHF bias on purpose.
• Scored: a point for being right, minus a penalty when the confidence doesn't match the outcome. This is my stand-in for RLCD.

I added two policies I didn't train, for comparison. One states the true odds for each clue, the best a language-blind model could ever do. The other just says 85% about everything.

Before running anything, I wrote down what would count as success and failure.

Six policies that agreed on every answer

All six policies got 86.8% of tickets right. They made exactly the same calls. On a leaderboard they'd be a six-way tie.

What changed was how sure they claimed to be, and that's what decides what gets automated.

Two things caught me off guard. I expected the correct-only version to be overconfident. Instead it was random. Its confidence never got any real training, so it ended up wherever the noise left it: one run auto-approved 10% of tickets, another 70%. If you'd tested only the lucky run, you'd have concluded it worked fine.

The second was the flat 85% policy, which I'd added almost as a joke. Its calibration error came out better than the trained scored model's, and it automated nothing at all, because it never reaches 90%. I hadn't expected a joke to make the point better than anything I'd planned.

Then I gave it Portuguese tickets

Next I switched to a queue that was half Portuguese, the kind of shift a growing team in a new market would see.

I'd expected the trained model to stumble. I didn't expect the perfect one to stumble with it, though it's obvious in hindsight. If the model can't tell the languages apart, it can only be honest about the average ticket it saw in training. A strong clue means 95% in English and about 71% in Portuguese. A model that can't see the difference says the same thing for both.

TypeSafe's docs actually say this, in their own way: calibration holds across many predictions taken together, not for any single one. "Taken together" always means a particular mix. My Portuguese tickets were a slice of that mix, not the whole of it.

Then I tried the obvious fix, which wasn't in my original plan, so treat it as exploring. I gave the model the language as part of its input. Jev takes JSON, so in real life that's just one more field. With it, the error on the mixed queue dropped to 6.6%, back under the line. On a queue of only Portuguese tickets it was still 19.5%, though. One ticket in ten during training wasn't enough for it to learn the Portuguese odds well.

The threshold I picked was the lucky one

The practical fix the docs hint at is to pick your own threshold from your own data. So I did that too. I took 500 labelled tickets from the mixed queue and chose the lowest confidence at which the errors in that sample stayed under 10%.

That one surprised me most, and it's probably the most useful result here. If you pick the lowest threshold that passes on a sample, you tend to pick one that happened to get lucky on that sample. The fix is old and dull: leave yourself a margin.

Where I'd still keep a person in the loop

Pulling the papers, TypeSafe's own list of weak spots, and my test together, these are the places I'd be careful.

Languages and customer groups that are a bigger share of your traffic than of the benchmarks. In the Bonn study, accuracy fell as low as 37.4% in Nigerian Fulfulde, and nobody has checked calibration language by language. I expect it moves with accuracy, but that's a guess.

Anything involving numbers, dates, or tricky wording. TypeSafe's own docs are blunt: "Jev is not a calculator," and comparing dates "is unreliable". A confident answer on one of those is a confident guess.

Many yes-or-no questions about the same input. That's where the Bonn study found its weakest calibration.

Long forms with lots of fields, and hard judgement calls. Accuracy drops on both, sometimes by a lot.

Text written to push the answer one way. TypeSafe lists "text that argues for its own classification" as a weakness. A refund request arguing for its own approval is exactly the one I'd want a person to read.

What I'd move to Jev this month

If I were running that refund team, I'd move the sorting and tagging over this week and keep a log of every probability Jev returns. For the first month I wouldn't let anything auto-approve. Once a few hundred labelled cases had built up, I'd check how the confidence holds up for each language or customer group, and only then pick a threshold, a bit stricter than I think I need.

Run the refund desk yourself

The test takes about three seconds and needs no API key or extra packages.

git clone https://github.com/shravan1996/generative-ai-jev-rlcd-calibrated-decisions
cd generative-ai-jev-rlcd-calibrated-decisions
python3 -m unittest discover -s tests -t .
python3 eval.py

Try changing how weak the Portuguese clue is, or the training mix, in data/world.json, and see which versions survive. If you have Jev access, the steps below work on its real output too.

1. 01

   Label 500 recent cases from your own queue

   Owner

   You

   Artifact

   Inputs, the right answers, and a column for language or customer group

   Signal

   If one group is under a tenth of the file, label more of that group

2. 02

   Run them through Jev and keep every probability

   Owner

   You

   Artifact

   The pick and the probabilities for each case

   Signal

   Accuracy close to what your current setup gets on the same cases

3. 03

   Check each group separately

   Owner

   You

   Artifact

   For each group: of the cases Jev called 0.9 or higher, how many were right

   Signal

   Each group close to what Jev claimed

4. 04

   Pick a threshold with some margin

   Owner

   You

   Artifact

   The lowest confidence where error stays a few points under your limit

   Signal

   It still holds on 200 fresh cases you didn't use to pick it

5. 05

   Spot-check auto-approvals every week

   Owner

   You

   Artifact

   Fifty auto-approved cases, reviewed by a person

   Signal

   Errors above your limit, or a fast-growing new group, means back to review

Split, for now

Split for moving a classification step to Jev in October 2026.

As far as I can tell, Jev is a genuinely good tool. I don't know a cheaper or faster way to make hundreds of small decisions a second, and the people building on it aren't wrong to be excited. RLCD might well be why its probabilities look as honest as the Bonn study found. I think that's plausible, but nobody outside TypeSafe can check it yet.

What hasn't been shown, by TypeSafe or anyone else, is that those probabilities stay honest for the particular slice of the world your queue lives in. My toy suggests that's exactly where an honest-on-average model goes wrong.

I'd change this to Use if:

1. TypeSafe, or someone independent, publishes calibration plots broken down by task and by language.
2. A study shows Jev's confidence separating right from wrong answers as well as a frontier chatbot's on a hard task, not just an easy one.
3. TypeSafe publishes the RLCD method, so the claim that it fixes overconfidence can be checked.

If you've run Jev on traffic in more than one language and checked its scores language by language, I'd love to see what you found, whichever way it points. It's the biggest open question in this piece, and I'd much rather replace my toy with your real numbers.

Acknowledgements

This piece builds on TypeSafe's launch post and documentation, the four independent evaluations listed below, Tang and Zheng's early evidence review, and reporting from TechCrunch, SiliconANGLE, and InfoQ. Thanks to every author who posted a Jev study with its limitations section intact; that made this possible inside two weeks. Errors in the toy and its reading are mine.

References

TypeSafe

• Introducing System One models and Jev. Diogo Almeida, TypeSafe, launched 15 September 2026, updated 28 September. RLCD named and described, "epistemically honest probabilities", 70 to 500 ms, pricing, evals reference labels.
• API reference and models. Request shape, Noul, Choice and Score, jev-1.13.0, context limits, pricing.
• System One concepts. Calibration described as holding across many predictions, not any single answer.
• Confidence. The confidence formula, the 0.9 and 0.5 suggestions, and "the correct threshold values depend on your domain".
• Model jaggedness: jev-1.13. Known weaknesses, last reviewed 17 September 2026.
• TypeSafe evals. Four workflows, accuracy as agreement with GPT-6 Astra and Fable 5.1 at high thinking.

Independent evaluations

• Evaluating and Benchmarking the System One Model Jev. Deußer, Sparrenberg, Sifa, University of Bonn and Fraunhofer IAIS, v1 29 September 2026. 37 datasets, pooled and per-dataset calibration, language results.
• Jev in Medicine: A Benchmark Evaluation. Preliminary Results. Madrid-García, Merino-Barbancho, v1 27 September 2026. Accuracy, calibration, AUROC, and coverage against GPT-6 Sol.
• Same Scores, Different Decisions: Evaluating JEV and Language Models for Legal Document Understanding. Zhang et al., v1 23 September 2026. ContractNLI accuracy, cost, and repeat consistency.
• Replacing Large Language Models with Jev Decision Models for Low-Latency Edge Service Orchestration. Li, Wang, Gong, Lang, Yu, University of Technology Sydney, v1 19 September 2026, v2 26 September. Figures here are from v2.
• Typed Decision Models: An Early Evidence Audit and Evaluation Checklist. Tang, Zheng, v1 26 September 2026. Review of 28 early Jev papers.

Reporting

• A new kind of AI model from a ChatGPT inventor is thrilling developers. TechCrunch, 18 September 2026. Synthetic training data, Vercel's classifier.
• TypeSafe AI exits stealth with $40M. SiliconANGLE, 16 September 2026.
• TypeSafe emerges from stealth. DCVC, 15 September 2026. Seed round.
• TypeSafe AI releases Jev. InfoQ, 1 October 2026.

Background

• Training language models to follow instructions with human feedback. Ouyang et al., 2022. InstructGPT and RLHF.
• Taming Overconfidence in LLMs: Reward Calibration in RLHF. Leng et al., 2024. Preference reward models favour confident answers.
• Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty. Damani et al., 2025. RLCR, a correctness reward plus a Brier term.
• RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignment. Yang et al., 2023. The unrelated method that shares the acronym.

This trial

• Source, world, rewards, and scorer
• Exact tested commit
• Five seeds, three rewards, two reference policies, three queues, 1 October 2026. No model API, no network.

Sources last checked: 2026-10-01