TypeSafe AI’s Jev decision model has been rebuilt from the outside. An independent research lab published an open reproduction over the weekend of 19 to 20 September without holding the model’s weights.
Benchmarks are not the only check on a technology company’s conduct. Amazon began issuing refunds to Prime members after a US$2.5 billion settlement with the US Federal Trade Commission.
Should AI vendors have to publish independently verified benchmark results before making accuracy claims?
Choose an answer, then select Vote.
Answers are not sent anywhere. This poll does not count or store responses.
The Reproduction
APUS AI Lab, the research arm of APUS Group, worked from public documentation and observed input and output behaviour. Its chief scientist, Zhang Xu, told 36Kr the team never held Jev’s weights, full architecture or training method. What was reproduced, he said, is the function.
The fast-browser-use repository, under an MIT licence, describes itself as an open-source reverse engineering of Jev’s core decision mechanics, on two Qwen3.5 base models at 9B and 35B-A3B. APUS-OpenJev-v1 adds 4B, 9B and 35B-A3B checkpoints. The interview names Google’s Gemma family, a detail neither the repository nor the model card carries; both name Qwen3.5 only.
No reproduction replicates TypeSafe’s training method, and the model card says so.
The Numbers and Whose They Are
TypeSafe published its own evaluation: Jev at 67.8 per cent aggregate accuracy against 74.1 per cent for the best comparator model. The gap is uneven: about 2.3 points on customer service triage, 17.3 points on invoice processing, where Jev scores 61.8 against 79.1. A support ticket is near a tie; an invoice is a job for a frontier model.
A company’s own figures are read against outside expectations. Tesla shares slid when its China sales hit a three-year low.
The reference labels were not human-verified ground truth. They were produced by averaging the outputs of two frontier models, which measures agreement between models, not correctness, and can inherit the blind spots those models share. That cuts both ways.
The speed and cost multiples are vendor-run. In its own workflow evaluations, TypeSafe reports that Jev runs up to 193.6 times faster and up to 444.6 times cheaper than large language models. It is priced at US$0.042 per million input tokens with output billed at zero, because it produces no output tokens, so part of the multiple compares two billing models, not two efficiency outcomes.
APUS’s own benchmark is more careful. On an 80-question panel, APUS reports 35B-A3B at 88.75 per cent, 9B at 85, 4B at 82.5 and the Jev API at 82.5. On a 1,000-question panel the same four come in at 82.2, 81.1, 80.5 and 77 per cent. The card states that panel differences “do not establish broad or statistically significant superiority”. The caveat is the lab’s own.
The card reports the 9B at 25 milliseconds at the median and 276 at the 99th percentile, against 281 at the median and 830 at the 99th percentile for the Jev API. The 79 millisecond figure in the 36Kr interview measures a single decision on an RTX PRO 6000, not a task. The repository’s benchmark gives the 9B a median of 4.055 seconds on a Wikipedia navigation task.
An Intelligent If Statement
The critique is that Jev is an intelligent if statement: the semantic condition a normal if cannot evaluate, such as whether a transaction looks suspicious. The sceptical read is an elaborate classifier whose edge is cost specialisation rather than capability. Where a condition is crisp, it cannot beat a plain if and else; where it is ambiguous, it loses to a general model.
The counter-argument is that the label undersells the construction. An ordinary classifier has a fixed label set and needs retraining when the labels change. Jev takes natural-language labels and criteria at runtime and scores options against them, closer to a cross-encoder reranker than to a classifier.
The model cannot emit broken JSON because the answer space is fixed before the call. TypeSafe’s zero hallucination claim is a type guarantee, not a correctness guarantee: the answer always falls inside the schema, and the wrong option can still be picked with confidence.
Context Is the Product
A developer’s test on the 2048 sliding tile game shows how much context decides. Shown the board state alone, the model produced a median of 706, worse than random by the developer’s reading, with 24 per cent no-op moves. Rules and move descriptions brought 1,656, a strategic tip 1,454, pre-computed outcomes 2,494, inside the 1,964 to 2,740 band fixed-rule heuristics reach. The developer called the test brief and probably incomplete.
TypeSafe’s own guidance warns that the text supplied to the model is not treated as hostile, so adversarial content can shift a typed judgement. An independent evaluation, pre-registered at 28 predictions across 123,805 requests, found crude direct injection mostly fails, moving one ticket in 200, while polite or structural injection works: text claiming a decision had already been taken moved the answer on 147 of 200.
What Stays Closed
TypeSafe’s own model remains closed. It publishes no weights, no parameter count and no architecture paper, and there is no self-host option. The fast-browser-use README says Jev is a cloud API “without publicly available model weights or internal implementation details”. The open artefacts are APUS’s. Jev is also not a chat model: a standard chat/completions call will not reach it, and it needs its own decisions endpoint, systemOne.create on OpenRouter or TypeSafe’s own /v1/systemone.
Pilot on Your Own Data
What the reproduction does not settle matters as much as what it shows. The training method was not reproduced, panel differences are not statistical significance, and the answer key behind the vendor’s own evaluation is model agreement rather than human-checked truth.
The rule that follows is procurement: run it on your own representative tasks, against whatever you use today, with a fair baseline and a labelled sample, before scaling. Accuracy here is built with your own context and calibration, on your own data.
Sources: APUS AI Lab, “fast-browser-use”, github.com/APUS-AI-Lab/fast-browser-use; APUS AI Lab, “APUS-OpenJev-v1” model card, huggingface.co/apus-ailab/APUS-OpenJev-v1; 36Kr interview with APUS AI Lab, 22 September 2026; TypeSafe AI published evaluation, evals.typesafe.ai; Xinhua, 21 September 2026; QbitAI, 20 September 2026; VentureBeat, on Jev prompt-injection risk, September 2026; willkelly/jev-evaluation adversarial evaluation of Jev, September 2026.
Illustration: AI-generated for WRP.
Illustration: an AI-generated conceptual image, not a photograph of any real system, product or company. The article reports an independent open-source reproduction of TypeSafe AI’s Jev decision model and the questions raised about its published benchmarks.