Competence Check

The identity check asks which model answers. The competence check asks whether it answers as well as the official deployment does. Serving a model below its official precision, compressing its memory, or cutting its reasoning budget leaves its preferences mostly intact and erodes its competence instead.

The check is experimental. Its power is moderate, it has been compared with real lower-precision hosts for one model so far, and "consistent" means not measurably worse rather than proven full quality.

When You Can Use It

Switch on Competence Check on the test form. It needs official competence data for the claimed model, which exists for GLM 5.3, GLM 5.3 Flash, DeepSeek V4.1 Flash and Kimi K3; the Baselines page marks them. For other models the switch is greyed out.

A one-logit degradation turns 90% right into about 77% right: a clearly worse model, but not a broken one.

What Degradation Looks Like

The Items

Each item has an answer a program can verify, so there is no human or AI judge. Items are generated fresh for every test, so a relay can't recognise them and send only those to the real model. Each kind of item has a difficulty dial, and the test uses the levels where the official model sometimes fails, since items it always or never gets right carry no signal.

The kinds include long multiplication and sums, counting letters, reversing and sorting strings, converting hex numbers, date arithmetic, Chinese financial numerals, and writing text that meets several rules at once.

How It Decides

• Accuracy. The items right are compared with what the official model would get on the same kinds and levels. Fewer than the line allows means Degraded.
• Reasoning length. A degraded model often keeps its accuracy by thinking longer. Reasoning at least 15% longer than official, beyond what chance explains, also means Degraded. Much shorter reasoning is only a note, since it can mean a lowered reasoning budget.

Each test allows a 0.5% false-alarm rate, so a genuine endpoint is called degraded in about 1% of tests. The result is only meaningful when the identity check confirms the model; otherwise the report marks it as indicative only.

Hidden Reasoning

The check doesn't need to see the reasoning. Items are graded from the visible answer, and the reasoning length comes from the output token count the endpoint reports. OpenAI's Responses API and Anthropic's Messages API include encrypted reasoning in that count, so it's measured like visible reasoning. The report shows how many answers had their reasoning hidden. When an endpoint reports no token count, the length test is skipped and accuracy decides alone. An endpoint that hides its reasoning and leaves it out of the count looks shorter; that's noted, never called degraded.

The Token Limit

Each item allows 6,000 tokens for reasoning and answer together, and an answer cut off at the limit counts as wrong. That limit is part of the test: many of the official model's own failures there are answers cut off mid-reasoning, so the check partly measures how much work fits in 6,000 tokens. A degraded copy that reasons longer runs out sooner.

A Real Comparison

GLM 5.3 from its official provider, at fp8, against three hosts that declare fp4, on the same 156 items:

A lower declared precision is not the same as degraded serving: two of the hosts were as good as official, and one was clearly worse, losing most on long sums, multi-rule writing and letter counting. The check flags the serving quality you get, not the label.

What a Degraded Verdict Means

The endpoint is measurably weaker than the official deployment. The usual causes are weights quantized below the official precision, a compressed memory cache, a lowered reasoning budget, a shorter context window or an injected system prompt. The check can't tell which.