Limits

Read these before acting on a verdict.

• A mismatch is evidence, not proof of intent. Answer distributions also change with a system prompt added by a relay, different sampling defaults, quantization, a new checkpoint from the vendor, or routing. Confirm any mismatch with a separate test at the Thorough tier.
• Baselines age. Vendors update hosted models. If the vendor's own API suddenly comes out as Unknown Model, the baseline is out of date, not the vendor.
• Models outside the baseline can't be named. They're caught when their answers don't fit the claimed model. A close lookalike can pass a Standard test.
• Simulated rates are averages. The numbers in the side panel and on the Baselines page come from simulations on the collected data. For the closest pairs, a real test can do somewhat worse.
• Token counts and model names in replies are hints. Relays can rewrite both. They become evidence only when requests with different counts also answer differently.
• Siblings can't be separated. Models that answer almost alike, such as MiMo-V2.5 and MiMo-V2.6-Flash, end Inconclusive rather than being called substitutes.
• Mixed routing needs a large share or different families. A quarter of the traffic sent to a close relative is mostly missed. See Mixed Routing.
• The competence check is experimental. It catches about half of clearly degraded deployments at Standard, and "consistent" doesn't prove full quality. See Competence Check.
• The refusal check is for complete removals. A copy that complies half the time gets a note, not a verdict, and telling modified weights from a jailbreak prompt rests on token counts. See Refusal Check.