Ventor-QTest Audits Whether Your Hosted LLM Endpoint Is Serving the Model It Claims — Without Logprobs

arXiv 2608.16391·high signalXBlueskyLinkedInCopy link

Third-party providers hosting open-weight models are now a large slice of inference, and this Tencent-affiliated work formalizes hosted model routing as a stochastic process to audit it black-box. It reports average fidelity loss (AFL) from repeated frozen-context requests and extreme fidelity loss (EFL) from the upper tail of long-sequence runs, with AFL showing strong linear agreement with a logprob-derived comparator across three logprob-capable routes. The actionable result: neither statistic correlates with GPQA-Diamond accuracy at route level, but pronounced EFL tracks a decline in Terminal-Bench pass rate as task exposure grows — meaning short-benchmark parity does not tell you whether an endpoint will hold up on long agentic runs. Implementation is open-source in Tencent's AI-Infra-Guard.