Jared Palmer ports Kev to Qwen3.5 for about $95 in rented H100 time and publishes the benchmark he failed
Kev shipped a Qwen3.5 generation on 20 September: 0.8B, 4B and 9B Apache-2.0 models that take application state plus a set of typed questions and return probabilities for yes/no, multiple-choice and ordered-score answers in one request. On the repo's locked new-source test set Kev-9B scored 0.837 against 0.780 for the prior Qwen3-based Kev-8B, while hosted Jev still leads at 0.857 versus 0.812 on the development comparison, and the repo publishes that its own pre-registered development criteria were not met rather than moving the gate. Palmer's execution log records roughly $95 of Modal H100 usage plus $0.03 in Jev API calls for the whole port, and flags a real regression: a five-question request on an Apple M5 takes 779ms on Qwen3.5 Kev-4B versus 174ms on the Qwen3 predecessor, so he recommends the older checkpoints until an MLX backend lands.
↳ Follow the thread