Repurpose the speculative-decoding module as a safety classifier and monitoring costs almost nothing

arXiv 2608.28099·medium signalXBlueskyLinkedInCopy link

Speculative Probing (arXiv 2608.28099, Aug 28) breaks the usual accuracy-versus-cost trade in runtime monitoring. Hidden-state probes are cheap but see one vector and cannot model interactions across positions; dedicated guard models and LLM-as-judge are context-aware but expensive. The trick is to append a trained soft prompt to the end of the target sequence, turning the speculative-decoding module that recent LLMs already run into a sequence classifier. Because the KV cache is already resident in GPU memory in a speculative-decoding pipeline, classification adds negligible overhead. Across four tasks and four models (Qwen3.5-4B/9B/27B, MiniCPM4.1-8B) the small probes beat zero-shot GPT-5.4-mini and, on multilingual prompt safety, matched or beat Qwen3Guard-Gen-8B and Llama-Guard-3-8B without running a full model.