A Fine-Tuned 30B Reasoning Triage Model Beats Frontier LLMs on SOC Alerts, but Only With a Trained Calibrator

arXiv·medium signalXBlueskyLinkedInCopy link

This work (arXiv 2607.28460, July 30) trains a chain-of-thought triage classifier on real human-labeled Windows endpoint detections using automated prompt optimization, self-training, and RL with verifiable rewards, reaching 82.6% test accuracy. The notable systems finding: CoT reasoning degrades the label-token probabilities that automated triage thresholds depend on, so the authors separately train a calibrator that reads the full reasoning trace and estimates verdict correctness — an untrained confidence judge collapses high-confidence recall to zero. At the high-confidence operating point governing automation, benign recall improves 43.0% and malicious recall 18.3% over a direct-label LLM classifier, and the fine-tuned 30B significantly outperforms frontier general-purpose models.