Locally Deployed Qwen3.6:27B Hits 0.803 Kappa Judging a Hand-Built 150-Report STIX 2.1 Threat Intel Dataset
This work (arXiv 2607.23312) releases a manually constructed dataset of 150 English CTI reports rendered as STIX 2.1 graphs, totaling 4,777 STIX entities, 5,817 relationships, and 1,273 attack-pattern entities mapped to 269 unique MITRE ATT&CK Enterprise techniques and sub-techniques. Twenty-five reports were independently assessed by two cybersecurity researchers with substantial inter-rater agreement, and disagreements adjudicated into a gold-standard reference. Four locally deployed open-source LLMs were then evaluated as automated judges against that reference; Qwen3.6:27B performed best with a maximum kappa of 0.803, micro-F1 above 92%, and false-positive rates under 5% — evidence that local open models can carry annotation-QA work, though the authors stress expert validation remains essential.
↳ Follow the thread