GitHub cut secret-scanning false positives 95% and published the eval methodology behind it

GitHub Blog·medium signalXBlueskyLinkedInCopy link

GitHub's August 25 post describes a three-tier eval structure for putting LLMs into secret scanning: false-positive reduction and precision as the primary outcome, recall held inside a defined guardrail as the safety constraint, and latency, cost, reliability and compatibility as operational guardrails. They report a 95% reduction in false positives on the offline dataset while keeping recall inside the guardrail. The transferable practices are versioning prompt, model and system config on every run, changing one major variable at a time against a known baseline, and manually classifying failures by source (model reasoning, prompt framing, input construction, pipeline logic, dataset quality, labeling error).