Four Frontier Models Score Above 0.81 BERT-F1 Extracting Architectural Decisions From Commits and Still Miss the Rationale

arXiv 2609.03721·low signalXBlueskyLinkedInCopy link

arXiv 2609.03721 tests whether LLMs can recover architectural design decisions hidden inside source code commits, using Gemini 3 Pro, DeepSeek R1, Kimi K2 and Qwen3 with zero-shot and few-shot prompting on 30 developer-written ADDs from open-source projects, scored with ROUGE-L, BLEU, METEOR and BERTScore plus manual review of the Gemini outputs. All models reached BERT-F1 above 0.81 and few-shot improved alignment (Gemini 0.828 to 0.847), but manual review found the generated ADDs were often too long, implementation-focused, and missing the rationale behind the decision. The gap between a strong similarity score and a useless output is the finding, and it is a caution for anyone trusting text-overlap metrics on documentation-generation tasks.