Four frontier models scored above 0.81 BERT-F1 extracting architectural decisions from commits and still missed the point.

arXiv 2609.03721 tested Gemini 3 Pro, DeepSeek R1, Kimi K2 and Qwen3 on 30 developer-written architectural design decisions from open-source projects, scored with ROUGE-L, BLEU, METEOR and BERTScore plus manual review of Gemini's outputs. Few-shot improved alignment (Gemini 0.828 to 0.847). Manual review found the generated ADDs were too long, implementation-focused, and missing the rationale, which is the only reason to write an ADD. A high similarity score on a useless output is the finding (arXiv).