Less Is More: Constraining LLM Delegation Improves Static Analysis Chatbot Accuracy

arXiv·medium signal

Study comparing three architectures along a spectrum of LLM delegation for translating natural language to Joern's query language: direct query generation, schema-constrained JSON intermediate representation, and minimal LLM involvement. Counterintuitively, architectures with less LLM delegation produced higher accuracy. For practitioners building LLM-powered developer tools, this suggests that constraining the LLM's role via structured intermediaries outperforms giving it full code-generation freedom.