Gemini Pro 1.5 With Prompt Chaining Reaches 25.99% Mutation Score on Solidity Properties vs 31.75% for Human Experts
Property-based testing plus fuzzing is the strongest pre-deployment defense for immutable smart contracts, but writing the properties is expensive expert work. This study (arXiv 2607.23308) evaluates state-of-the-art LLMs generating Solidity properties, scoring quality by mutation testing across zero-shot, few-shot, and prompt-chaining regimes. Gemini Pro 1.5 with prompt chaining posts the best average mutation score at 25.99% against a human-written benchmark of 31.75% — close but not equal. Per-contract variance is the real finding: on the LibBit contract the same configuration scores 74.34% versus 74.83% for human properties, showing LLMs can match expert-level generation on some contracts while averages conceal the spread.
↳ Follow the thread