Six VLMs Tested as Game-QA Bug Detectors: All Produce Heavy False Positives, Gemini-3.1-Flash Most Prompt-Robust

arXiv 2607.25921·medium signalXBlueskyLinkedInCopy link

A paper posted 2026-07-28 (arXiv 2607.25921) evaluates six recent VLMs — Gemini, GPT, Qwen, Gemma, Llama, and Ministral — on zero-shot geometry-clipping detection inside an agent-driven game QA pipeline, where a custom exploration agent navigates a level and an automatic annotation pipeline supplies frame-level labels, removing manual annotation from the loop. Under four prompt variants, every model captures visual cues associated with clipping but produces substantial false positives on ambiguous frames such as near-contact geometry and partial occlusions. Gemini-3.1-Flash achieves the best overall accuracy and is most robust to prompt variation, while open-source models swing widely on precision-recall depending on prompt design. The authors' conclusion is the practical one: use current VLMs as high-recall candidate filters in multi-stage pipelines, not standalone bug detectors.