ERUnderstand: VLMs Recover Common ER-Diagram Elements at F1 > 0.74 but Collapse to 0.07 F1 on N-ary Relationships
ERUnderstand (arXiv 2607.24707, July 27) is the first large-scale benchmark for structured understanding of Entity-Relationship diagrams, with 2,960 diagrams from curated educational sources, real-world schemas, and synthetic generation spanning multiple notations, complexity levels, and Extended-ER constructs, each paired with a machine-readable target schema. State-of-the-art VLMs handle common elements reliably (F1 > 0.74) but fall off a cliff on weak entities (0.28 F1), multivalued attributes (0.14 F1), and N-ary relationships (0.07 F1). Reasoning-augmented models gain 15–25% overall yet stay sensitive to linguistic priors and diagram complexity. Benchmark, dataset, evaluation toolkit, and generation code are released at github.com/salinaria/ERUnderstand — a concrete caution for anyone building schema-extraction-from-screenshot workflows.
↳ Follow the thread