Production APIs used as agent tools fail silently: prose-only constraints returned a wrong-looking 200 in 44 of 61 cases

arXiv·high signalXBlueskyLinkedInCopy link

SilentProbe audited 721,320 parameters across 2,501 independently published OpenAPI documents and found only 7.5% declare an enumeration and 15.2% declare any machine-checkable constraint, while 40.1% of documents state a constraint in prose that the schema never encodes. Executing 219 schema-derived perturbations against live commercial endpoints from 27 vendors, machine-checkable constraints produced an honest error in 111 of 111 cases while prose-only constraints failed silently in 44 of 61 (p = 2e-13). Across twelve models from eight families, a vocabulary merely exemplified in the description was missed on 88 of 88 attempts, while fully written-out vocabularies were used correctly 88-91% of the time.