Tool retrieval benchmarks penalize correct answers, paper finds

A new benchmark re-annotation found that 67.9% of queries in a widely used tool-retrieval test have valid answers the scoring never counted. Tool retrieval benchmarks usually label each query with exactly one correct tool combination, so any other working combination gets marked wrong.

That matters because the scores drive real decisions. Teams use them to decide which retriever ships and whether a fine-tuning pass was worth the compute.

The method behind the finding, called ToolEX, automatically finds tool combinations that work as well as the labeled answer but score as failures under single-label grading. Applied to Tool-DE, a 7,360-query benchmark, it expanded the ground truth to an average of 5.3 valid combinations per query, according to the ToolEX paper.

The researchers then re-ran eight base retrievers and two fine-tuned variants against the expanded benchmark, called ToolEQ. Between 30% and 47% of the reported fine-tuning gain disappeared. The models weren't improving at the rate anyone thought. The benchmark was wrong about what counted as a miss.

The same one-label problem showed up on skill retrieval too. That's a direct hit if you're tuning a skill or tool selector against a single-answer set and trusting the delta between runs to tell you the fine-tune worked.

The paper doesn't say whether other widely used tool-retrieval benchmarks beyond Tool-DE carry the same rate of unlabeled equivalents, so how far this generalizes past the one re-annotated dataset is unclear. If a retriever change looks too clean, check whether the eval set has more than one right answer per query before trusting the number.