CLIP Style Classification Accuracy Drops 0.87→0.77 Once You Stop Letting the Model Recognize the Artist

arXiv 2608.14435·low signalXBlueskyLinkedInCopy link

Standard evaluation of frozen-embedding style classification uses random splits where works by the same artist appear on both sides, so a classifier can score well by identifying the painter instead of the movement. Under an artist-disjoint protocol on 320 paintings across four twentieth-century movements, 5-NN accuracy falls from 0.87 to 0.77, and unevenly — Impressionism and Cubism barely move while Surrealism drops twenty points. The pattern holds across four encoders including a vision-only self-supervised model, placing the effect in visual structure rather than language, and it is a clean template for the broader trap of leakage-by-entity in any embedding benchmark.