System Prompts Leave Detectable Behavioral Fingerprints, but One Formal-Tone Sentence Collapses Detection From 0.978 to 0.547 AUC

arXiv 2608.24461·medium signalXBlueskyLinkedInCopy link

System prompts can be extracted from commercial LLMs with over 80% success and redeployed for free, leaving owners no way to prove a suspected deployment is a clone. Black-Box Behavioral Fingerprinting registers a behavioral signature from outputs and later tests a suspect deployment against it using only API access. In a study across 4 model families, 8 benchmarks and 288,000 responses, prompt choice explains 24.4% of output variance and same-model detection reaches AUC 0.876, though cross-model performance depends heavily on which model does the detecting (0.845 with Claude as detector down to 0.665 with Qwen, mean 0.725). It survives non-adaptive paraphrasing at AUC ≥0.889, but a single-sentence formal-tone prefix collapses MNLI detection from 0.978 to 0.547.