Permutation-Based Model Confidentiality in Hybrid FHE Breaks in d+1 Queries Per Layer

arXiv 2609.12911·medium signalXBlueskyLinkedInCopy link

Hybrid FHE inference has the server evaluate linear layers homomorphically while the client decrypts and applies nonlinearities, and recent schemes try to hide the model by returning noisy output-permuted responses backed by shuffle-model differential privacy. This paper shows that fails inside the correctness regime those systems need: d+1 admissible queries to a d-input linear layer exactly recover a permutation-invariant layer summary, which is enough for perfect model distinguishability. They recover every linear layer of a ResNet-20 from TFHE transcripts with zero error in 5,712 queries and confirm exact per-layer recovery on ImageNet-scale CNNs and ViT-B/16.