Combining Hessian-Guided Block Pruning, Quantization and Distillation Beats Applying Them in Isolation

arXiv 2609.05334·low signalXBlueskyLinkedInCopy link

arXiv 2609.05334 notes that ViT compression work usually studies pruning, quantization and knowledge distillation separately, leaving their interactions unexplored. The paper proposes a unified framework pairing Hessian-Balanced Adaptive Block Pruning, guided by second-order sensitivity estimation, with quantization and attention-based knowledge distillation, then sweeps configurations within each combination. The application is on-device chilli plant disease detection in Indian field conditions, but the combined-compression ablation is the transferable part for anyone shrinking a transformer for edge deployment.