Treating layer pruning as an Ising problem beats block-influence heuristics by 23 MMLU points

Hugging Face Blog·medium signalXBlueskyLinkedInCopy link

Multiverse Computing posted a method that turns "which transformer blocks do I delete" into a constrained binary optimization on an Ising glass, using a second-order Taylor expansion of the loss so the Hessian's off-diagonal entries become pairwise couplings between blocks — the part heuristics ignore. Removing 40 of 80 blocks from Llama-3.3-70B-Instruct, their correlated method holds 76.9 MMLU against 82.2 for the full model, while block-influence pruning drops to 54.0. Small cases are solved by GPU brute force over up to ~29 billion configurations, larger ones by an open-source tabu solver in seconds, and they also pruned Qwen3-14B and NVIDIA's heterogeneous Nemotron-3-Nano-30B-A3B-FP8.