Odin Runs All 32 Llama-3-8B Transformer Layers Under FHE in 366 Seconds on One H100, 4.51x Faster Than THOR

arXiv 2609.12378·medium signalXBlueskyLinkedInCopy link

Odin co-designs CKKS ciphertext packing with model execution, using a feature-major cross-layer layout to unify residual connections and layer interfaces and transient intra-operator layouts for linear projections and attention, so QK^T scores feed softmax directly and PV consumes the probabilities without intermediate repacking. Nonlinear ops use minimax polynomial approximation with input-range control and joint error allocation guided by model quality, cutting polynomial degree and multiplicative depth. With Llama-3-8B weights and a 128-token input it takes 366.4 s and 58.9 GiB peak device memory on a single H100 80GB against THOR's 1651.9 s, and the authors call it the first open-source end-to-end GPU CKKS implementation of Llama-3.