Fetching from the wire…
Public story · 2026-09-10 · high
FSDP2 checkpointing was wrapping attention, MLP and norm layers separately instead of the transformer layer as a whole, keeping activations between them in memory during backward.
Why now: Hugging Face published the fix in the 1.15.0 release notes on September 9, 2026.
Hugging Face shipped accelerate 1.15.0 on September 9, fixing an activation checkpointing bug specific to FSDP2. The wrapping logic was applying checkpointing to self_attn, mlp and the norm modules individually instead of the transformer layer that contains them. That meant every activation passed between those child modules stayed resident in memory for backward, which defeats a chunk of what checkpointing is supposed to save.
PR #4172, referenced in the 1.15.0 release notes, wraps the layer itself instead of its children. The fix is narrow: it only matters if you're using FSDP2 with activation checkpointing on a model where the layer boundary and the child module boundaries don't coincide, which is most transformer implementations.
The same release adds activation_checkpointing_offload, an FSDP2-only option that moves per-layer checkpoint inputs to pinned CPU memory. Hugging Face says the gradients it produces are identical to plain checkpointing, just with the inputs parked off-GPU between forward and backward.
A second fix in this release addresses FSDP2 plus PEFT plus FULL_STATE_DICT, which was dropping every rank's adapter shard except rank 0 when saving. That's a separate bug from the checkpointing one, but anyone who's been debugging missing LoRA weights after a distributed save should check #4206.
Neither fix changes the training math. They change how much memory a correct FSDP2 setup actually uses, which is the whole point of running activation checkpointing in the first place.
Each link below shares sources, entities, or timing with this story.
Released September 9, the bug wrapped self_attn, mlp and the norms individually rather than the matched transformer layer, so every activation between those children stayed alive for backward. PR #4172 wraps the layer. The release also fixes FSDP2 + PEFT + FULL_STATE_DICT drop...
The open-source project replaces vector search with ls and find commands, and its LoCoMo scores go from 24-57% to 80-83% accuracy.
Grep and Glob had the same gap, and the release also blocks project settings from logging raw API request bodies.
Boundary-Bench ran 12 agent harnesses through real firewall and filesystem locks, and costs climbed as much as 167 percent as those restrictions tightened.
Self-hosted setups on the bundled vector store jump twelve minor versions, and skipping the staged path corrupts search with no error.
A failed re-ingest could erase chunks that were still searchable in the old version.
MindPattern daily
One email a day at 7 AM. Sources and a take on every story. Unsubscribe anytime.