Micron: AI compute triples while memory bandwidth barely doubles

Accelerator compute keeps growing about 3x every two years while the bandwidth attached to it climbs under 2x, Micron said at its Hot Chips session on August 23. The gap means GPU packages keep getting physically bigger and pricier just to keep memory close to the compute.

Micron's talk traced the HBM line from HBM1 through HBM4 to show how vendors have tried to close it. Bank counts per die are one lever: HBM3E has 128 banks per die, and HBM4 doubles that to 256. More parallelism, not more raw bandwidth.

The physical cost shows up fast. A typical GPU package built with eight HBM4 stacks now tops 12,000 square millimeters, and matching the capacity of a DDR5 module with HBM3E takes about 3x the silicon, per ServeTheHome's account of the session.

Micron's numbers don't say which chipmaker closes the gap first. They say the gap is now measured instead of assumed.