Micron Says AI Memory Wall Worsens as Compute Outpaces HBM Bandwidth 3x Every Two Years
This digest was compiled by AI from multiple sources — links to the originals are below.
Micron says the AI memory wall is worsening as compute scales 3x every two years while HBM bandwidth scales under 2x. The company presented the analysis at Hot Chips 2026, where it also outlined packaging and process innovations to address rising thermal constraints. The gap persists even as HBM stack heights have grown from 4-high to 16-high.
Key Facts
- Micron says compute capabilities advance at a rate of 3x every two years, while memory advances at less than 2x every two years.
- In a typical GPU SIP with four 12-high HBM stacks, memory silicon accounts for about 90% of the total silicon, or 8x the GPU silicon.
- Meta's Llama 3 training paper attributed 17% of unintended interruptions to HBM.
- Micron's HBM4 offers up to 2800 GB/s bandwidth, twice the IO, and double the channels of HBM3E.
Memory Wall Analysis
Micron Design Architecture Fellow Raghu Sreeramaneni presented the memory wall analysis at Hot Chips 2026. The company states that compute capabilities are advancing at a rate of 3x every two years, while memory is only advancing at less than 2x every two years. AI workloads are often memory-bound, and higher HBM bandwidth shifts the roofline upward, unlocking more of the processor's peak performance. Micron identifies three memory architectures: 2.5D attached memory, 2.5D advanced memory, and processing-in-memory DRAM.
HBM Scaling Challenges
Stack heights have grown from 4-high to 16-high, with a path to 20-high facing major thermal and mechanical hurdles. As density increases per DRAM, efficiency in pJ/bit goes down, requiring disruptive process and packaging technologies. Micron says the industry needs advanced D2D PHY capability to offload from the GPU and support custom features. HBM is the most cross-functionally complex memory solution, combining packaging, process, and design in a form factor the size of a postage stamp.
1 source
Micron Says AI Memory Wall Worsens as Compute Outpaces HBM Bandwidth 3x Every Two Years
