Why AI needs memory under the chip

Every word a language model writes means reading its weights, and its memory of the conversation, from memory. Raptor, the first commercial 3D-DRAM accelerator for generative inference, puts that memory directly beneath the processor.

The bottleneck

Try it: pick how long the conversation is and how many people are using the card, and watch how long one new word takes to read.

Conversation length

Each new word reads about 80 GB: the model plus 10 GB of conversation memory (KV cache).

HBM4 card ~18 TB/s
4.4 ms
Raptor 3D-DRAM ~100 TB/s
0.8 ms

Idealized: Llama-3.1-70B at 8-bit (about 70 GB) plus its KV cache (about 0.16 MB per token per user), streamed once at peak bandwidth. Real deployments split the work across cards; the paper reports measured results.

Beside, or underneath

package logic HBM stack narrow edge DRAM logic thousands of short vertical bonds across the whole face
HBM sits beside the processor, so all of its data must cross a narrow edge.

On silicon

  • up to 100 TB/sof 3D-DRAM bandwidth per card
  • 4.71×higher throughput than HBM
  • 2.44×higher throughput than SRAM

Across Llama-3.1 70B, DeepSeek-V3, Kimi K2, GPT-OSS, Whisper, and Canary, from Early Silicon of Raptor: The First 3D-DRAM Accelerator for Generative Inference (ISCA 2026).

Go deeper