Memory for AI: why inference and training wait on memory
Generative inference reads the model weights and the KV cache for every output token, and recommendation training reads and updates embedding tables that do not fit on a GPU. This page explains, with four of our papers, how the memory system sets the performance of both, and what can be done about it.
Part IGenerative inference and 3D-DRAM
Serving a large language model is predominantly memory-bound. This part explains why, quantifies the bound with a roofline, and then describes Raptor, the first commercial accelerator for generative inference with 3D-stacked DRAM (3D-DRAM). A roofline is a bound on the speed of a workload from the compute rate and the memory bandwidth of the hardware. This part follows our ISCA 2026 paper and the keynote Scaling the Memory Wall.
Two phases
Key idea Generative inference has two phases. Prefill processes all input tokens in parallel and is compute-bound. Decode generates one token at a time, and every token reads the model weights and the key–value (KV) cache of all earlier tokens. Decode is therefore limited by memory bandwidth and memory capacity.
Decoder-only Transformers generate a sequence autoregressively. They factorize the joint probability of $T$ tokens as:
Each token depends on all prior tokens, so decoding is inherently sequential. This dependency leads to two phases with different system behavior. The prefill phase processes the input context with large, highly parallel matrix multiplications and exhibits strong weight reuse. The decode phase produces one token at a time and repeatedly reads and writes the KV cache, which drives high memory traffic with limited reuse. About 90% of the wall-clock time of inference is spent in decode.
Two metrics capture the performance of decode. Time-per-Output-Token (TPOT) is the latency of one decode step. Its inverse, the tokens per second that one user receives, is the interactivity of a session. Throughput is measured per user and per accelerator card. For $U$ concurrent users on a deployment of $N$ cards:
The paper writes the second relation for one card. The memory system constrains both metrics: its bandwidth sets how quickly the bytes of one decode step can be moved, and its capacity sets how many sessions fit on a given number of cards.
The KV cache
Key idea Every generated token appends one key vector and one value vector per key–value head in every layer. The KV cache therefore grows linearly with the context length and with the number of concurrent users, while the weights stay fixed.
For a model with $L$ layers, $H_{KV}$ key–value heads, a head dimension of $d_{head}$ and $q$ bytes per element, each token adds:
A Llama-class model with 70 billion parameters occupies about 70GB at 8-bit precision. It has 80 layers and uses grouped-query attention, in which several query heads share one key–value head, with roughly 8 KV heads and a head dimension of about 128. At 8-bit storage, Equation $\eqref{eq:kvtoken}$ gives about 0.16MB per token per user. The KV cache of one user is then nearly 1.25GB at a context of 8K tokens, 5GB at 32K and 20GB at 128K. With concurrent sessions the footprint multiplies: in the example of the paper, the KV cache of this model increases from 10GB at a batch of one to 320GB at a batch of 32. This imposes a hard cap on the number of sessions per card.
Figure 2. Equation $\eqref{eq:kvtoken}$ for Llama-3.1 70B. Move the sliders to change the context length and the number of concurrent users. The dashed lines mark the capacity of one Raptor card with 3D-DRAM (32GB) and of one card with high-bandwidth memory, HBM (192GB); both are described below. The weights need a further 70GB. As in the paper, the KV cache is counted in units of $2^{30}$ bytes.
KV traffic alone needs several TB/s
A lower bound on the required DRAM bandwidth follows from the KV reads alone, ignoring the weights and the other activations. If attention reads $S$ prior tokens and writes one new token per step, the KV bytes moved per output token are:
The bandwidth needed to sustain a decode rate of $r$ tokens per second is $r \cdot D_{KV}$. With 16-bit KV storage, each token of history contributes 320KB, so a 64K context moves about 20GB of KV data for every output token.
Figure 3. Equation $\eqref{eq:dkv}$ for Llama-3 70B with 16-bit KV storage, converted to the bandwidth needed for a target decode rate. The dashed lines are the peak bandwidth of an HBM3 accelerator (about 3.35 TB/s), an HBM4 card (about 18 TB/s) and a Raptor card with 3D-DRAM (about 100 TB/s). Move the sliders to change the context length and the decode rate.
The KV cache sets the capacity that a deployment needs, and the weights and the KV cache together set the bytes that each token moves. The next section compares those bytes with the arithmetic that each token needs.
A roofline for decode
Key idea The arithmetic intensity of a workload is the number of floating-point operations it performs per byte it moves. An accelerator has a balance point: its peak compute divided by its memory bandwidth. A workload whose intensity is below the balance point cannot keep the compute units busy, and its speed is set by the memory bandwidth alone.
Let one decode step perform $F$ floating-point operations and move $D$ bytes between memory and logic. Let the accelerator provide a peak compute rate of $C$ operations per second and a memory bandwidth of $B$ bytes per second. Then:
The step takes at least $D/B$ seconds to move its data and at least $F/C$ seconds to compute, so:
Equation $\eqref{eq:roof}$ is the roofline. For $AI \lt AI^{*}$ the attained compute rises in proportion to the bandwidth, and the compute units are idle for a fraction $1 - AI/AI^{*}$ of the time. For $AI \ge AI^{*}$ the workload is compute-bound.
For Llama-70B, the keynote uses about 140 GFLOP ($140 \times 10^{9}$ floating-point operations) per token of decode and 75GB moved per token. These values correspond to two operations per parameter, the 70GB of weights and the 5GB KV cache of a 32K context. The arithmetic intensity is $140\ \text{GFLOP} \div 75\ \text{GB} = 1.9$ operations per byte. At 5 PFLOPS ($5 \times 10^{15}$ operations per second) of compute, a card with HBM at 18 TB/s has a balance point of 278, and a card with 3D-DRAM at 100 TB/s has a balance point of 50. Decode at a batch of one operates $146\times$ below the HBM balance point and $26\times$ below the 3D-DRAM balance point. More bandwidth lowers the balance point. While decode is memory-bound, TPOT falls in proportion to the bandwidth.
Figure 4. The roofline of Equation $\eqref{eq:roof}$ for Llama-70B decode, on a single card that holds the whole model, computed for this page from the numbers of the keynote. The solid line is the chosen bandwidth, and the dashed lines are HBM and 3D-DRAM at the chosen compute. The dot is the operating point. Move the sliders to change the bandwidth, the compute, the batch size and the context length. The keynote uses 5 PFLOPS, and the evaluation in the paper uses 10 PFLOPS. The keynote rounds the arithmetic intensity to 1.9, which gives $146\times$ and $26\times$.
The roofline shows one way to use the compute: raise the arithmetic intensity to the balance point. Batching raises it, up to the three limits described next.
Batching and its limits
Key idea A batch of $b$ users reads the weights once and reuses them $b$ times, so the arithmetic intensity rises with $b$. The KV cache does not amortize: each user has a separate cache that must be stored and read. Batching therefore trades capacity and interactivity for throughput, and it cannot be raised freely.
With a batch of $b$ sequences, one decode step performs $b \cdot F$ operations and moves the weights $W$ once and $b$ KV caches:
Three effects limit the batch size. The first is capacity: the memory must hold $W + b \cdot KV_{user}$, which is $70 + 8 \times 20 = 230$GB for eight users at a 128K context. The second is the ceiling of Equation $\eqref{eq:aib}$: at long contexts the KV traffic dominates, and the intensity stops rising. The third is the arrival process, because a batch can only contain the requests that are present.
We model request arrivals as a Poisson process in front of a batched compute engine. Whenever the engine becomes idle, it forms a batch from the pending requests. An arrival rate of 110 requests per second, about 9.5 million requests per day, is a representative production load. With 1 ms of latency per batch, the average batch at this rate is 1.00.
Figure 5. Average batch size under Poisson arrivals with a per-batch latency of 1 ms, from the simulation in the paper. The maximum batch sizes observed are 1, 3, 5 and 27. The average batch stays well below 32 even at nine times the representative load, and the paper uses 32 as a practical upper operating point.
Mixture-of-experts (MoE) models are further from the balance point. Each token activates only a subset of the experts, so different tokens in a batch read different weights, and the arithmetic intensity rises sub-linearly with the batch size.
Figure 6. Arithmetic intensity against the batch size, read from a figure of the keynote; values between the read points are interpolated. The solid curve is the decode of Kimi K3, a mixture-of-experts model in which each token activates 16 of 896 experts. The dashed line is a dense reference with 104 billion active parameters whose weights are read once per step, drawn as the line $1.5\,b$. The horizontal lines are the balance points of HBM and 3D-DRAM at 5 PFLOPS. Move the slider to change the batch size.
MoE decode needs about 400 concurrent tokens to reach a balance point of 50, far more than the batches that the arrival process provides. Batching cannot raise the arithmetic intensity to the balance point. The remaining option is a memory system with more bandwidth.
Bandwidth or capacity
Key idea On-die SRAM offers very high bandwidth but only a few GB. High-bandwidth memory (HBM) offers hundreds of GB, but its bandwidth and its I/O energy are limited by the interposer between memory and logic. 3D-stacked DRAM places the memory die directly under the logic die and connects the two with dense vertical links, which provides most of the bandwidth of SRAM with far more capacity.
Commercial accelerators primarily rely on two memory substrates. On-die SRAM is fast, but a six-transistor SRAM cell is 100 to 150 times larger than the one-transistor, one-capacitor cell of DRAM, so its capacity cannot scale to large weights and KV caches. Off-die DRAM is dense but is constrained by interface bandwidth and I/O energy. HBM places stacks of DRAM dies beside the logic on an interposer. Each HBM generation has raised bandwidth by increasing pin speed and interface width: HBM3 keeps a 1,024-bit data interface and doubles the channels to 16, HBM3E pushes pin speeds beyond 9.2 Gb/s to exceed 1.2 TB/s per placement, and HBM4 doubles the interface to 2,048 bits for up to 2 TB/s per stack. The energy per bit of HBM is not scaling at the same rate, and decode is dominated by bytes moved.
| Card memory | Bandwidth | Capacity | Limitation |
|---|---|---|---|
| SRAM, on die | 150 TB/s | 4GB | Capacity. Llama-70B needs at least 18 cards for the weights alone. |
| HBM | 18 TB/s | 192GB | Bandwidth and I/O power. 75GB per token takes 4.2 ms to read. |
| 3D-DRAM (Raptor) | 100 TB/s | 32GB | Integration: mapping, I/O power, yield and temperature. |
The bandwidths and capacities are the three configurations that the paper evaluates, each paired with the same accelerator logic at 10 PFLOPS. The remarks in the last column are from the keynote. A decode step that takes 4.2 ms on reads alone consumes 42% of a TPOT budget of 10 ms and 8% of a budget of 50 ms before any arithmetic is done.
Figure 8. Capacity and bandwidth of the three card memories for Llama-3.1 70B at a chosen context and number of users, computed for this page from the table above. The read time is the time to move the bytes of one decode step through the interface of a single card, and the number of cards is the minimum for capacity alone. The energy uses 0.376 pJ/bit for 3D-DRAM and six times the unflipped 0.45 pJ/bit for HBM3, as measured and cited in the paper, and about 0.5 pJ/bit for SRAM, from the keynote.
In the keynote's comparison per unit of silicon area, Raptor's 3D-DRAM provides about 20 times the bandwidth per mm2 of HBM4 and 13.5 times better power per GB/s, at about half the capacity per mm2. This trade favors memory-bound decode.
Placing the DRAM under the logic creates four problems. The rest of this part describes them and how Raptor solves each.
RaptorISCA 2026
Key idea Raptor bonds a logic die face to face on a 3D-DRAM die and gives every tensor engine its own DRAM channel. The 840 DRAM banks of a chiplet are grouped into 256 independent channels, so a maintenance operation on one bank, such as a refresh, does not stall the other channels.
Each accelerator card integrates two to four multi-chip modules (MCMs), and each MCM contains four 1.2 GHz chiplets. A chiplet stacks a logic die, built in TSMC N4P, on a 3D-DRAM die through a µbump array with a pitch of 36 µm. The logic is organized as a hierarchy of slices, gangs and chiplets.
- Slice. The unit of compute and memory co-location. Its 16 tensor engines (TEs) perform the multiply-accumulate operations. Activations are staged in slice-level SRAM. The 3D-DRAM stores the weights and the KV cache and feeds each weight buffer through a dedicated channel. A SIMD core handles auxiliary vector and transcendental operations.
- Gang. Four slices that jointly process a wider tensor shard and share work across nearby channels, without imposing chiplet-wide control on every local operation.
- Chiplet. Four gangs, with 16 channels per slice and 256 per chiplet.
- MCM and card. Chiplets within an MCM communicate over die-to-die links at 32 Gbps per lane. Each MCM targets about 422 W at junction temperatures up to 105 °C, and carries eight LPDDR5X devices that provide 128GB as a secondary memory tier.
Each 3D-DRAM bank is an array of 1,364 rows and 124 columns. The row buffer, which holds the row that is currently open, spans all 124 columns, and each column access returns 256 bits, which is 32 bytes. Four problems follow from this organization: how to map banks to channels, how to save I/O power without the mechanisms of commodity DRAM, how to tolerate faulty banks, and how to remain reliable at 105 °C.
Stream blocking
Key idea A weight buffer consumes data in units of 128 bytes, called flits, but a channel of three banks returns 96 bytes per access. Stream blocking splits each flit into a 96-byte aligned part and a 32-byte partial, and packs the partials of three consecutive flits into one shared access. Four accesses then deliver exactly three flits, and no fetched byte is discarded.
Each slice has 16 weight buffers and 16 DRAM channels, and each channel delivers a 128-byte flit to its buffer. One column access to one bank supplies 32 bytes, so a 128-byte flit needs four co-accessed banks. The 256 channels of a chiplet would then need 1,024 banks, but the die implements 840. After reserving 72 banks as spares, 768 remain, which is exactly three banks per channel.
A flit then requires two accesses, which fetch 192 bytes for 128 bytes of payload: one third of the fetched bytes are discarded, and about 33 TB/s of a card's raw bandwidth is lost. Staggering the column indices across banks removes the overfetch, but the controller must then buffer 192 bytes, shift unaligned fragments into flits and track a column pattern per address. This complicates the datapath and hinders timing closure.
Stream blocking instead relies on the software stack. The stack tiles the weights and the KV cache into per-channel blocks that are read as near-unit-stride streams of flits. On a read, the controller first accesses the partial region and caches 96 bytes, which are the 32-byte fragments of three consecutive flits. Each following access fetches one 96-byte aligned region and merges it with a buffered partial. Both kinds of access follow a fixed pattern with two 96-byte buffers.
Figure 10. Column accesses needed to deliver a stream of 128-byte flits from a channel of three banks. Each group of three cells is one access, and each cell is the 32 bytes that one bank returns. Choose a scheme and move the slider to change the number of flits.
A layer of KV cache on one slice
Consider one attention layer of Llama-3.1-70B with 16-bit storage. Each token produces $2 \times 8 \times 128 \times 2 = 4$KB of KV state, so the layer cache is 16MB at a 4K context. The stack partitions it into 1,024 stream-blocked tiles of 16KB and spreads them evenly across the 16 channels of the slice, 64 tiles per channel. A tile is 128 flits and spans roughly two rows per bank. Successive flits read consecutive columns of an open row, and the row buffer is fully read before the next activation. The layer then occupies about 128 of the 1,364 rows of each bank, which is under 10%. New KV entries are appended to the tail of a tile. The 16KB tile matches the page sizes of paged attention, so allocation and eviction occur at page boundaries.
Figure 11. The footprint of one attention layer of Llama-3.1-70B on one slice, following the example in the paper at a 4K context and scaled linearly, for this page, to other context lengths. Move the slider to change the context length. The bar shows the share of each bank's 1,364 rows that the layer occupies.
Stream flipping
Key idea The I/O energy increases with the number of bits that toggle on the µbumps. Stream flipping compares each flit with the previous flit on the same channel and stores it inverted when more than half of its bits would toggle. One metadata bit per flit records the choice. The switching rate can then never exceed 50%, and no pin is added.
The short vertical links of the 3D stack reduce the I/O energy to 0.45 pJ/bit, about six times lower than HBM3. The much higher aggregate bandwidth still amplifies the total switching power:
Commodity DRAM interfaces reduce transitions with data bus inversion (DBI). For each transfer of a multi-cycle burst, the physical-layer interface (PHY) decides whether to invert the word and signals the choice on a dedicated pin. Raptor's 3D-DRAM transfers a single-cycle 256-bit word per bank. The interface has no burst and no DBI pin, so the same effect must be obtained at the architectural level.
Stream flipping exploits the layout that stream blocking creates: each channel carries long streams of flits, and the software stack controls their placement. On a write, the memory controller compares each flit with the previously written flit on the same channel, inverts it if that reduces the transitions, and records one metadata bit in a compact side region. On a read, the controller fetches the metadata bit with the flit and inverts the flit if the bit is set.
Figure 12. Stream flipping on a stream of flits, shown with 24 bits per flit instead of 1,024. A filled cell is a bit that toggles relative to the previous flit on the wire. Move the sliders to change how different consecutive flits are and the bandwidth of the card. The energy per bit is the line, drawn for this page, through the measurements at 3% and at 100% switching with eight active banks; the measurement with stream flipping, 0.376 pJ/bit, corresponds to 48% on this line. The toggle counts are an idealized model in which every flit differs from the previous flit in the same number of bits, and the drawing rounds to 24 bits. The paper measures 40% to 48% effective switching on real data. The 360 W of the paper and the 296 W of the keynote use the rounded values of 0.45 and 0.37 pJ/bit.
Figure 13. Measured I/O energy against the number of active banks at 500 MHz, read from the paper. At eight banks the worst case of 100% switching costs 0.455 pJ/bit. Stream flipping reduces the effective switching to between 40% and 48% and the energy to 0.376 pJ/bit, an 18% reduction.
- 18%lower 3D-DRAM I/O energy, with no change to the DRAM PHY
- 360 W to 296 Wof I/O power per card at 100 TB/s, from the keynote
- 1 bit per flitof metadata, stored with the error-correcting code
Redundancy that preserves the topology
Key idea Channels feed the weight buffers in lockstep, so every channel must keep its full width. Raptor places spare banks in line with the regular banks. A chain of $N$ functional and $M$ redundant banks can skip any $M$ faulty banks with $M$ levels of multiplexing, and every channel keeps its full width.
A primary reliability concern is bank failure during manufacturing. Each bank includes spare rows and columns, but if these cannot repair all defects, the bank is marked faulty. In a die with 840 banks, discarding every die that has a few faulty banks is uneconomical. Disabling the faulty banks is not sufficient either: the affected channel would run at a reduced width or capacity, and an under-provisioned channel becomes the bottleneck.
A naive scheme would place all redundant banks at the edge of the die and route around faulty banks with long wires. This increases metal usage, latency and power on an interface that is already power-sensitive. Raptor instead distributes the spare banks alongside the regular banks and chains them. Lightweight multiplexers on the logic die select the non-faulty banks of a chain in order, and consecutive groups of three form the channels. For example, $N = 24$ and $M = 2$ form eight channels of three banks from any 24 non-faulty banks in a chain of 26. The routing stays local to the tensor engine that the channel feeds.
Figure 14. A chain of 24 functional banks and a chosen number of redundant banks. Press a bank to mark it faulty or to repair it. The remaining banks are assigned to channels in order. The chain fails only when fewer than 24 banks are usable.
We model the failure as follows. A chain fails when more than $M$ of its $N + M$ banks are faulty. If banks fail independently with probability $p$, and a die has $G$ chains:
Figure 15. Probability that a DRAM die cannot form all of its channels, against the bank fault rate. The curves are Equation $\eqref{eq:chain}$, computed for this page with 12 functional banks per redundant bank and 768 functional banks per die. With these parameters the equation reproduces the four curves of the corresponding figure in the paper. The parameters are fitted to those curves; the paper does not print them. They imply 64 redundant banks per die, not the 72 of the Raptor die, and a number of chains that is not an integer for $M = 3$. Move the sliders to change the redundancy and the fault rate.
The paper does not disclose absolute yield. The redundant banks are used in both wafer-level and stack-level repair flows, as is standard in high-volume DRAM manufacturing.
Thermal behavior
Key idea The heatsink is on the logic side of the stack. The thinned die stack, logic and DRAM together, adds only about 1.5% of the total thermal resistance, so stacking does not measurably raise the junction temperature. Because the logic sits above the DRAM, the DRAM runs about 3.5 °C cooler than the logic.
The temperature can reach 105 °C at the interface between the logic and the 3D-DRAM. The path from junction to ambient is analyzed as a series resistor network of five layers: the thinned die stack of 0.62 mm, the first thermal interface material, the copper lid, the second thermal interface material, and the heatsink with its airflow. The cooling solution contributes about 80% of the total thermal resistance $R_\theta$. The entire die stack, including the 3D-DRAM, adds about 0.003 °C/W.
Power-rail analysis shows that the 1.1 V array rail accounts for 87% of the active power of the memory, which centers thermal management on the activity of the DRAM array. At the operating point of about 106 W per chiplet, a quarter of the power of the MCM, and with the optimized heatsink, the tensor-engine arrays form localized hotspots at about 92 °C. The SRAM buffers, the inter-gang crossbar and the PHY regions are 4 to 6 °C cooler. In a conventional HBM stack the DRAM sits above the logic and absorbs the conducted heat. In Raptor the order is inverted.
Figure 16. Peak junction temperature against the power of one chiplet stack for three cooling configurations, read from the analytical model in the paper. Choose a cooling configuration and move the slider to change the power. The refresh interval is 4 ms above 85 °C and 16 ms otherwise, as in the paper's measurement, and the bandwidth loss follows the refresh model of the next section.
- about 62 Wper chiplet stays below 105 °C with baseline air cooling, read from the figure; the text of the paper gives 63 W
- about 93 °Cpeak at 106 W with the optimized heatsink, 12 °C below the limit
- under 60 °Cat 106 W with liquid cooling
Refresh at 105 °C
Key idea At a junction temperature of 105 °C, DRAM cells must be refreshed every 4 ms, eight times more often than the nominal 32 ms of HBM devices. The cost of refresh is proportional to the number of rows in a bank. A Raptor bank has 1,364 rows, 16 to 32 times fewer than a conventional bank, so the faster refresh costs 1.37% of the bandwidth.
At runtime the dominant reliability stressor is temperature. While a bank refreshes its rows it cannot serve its channel. Because the channels are independent, a refresh in one bank does not stall the others. We model the bandwidth lost as the fraction of time each bank spends refreshing:
where $R$ is the number of rows per bank, $t_{row}$ is the time to refresh one row and $t_{REF}$ is the refresh interval. The measured overheads at 700 MHz are 5.46% at 1 ms, 2.70% at 2 ms and 1.37% at 4 ms, which leave 99.3, 102.1 and 103.6 TB/s per card. These values correspond to $t_{row}$ of about 40 ns.
The same interval also mitigates Rowhammer. Raptor's DRAM uses an older technology node with a Rowhammer threshold of 200K. At a row cycle time $t_{RC}$ of 44 ns, back-to-back activations need 8.8 ms to reach this threshold. Every row is refreshed within 4 ms, so no row can accumulate enough activations between two refreshes. The Rowhammer explainer describes the general problem.
Figure 17. Equation $\eqref{eq:refresh}$ with $t_{row} = 40$ ns, fitted for this page to the three overheads measured in the paper. Move the sliders to change the refresh interval and the number of rows per bank. The last tile compares the activations that fit in one interval, at $t_{RC} = 44$ ns, with the threshold of 200K. The cases with more rows per bank are computed for this page.
Interleaved ECC
To harden the memory against manufacturing and runtime faults, each bank uses a blocking error-correcting code (ECC), which stores redundant symbols with the data so that errors can be detected and corrected. The last eight columns store a pair of Reed–Solomon $[144, 140]$ codewords over 8-bit symbols, interleaved to match the subarray mapping. The ECC is co-located with the stream-flipping metadata. On a read, the controller first fetches the ECC and the metadata from all banks in a channel, then the data flits. It checks the ECC, corrects erroneous data if needed, and then applies the conditional inversion. Writes follow the reverse order. Periodic background scrubbing, which reads the data and rewrites corrected values, repairs transient errors before they accumulate.
Measured on silicon
The paper measures the bandwidth, the latency, the I/O energy and the refresh overhead on early silicon. Because each row holds multiple flits, streaming an entire row amortizes the row activation and reduces the latency per flit. With three banks per channel at 700 MHz, the 3D-DRAM delivers an average flit latency of about 2.5 ns and about 105 TB/s per card. This is about 12.5 times the bandwidth of HBM3 cards. Even if HBM4 doubles the bandwidth of HBM, 3D-DRAM retains a 6.25-fold advantage.
Figure 18. Measured 3D-DRAM bandwidth per card and latency per flit against the DRAM frequency, read from the paper, for two, three and four banks per channel. The design with two banks keeps 256 channels and uses the remaining banks for redundancy. The design with four banks is hypothetical and assumes a DRAM die with 33% more banks. Values between the measured frequencies are interpolated. Choose a grouping and move the slider to change the frequency.
One card provides 32GB at about 100 TB/s. A large model has hundreds of GB of weights, so it must be partitioned across cards. The next sections evaluate complete deployments.
From one card to a deployment
Key idea A model that does not fit on one card is partitioned across cards, and every partition adds collective communication, in which a group of cards exchanges or combines data, to each token. The number of cards is set by capacity. A card with more capacity therefore needs a lower degree of parallelism and spends less time in collectives.
Raptor has a three-level interconnect. Within a chiplet, the four gangs communicate over an on-chip network. Within an MCM, the four chiplets use die-to-die links. Across MCMs, cards and nodes, every MCM connects to every switch tray through PCIe Gen 7 or the Ethernet Scale-Up Network, which forms a one-hop tree with all-to-all connectivity within a rack. The aggregate bandwidth at each level scales with the number of links, so the primary distinction between the levels is latency. The paper models the network between cards as a flat fabric with a one-way latency, swept from 0.01 to 10 µs, and an aggregate bandwidth, swept from 32 GB/s to 4 TB/s.
Three forms of parallelism partition a model:
- Tensor parallelism (TP) shards attention heads and feed-forward (FFN) matrices across cards. Each Transformer layer then triggers two all-reduce operations, which sum partial results across the cards, one after the attention projection and one after the FFN, and each transfers $O(h \cdot b)$ bytes for a hidden dimension $h$ and a batch $b$. The paper caps TP at 8.
- Pipeline parallelism (PP) partitions the layers across additional cards, and is used only when a single card cannot hold the required weights and KV cache.
- Expert parallelism (EP) spreads the experts of a mixture-of-experts model across cards. A small TP group handles attention, and each layer adds a dispatch and a combine step that route tokens to the expert cards and back.
Dense models use a unified deployment, in which every layer is distributed across the same set of cards. MoE models use a disaggregated deployment with separate attention and expert pools. The minimal-card deployment uses the fewest cards that can hold the weights and the KV cache.
Figure 19. Cards in the minimal-card deployment of four mixture-of-experts models, from the configurations in the paper. The number of cards is the aggregate memory of the deployment divided by the capacity of one card. Bars are drawn on a logarithmic scale. Choose a model.
SRAM's capacity of 4GB per card forces extreme parallelism: Kimi K2 needs 1,584 cards. HBM needs the fewest cards, but each card is limited to 18 TB/s. 3D-DRAM lies between the two in card count and provides 100 TB/s on every card.
Throughput against interactivity
Key idea Increasing the batch size raises the tokens per second of a card and lowers the tokens per second of each user. The set of batch sizes traces a curve, and the useful part of the curve is its Pareto frontier: the operating points for which no other point has both higher throughput and higher interactivity. A memory with more bandwidth moves the memory-bound part of the frontier toward the upper right.
The evaluation pairs the same accelerator logic with SRAM, HBM and 3D-DRAM, and sweeps the batch size for each. It covers a dense model (Llama-3.1 70B), four mixture-of-experts models (GPT-OSS 20B and 120B, DeepSeek-V3 and Kimi K2) and two speech models (Whisper and Canary). The network between cards has a latency of 0.5 µs and a bandwidth of 1 TB/s. To separate the effect of bandwidth from that of capacity, four further 3D-DRAM design points scale the bandwidth alone by $2\times$ and $4\times$, or the bandwidth and the capacity together.
Figure 20. Throughput per card against interactivity in minimal-card deployments, read from figures of the paper. Each curve is a sweep of the batch size, which increases from the lower right to the upper left. The markers are those that could be read from the figure; a few batch sizes that fall on a straight segment are not marked. Choose a model, a context length and the set of memories to compare.
Dense models show two regimes. At small batch sizes decode is memory-bound, and a larger batch amortizes the transfer of the weights and raises the throughput per card. At large batch sizes the arithmetic units saturate, the throughput per card reaches a plateau and TPOT rises. MoE models show up to four regimes, because attention and expert loading saturate at different batch sizes. HBM matches the throughput of 3D-DRAM only in the compute-bound regime, and with significantly worse interactivity. At a 64K context the larger KV cache reduces the maximum batch size, and DeepSeek-V3 remains in the regime in which attention is memory-bound.
- 4.71×higher throughput per card than HBM, on average
- 2.44×higher throughput per card than SRAM, across the evaluated models
- 9.96×lower TPOT than HBM, on average
At a 4K context, a batch of 32 and a realistic network of 0.5 µs and 1 TB/s, 3D-DRAM improves the throughput per card across the models by $4.38\times$ over HBM and $3.15\times$ over SRAM. The throughput is flat for network latencies below 0.1 µs. Beyond this point the collectives of tensor and pipeline parallelism become significant. SRAM degrades fastest, because its limited capacity forces higher degrees of parallelism and larger collective volumes. For the speech models, which fit on a single card and have a context of 448 tokens, bandwidth rather than capacity determines performance: SRAM delivers the highest throughput per card, followed by 3D-DRAM and then HBM.
A first-order model of the frontier
The shape of these curves follows from the roofline. Consider a dense model with weights $W$ on $N$ cards, of which $TP$ cards operate in parallel on each layer. With a batch of $b$, one decode step moves $W + b \cdot KV_{user}$ bytes, performs $b \cdot F$ operations and, if $N \gt 1$, executes two collectives in each of the $L$ layers:
Here $t_{net}$ is the one-way network latency, $B_{net}$ is the network bandwidth, $h$ is the hidden dimension with two bytes per activation, and $N_{eff} = TP$. The last relation is inferred from the paper's SRAM curve, where 32 cards with $TP = 8$ and a pipeline depth of 4 produce one eighth of the tokens per second of a user at a batch of one; it is consistent with pipeline stages that work on different batches at the same time. Memory-bound operation is the first term of the maximum: the interactivity falls slowly with $b$ while the throughput rises almost in proportion to $b$. Compute-bound operation is the second term. On a single card the throughput is then constant at $C/F$ and the interactivity falls as $1/b$, so a larger batch reduces interactivity without raising throughput. With more than one card the fixed latency of the collectives is spread over a larger batch, and the throughput continues to rise, more slowly, toward $C/F$.
Figure 21. Equations $\eqref{eq:tpot}$ and $\eqref{eq:frontier}$ for Llama-3.1 70B, computed for this page with $C = 10$ PFLOPS, $B_{net} = 1$ TB/s, $h = 8{,}192$ and $F = 147.5$ GFLOP per token. $F$ is fitted to the compute-bound plateau of the paper's curve for four times the bandwidth and capacity; the keynote's roofline uses 140 GFLOP. The model reproduces the corresponding curves of the paper at a 4K context to within 6%; the circles are the paper's points and appear when a configuration of the paper is selected. Move the sliders to change the memory of the card, the context, the network and the batch size. The number of cards is the smallest power of two that holds the weights and one KV cache. With a pipeline depth of $PP$, that many batches are in flight at the same time, so the memory must hold $PP \times b$ KV caches.
The rack: tokens per second per kW
Key idea At the scale of a rack, power limits the number of tokens per second. The two axes become efficiency, in tokens per second per kW, and interactivity, in tokens per second per user. A rack of 3D-DRAM accelerators extends the Pareto frontier to higher interactivity at a comparable peak efficiency.
This section follows the keynote, which describes a rack built from Raptor MCMs and evaluates Kimi K3; the paper evaluates Kimi K2. A 1U compute tray holds eight MCMs, which is four cards, with 128GB of 3D-DRAM and 400 TB/s of bandwidth, and is cooled by liquid cold plates. Scale-up switch trays connect any MCM to any other MCM in one hop, and every collective traverses them.
- 144 MCMsin 18 compute trays of 8
- 7.2 PB/sof memory bandwidth, at 50 TB/s sustained per MCM
- 2.3TBof 3D-DRAM capacity, at 16GB per MCM
- about 200 kWpower envelope, liquid-cooled
The frontier of a rack is the set of operating points that are Pareto-optimal over the batch size and the deployment. A larger batch gives higher efficiency and lower interactivity. For Kimi K3, a mixture-of-experts model, at a 64K context, the simulated frontiers of a GPU rack and of the Raptor rack are as follows.
Figure 22. Simulated Pareto frontiers of efficiency against interactivity for Kimi K3 at a 64K context, read from the keynote. Interactivity is $1000/TPOT$ with TPOT in milliseconds. Move the slider to set the interactivity that each user requires. The readouts give the most efficient operating point of each rack that still meets it. The last readout is computed for this page from the power envelope of about 200 kW.
The peak efficiencies are comparable, at 4,614 tokens/s per kW for the GPU rack and 4,269 for the Raptor rack. The frontiers differ in their extent. The GPU rack's frontier ends near 750 tokens/s per user, and the Raptor rack's frontier extends to about 1,100, which is 1.5 times the interactivity. At 500 tokens/s per user, the Raptor rack is more than 10 times as efficient. At a long context the KV cache dominates the memory traffic, and the bandwidth of the rack shortens each decode step.
Sensitivity to the scale-up network
A rack reaches the bandwidth of 3D-DRAM only if the latency and the bandwidth of the scale-up network do not limit the collectives. Every collective incurs the network latency, and larger batches and models move more data per collective.
Figure 23. Simulated frontiers for Kimi K3 on four 144-MCM Raptor racks at a 64K context, read from the keynote, for five scale-up latencies and four scale-up bandwidths. Choose the parameter to sweep and move the slider to select a value. The other values are drawn in gray.
Raising the latency from 0.6 to 1.4 µs lowers the interactivity ceiling from about 1,750 to about 1,000 tokens/s per user. Raising the bandwidth from 600 to 1,200 GB/s raises the peak efficiency from about 4.5K to about 5.3K tokens/s per kW. This motivates a single-hop, high-radix scale-up network.
The frontiers above use one kind of accelerator for the whole decode loop. A deployment can also assign each part of the loop to the hardware that matches its bottleneck.
Heterogeneous deployments
Production serving stacks increasingly partition the decode loop across hardware pools with complementary strengths. Two directions pair a 3D-DRAM accelerator with an HBM-class GPU.
- Attention–FFN disaggregation splits a single MoE layer between a GPU that runs attention and Raptor that runs the FFN and expert blocks. Attention traffic is dominated by the KV cache, which fits in HBM capacity. FFN traffic is dominated by expert weight loading, which scales with the batch and benefits from the highest available bandwidth. The activation exchange fuses with the all-to-all collectives that expert parallelism already uses.
- Speculative decoding uses a small draft model to propose $K$ candidate tokens and a larger target model to verify them in one parallel forward pass. Verification amortizes the target weights across $K$ tokens and is compute-bound. Drafting is $K$ back-to-back autoregressive steps and is memory-bound. The draft model therefore runs on Raptor and the verification on a GPU.
Related designs
Related proposals embed compute in or near the memory. H2-LLM places near-memory processing in hybrid-bonded DRAM for edge inference, with 0.4 TB/s of channel bandwidth and 0.28 PFLOPS on a single chip. Stratum stacks 1,024 monolithic-3D DRAM layers and reaches high internal bandwidth, but its external interface delivers about 0.82 TB/s per chip, so its effective multi-chip bandwidth is an estimated 10 to 35 TB/s. Both are simulation studies. Raptor decouples compute from memory: a 100 TB/s DRAM subsystem feeds a 10 PFLOPS accelerator on a leading-edge logic process, and the evidence comes from silicon.
Early Silicon of Raptor: The First 3D-DRAM Accelerator for Generative Inference · ISCA 2026 · Keynote slides: Scaling the Memory Wall · Illustrated guide by Kevin Quimbo
Raptor raises the bandwidth that is available to decode. The complementary approach is to reduce the bytes that decode must move. The KV cache grows with the context, and the next paper reduces it.
Part IIMoving fewer bytes per token
The KV cache grows with every token, and each decode step reads all of it. If most of the attention weight falls on a subset of the tokens, the keys and values of the other tokens can be discarded. This reduces both the size of the KV cache and the data that each decode step moves.
KeyformerMLSys 2024
Key idea During generation, about 90% of the attention weight falls on a subset of the tokens, called key tokens. Keyformer keeps only the key tokens and a window of recent tokens in the KV cache. It identifies key tokens with a score function that adds Gumbel noise to the attention logits and raises a temperature as tokens are discarded. The KV cache then has a fixed size, and the model needs no retraining.
Token generation is constrained by memory bandwidth, because the weights and the KV cache must be transferred from the memory system to the compute units. For the language model MPT-7B on an NVIDIA A100 GPU, increasing the sequence length by $16\times$, from 512 to 8K, increases the inference latency by more than $50\times$, and approximately 40% of the inference time is consumed by KV cache data movement. Multi-query and group-query attention reduce the KV cache, but they typically require retraining or fine-tuning. An inference-time method must also preserve accuracy: the MLPerf benchmark mandates that an optimization maintains between 99% and 99.9% of the baseline accuracy.
Attention is concentrated on key tokens
The attention score of a decoding step is the softmax of the scaled logits between the query $Q$ and the keys $K$:
where $d_k$ is the dimension of the keys. Across GPT-J, Cerebras-GPT and MPT, 90% of the attention score is dedicated to about 40% of the tokens. The attention heat maps show that the sparsity of the attention scores is dispersed across layers and heads without a distinct pattern, so the key tokens must be identified during inference.
Figure 24. Average attention score retained against the fraction of the KV cache that is kept, on summarization with CNN/DailyMail, read from a figure of the paper; the values are approximate. Move the slider to change the fraction.
Why the highest attention scores are not enough
The natural score function is the accumulated attention score, as in H2O, a prior method that keeps the tokens with the highest accumulated attention scores. Two effects limit it. First, both key tokens and recent tokens are needed. Keeping only the tokens with the highest accumulated scores loses the recent context, and keeping only a recent window, as window attention does, loses the earlier context that the model needs; each alone performs below full attention at a 50% cache, and for Cerebras-GPT and MPT far below it. Second, discarding tokens changes the score function itself. With $n$ tokens and unnormalized logits $x_i$, the score function is $f_\theta(x_i) = e^{x_i} / \sum_{j=1}^{n} e^{x_j}$. Once the cache holds $k$ tokens, the normalization runs over $k$ terms only:
The attention weight of the discarded tokens is redistributed unevenly among the remaining tokens, and a token that was discarded cannot be reintroduced without recomputing its keys and values. Scaling the scores by a constant damping factor does not restore the quality of full attention.
The Keyformer score function
Keyformer regularizes the logits. It adds noise $\zeta_i$, drawn from a Gumbel distribution, to each unnormalized logit $x_i$, and it divides by a temperature $\tau$:
The Gumbel distribution characterizes the maximum of a set of samples, and the paper selects it because, after normalization, it yields, in expectation, a more uniform distribution than the unregularized logits. The temperature accounts for the tokens that have been discarded. A higher $\tau$ gives a more uniform score, and a lower $\tau$ gives a sharper one. Keyformer raises $\tau$ linearly over the $T$ tokens that are generated:
where $t$ is the index of the decoding step, $\tau_{init} = 1$ and $\tau_{end} = 2$. The score is accumulated over the decoding steps, separately for each layer and head, so that a key token is one that receives a high score across most decoding steps.
- Prompt phase. Keyformer computes the keys and values of all $n$ prompt tokens. It keeps the $w$ most recent tokens and selects the $k - w$ tokens with the highest scores from the other $n - w$. These $k$ tokens form the reduced KV cache.
- Generation phase. Each new token attends only to the $k$ cached tokens. The recent window shifts right by one token, the scores accumulate, and the $k - w$ key tokens are selected again from a window of $k + 1 - w$. One token is added and one is removed, so the cache stays at $k$ tokens.
Figure 25. One selection step of Equation $\eqref{eq:kf}$ on the eight-token example of the paper, computed for this page. The logits are the logarithms of the full-attention scores that the paper prints for these tokens. The noise is one fixed sample, and the score is not accumulated over decoding steps, so the figure shows how the selection is formed and not the accuracy result of the paper. In a single step the temperature changes the scores but not their order. Move the sliders to change the cache size $k$, the recent window $w$ and the position in the generation, which sets $\tau$ by Equation $\eqref{eq:tau}$. Choose whether Gumbel noise is added.
Accuracy against the size of the KV cache
The paper evaluates GPT-J, Cerebras-GPT and MPT, which use three different position encodings, on summarization (CNN/DailyMail, GovReport) and conversation (SODA). The baselines are full attention, window attention and H2O.
Figure 26. ROUGE-2 against the KV cache budget, as a percentage of the prompt length, read from a figure of the paper; the values are approximate. ROUGE-2 measures the overlap of bigrams between the generated and the reference text. The dashed lines are full attention and 99% of full attention. The Keyformer series of the paper ends at the first budget that reaches the target. Choose a model and a task, and move the slider to change the budget.
On summarization with CNN/DailyMail, Keyformer attains the target with 70% of the KV cache, whereas window attention and H2O fall short of it even with 90%. For long-context summarization with GovReport, at a sequence length of 8K, Keyformer maintains the 99% threshold with a 50% KV cache. On summarization it also exceeds the accuracy of full attention by up to 1.73%, which the paper attributes to the regularizing effect of the noise. The paper also compares Gumbel noise with three other logit adjustments at a 60% KV cache.
Figure 27. ROUGE-2 of MPT-7B on summarization with a 60% KV cache, for four choices of logit adjustment. Gaussian noise with the same mean and variance and a constant offset both perform worse than no adjustment. The corresponding values for GPT-J are 19.44, 18.87, 14.53 and 12.49.
Performance
A smaller KV cache helps in two ways. It reduces the data movement from the off-chip memory of the GPU, and it reduces the scaled dot product of attention from $n$ to $k$ tokens. For sequences of 4K tokens and a 50% cache, Keyformer decreases the KV cache data movement by $2.9\times$ and reduces the time of the scaled dot product by $1.3\times$. The Gumbel softmax adds some overhead.
Figure 28. Generation throughput of MPT-7B for three prompt and generation lengths, from the paper. H2O uses a 90% KV cache and Keyformer a 50% KV cache. Choose a sequence length.
- up to 2.1×lower inference latency with a 50% KV cache at the same batch size, for MPT-7B with 4,096 + 4,096 tokens
- up to 2.4×higher token generation throughput, with a batch of two
- No retrainingapplied at inference time to existing models
Keyformer and 3D-DRAM address the same bottleneck. Keyformer moves fewer bytes per token, and 3D-DRAM moves bytes faster. Training can be limited by memory in the same way. Part III covers recommendation models, whose embedding tables often exceed the memory of a GPU.
Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference · MLSys 2024
Part IIITraining recommendation models
Recommendation models store numerical representations of users and items in embedding tables that reach tens to hundreds of GB, and these tables often do not fit in the memory of a GPU. Training them is therefore memory-bound. The two papers in this part use the semantics of the training data. FAE uses which embeddings are accessed most to place them in GPU memory, and Slipstream uses which embeddings have stopped changing to skip the inputs that train them.
Embedding tables
A deep-learning recommendation model takes two kinds of input. Sparse inputs denote categorical preferences of a user and index into embedding tables, which reduce each categorical value to a vector. Dense inputs are continuous and feed a bottom multi-layer perceptron (MLP). The embedding vectors and the output of the bottom MLP are combined by a feature-interaction layer, and a top MLP produces the click-through rate.
The neural networks benefit from GPU acceleration, but the embedding tables often cannot fit in the device memory of a GPU. The common remedy is a hybrid mode: the CPU stores and processes the embeddings, and the GPUs execute the neural networks in a data-parallel fashion. This mode transfers intermediate results and gradients between the CPU and the GPUs, and it executes every computation that involves embedding data, including the optimizer, on the CPU. The CPU–GPU communication can take up to 22% of the total training time.
FAEVLDB 2022
Key idea Accesses to embedding tables are highly skewed: a few entries are accessed up to 10,000 times more often than the others. The frequently accessed, or hot, entries occupy a few hundred MB, yet most training inputs access only hot entries. FAE stores the hot entries on every GPU and trains the inputs that need only hot entries entirely on the GPUs.
For the Criteo Kaggle dataset, the top 6.8% of the embedding entries observe at least 76% of the total accesses. For Criteo Terabyte, the hot entries occupy about 400MB of the embedding tables, which are 61GB in the text of the paper and 63GB in its table of workloads, and serve 81.6% of the inputs. An entry is classified as hot if its share of the total accesses exceeds an access threshold $t$, and an input is hot only if all of its embedding accesses are to hot entries.
Figure 30. Size of the hot embedding entries and share of the training inputs that access only hot entries, against the access threshold. The shares of hot inputs and of hot entries at the thresholds that the paper uses are printed in the paper, as is the size of about 400MB for Criteo Terabyte. All other values are read from its figures and are approximate. The paper allocates 512MB of GPU memory to hot embeddings. Choose a dataset and move the slider to change the threshold.
Lowering the threshold has diminishing returns: the number of hot entries grows more steeply than the share of hot inputs. The threshold must also be found cheaply, because profiling an entire dataset could mean processing terabytes of data.
Finding the threshold by sampling
FAE samples $x = 5\%$ of the inputs, and an embedding logger records the access count of each embedding entry. Because the logger observes only a sample, the cutoff for a hot entry is scaled accordingly. For table $z$ with $T$ total accesses:
FAE then picks $n = 35$ random chunks of $m = 1{,}024$ logger entries and counts, in each chunk, the entries $C_i$ whose access count is at least $H_{zt}$. The mean $\bar{y}$ of these counts, divided by $m$, estimates the fraction of the table that is hot, and its confidence interval follows from the Student's t-distribution:
Here $k_j$ is the access count of entry $j$, $s$ is the standard deviation of the counts, $N$ is the number of chunks in the logger, and $t_{\alpha/2}$ is the critical value of the t-distribution for a confidence level of $100(1-\alpha)\%$. A statistical optimizer repeats this estimate for successive thresholds until the hot entries fit in the GPU memory that the user allocates. At a confidence level of 99.9% the estimated size is within 10% of the actual size. Input sampling reduces the profiling latency by $19\times$ to $55\times$. Sampling chunks of the logger reduces the latency of each scan by $14.5\times$ to $61\times$, to under 25 seconds for all tables.
Figure 31. Equation $\eqref{eq:hzt}$. With 60.5 million training samples and a threshold of $10^{-2}$%, an entry needs 6,050 accesses to be hot, and 302.5 accesses in a 5% sample. Move the sliders to change the threshold and the sampling rate.
Hot and cold mini-batches
Once the threshold is fixed, FAE classifies every input as hot or cold. A single input that accesses a cold entry stalls the GPU while its embedding is fetched from the CPU. Mixing hot and cold inputs in one mini-batch is therefore ineffective: even if 99% of the inputs are hot, the probability that every input of a mini-batch is hot falls quickly with the size of the mini-batch.
Figure 32. Probability that a randomly formed mini-batch contains only hot inputs, computed for this page as $p^{B}$ for a share $p$ of hot inputs and a mini-batch of $B$ independent inputs. For $p = 99\%$ it matches the curve in the paper. Move the sliders to change $p$ and $B$.
FAE therefore pre-processes the dataset once into mini-batches that are either entirely hot or entirely cold. Hot mini-batches execute entirely on the GPUs, which hold replicas of the hot embeddings. Cold mini-batches fall back to the hybrid mode.
The Shuffle Scheduler
Training on all hot mini-batches and then all cold mini-batches would need only one synchronization of the hot embeddings between the GPU and CPU copies, but it reduces the randomness of training and affects accuracy. Interleaving the two kinds restores the randomness, and each swap costs one synchronization over PCIe. The Shuffle Scheduler sets the interleaving rate from the test loss. A rate of $R(100)$ issues all cold mini-batches before the first hot one, and $R(1)$ swaps after every mini-batch. The scheduler begins with cold inputs at $R(50)$. After each swap it halves the rate if the test loss has increased, down to $R(1)$, and doubles the rate if the loss has decreased for $u = 4$ consecutive swaps, up to $R(100)$. Otherwise the rate is unchanged.
Figure 33. The rate rule of the Shuffle Scheduler, computed for this page from its description in the paper. Each cell is one swap between cold and hot mini-batches. Press a cell to change whether the test loss rose or fell after that swap.
Results
The evaluation uses DLRM (Deep Learning Recommendation Model) and TBSM (Time-based Sequence Model) on the Criteo Kaggle, Taobao Alibaba, Criteo Terabyte and Avazu datasets, on a server with up to four NVIDIA V100 GPUs of 16GB each. The baseline is XDL, an industrial training framework for recommendation models. FAE reaches the baseline accuracy on every dataset to within 0.2 percentage points.
Figure 34. Training time for 10 epochs and the data transferred between the CPU and the GPUs, from the tables of the paper. Choose a dataset and the number of GPUs.
- 2.3×lower training time on average, for FAE with four GPUs against XDL on CPUs only
- 1.52×lower training time on average than XDL with four GPUs
- 1.54×less data transferred between the CPU and the GPUs on average
FAE decides where each embedding is stored from how often it is accessed. The next paper observes that the values of the frequently accessed embeddings also stop changing first.
Accelerating Recommendation System Training by Leveraging Popular Choices · VLDB 2022
SlipstreamDATE 2025
Key idea Hot embeddings receive most of the accesses, so they are trained quickly and their values stabilize early. An embedding whose value no longer changes is stale, and updating it again moves data and performs computation without changing the model. Slipstream identifies stale embeddings at runtime from snapshots of the hot embeddings and skips the inputs that train them.
The embedding operations of training, which are the lookups in the forward pass, the updates in the optimizer and the CPU–GPU communication, can collectively consume up to 75% of the training time on large datasets such as Criteo Terabyte. Hot embeddings saturate in under 25% of the training iterations. The decision of what to skip must be made per embedding: skipping the updates of all hot or of all cold embeddings loses 4% to 6% of accuracy.
- Snapshot block. After a warm-up of 2K iterations, Slipstream periodically captures snapshots of the hot embeddings on the GPUs. The hot embeddings are small: with the thresholds of this paper, they occupy 500MB of the 63GB of embedding tables for Criteo Terabyte and serve approximately 75% of the inputs.
- Sampling block. An entry is stale if the Euclidean distance between its values in consecutive snapshots is at most a threshold $T$. Comparing all hot entries for many candidate thresholds would cost more than $30\times$ the training time for Criteo Kaggle and more than $500\times$ for Criteo Terabyte. Slipstream instead evaluates each candidate threshold on a random 0.1% of the hot inputs and searches for $T$ by binary search.
- Input classifier block. In a single pass, each hot input is classified by the embeddings it accesses. An input is dropped if at least $\alpha$ of its sparse features access stale embeddings, where $\alpha$ is a hyperparameter, and training continues on the remaining inputs.
For a hot entry with consecutive snapshots $\hat{e}^{\,n}$ and $\hat{e}^{\,n-1}$, the indicator $\delta$ records whether it is still varying. Slipstream evaluates the indicator for the entries that the sampled inputs $\mathcal{S}$ access and obtains the fraction $D$ of the sampled inputs that the threshold drops. The estimate $\bar{D}$ for all hot inputs has the confidence interval:
Here $sd$ is the standard deviation of the per-input outcome, $m = |\mathcal{S}|$ is the number of sampled inputs and $I$ is the number of inputs. For a confidence level of 99.9%, $t_{\alpha/2}$ is approximated as 3.340. A precise threshold is not needed: a sample of 0.1% gives a narrow interval and reduces the cost of the search by nearly $1{,}000\times$.
Figure 36. The staleness rule on 48 hot embedding entries with synthetic values, computed for this page as an illustration. The change of each entry between snapshots decays as training proceeds, at a rate that depends on how often the entry is accessed. Move the sliders to change the threshold $T$ and the training progress.
Figure 37. The confidence interval of Equation $\eqref{eq:slip}$ for the fraction of inputs dropped, computed for this page with $t_{\alpha/2} = 3.340$. Because each sampled input is either dropped or not, $sd^{2} = D(1 - D)$. Move the sliders to change the number of hot inputs, the sampling rate and the measured fraction.
Because the threshold holds only with a certain confidence, a small fraction of varying embeddings may be classified as stale. Slipstream adds feature normalization, implemented as a layer normalization (LayerNorm) layer, so that dense and sparse vectors have zero mean and unit variance. This stabilizes the feature distribution, improves the detection of stale embeddings and hastens convergence.
The preprint of the paper evaluates four models, RM1 to RM4, on the four datasets that FAE uses, on a server with four NVIDIA V100 GPUs. The baselines are XDL, an open-source implementation of DLRM and TBSM that the paper calls Intel-optimized DLRM, FAE, and Hotline, a hardware accelerator for recommendation training.
Figure 38. Results per model, from the tables of the preprint. The fraction dropped is for $\alpha = 4$. AUC is the area under the receiver operating characteristic curve. Choose a model and whether LayerNorm is used.
The numbers in this section are from the arXiv preprint of the paper. The abstract of the DATE 2025 version reports a training time reduction of approximately $2\times$ over the XDL, Intel-optimized DLRM, FAE and Hotline baselines.
- about 2×lower training time than XDL, and 2.4× lower than Intel-optimized DLRM
- 1.2×lower training time than FAE, and 1.175× lower when Slipstream is added to Hotline on one GPU
- at most 5%of the execution time is the overhead of Slipstream, for Criteo Kaggle and Avazu
Slipstream removes the data movement and the computation of inputs that no longer change the model. The table below places the four papers side by side.
Slipstream: Semantic-Based Training Acceleration for Recommendation Models · DATE 2025
At a glance
The four papers share one premise. When a workload moves far more bytes than it computes on, its speed is set by the memory system, and the remedies are to move bytes faster, to move fewer bytes, or to keep the most frequently used bytes in the fastest memory.
| Paper | Workload | Observation | Mechanism | Result |
|---|---|---|---|---|
| Raptor | Generative inference, decode | Each token reads the weights and the KV cache; arithmetic intensity is about 1.9 operations per byte | Logic stacked on 3D-DRAM; stream blocking, stream flipping, bank chaining, thermal-aware refresh, interleaved ECC | About 100 TB/s per card; 4.71× the throughput per card of HBM and 2.44× that of SRAM |
| Keyformer | Generative inference, decode | About 90% of the attention weight falls on about 40% of the tokens | Keep key tokens and a recent window; score with Gumbel noise and a rising temperature | Up to 2.1× lower latency and 2.4× higher throughput with a 50% KV cache |
| FAE | Recommendation training | A few hundred MB of hot embeddings serve about 75% to 92% of the inputs | Hot embeddings on every GPU; hot and cold mini-batches; Shuffle Scheduler | On average 2.3× and 1.52× lower training time than XDL on CPUs only and on four GPUs |
| Slipstream | Recommendation training | Hot embeddings saturate in under 25% of the iterations | Snapshots, a sampled threshold, and dropping the inputs that train stale embeddings | About 2× lower training time than XDL, and 1.2× lower than FAE in the preprint |
The papers
- Early Silicon of Raptor: The First 3D-DRAM Accelerator for Generative InferenceISCA 2026
- Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative InferenceMLSys 2024
- Accelerating Recommendation System Training by Leveraging Popular ChoicesVLDB 2022
- Slipstream: Semantic-Based Training Acceleration for Recommendation ModelsDATE 2025
The roofline and the rack-level results are from the keynote Scaling the Memory Wall: Towards 3D-DRAM-Based Accelerators for Efficient Generative Inference, given at the IBM IEEE CAS/EDS AI Compute Symposium in October 2026. For the security of DRAM itself, see the Rowhammer explainer.