HBM: the fast memory on data-centre GPUs
HBM, high bandwidth memory, is DRAM stacked beside the GPU chip, giving far more bandwidth than the GDDR memory on graphics cards. An H100 SXM reads 3.35TB a second; an L40S reads 864GB.
See the H200Checked 31 Oct 2026 · ZenoCloud Team
HBM or GDDR?
Same job, different build. HBM stacks chips and sits close to the GPU, so it moves more data per second. GDDR is cheaper and sits further away.
| Card | Memory | Type | Bandwidth |
|---|---|---|---|
| L4 | 24GB | GDDR6 | 300GB/s |
| L40S | 48GB | GDDR6 | 864GB/s |
| A100 80GB (SXM) | 80GB | HBM2e | About 2.0TB/s (PCIe about 1.9TB/s) |
| H100 (SXM) | 80GB | HBM3 | 3.35TB/s |
| H200 | 141GB | HBM3e | 4.8TB/s |
| B200 | 180GB | HBM3e | About 8TB/s (HGX datasheet: 7.7TB/s) |
NVIDIA product pages. B200 per card is our arithmetic from the DGX B200 page: 1,440GB and 64TB/s across eight GPUs.
Why does bandwidth matter for LLMs?
Each generated token reads the weights from memory. NVIDIA calls generation memory-bound: memory speed, not arithmetic, sets the pace.
The one-user ceiling is bandwidth divided by bytes read. An 8-billion model at FP16 is 16GB: an H100 SXM’s simplified ceiling is near 209 tokens a second.
Real speeds are lower: this ignores overhead, and it assumes one user and a dense model.
Is HBM the same as VRAM?
VRAM is the role: the GPU’s own memory. HBM and GDDR are two technologies that can fill it.
Questions people ask
Is HBM better than DDR?
For bandwidth, yes, by a wide margin. It is also costlier to build, which is why it sits on data-centre GPUs and not on ordinary computers.
What is the difference between DRAM and HBM?
HBM is a kind of DRAM. The chips are the same family, but HBM stacks them vertically and connects them to the GPU over a very wide interface.
Does more bandwidth always mean faster AI?
For generating tokens for a few users, mostly yes. For training and large batches, compute matters more. Both limits apply, and the lower one wins.
Choosing a card for bandwidth
For single-user generation, read the bandwidth column first. Our H200 141GB starts at ₹2,20,000 a card a month, host included, before tax.
See the H200Sources
- NVIDIA H100 Tensor Core GPU (80GB, 3.35TB/s)
- NVIDIA H200 Tensor Core GPU (141GB HBM3e, 4.8TB/s)
- NVIDIA A100 Tensor Core GPU (80GB HBM2e)
- NVIDIA L40S and L4 product pages
- NVIDIA DGX B200
- NVIDIA, Mastering LLM Techniques: Inference Optimization (decode is memory-bound)
- ZenoCloud price book, checked 14 Oct 2026