VRAM: the memory your GPU keeps models in
VRAM is the memory built into a GPU. A model’s weights and its working memory usually have to fit in it. Weights take parameters times bytes each: a 70B model needs 140GB at FP16, 70GB at FP8.
See AI inference hostingChecked 31 Oct 2026 · ZenoCloud Team
VRAM for a 70B model
Weights first, then the KV cache. This table adds one user at 8,192 tokens of context to Llama 3.3 70B.
| Format | Weights | With KV cache | Card that fits |
|---|---|---|---|
| FP16 | 140GB | 142.7GB | B200 180GB |
| FP8 | 70GB | 72.7GB | H100 80GB (tight) |
| 4-bit, ideal | 35GB | 37.7GB | L40S 48GB |
| Q4_K_M file (Llama 3.1) | 43.1GB | 45.8GB | L40S 48GB (tight) |
Our estimates: 70 billion parameters × bytes (the model has about 70.6 billion); KV cache from its published config. Q4_K_M size from llama.cpp. Runtime overhead is extra, so test on the real card.
What is the KV cache?
While answering, the model keeps keys and values for every earlier token. This is the KV cache. It grows with context length and with each user.
Llama 3.3 70B has 80 layers and 8 KV heads of size 128. At FP16: 2 × 80 × 8 × 128 × 2 = 327,680 bytes a token.
At 32,768 tokens that is 10.7GB for one user. Two users need twice as much.
VRAM or system RAM?
A model that spills into system RAM still runs, but those layers are read at RAM speed and generation slows. Keep the whole model in VRAM when you can.
Training needs far more VRAM
Training adds gradients and optimizer state: 16 bytes per parameter with mixed-precision Adam. For 70 billion parameters that is 1,120GB before activations.
Questions people ask
Is 16GB of VRAM enough for AI development?
For inference on an 8-billion-parameter model at 4-bit, yes: the weights are 4GB (our arithmetic) and the rest goes to context. A 70B model does not fit.
What is VRAM vs RAM?
RAM is the CPU’s memory. VRAM sits on the GPU, next to its cores, with much higher bandwidth. A model must fit in VRAM to run at full speed.
Can a model use VRAM and system RAM together?
Yes. Runtimes such as llama.cpp can keep some layers in RAM. Every layer left there is read at RAM speed, so generation gets slower.
Matching the model to a card
Add weights and KV cache, pick a card, then test on it. An H100 80GB starts at ₹1,80,000 a card a month, host included, before tax. Our inference page has the full sizing table.
See AI inference hostingSources
- NVIDIA, Mastering LLM Techniques: Inference Optimization (KV cache formula)
- Llama 3.3 70B config.json (Hugging Face)
- llama.cpp, quantize tool documentation
- Rajbhandari et al., ZeRO: Memory Optimizations (16 bytes per parameter)
- Kwon et al., PagedAttention (KV cache memory)
- ZenoCloud price book, checked 14 Oct 2026