Quantization: storing model weights in fewer bits
Quantization stores a model’s weights in fewer bits, such as 8 or 4 instead of 16, so it needs less memory and fits smaller GPUs, at some cost in accuracy. GGUF is a file format for quantized models.
See the L40SChecked 31 Oct 2026 · ZenoCloud Team
How much memory does it save?
Each halving of bits halves the weights. A real 4-bit file is larger than the ideal: some tensors stay at higher precision and blocks store scale data.
| Format | Bytes per weight | Weights |
|---|---|---|
| FP32 | 4 | 280GB |
| FP16 | 2 | 140GB |
| FP8 or INT8 | 1 | 70GB |
| 4-bit, ideal | 0.5 | 35GB |
| Q4_K_M file (Llama 3.1 70B) | about 0.6 | 43.1GB |
Rows 1 to 4: parameters times bytes. Last row: size listed in the llama.cpp quantize documentation.
Does quantization hurt quality?
It may. Loss varies with model, method and task, and usually grows as bits fall. llama.cpp measures it as perplexity or KL divergence. Test your own task.
One unreviewed user test on r/LocalLLM, 30,000 generations, saw no drop in JSON reliability at Q8 and a drop at Q3.
What do Q4_K_M and GGUF mean?
GGUF is the file format llama.cpp uses for quantized models. Q4 means about four bits per weight. K marks the k-quant method; S, M and L are size variants.
Which GPUs support FP8?
Among NVIDIA GPUs, native FP8 starts with Ada and Hopper; Ampere (A100) lacks it. The A100 has no FP8 Tensor Cores, so it serves FP16, INT8 or 4-bit.
Questions people ask
What is quantization in simple terms?
Rounding. The model’s numbers are stored with fewer digits, so the file shrinks and fewer bytes are read per token.
Q4 or Q8?
Q8 is closest to the original and about half the size of FP16. Q4_K_M is the common compromise. If both fit your card with room for context, take the larger.
Can the KV cache be quantized too?
Yes. llama.cpp can store it at 8 or 4 bits, which frees memory for context. Users report mixed quality results, so test it.
Quantization and the GPU you rent
A 4-bit 70B file can fit a 48GB card with a short context; see VRAM. The L40S 48GB starts at ₹55,000 a card a month, host included, before tax.
See the L40S