Inference: running a trained model on new input
Inference is running a trained AI model on new input to get an answer. Training builds the model; inference uses it. A chatbot’s reply is produced by inference.
See AI inference hostingChecked 31 Oct 2026 · ZenoCloud Team
Inference vs training
Training adjusts the weights. Inference leaves them fixed. Training needs far more memory per parameter, and inference repeats on every request.
| Aspect | Training | Inference |
|---|---|---|
| Compute per token (dense LLM) | About 6N FLOPs | About 2N FLOPs |
| Memory per parameter | 16 bytes with mixed-precision Adam | 2 bytes at FP16, plus KV cache |
| How often | Once per model version | On every request |
| Speed limit when generating | Compute | Often memory bandwidth (decode) |
N is the parameter count of a dense transformer; sparse models differ. Sources: Kaplan et al. (compute), the ZeRO paper (16 bytes), NVIDIA (decode is often memory-bound).
Does it learn from my prompts?
Not during inference: the weights are fixed. An app can add stored chat history to the next prompt. Whether a provider trains on your data later is a policy.
What does inference need?
Reading the prompt (prefill) runs in parallel. Generating tokens (decode) is often limited by memory. So you need memory for the weights and KV cache, then bandwidth.
vLLM or Ollama?
vLLM is a serving engine built for many users at once; its PagedAttention manages KV cache memory. Ollama is a simple way to run a model on one machine.
Questions people ask
Is inference or training more expensive?
One training run costs far more than one request. But inference repeats on every request, so its total depends on your traffic.
Is ChatGPT an inference engine?
No. ChatGPT is a product. The model behind it performs inference. An inference engine is software, such as vLLM, that serves models.
What is the difference between generative AI and inference AI?
They are not rival types. Generative describes what a model creates. Inference describes using any trained model.
Renting a GPU for inference
Size the card by weights plus KV cache. The sizing table lives on our inference page. An L40S 48GB starts at ₹55,000 a card a month, host included, before tax.
See AI inference hostingSources
- NVIDIA, What’s the difference between deep learning training and inference?
- NVIDIA, Mastering LLM Techniques: Inference Optimization
- Kaplan et al., Scaling Laws for Neural Language Models (2020)
- Rajbhandari et al., ZeRO: Memory Optimizations (2019)
- Kwon et al., Efficient Memory Management for LLM Serving with PagedAttention (2023)
- ZenoCloud price book, checked 14 Oct 2026