RAG: letting an LLM look up your documents
RAG, retrieval-augmented generation, looks up relevant passages from a knowledge source, such as your own documents, and adds them to the prompt before the model answers. Unlike fine-tuning, it does not change the weights.
See LLM hostingChecked 31 Oct 2026 · ZenoCloud Team
How does RAG work?
In the common setup, a question is turned into numbers, matched against an index of your documents, and the best passages go into the prompt. Then the LLM answers.
RAG or fine-tuning?
Use RAG for facts that change or must be cited. Use fine-tuning to change style, format or behaviour. Many systems use both.
| Aspect | RAG | Fine-tuning |
|---|---|---|
| Changes the model’s weights | No | Yes |
| New information arrives | Update the index | Train again |
| Best for | Facts, documents, citations | Tone, format, task skill |
Red Hat: RAG works without modifying the underlying LLM; fine-tuning adjusts its weights. The last two rows summarise the same trade-off.
Is RAG a type of LLM?
No. RAG is a pattern around an LLM. The 2020 paper that named it paired a retriever with a generator.
Do I need a vector database?
Not always. A vector database stores embeddings for similarity search. A small collection can sit in a library or in Postgres with pgvector.
Questions people ask
What is the difference between an LLM and RAG?
An LLM generates text from what it learned in training. RAG adds a lookup step so the answer can use documents the model never saw.
Is RAG still relevant with long context windows?
Yes. You still have to choose what to send, and a longer prompt costs more time and GPU memory, because the KV cache grows with it.
Does RAG need a GPU?
Not by itself. The answering model needs compute: a rented GPU, a CPU for small models, or a hosted API. Index search is a lookup.
Hosting a RAG system
If you host the answering model yourself, size GPU memory to its weights plus prompt. An L40S 48GB starts at ₹55,000 a card a month, host included, before tax.
See LLM hosting