Retrieval-augmented generation, usually shortened to RAG, combines a search step with a language model: before the model answers, a retriever pulls relevant passages out of your own knowledge base and puts them in front of it as context. The answer is then written from those passages rather than from whatever stuck during training. The term comes from a 2020 research paper and now describes the standard shape for anything that connects a large language model to company content.
The practical appeal is that the knowledge can change without touching the model. New document in, index updated, done.
Why a model on its own is not enough
A language model only knows what was in its training data. Three gaps follow from that:
- The cut-off: anything after training ended is missing. Prices, headcount, the process that changed last week.
- Internal content: handbooks, contracts, tickets, wiki pages. None of it was ever public.
- Evidence: a model cannot say where a statement came from. It has no sources, it has weights.
With no grounding, the model fills the gap with something that sounds plausible. That is exactly an AI hallucination, and it is the main reason plain chat models are so weak at clerical work.
How a RAG application is built
There are two processes, and they run at different times.
Preparing the knowledge
Collect the documents, cut them into passages (chunking), compute a vector for each passage (embedding) and write both into a database. Metadata belongs with it: source, date, version, who is allowed to read it. This runs in the background and is repeated whenever the source documents change.
Answering the question
- Take the question and rewrite it if needed, for instance resolving pronouns from the conversation so far.
- Find candidates: by vector similarity, by classic keyword search, or by both together.
- Re-sort the hits (reranking) and keep only the best ones.
- Hand question and passages to the model as a prompt, with the instruction to answer only from them and to name the source.
- Deliver the answer together with its evidence.
| Part | Job | Typical implementation |
|---|---|---|
| Chunking | cut documents into retrievable pieces | along headings and paragraphs, with overlap |
| Embedding model | turn text into vectors | an API model or an open-source LLM, chosen to suit the language |
| Vector store | similarity search across millions of pieces | pgvector, Qdrant, Weaviate, Elasticsearch |
| Retriever | find candidates | hybrid of vector and keyword search |
| Reranker | order candidates by real relevance | a cross-encoder model |
| Generator | write the answer | the language model with a strict system prompt |
The quality of a RAG application almost always hangs on retrieval, not on the model.
RAG, fine-tuning or simply a large context window
The three approaches solve different problems and do not rule each other out.
| Approach | What it is good for | What it is not |
|---|---|---|
| RAG | current knowledge, citations, per-user access rights | tone, output format, understanding the task |
| Fine-tuning | style, format, recurring task patterns | fresh facts, because every change means training again |
| Large context window | one-off questions about a handful of known documents | collections beyond the window, and the cost per request |
Everything you put in the context is paid for and processed on every single request, see AI token. Retrieving five targeted passages is cheaper and usually more accurate than tipping the whole handbook in.
Where RAG fails in practice
A prototype takes an afternoon. The distance between that prototype and something a department actually uses is the real project.
Badly cut documents
Cut strictly by character count and a table falls apart into fragments, a condition gets separated from its exception. The model then answers correctly on the basis of half a sentence. Structure beats character count: cut at headings and write the section title into every piece.
Questions no document answers
When the knowledge base does not contain the answer, retrieval still returns the five most similar pieces. Without explicit permission to know nothing, the model builds an answer out of them. A sensible relevance threshold and an allowed “that is not in here” matter more than any round of prompt polishing.
Permissions
The index knows nothing about roles unless you teach it. Put personnel files and the wiki in the same store and you have built a search engine that bypasses every permission in the source systems. Rights belong in the query as a filter, not in the prompt.
A stale index
A document changes, the index does not. From then on the system answers confidently with the state of the day before yesterday. Syncing needs a fixed rhythm and a visible indication of how old the content is.
What makes a RAG application production ready
- Evidence in every answer. Without a link to the source nobody can check whether the answer is right.
- A test set of real questions. Fifty questions from the support inbox with known correct answers say more about quality than any feeling while trying it out.
- Retrieval and generation measured separately. Did the right passage make it into the context at all? If not, a better model will not help.
- Protection against tampered sources. Anyone indexing documents from the web or from users has to expect prompt injection, because retrieved text ends up in the same prompt as the instruction.
- Logs. Which question, which passages, which answer. Without them, misbehaviour cannot be reconstructed.
Frequently asked questions about retrieval-augmented generation
Do I need a vector database for this? Not necessarily. For moderate volumes an existing database with a vector extension, such as PostgreSQL with pgvector, is plenty, and a good keyword search often beats pure vector search anyway. Specialised software only becomes interesting with very large collections or high query load.
Does our data stay in house? That depends entirely on where the embedding model and the language model run. With an API provider the retrieved passages leave the building, on every request. If you do not want that, you run both yourself, which is possible but more expensive and slower. This question belongs at the start of the project.
Does RAG prevent hallucinations? It reduces them considerably, because the model works from text put in front of it rather than from memory. It does not remove them: contradictory passages, questions with no evidence in the collection and summaries across several passages still produce wrong statements. Hence the citation.
How much work is it to get started? A first end-to-end run with a handful of documents is quick. The effort sits in the formats (scanned PDFs, tables, presentations), in the permissions and in operating the index. As a rule of thumb: preparing the data is the project, the model is a line of configuration.
Conclusion
RAG is the least spectacular and at the same time most durable way to tie a language model to your own knowledge. The gain does not come from the model, it comes from well-prepared documents, good retrieval and the discipline of making every answer verifiable. Do that properly and you get a system whose mistakes you can see rather than hunt for. How we build software around requirements like these is described on our page about custom software development, and what actually works with AI in our project practice is written up in the article AI in software development. A finished product built on this, from the document base to the source shown under the answer, is described on our page about AI chatbot development. For anything beyond that there is a free consultation.