RAG architecture explained: how retrieval-augmented generation works in practice

Language models know a lot, but not your policies, contracts or product details. Retrieval-augmented generation (RAG) closes that gap by finding relevant content first and asking the model to answer from it.

How it works

1. Ingest. Documents are collected, cleaned and split into passages.
2. Index. Each passage is converted to a numerical representation (an embedding) and stored for search, alongside its metadata and access rules.
3. Retrieve. When a user asks a question, the system finds the most relevant passages.
4. Generate. The model receives the question and those passages and writes an answer, ideally with citations.
5. Evaluate. Answers are checked against reference questions to measure quality.

Design choices that matter

• Chunking. Passages that are too long dilute relevance. Passages that are too short lose context.
• Hybrid search. Combining keyword and semantic search usually retrieves better than either alone.
• Access control. Retrieval must respect who is allowed to see each document.
• Freshness. Plan how the index updates when documents change.
• Citations. Showing sources lets users verify answers.

Common mistakes

• Indexing poor or duplicated content, which gives poor answers.
• Skipping evaluation, so nobody knows whether quality improves.
• Ignoring permissions, which can expose confidential documents.
• Sending sensitive data to services that do not meet residency requirements.

Where to start

Pick one well-defined use case with trusted content, build an evaluation set, then improve retrieval before changing the model. Our Machine Learning Engineering team can help design and test a RAG system for your data.

Frequently asked questions