What Is RAG? Retrieval-Augmented Generation, Explained

RAG (retrieval-augmented generation) is an architecture that pairs a large language model with retrieval over your own documents at answer time: the system finds the passages most relevant to a question, hands them to the model, and the model writes an answer grounded in that evidence, usually with citations. The model's built-in training stays generic; your knowledge arrives fresh with every query. That single design choice is why RAG has become the default way businesses point language models at their own information.

How does RAG work, step by step?

The pipeline has two halves: preparation, done once and refreshed as documents change, and answering, done per query.

Preparation. Your documents — help articles, policies, product specs, past proposals — are split into passages a few paragraphs long, a step called chunking. Each chunk is run through an embedding model that converts it into a vector: a numerical fingerprint of its meaning, built so that passages about similar things end up near each other. The vectors go into an index designed for fast similarity search.

Answering. When a question arrives, it gets the same fingerprint treatment. The index returns the chunks closest in meaning, a reranking step reorders that shortlist by true relevance to the specific question, and the winning passages are placed into the model's context with instructions to answer from them and cite. The user gets an answer with sources; the model never had to memorize your business.

Every stage hides a tuning decision that shows up in answer quality later: chunk size and overlap, whether to combine vector search with classic keyword search (hybrid retrieval catches product names and SKUs that embeddings blur), how many passages to retrieve, how aggressively to rerank. The operating rule is blunt: retrieval sets the ceiling. If the right passage never reaches the model, no amount of model quality rescues the answer.

Why does RAG beat fine-tuning for knowledge tasks?

Fine-tuning — continuing a model's training on your own examples — sounds like the obvious way to teach it your business. For knowledge specifically, it loses to RAG on four fronts:

• Freshness. A price change or policy update reaches a RAG system by re-indexing the changed document, in minutes. A fine-tuned model stays wrong until the next training run.
• Citations. RAG answers can point at the exact passage they came from, which turns trust from a feeling into a check. Knowledge baked into weights cites nothing.
• Access control. Retrieval can filter by user permission, so the intern's assistant genuinely cannot see the board deck. A fine-tuned model cannot forget selected pieces of what it learned for selected people.
• Cost profile. RAG is built once and paid per query; fine-tuning adds training runs and often dedicated hosting on top.

Fine-tuning keeps a real job: teaching tone, output format, and domain vocabulary. Plenty of production systems fine-tune for voice and retrieve for facts. The decision framework, with the edge cases, is in our fine-tuning vs RAG comparison.

Does RAG stop hallucinations?

It shrinks them without abolishing them, and the honest accounting matters because vendors routinely oversell this point. The remaining failure modes:

• Retrieval misses. The user asks in words the documents never use, the right chunk ranks eleventh when ten are retrieved, and the model answers from the wrong material.
• Bad ground truth. If the indexed documents are outdated or contradict each other, the system produces confident, well-cited, wrong answers. RAG faithfully amplifies your knowledge base, including its flaws.
• Context disobedience. Models sometimes blend retrieved passages with their own training memory, or answer past the evidence. Instructions like answer only from the provided context help; they are seatbelts rather than physics.

The mitigations are workmanlike: hybrid search and reranking to lift retrieval quality, refusal behavior when retrieval confidence is low, citation enforcement so every claim maps to a passage, and a golden-question evaluation set that gets replayed whenever anything in the pipeline changes. What RAG really buys you is a better class of failure — a wrong answer usually traces to a specific retrievable page rather than to opaque model memory, which makes the system debuggable in a way a bare model never is.

What do businesses actually use RAG for?

The pattern behind every strong use case is the same: a body of documents that changes, questions that arrive constantly, and answers that must match your sources rather than the internet's average opinion.

• Support assistants grounded in help documentation, deflecting tickets with cited answers and escalating cleanly when retrieval comes back thin.
• Internal knowledge copilots over policies, playbooks, and past work — the end of asking Slack whether anyone has the latest version of the deck.
• Sales enablement: proposals and answers to security questionnaires drawn from approved case studies and current pricing rules instead of last year's folder.
• Marketing ops assistants grounded in brand guidelines and metric definitions. This one is underrated: a reporting copilot that explains performance using your definitions — what your team counts inside ROAS, why blended MER is the guardrail, how CAC is calculated for the board — gives every teammate the same answer instead of five private interpretations.

Worth placing on the map: classic marketing automation moves data along predefined rules, RAG answers questions from documents, and agentic AI is where they converge — an agent executes multi-step work while using RAG as its knowledge layer, consulting your policies mid-task the way a careful new hire would.

What does a RAG system cost, and what drives the price?

Directional published market rates put a working proof-of-concept at $10k-50k and production systems at $50k-250k+. The spread has surprisingly little to do with the model and almost everything to do with the surroundings:

• Data readiness is the dominant driver. Clean, current, well-structured documents make the build quick; a decade of duplicated shared drives means the project starts with archaeology.
• Integrations — where the documents live, where the answers surface, single sign-on, permission systems.
• Evaluation depth. A demo needs none; a system your support team trusts needs golden questions, regression runs, and monitoring.

Ongoing costs are per-query inference, periodic re-indexing, and monitoring — typically modest next to the build. Before commissioning anything, two free tools shortcut the scoping conversation: our AI Readiness Scorecard grades whether your documents, data, and processes are ready for a retrieval system, and the AI ROI calculator turns deflected tickets and saved hours into a defensible payback estimate.

When is RAG the wrong tool?

A short list that saves long projects:

• The goal is voice or format. Teaching a model to write like your brand is a fine-tuning job; retrieval fetches facts rather than style.
• The corpus is tiny and stable. Twenty pages that never change fit in a prompt; a pipeline would be ceremony.
• The task is calculation or logic. Models should call tools — calculators, queries, code — for deterministic work. Retrieving documents about math produces essays about math.
• There is nothing worth retrieving. If the knowledge base is thin or wrong, fix that first; RAG is an amplifier, and it amplifies whatever it finds.

Scoping this boundary honestly is most of the value of working with an AI and machine learning practice: the engagement starts by evaluating retrieval quality on your actual documents and golden questions, so the build decision rests on evidence instead of a demo. And for every other term in this series, from embeddings' neighbors to agentic AI, the growth marketing glossary keeps the full set of definitions in one place.