What Is RAG, Actually? A Plain-English Breakdown

"RAG" gets thrown around a lot in AI conversations, often as if saying it explains anything. Here's what it actually means, without the diagram full of arrows.

The problem RAG solves

A language model only knows what it was trained on, up to a cutoff date, and it doesn't know anything about your specific documents, product, or company. Ask it a question about your internal policy document and it will either say "I don't know" or — worse — confidently make something up.

The fix: give it the answer before it answers

RAG stands for retrieval-augmented generation. Instead of relying only on what the model memorized during training, you retrieve the relevant piece of your own data first, and hand it to the model along with the question. The model's job shifts from "recall a fact" to "summarize what's right in front of you" — a much easier and more reliable task.

How retrieval actually works

Your documents get split into chunks, each chunk is converted into a vector (a list of numbers representing its meaning) using an embedding model, and those vectors get stored in a vector database. When a question comes in, it's converted into a vector too, and the database returns the chunks whose vectors are closest in meaning — not just matching keywords.

Where it breaks

RAG isn't magic. Bad chunking (splitting a document in the wrong places) produces bad retrieval. Retrieving the wrong chunks produces a confidently wrong answer, just now with a fake source. Most of the actual engineering work in a RAG system is in getting the chunking and retrieval right — not in calling the LLM API, which is the easy part.

When you actually need it

If your use case is "answer questions about our own documents, policies, or product," you probably need RAG. If it's "write a poem" or "summarize this text I'm pasting in," you probably don't — you're already giving the model the context it needs directly.