Retrieval-Augmented Generation (RAG)

What is Retrieval-Augmented Generation (RAG)?

RAG solves a basic problem with a large language model: it does not know your company's information, and it may invent an answer when it is unsure. Instead of relying on what the model absorbed in training, a RAG system looks up the facts at the moment a question is asked.

It works in two steps. First, retrieval: the question is used to search an indexed collection of documents, such as help articles, policies, contracts or product data, and the most relevant passages are selected. Second, generation: those passages are placed in the prompt with an instruction to answer only from them, and the model writes a response, ideally with references to the sources it used.

RAG is behind most "chat with your documents" features: support assistants that answer from the help center, internal tools that answer HR or policy questions, sales assistants that draw on product sheets, and search boxes that return a direct answer instead of a list of links.

How does RAG work under the hood?

Before any question is asked, documents are prepared. They are split into chunks, such as sections or paragraphs, and each chunk is turned into embeddings, numeric representations of its meaning. These are stored in a vector database or search index, along with metadata such as source, date and access rights.

When a question arrives, it is converted into an embedding too, and the system finds the chunks closest in meaning. Many systems combine this with keyword search and a re-ranking step to improve precision. The top passages go into the prompt, the model writes the answer, and the system returns it with links to the passages used.

Each stage can be tuned: how documents are chunked, how many passages are retrieved, and how the prompt tells the model to handle missing information.

How is RAG different from fine-tuning?

RAG gives a model the facts at answer time. Fine-tuning changes the model itself by training it further on your examples. They solve different problems. RAG is the right choice when answers depend on information that changes, needs to be traceable to a source, or is too large to build into a model: product catalogs, policies, knowledge bases, contracts. Updating an answer is as simple as updating the document.

Fine-tuning is better for teaching a model a style, a format or a narrow skill, such as writing in a brand voice or sorting tickets into your categories. It is a poor way to store facts, because they are hard to update and impossible to cite. Many production systems use RAG alone. Some combine both: a fine-tuned model for tone and format, with retrieval supplying the facts.

Why do RAG systems give wrong answers?

Most RAG failures are retrieval failures. If the right passage is not found, the model either says it does not know or, worse, fills the gap itself. Common causes are documents split in the wrong places, outdated or duplicate content in the index, questions phrased differently from the documents, and access rules that hide the relevant file.

Other failures happen at generation: the model blends two sources, misreads a table, or ignores the instruction to stay within the retrieved text.

The fixes are mostly unglamorous: clean and current source content, sensible chunking, hybrid keyword and semantic search, prompts that tell the model to say when the answer is not in the sources, and citations so users can check. A test set of real questions with known answers shows which change actually helped.

How do you measure a RAG system?

Measure retrieval and answers separately. For retrieval, check whether the passages that contain the answer appear among the results for each test question. For answers, check whether they are correct, whether they stay within what the sources say, and whether the citations actually support the claims.

Build the test set from real questions users ask, including ones the documents cannot answer, since a good system should decline those rather than guess. Automated scoring with a second model makes frequent checks practical, but spot-check by hand, especially early on.

In production, track how often users rate answers as helpful, how often the system says it does not know, and which questions get escalated to a person. Those gaps point to missing or unclear content as often as to technical problems.

How does Vizio AI build RAG systems?

Our Venture Studio builds RAG into AI-enabled product features and internal tools: assistants that answer from a help center, search that returns direct answers, and tools that let staff query policies, contracts or product data. We handle the AI integration end to end, from preparing and indexing content to retrieval, prompts, citations and the test set that tracks answer quality, and we design the architecture so answers stay current as documents change. When the goal is a new product, we validate the use case with an MVP before a larger build.

We usually start by gathering real questions and the documents that should answer them, then testing whether retrieval finds the right passages.