Retrieval-augmented generation (RAG) works for company documents when the answer is written down in a few current, text-readable files that the person asking is allowed to see. It struggles when files are outdated or duplicated, when PDFs are scanned images, when permissions are ignored, and when the question needs counting or totals across many records.

Picture an operations manager at a 60-person services firm. The team asks the same questions every week: what is the travel policy for client visits, which contract template covers a fixed-price project, who approves a refund above the usual limit. The answers sit in a shared drive with three versions of the handbook, a folder of scanned contracts and a few policies that only HR can open. A chat assistant over that drive will answer quickly. Whether it answers correctly depends on the drive, not the model.

What is RAG, and how does it answer from company documents?

RAG is a pattern where an AI system first searches your documents for the passages most related to a question, then gives those passages to a language model to write the answer. The model answers from your content instead of only from what it learned in training.

The term comes from a 2020 research paper by Patrick Lewis and colleagues, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, which describes models that "combine pre-trained parametric and non-parametric memory." Microsoft's Azure AI Search documentation puts it in business terms: RAG "extends LLM capabilities by grounding responses in your proprietary content."

The four steps in a typical setup

1. Ingest. Files are collected from drives, wikis or ticket systems and converted to text.

2. Chunk. Long documents are split into smaller passages so each part can be matched on its own.

3. Retrieve. When someone asks a question, a search (by meaning, by keyword, or both) picks the most relevant passages.

4. Generate. The model writes an answer using only those passages, ideally with links back to the source.

Hosted tools package these steps. OpenAI's file search tool, for example, lets a model "retrieve information in a knowledge base of previously uploaded files through semantic and keyword search."

When does RAG work well for business documents?

RAG works well when the answer exists in writing, in one or a few places, and the question asks what a document says rather than asking for a calculation. Policies, procedures, product documentation, contract clauses and past proposals are good fits.

• Policy and HR questions. Fit: strong. Why: answers are short passages in a handful of official documents.

• Product and support documentation. Fit: strong. Why: content is already written to explain things and is usually kept current.

• Contract lookups. Fit: good, with care. Why: clauses are findable, but the assistant must say which contract and version it used.

• "How many" or "how much" questions. Fit: weak. Why: totals across records need a database query, not a few passages.

• Questions that need judgement. Fit: weak. Why: RAG can bring the relevant documents together, but a person still has to decide.

If your questions are mostly about numbers, such as revenue by region or tickets per week, you need structured data and defined metrics, not document search. Our post on a semantic layer for AI covers that side.

Why does RAG give wrong or incomplete answers?

Most wrong RAG answers come from retrieval, not from the model: the system found the wrong passage, an outdated one, or none at all, and the model answered from what it was given. Fixing the documents and the search usually helps more than changing the model.

The common failure modes

• Stale and duplicate files. Three versions of the same handbook means three different answers. The search has no way to know which one is current unless you tell it.

• Scanned PDFs. A scanned contract is a picture of text. Without optical character recognition (OCR), there is nothing to search. Microsoft's OCR skill, for instance, "recognizes printed and handwritten text in image files," including images embedded in PDFs. Check what your tool does with images before you rely on it; OpenAI's file search page lists PDF, Word, PowerPoint, Markdown, text and HTML among its supported formats, but no image formats.

• Chunks that lose their context. A passage that says "the limit is raised to the amount above" means little without the heading and the paragraph before it. Anthropic's write-up on contextual retrieval (September 2024) notes that traditional RAG solutions "remove context when encoding information." In its own tests, adding a short context to each chunk and combining meaning-based and keyword search reduced top-20 retrieval failures by 49%, and by 67% with reranking.

• Tables and spreadsheets. Rows split across chunks lose their column headers. Tabular data usually belongs in a database the assistant can query.

• Questions with no answer in the files. If nothing relevant is found, a well-built assistant should say so, not fill the gap.

How do you keep a RAG assistant from showing people documents they shouldn't see?

Filter results by the asking user's permissions before anything reaches the model. If a person can't open a file in the drive, the assistant must not retrieve it for them, quote it or summarise it.

This is called security trimming or document-level access control. Microsoft's document-level access documentation describes the simplest form: "Your application passes in a user or group identity as a string, which populates a filter on a query, excluding any documents that don't match on the string." Its newer approach, still marked as preview, reads permission metadata from the indexed content itself.

What to check

• Every indexed document carries the groups or users allowed to read it.

• The filter is applied in the search query, not after the model has seen the text.

• Permission changes in the source system reach the index on a schedule you know.

• Highly sensitive folders, such as payroll or legal disputes, are left out entirely until the rest works.

The same rule applies when an agent reaches documents through a connector. Our guide to securing an MCP server covers scoped access in more detail.

How should you prepare documents before building RAG?

Clean the source before you index it: pick one current version of each document, convert scans to text, add clear titles and owners, and decide who may see what. An hour spent on the drive saves days of tuning prompts.

1. List the questions. Collect 30 to 50 real questions people ask, with the correct answer and the document it comes from. This becomes your test set.

2. Pick the sources. Start with the folders that answer most of those questions, not the whole drive.

3. Remove duplicates and archive old versions. Keep one current file per policy, with a date and an owner.

4. Make files readable. Run OCR on scans and check a sample by eye.

5. Tag permissions. Record which groups may read each source.

6. Test before launch. Run the test set, check that each answer cites the right document, and fix the documents behind the wrong ones.

When is RAG the wrong tool?

RAG is the wrong tool when the answer has to be calculated, when it lives in a system that changes by the minute, or when the task is to take an action rather than answer a question. In those cases a database query, a direct integration or an agent with tools fits better.

• Calculations and reports. Use the database or BI layer, and let the AI write or explain the query.

• Live status, such as an order or a ticket. Call the system's API at question time instead of indexing a copy.

• Actions, such as booking, refunding or updating a record. That is the job of an agent with permissioned tools; our explainer on what an AI agent is covers the difference.

Many useful assistants combine these: RAG for the policy, an API call for the customer's current order, and a person for the decision.

How we approach it at Vizio AI

We start with the questions, not the documents. Before indexing anything, we collect real questions from the team, the correct answers and where those answers live, and we use that list to decide which sources go in and to test the assistant before anyone relies on it.

We then clean the sources with the people who own them, apply permissions in the search itself, and make every answer cite the document and section it used, so a wrong answer leads straight to the file that needs fixing. Where questions turn out to need numbers or live data, we connect the right system rather than stretching document search. This work sits in our DX Studio.

Sources

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv, 2020): https://arxiv.org/abs/2005.11401

• Microsoft Learn, RAG and Generative AI, Azure AI Search: https://learn.microsoft.com/en-us/azure/search/retrieval-augmented-generation-overview

• Microsoft Learn, Document-Level Access Control, Azure AI Search: https://learn.microsoft.com/en-us/azure/search/search-document-level-access-overview

• Microsoft Learn, OCR Skill, Azure AI Search: https://learn.microsoft.com/en-us/azure/search/cognitive-search-skill-ocr

• Anthropic, Introducing Contextual Retrieval (19 September 2024): https://www.anthropic.com/news/contextual-retrieval

• OpenAI, File search: https://developers.openai.com/api/docs/guides/tools-file-search

• Vizio AI, Semantic Layer First: Getting Your BI Ready for AI Agents: https://www.vizio.ai/resources/blog/semantic-layer-for-ai

• Vizio AI, What Is an AI Agent, Really? Agents vs Chatbots vs Workflows: https://www.vizio.ai/resources/blog/what-is-an-ai-agent

• Vizio AI, Connecting an AI Agent to Your CRM? Secure the MCP Server First: https://www.vizio.ai/resources/blog/mcp-security-best-practices

• Vizio AI, DX Studio: https://www.vizio.ai/studios/dx-studio