Skip to content

Guides

Company data

RAG or fine-tuning: how should you use your company data with an AI model?

Use RAG when answers must come from current company documents; fine-tune when you need a fixed format or behaviour. When to combine them, or use a long prompt.

6 min read

Reviewed

logo of Facebook
logo of X
logo of LinkedIn
logo of Reddit
logo of Telegram

RAG, fine-tuning, both, or a long prompt

01RAGBest forAnswers that must come from your current documents, with a source a person can check02Fine-tuningBest forA fixed format, tone or classification task that prompting alone does not get right03BothBest forA specialised task that also needs facts that change04Long-context promptingBest forA small, stable set of documents that fits in one request
Compared on01RAG02Fine-tuning03Both04Long-context prompting
Typical costEmbeddings, vector storage and the extra tokens retrieved text adds to each requestPreparing labelled examples and training runs, then use of the tuned modelThe costs of both, plus keeping them in stepTokens for the whole document set on every request; prompt caching lowers repeat costs
When your data changesRe-index the changed documentsTrain again on new examplesRe-index facts; retrain only when behaviour must changeEdit the documents you send
Data and securityAccess rules can be applied when passages are retrievedTraining data shapes the model; removing one record means training againBoth sets of controls applyEverything in the prompt goes to the model on each call
Time to launchShort for a first versionLongest of the single options: needs data, training and evaluationLongest overallShortest
Main riskRetrieving the wrong passage, so the answer is confidently wrongThe model learns from stale or messy examplesTwo systems that drift apartAccuracy dropping as the prompt grows

Fine-tuning changes how a model behaves. It does not give the model a way to look up facts that changed after training, so most company-data projects start with RAG or a long prompt.

When RAG is the right choice

Use RAG when the answer is already in your documents and you need to show where it came from. Retrieval-augmented generation finds the passages that match a question and adds them to the prompt, so the model answers from your text rather than from memory.

  • Your content changes. Policies, manuals, contracts, tickets or product data that are updated every month.

  • Different people may see different things. Retrieval can follow the access rules you already have, so a user only gets passages they are allowed to read.

  • You need citations. Each answer can point to the document it used, which makes errors easier to catch.

Under the hood, documents are split into chunks, turned into embeddings and stored in a vector database. OpenAI's retrieval guide, for example, splits files into 800-token chunks with 400 tokens of overlap by default, and charges $0.10 per GB a day for vector storage beyond the first 1 GB. Anthropic does not offer its own embedding model and points developers to Voyage AI.

When fine-tuning is worth it

Fine-tune when you need the model to do one task the same way every time and better prompts have stopped helping. Google's Vertex AI docs recommend supervised fine-tuning for classification, sentiment analysis and entity extraction, and suggest prompting instead when you have little labelled data. OpenAI's guide lists output formats and instruction-following failures as good fits, and notes that a tuned model can work with shorter prompts, which saves tokens at scale.

Check the vendor before you plan. As of 17 November 2026, OpenAI says it is winding down its fine-tuning platform: new users can no longer access it, and existing users can create training jobs only for the coming months. Google's Vertex AI still lists supervised, preference and reinforcement tuning for several Gemini models.

Use both when a tuned model handles the format or tone and RAG supplies the facts. A support assistant that must answer from today's policy, in your house style, is a typical case.

Is a long prompt enough?

Sometimes, yes: if your material is small and stable, put it in the prompt and skip retrieval. Anthropic wrote in September 2024 that a knowledge base under 200,000 tokens, about 500 pages, can go straight into the prompt, and that prompt caching makes repeat requests faster and cheaper. Its docs now list a context window of up to 1M tokens, depending on the model.

More is not always better. Anthropic's own docs warn that accuracy and recall degrade as the token count grows. Start with the long prompt, measure it against questions with known answers, and add retrieval when answers slip or costs climb.

What goes wrong six months after go-live

  • The index goes stale. New documents are never added and deleted ones are still found. Re-index on a schedule and whenever a source system changes.

  • Permissions leak. A document moved to a restricted folder is still in the index. Sync access rules, not just files.

  • Quality drops and nobody notices. Without a fixed set of test questions with known answers, you only hear about errors from users. OpenAI's guide lists writing evals as the first step.

  • The tuned model's base is retired. OpenAI says fine-tuned models stay available until their base models are deprecated. Plan for a retrain or a move.

  • Made-up answers come back. When retrieval finds nothing useful, the model may still answer. Tell it to say when the documents do not cover the question, and watch for hallucinations in your test set.

How we approach it at Vizio AI

We start with the data, not the model. In a Free Data Assessment we look at where your documents and records live, how clean and current they are, who may see what, and which questions people actually need answered. Then we build a small test set, try the simplest approach that could work, often a long prompt or basic RAG, and measure it before adding anything more complex. Fine-tuning comes in only when the test set shows a behaviour problem that prompting and retrieval cannot fix.

When Vizio AI is the wrong choice

  • Your staff only need to ask questions of files that already sit in Microsoft 365 or Google Workspace. Try the assistant built into that suite first. For agents inside Microsoft 365, see Copilot Studio or n8n.

  • You have your own machine learning engineers and a clear evaluation set. You may need a second opinion, not a build partner.

  • You do not yet have the right to use the data, or it has not been collected. Sort out access and consent first.

FAQs

01

02

03

04

05

Sources

  1. OpenAI: Model optimization guidefine-tuning methods and uses; notice that the fine-tuning platform is winding down; read 17 November 2026
  2. OpenAI: Retrieval guidedefault chunk size and overlap, vector storage pricing
  3. Google Cloud: Tune Gemini models in Vertex AIwhen tuning fits, supported tuning methods
  4. Anthropic: Context windowscontext windows of up to 1M tokens; accuracy and recall degrade as context grows
  5. Anthropic: Embeddingsno Anthropic embedding model; Voyage AI suggested
  6. Anthropic: Introducing Contextual Retrievalpublished 19 September 2024; knowledge bases under 200,000 tokens can go in the prompt

More guides

  • Workflow automation

    Microsoft Copilot Studio or n8n: where should you build AI agents and automations?

    Choose Microsoft Copilot Studio if your company runs on Microsoft 365, your agents mainly serve your own staff, and IT wants Microsoft's admin and billing controls. Choose n8n if your tools come from many vendors, you want to self-host or pay per workflow run, and someone on your team will own its upkeep.

    Reviewed

  • AI adoption

    Should you hire an AI consultant, an AI agency, or build an in-house AI team?

    Hire a consultant if you need priorities and a readiness check before spending on builds; choose an agency or studio if you need working systems live within months and have no team to build them. Build in-house when AI is core to your product and you can hire and keep specialists for years.

    Reviewed

  • AI coding agents

    Claude Code, Cursor or Codex: which AI coding agent should your team use?

    Choose Claude Code if your team wants an agent it can run from a terminal, desktop app or browser; choose Cursor if the people building want to work inside an editor and accept each change. Choose Codex if your company already pays for ChatGPT Business or Enterprise. Whichever you pick, name one person who reviews and owns the code.

    Reviewed

Think AI-first.
Move future-fast.

Let’s build what’s next! From product ideas and intelligent workflows to data-driven systems and scalable growth operations.

Book a Strategy Call