RAG, fine-tuning, both, or a long prompt
| Compared on | 01RAG | 02Fine-tuning | 03Both | 04Long-context prompting |
|---|---|---|---|---|
| Typical cost | Embeddings, vector storage and the extra tokens retrieved text adds to each request | Preparing labelled examples and training runs, then use of the tuned model | The costs of both, plus keeping them in step | Tokens for the whole document set on every request; prompt caching lowers repeat costs |
| When your data changes | Re-index the changed documents | Train again on new examples | Re-index facts; retrain only when behaviour must change | Edit the documents you send |
| Data and security | Access rules can be applied when passages are retrieved | Training data shapes the model; removing one record means training again | Both sets of controls apply | Everything in the prompt goes to the model on each call |
| Time to launch | Short for a first version | Longest of the single options: needs data, training and evaluation | Longest overall | Shortest |
| Main risk | Retrieving the wrong passage, so the answer is confidently wrong | The model learns from stale or messy examples | Two systems that drift apart | Accuracy dropping as the prompt grows |
Fine-tuning changes how a model behaves. It does not give the model a way to look up facts that changed after training, so most company-data projects start with RAG or a long prompt.
When RAG is the right choice
Use RAG when the answer is already in your documents and you need to show where it came from. Retrieval-augmented generation finds the passages that match a question and adds them to the prompt, so the model answers from your text rather than from memory.
Your content changes. Policies, manuals, contracts, tickets or product data that are updated every month.
Different people may see different things. Retrieval can follow the access rules you already have, so a user only gets passages they are allowed to read.
You need citations. Each answer can point to the document it used, which makes errors easier to catch.
Under the hood, documents are split into chunks, turned into embeddings and stored in a vector database. OpenAI's retrieval guide, for example, splits files into 800-token chunks with 400 tokens of overlap by default, and charges $0.10 per GB a day for vector storage beyond the first 1 GB. Anthropic does not offer its own embedding model and points developers to Voyage AI.
When fine-tuning is worth it
Fine-tune when you need the model to do one task the same way every time and better prompts have stopped helping. Google's Vertex AI docs recommend supervised fine-tuning for classification, sentiment analysis and entity extraction, and suggest prompting instead when you have little labelled data. OpenAI's guide lists output formats and instruction-following failures as good fits, and notes that a tuned model can work with shorter prompts, which saves tokens at scale.
Check the vendor before you plan. As of 17 November 2026, OpenAI says it is winding down its fine-tuning platform: new users can no longer access it, and existing users can create training jobs only for the coming months. Google's Vertex AI still lists supervised, preference and reinforcement tuning for several Gemini models.
Use both when a tuned model handles the format or tone and RAG supplies the facts. A support assistant that must answer from today's policy, in your house style, is a typical case.
Is a long prompt enough?
Sometimes, yes: if your material is small and stable, put it in the prompt and skip retrieval. Anthropic wrote in September 2024 that a knowledge base under 200,000 tokens, about 500 pages, can go straight into the prompt, and that prompt caching makes repeat requests faster and cheaper. Its docs now list a context window of up to 1M tokens, depending on the model.
More is not always better. Anthropic's own docs warn that accuracy and recall degrade as the token count grows. Start with the long prompt, measure it against questions with known answers, and add retrieval when answers slip or costs climb.
What goes wrong six months after go-live
The index goes stale. New documents are never added and deleted ones are still found. Re-index on a schedule and whenever a source system changes.
Permissions leak. A document moved to a restricted folder is still in the index. Sync access rules, not just files.
Quality drops and nobody notices. Without a fixed set of test questions with known answers, you only hear about errors from users. OpenAI's guide lists writing evals as the first step.
The tuned model's base is retired. OpenAI says fine-tuned models stay available until their base models are deprecated. Plan for a retrain or a move.
Made-up answers come back. When retrieval finds nothing useful, the model may still answer. Tell it to say when the documents do not cover the question, and watch for hallucinations in your test set.
How we approach it at Vizio AI
We start with the data, not the model. In a Free Data Assessment we look at where your documents and records live, how clean and current they are, who may see what, and which questions people actually need answered. Then we build a small test set, try the simplest approach that could work, often a long prompt or basic RAG, and measure it before adding anything more complex. Fine-tuning comes in only when the test set shows a behaviour problem that prompting and retrieval cannot fix.
When Vizio AI is the wrong choice
Your staff only need to ask questions of files that already sit in Microsoft 365 or Google Workspace. Try the assistant built into that suite first. For agents inside Microsoft 365, see Copilot Studio or n8n.
You have your own machine learning engineers and a clear evaluation set. You may need a second opinion, not a build partner.
You do not yet have the right to use the data, or it has not been collected. Sort out access and consent first.
FAQs
Sources
- OpenAI: Model optimization guidefine-tuning methods and uses; notice that the fine-tuning platform is winding down; read 17 November 2026
- OpenAI: Retrieval guidedefault chunk size and overlap, vector storage pricing
- Google Cloud: Tune Gemini models in Vertex AIwhen tuning fits, supported tuning methods
- Anthropic: Context windowscontext windows of up to 1M tokens; accuracy and recall degrade as context grows
- Anthropic: Embeddingsno Anthropic embedding model; Voyage AI suggested
- Anthropic: Introducing Contextual Retrievalpublished 19 September 2024; knowledge bases under 200,000 tokens can go in the prompt
