RAG vs Fine-Tuning: When to Use Each (and When to Use Neither)

Use RAG when the model needs to know something, and fine-tuning when it needs to behave differently. Retrieval-augmented generation fetches your documents at question time and puts them in the prompt, so answers stay current and can cite sources. Fine-tuning trains the model on examples, which fixes format, tone and narrow tasks but is a poor way to teach it new facts. Most teams should try prompting first, then RAG, and fine-tune only when a measured behaviour gap remains.

Updated October 2026.

Quick answer

One thing changed in 2026 that most comparison articles have not caught up with: hosted fine-tuning has shrunk. OpenAI is winding down its self-serve fine-tuning platform, and the Gemini API no longer offers a tunable model. More on that below, because it changes the decision for many teams.

What RAG and fine-tuning actually do

RAG: the open-book exam

The term comes from a 2020 paper by Lewis and colleagues at Facebook AI Research, UCL and NYU, which described models that "combine pre-trained parametric and non-parametric memory for language generation." The parametric memory is the model's weights. The non-parametric memory is an index of documents searched with a retriever. The paper's motivation still holds: "providing provenance for their decisions and updating their world knowledge remain open research problems" for models that only use their weights.

A modern RAG pipeline, as Google's RAG Engine docs lay it out, is: ingest documents, chunk them, create embeddings, index them (usually in a vector database), retrieve the best chunks for each question, and generate an answer with those chunks in the prompt.

Fine-tuning: changing the student

Fine-tuning continues training a pre-trained model on your examples. The common methods in vendor docs today are:

• Supervised fine-tuning (SFT): input and ideal output pairs. OpenAI lists it for classification, nuanced translation, a specific format and fixing instruction-following.
• Preference tuning (DPO): pairs of better and worse answers, for tone, style and summary focus.
• Reinforcement fine-tuning (RFT): a grader scores each candidate answer; OpenAI positions it for complex domain reasoning and notes it needs expert graders.

Parameter-efficient methods make this affordable. The LoRA paper reports that, compared with fully fine-tuning GPT-3 175B, LoRA "can reduce the number of trainable parameters by 10,000 times and the GPU memory requirement by 3 times," with no added inference latency. QLoRA pushed it further: fine-tuning "a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance."

OpenAI's accuracy guide uses an exam analogy that is worth keeping: RAG is the textbook you bring into the exam (in-context memory), fine-tuning is the classes you attended (learned memory). It calls the two "additive, not exclusive."

Why fine-tuning is the wrong way to add knowledge

This is the most common mistake, so it deserves its own section. A Microsoft research paper (Ovadia et al., 2024) compared the two directly and found that "while unsupervised fine-tuning offers some improvement, RAG consistently outperforms it, both for existing knowledge encountered during training and entirely new knowledge." The authors add that "LLMs struggle to learn new factual information through unsupervised fine-tuning."

OpenAI's guide frames it the same way. Context optimisation (RAG, better prompts) fixes missing, outdated or proprietary knowledge and "maximizes response accuracy." LLM optimisation (fine-tuning) fixes inconsistent formatting, wrong tone or reasoning that isn't followed, and "maximizes consistency of behavior." The guide includes a case where adding RAG after fine-tuning "actually decreased accuracy, dropping four points," because the remaining problem was behaviour, not knowledge.

Microsoft's Foundry docs state the limit plainly: fine-tuning "doesn't replace retrieval for current information or application-level safety controls." Its classic guidance adds a warning many teams learn the hard way: "Fine-tuning with bad data makes the base model worse."

Cost and effort compared

Some real numbers from vendor pages, as of October 2026:

• Prompt caching. Anthropic bills cache reads at 0.1x the base input price (and 0.05x on Opus 5.5 and Sonnet 5.5), with cache writes at 1.25x for the 5-minute cache. OpenAI says cached input is "discounted up to 95%" and that, for GPT-5.6 and later, writing a prefix once and reusing it once costs 1.35x its ordinary input cost instead of 2x.
• Hosted fine-tuning (OpenAI, legacy models). The OpenAI pricing page lists training at $25 per 1M tokens for gpt-4.1, $5 for gpt-4.1-mini and $1.50 for gpt-4.1-nano, and $100 per hour for o4-mini reinforcement fine-tuning. Fine-tuned gpt-4.1-mini then costs $0.80 input and $3.20 output per 1M tokens.
• Azure. Microsoft's classic overview says serverless fine-tuning starts "at $1.70 per million input tokens," and notes "additional hourly costs for hosting" a fine-tuned model.
• AWS Bedrock. Customisation is billed on "number of tokens in training data corpus × number of epochs," plus monthly storage of the custom model.
• Your own GPU. Unsloth's published minimums for QLoRA (4-bit) are about 6 GB of VRAM for an 8B model, 8.5 GB for 14B and 22 GB for 27B; 16-bit LoRA needs roughly three to four times as much. Unsloth calls these "the absolute minimum." See our guide to local LLM hardware.

The option most guides skip: no retrieval at all

Anthropic's Contextual Retrieval post makes a point worth acting on: "If your knowledge base is smaller than 200,000 tokens (about 500 pages of material), you can just include the entire knowledge base in the prompt" with no need for RAG, and prompt caching cuts latency by more than 2x and cost by up to 90%. With today's large context windows, a lot of internal-docs chatbots never needed a vector database.

What you can still fine-tune in October 2026

This is where 2026 differs from older advice.

The practical result: if you want a fine-tuned model you can keep, an open-weight model is now the safest base. You own the adapter, nobody can deprecate it, and you can run it locally.

How to decide, step by step

1. Write evals first. OpenAI's fine-tuning guide says it bluntly: "Good evals first!" Collect 30 to 100 real questions with good answers before you change anything.
2. Prompt properly. OpenAI notes that "many of our largest customer deployments at OpenAI were done using only prompt engineering and RAG." Google recommends "starting with prompting to find the optimal prompt" before tuning. Good context engineering fixes more than people expect.
3. If knowledge is missing and fits in context, put it in the prompt and cache it.
4. If it doesn't fit, or changes often, build RAG. Measure retrieval separately from answers. Anthropic's tests showed that adding context to each chunk, mixing embeddings with BM25 keyword search and reranking cut failed retrievals by 67%.
5. If behaviour is still wrong, fine-tune. Start small: OpenAI says to "start with 50 well-crafted demonstrations," and "If 50 examples have no impact, rethink your task or prompt." It also says you can expect a similar gain each time you double the examples, and that a small amount of high-quality data beats a large amount of low-quality data.
6. Re-run the evals after every change. Fine-tuning can make things worse, and you will only know if you measure.

Combining RAG and fine-tuning

The strongest systems often use both, each for its own job:

• RAG for facts, fine-tuning for form. Retrieval supplies the policy text; a tuned model always answers in your support team's format and tone.
• Fine-tune on examples that include retrieved context. OpenAI's guide: "if you have a RAG application, fine-tune the model with RAG examples in it." Microsoft lists "use retrieved context effectively" as a fine-tuning benefit.
• RAFT. The RAFT paper (UC Berkeley) trains a model with relevant documents mixed with "distractor documents," teaching it to quote the right passage and ignore the rest. It reports consistent gains on PubMed, HotpotQA and Gorilla.
• Distil a small model. Use a large model with RAG to produce good answers, then fine-tune a small model on them so it runs cheaper and faster, which is one of the efficiency reasons OpenAI gives for fine-tuning.

A minimal open-weight setup with Hugging Face TRL and LoRA looks like this:

from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

# Each row: {"messages": [{"role": "user", ...}, {"role": "assistant", ...}]}
dataset = load_dataset("json", data_files="train.jsonl", split="train")

trainer = SFTTrainer(
    model="your-open-weight-base-model",   # a Hugging Face model id
    train_dataset=dataset,
    args=SFTConfig(output_dir="out", assistant_only_loss=True),
    peft_config=LoraConfig(r=16, lora_alpha=32, target_modules="all-linear"),
)
trainer.train()

If your RAG system also needs live data from tools and apps rather than documents, that is a different problem: see MCP vs RAG.

The bottom line

RAG and fine-tuning are not rivals. RAG answers "what does the model know right now?" and fine-tuning answers "how does the model behave?" Start with evals and prompts, use a long cached prompt when the data is small, build RAG when it isn't, and fine-tune only for a behaviour gap you can measure. In 2026, with hosted fine-tuning shrinking, plan any serious fine-tuning around open-weight models you control.