RAG vs. Fine-Tuning: Why Your Company Knowledge Doesn't Need Model Training

Up front, so you can frame this article correctly: if an AI should answer questions about your contracts, manuals, or process documents, you almost never need model training. You need an architecture that looks your knowledge up at runtime. That's exactly what RAG does. In our happycoding projects, one sequence has proven itself: build RAG properly and max it out first, then talk about fine-tuning — in most cases that second conversation never happens.

Both terms in two sentences each. RAG (Retrieval Augmented Generation): the language model stays unchanged; on every request, the system retrieves the relevant passages from your documents and hands them to the model as context. The model formulates its answer from what it has just read. Fine-tuning: you change the weights of the model itself, using hundreds to thousands of your own training examples. What the model mainly learns is behavior: tone, output format, terminology.

This article is the deep dive into stage 2 of our decision tree for custom AI models. The complete four-stage tree, from prompt optimization to training your own model, is in the hub article Training your own AI model, not repeated here. One level more fundamental: Adopting AI in your company sorts out the questions that come first. Here: the one fork where we see most AI budgets take the wrong turn, RAG vs. fine-tuning.

What you get in the next few minutes: the difference that actually matters, a reference architecture with pgvector, a cost comparison with numbers and retrieval dates, the reason the best-known fine-tuning route is disappearing right now — and the three limits where RAG honestly fails, along with countermeasures.

The question "RAG or fine-tuning?" sounds like a duel between two equivalent tools. It isn't: the two solve different problems. Four properties make the difference in everyday use, and three of them are missing from almost every comparison article on the topic.

Freshness: with RAG, you change a document, the pipeline re-indexes it, and the next answer knows the new state: a matter of minutes. A fine-tuned model only knows the knowledge state of its training run. If your price list changes, you retrain — or the model answers incorrectly, with full conviction. For a product catalog that changes weekly: with RAG, the update is an automated job; with fine-tuning, a weekly training project including sign-off.

Source attribution: a RAG system can say for every answer which document and which section it came from. For you as a decision-maker, this is the underrated point: your people can check any answer against the original in seconds, and that is exactly what decides whether they trust the system. A fine-tuned model cannot do this by design: its knowledge sits in billions of weights, not in addressable passages.

Deletability: if someone demands the deletion of their data, with RAG you remove the document along with its index entries: verifiable and complete. With today's methods, a single piece of information cannot be selectively removed from trained model weights. We've written up what that means for personal data and third-country transfers in our practical guide to GDPR-compliant AI.

Catastrophic forgetting: fine-tuning carries a risk that hardly any comparison article mentions: a model you train on your data can unlearn capabilities it had before. In practice, that looks like this: your model now knows your product names, but suddenly writes worse emails or loses language quality. The only defense is systematic evaluation, and that costs effort many project plans simply forget.

The often-overlooked point behind all this: fine-tuning is simply a poor knowledge store. A model that saw your 500 pages of policies during training has not filed them away, it has digested them statistically: it can reproduce content from them, but reliable recall is not guaranteed. For the question "What notice period applies to our supplier contracts?", you don't want a statistical memory, you want the passage from the current contract template, with a citation.

Most comparison articles stop at the concept. We'll show you instead the stack we actually build at happycoding: PostgreSQL with the pgvector extension, running on Supabase or directly on EU servers at Hetzner. Your documents pass through five stations in this system.

Ingestion: a pipeline in TypeScript collects your sources: PDFs, Word files, wiki pages, ticket system, ERP exports. It extracts the text and keeps the metadata: source, date, version, permissions. A lot is decided right here: what the pipeline doesn't extract cleanly, no search can find later. Scanned PDFs without a text layer, for example, need an OCR stage, otherwise they stay invisible to the system.

Chunking: the texts are split into sections of a few hundred words, along the document structure: chapters, paragraphs, lists. Our takeaway from projects: bad chunking is the most common cause of weak RAG answers, ahead of the choice of language model.

Embedding: an embedding model translates each section into a numeric vector that captures its meaning. Similar content gets similar vectors: that's how the search later finds matches that use different words than your question.

Storage in pgvector: the vectors land in the same PostgreSQL that can also hold your application data. That is the central architecture argument: no second specialized database like Pinecone, Weaviate, or Qdrant, no second backup concept, no extra operational burden, and no additional data processor. For the data volumes typical of mid-sized companies, from tens of thousands to a few million sections, pgvector is easily enough in our experience.

Retrieval and answer: your question is translated into a vector as well. The database returns the most similar sections, and the language model formulates the answer from them and names the sources.

Two properties of this setup matter more to you than any benchmark result. First, access control: with Row Level Security, the database itself filters which sections a user may see, before the model ever sees them. Sales doesn't get answers from HR documents: as a database rule, not a polite prompt instruction. Second, data sovereignty: the entire index runs on EU infrastructure, and you can swap the language model behind it without rebuilding your knowledge.

Two practical questions often get lost in stack selection. First, language: not every embedding model separates German compound nouns and technical terms cleanly. That's why we test the candidates against real questions from your company before the decision is made.

Second, measurability: a RAG system without reference questions is flying blind. At the start of a project, we define twenty to fifty questions with known answers, signed off by the relevant department, and measure every change to chunking, embedding, or prompt against that list. That turns "feels better" into a percentage you can base decisions on.

Of the articles ranking for this question, not one names a single price. We do, with retrieval dates, because these prices change.

What RAG costs: the biggest block is one-off: building the pipeline, chunking, and evaluation — for us, depending on the state of your sources, a project of a few weeks (happycoding experience). In operation, three items remain: hosting the database, embedding costs during indexing, and the retrieval overhead: every request carries the retrieved sections as additional tokens in the prompt, typically a few thousand per question. That overhead is the honest running price of RAG.

Important for the assessment: with it, you're buying exactly the properties fine-tuning fails at, namely freshness and source attribution. Comparing token prices alone means comparing right past the value.

What fine-tuning costs: in 2026, the useful reference is no longer OpenAI (more on that in a moment) but a provider like Together AI, which trains open models.

Their price list (retrieved August 18, 2026) charges per million training tokens and by model size: LoRA training costs $0.48 for models up to 16 billion parameters, $1.50 in the 17-to-69-billion class, and $2.90 in the 70-to-100-billion class, with a minimum fee of $4 per training job. A training run with 10 million tokens on a 70-billion model therefore costs around $29.

And there lies the surprise: the raw compute price is not the problem. The real costs of a fine-tuning project sit right next to it.

Data preparation: hundreds to thousands of verified question-answer pairs that someone in your business unit has to create and sign off. Evaluation: a set of reference questions plus regression tests against catastrophic forgetting. Re-training: due at every relevant knowledge change, including renewed evaluation. Operation: a fine-tuned open-weight model is yours to host. We work through what GPU rental and self-hosting cost in the sibling article What does it cost to self-host an LLM?.

And the question "What does a pilot cost?" that SMB guides love to raise and never answer? Our honest answer: the price depends on your sources, not the AI. Clean PDFs and a well-kept wiki connect quickly; a file server grown for twenty years is its own project. That's why we start pilots with a deliberately small but real document set: it proves feasibility on your data, without tidying up your entire archive first.

You can check the break-even as a formula: fine-tuning only pays off against RAG once the context tokens saved per request, multiplied by the requests per month and the token price, exceed the monthly share of training, data maintenance, and hosting. Plug in your numbers: at a few thousand requests per month, as we typically see in mid-sized companies, this calculation practically never comes out in favor of training (happycoding experience).

If you're checking the fine-tuning route anyway, here's a fact that has changed the map: OpenAI is winding down its self-serve fine-tuning, in three stages (developers.openai.com, retrieved August 18, 2026).

Since May 7, 2026, organizations without a prior fine-tuning history have been unable to start new training jobs. Since July 2, 2026, the block also applies to organizations with no inference on a fine-tuned model in the last 60 days. From January 6, 2027, the API accepts no new training jobs at all, from anyone. No successor product has been announced.

To be fair about it: existing fine-tuned models keep running until their respective base model is deprecated. It's a wind-down, not a shutdown. For new projects, the door is closed all the same, and in 2026 fine-tuning therefore means, in practice: LoRA or related PEFT methods on open-weight models like Llama or Qwen, on rented GPUs or via providers like Together AI.

For your decision, the direction matters more than the motive: the world's best-known AI API provider considers self-serve fine-tuning dispensable. Any vendor who wants to sell you model training as a first step in 2026 should have to explain that to you.

And the episode shows the structural risk: a fine-tune ties you to one base model and its lifecycle. Anyone who put budget into GPT-3.5 fine-tunes in 2024 is writing that investment off right now: OpenAI is shutting down fine-tuned GPT-3.5 models on October 23, 2026, together with the base model (developers.openai.com, retrieved August 18, 2026). A RAG index has no such risk: your knowledge lives in your database; the language model behind it is replaceable.

Run the timeline for yourself: between the first blocking stage in May 2026 and the complete closure in January 2027 lie eight months. Anyone who had planned a project on OpenAI fine-tuning in early 2026 had to switch providers mid-project. That dependency on the product decisions of a single provider is exactly what you avoid with a knowledge store you own.

We earn money with RAG systems. That's exactly why we'll tell you where the approach is structurally weak: three edge cases keep showing up in our projects, and none of them appears in the usual comparison articles. If you know these cases before the project starts, you can catch them in the architecture; if they surprise you in production, they cost trust with exactly the people who are supposed to use the system.

Table knowledge: RAG searches for semantically similar passages. A question like "Which customers had more than €100,000 in revenue in 2025?" is not answered by any text search: the answer doesn't sit in any single passage, it only emerges through filtering and aggregating across many rows. You'll recognize the pattern by words like "all", "more than", or "on average" in your users' questions.

The countermeasure: query structured data in a structured way. Via function calling, we give the model access to defined SQL queries, so it interrogates your database instead of guessing from text snippets.

Arithmetic: language models don't calculate, they produce plausible text. Even with perfect retrieval: as soon as the answer requires a calculation (sums, deadlines, margins, tiered prices), the result is unreliable. The typical picture from practice: the model cites the correct deadline rule and still calculates the wrong date. The countermeasure: outsource the math. The model calls a tool, say a calculation function or a SQL aggregate, and only phrases the verified result.

Contradictory sources: RAG finds what's there. If the travel expense policy from 2019 and the one from 2024 are both in the index, the model may get both as context and blend them into an answer that appears in no document at all. That's not a model error, it's a data problem. The countermeasure: source governance. One authoritative source per topic, validity dates as metadata in the index, outdated versions removed or down-ranked in search.

Important for your decision: fine-tuning solves none of these three limits. A trained model calculates just as little and aggregates your tables just as badly. Reaching for training because of these limits trades a solvable architecture problem for an unsolvable one.

Cutting across all three limits: how do you establish quality at all? Our answer is unspectacular: the reference questions from the stack section, plus hard prompt rules ("answer only from the provided context; otherwise say you don't know") and regular spot checks by the business unit. If you don't measure hallucinations, you have them: the difference between a usable and an embarrassing system rarely lies in the model, almost always in data quality and evaluation.

So you don't misunderstand us: legitimate fine-tuning cases exist, and we build them too. Three patterns come up again and again.

Style, format, terminology: if the model needs to write in a fixed tone, keep to an exact output format, or hit the technical language of your industry, training works where prompts reach their limits. One example: an expert-report style with fixed phrasings and outlines is hard to force through a prompt, but training anchors it reliably. That's a behavior problem, so it's exactly fine-tuning's territory.

High volumes of the same task type: if a system handles tens of thousands of similar tasks a day, say classifying emails or extracting fields from receipts, a small specialized model can beat a large generic one: faster, cheaper per request, without retrieval overhead.

Edge and offline: if the model has to run on your own hardware without internet access, there's hardly a way around a small, trained open-weight model.

The combination, by the way, is not an either-or: in hybrid setups, RAG delivers the current knowledge while a fine-tuned model holds tone and format. For most mid-sized companies, that's the second expansion stage, not the starting point: only once RAG is in place and a measurable style problem remains does the double effort pay off.

How you recognize these cases, how much data you need, and how a training project runs: we go deeper in the sibling article When does LLM fine-tuning pay off?.

One test question additionally belongs on the table, and as an open one: could your own training possibly make you a provider of a GPAI model within the meaning of the AI Act? The European Commission's non-binding GPAI guidelines of July 18, 2025 name one third of the base model's training compute as an indicative figure; this threshold is not legally binding, and typical LoRA training stays far below it.

Only a legal review can settle that bindingly: we've sorted out the obligations in our article on the EU AI Act for software buyers.

If you're facing this decision right now, we suggest three steps. None of them requires a big budget approval: all three are preparation you can do internally. First: clarify whether your problem is a knowledge problem or a behavior problem; the rule of thumb from section two is usually enough for that.

Second: start small and measurable: a well-defined set of documents, twenty reference questions with known answers, a RAG pilot against them. That way, after a few weeks you know what the architecture delivers in your case, instead of having to take it on faith. Third: check the three limits from this article against your use cases: if tables and calculations dominate, plan function calling in from the start, not as a retrofit.

How we build such systems, from the ETL pipeline in TypeScript to operation on your EU infrastructure, is on our process automation page. And if you'd rather talk through your specific case directly: book 30 minutes with me — no pitch, just a first architecture assessment with a clear recommendation, even if it's "no model training".