LLM fine-tuning: when it pays off — and when you actually need RAG

"We want to train an LLM on our data": that is how many of our first conversations start. In our experience, this almost always hides a knowledge problem. The model is supposed to know product data, contracts, or internal documentation, and answer correctly from them.

Fine-tuning is exactly the wrong tool for that: it changes a model's behavior, meaning tone, response format, and task routine. Knowledge belongs in a database the model reads from at runtime — that is what Retrieval-Augmented Generation, RAG for short, delivers.

You will find the full trade-off between the two approaches in the sibling article RAG vs. fine-tuning. Where fine-tuning sits in the overall hierarchy of model adaptation is shown by the decision tree in our overview Training your own AI model. This article does one thing only: it goes deep on the fine-tuning stage — what fine-tuning means technically in 2026, the three cases where it pays off, what it really costs, and when to keep your hands off it.

My thesis up front: fine-tuning is the most overrated stage of model adaptation. Most people who ask for it need RAG — and those who genuinely need it almost always underestimate the decisive part: not the training, but the data.

For context: anyone planning fine-tuning today no longer plans it at OpenAI. The official deprecation overview (retrieved August 18, 2026) names three dates.

Since May 7, 2026, organizations that never used fine-tuning cannot start new training jobs. Since July 2, 2026, the same applies to organizations with no inference on a fine-tuned model in the previous 60 days. And from January 6, 2027, nobody can create new jobs, active existing customers included. OpenAI has announced no successor.

One nuance matters: existing fine-tuned models keep running until their respective base model is shut down. But you can no longer start fresh there. If your automation is built on a fine-tuned GPT model, plan the migration now, not when the base model gets retired.

The practical consequence: fine-tuning in 2026 means open-weight models like Llama, Qwen, or Mistral, trained on your own or rented infrastructure. And it almost always means LoRA instead of full fine-tuning. LoRA (Low-Rank Adaptation) freezes the original weights and trains only small additional matrices that overlay the model at runtime.

The numbers from the original paper are remarkable: compared to full fine-tuning with Adam, LoRA reduces trainable parameters by a factor of 10,000 and GPU memory requirements by a factor of 3, measured on GPT-3 with 175 billion parameters (Hu et al., 2021). Quality lands at the level of full training or above.

For you, that means two things: a training run fits on a single rented GPU. And the result is a small adapter file of a few hundred megabytes, not a complete model of several hundred gigabytes.

So which model do you pick instead? The short answer: the smallest one that handles your task. For extraction and classification, models with 3 to 8 billion parameters are often enough; for free-form text in your brand voice, you will reach for the class between 20 and 70 billion.

Check two things before choosing: the license, because not every open-weight model allows every commercial use without conditions, and performance in your working language (German, in most of our projects), because it varies considerably between model families. Test both before training, with your own examples in plain prompting mode: that costs an afternoon and saves you a training run on the wrong foundation.

So when does it pay off? In our project experience, there are exactly three patterns where fine-tuning delivers a measurable edge over prompting and RAG.

Tone and format. Your model should write in your brand voice or stick to a strict output format: support replies in the house style, reports with a fixed structure, JSON without a single outlier. Behavior like this is tedious to force through prompts, because every format rule in the prompt costs context space and still gets ignored occasionally. Trained in, it holds reliably.

Extraction and classification at scale. The same narrow task, hundreds or thousands of times a day: reading out invoice line items, sorting emails into twelve categories, turning free text from forms into structure. Here a fine-tuned small model often beats a large generic one at a fraction of the cost per call. A concrete anchor: 5,000 documents a day through a large API model means paying anew every month. A specialist model, trained once, runs the same series on your own hardware at a fixed price.

A small specialist model instead of a large generalist. When latency or unit costs bite, the swap pays off: on a clearly bounded task, a fine-tuned 8B model often reaches the quality of a much larger model, answers faster, and runs on cheaper hardware. Exactly this pattern carries many of our automation projects: one small model, one task, high volume.

That leaves the question of data volume. The most concrete published thresholds come from OpenAI's supervised fine-tuning documentation (as of 08/2026): the minimum is 10 examples, visible improvements start at 50 to 100, and the recommended start is 50 carefully built demonstrations. The platform behind them is being wound down; as a guideline for LoRA on open-weight models, the values still hold. Quality matters more than quantity: 50 clean, uniformly formatted examples beat 500 scraped-together ones.

Now for the numbers that hardly any article on the topic mentions. Together AI, one of the largest providers for fine-tuning on open-weight models, publishes a price list (retrieved August 18, 2026): LoRA training costs $0.48 per million training tokens for models up to 16 billion parameters, $1.50 for 17 to 69 billion, and $2.90 for 70 to 100 billion. Full fine-tuning costs two and a half times as much in each tier: $1.20, $3.75, and $7.25.

The minimum fee per training job is $4.00. If you prefer to book dedicated compute: an H100 GPU costs $5.49 per GPU-hour there, and a LoRA run for an 8B model is a matter of hours on it, not days.

Let's run the numbers on a typical project. 75 training examples at around 2,000 tokens each make 150,000 tokens; with three training passes (epochs), that is 450,000 tokens, less than half a million. The LoRA run for an 8B model stays under the minimum fee: the training costs $4. Not a typo. The training run is the cheapest line item of the entire project.

Why do we still quote such projects in person-days? Because the effort sits elsewhere: selecting examples, cleaning them, formatting them consistently, setting aside a test set, and building an evaluation pipeline that makes success measurable.

In practice, we hold out part of the examples as a test set before training and define one metric per task: field accuracy for extraction, hit rate for classification, a scoring rubric for free-form text. In our project experience, the large majority of the hours goes into data preparation and evaluation, not training: that is a figure from our own experience, not a publisher number.

For the business case, one simple formula is enough: payback time in months = one-off costs for data, training, and evaluation, divided by the monthly savings from avoided API costs and avoided manual work. If the result lies beyond your planning horizon, drop the project. If it comes in under a year, run the numbers in more detail: then fine-tuning is rarely the overrated stage, but an investment with a clear return.

Radically honest, even against our own sales pitch: in four situations we advise against fine-tuning, even when the request sounds like it.

Your knowledge changes. Price lists, stock levels, legal positions: anything that goes stale does not belong in model weights. A fine-tuned model would need retraining after every change; a RAG system simply reads the current state from your database at runtime.

You do not have 50 good examples. Below that threshold, the model mostly learns the quirks of your small sample. The result looks convincing in the demo meeting and fails in production at the first input that looks different from the training data.

You cannot measure success. Without a held-out test set and a defined metric, you will not know after training whether it did any good. Fine-tuning without evaluation is a blind flight with a budget.

You need the model's general abilities. Training on a narrow task can degrade other capabilities; the phenomenon is called catastrophic forgetting. LoRA softens the risk because the original weights stay frozen, but it cannot rule it out. A model that is also supposed to write freely, translate, and summarize is better left untuned for a niche.

The order therefore always stays the same: exhaust prompting first, then RAG, and only then fine-tuning. How you anchor this order in your company, instead of relitigating it on every project, is described in our guide Adopting AI in your company.

A fine-tuned open-weight model belongs to you: you receive the LoRA weights as a file and decide where they run. Two routes are common. Either an inference provider loads your adapter and bills per token: little operational effort, but ongoing costs and a third party in the chain.

Or you run the model yourself, say on a dedicated GPU server in Germany: Hetzner's GEX131 with 96 GB of GPU memory costs €1,197.30 net per month plus €599 setup after the price adjustment of June 15, 2026 (Hetzner price list, retrieved August 18, 2026). What pays off at which utilization, which models fit on which card, and where the hidden costs sit: we work all of that out in the sibling article What does it cost to self-host an LLM?.

That leaves a question managing directors ask us regularly: "Does fine-tuning legally turn me into a model provider?" The European Commission's non-binding GPAI guidelines of July 18, 2025 name an indicative figure for this: whoever puts more than one third of the original training compute into their modification counts as the provider of their own GPAI model. This is not legally binding. Technically, only one thing can be said: a typical LoRA run consumes a tiny fraction of the original training compute.

The legal assessment of your specific case still does not belong in a technical article: what the AI Act means for you as a software buyer and which deadlines apply is compiled in our post EU AI Act for software buyers.

Fine-tuning is never our first offer: we start by checking whether your problem is a knowledge problem, because then a RAG system on EU infrastructure is enough. If a specialist model does turn out to be the answer, we build it with LoRA on an open-weight model: training on a rented GPU, operation on your servers, evaluation from day one. How such models are then embedded into automated workflows, from document extraction to quote generation, is shown on our process automation page.

If you want to know which stage of model adaptation your use case really needs: book a free initial consultation. Bring two things: ten typical examples of your task and one number, namely what manual processing costs you today. A first assessment needs nothing more.

In half an hour, we will work out together whether your task calls for behavior or for knowledge, which model class is a candidate for it, and what the payback calculation looks like in your case. We will give you the honest result — even if it reads: skip it.