The knowledge is there, nobody can find it
Hardly any company has too little knowledge. It is scattered: in manuals, contracts, wiki pages, tickets and in the heads of people who have been around for twenty years. Anyone who needs an answer asks in the corridor or spends half an hour searching the file system.
An AI knowledge base changes that. You ask a question in plain language and get an answer from your own documents, with the source. The technology is mature in 2026. Two other questions remain open: how do you implement it so that it holds up day to day? And how do you keep control over your knowledge?
That is what this article is about. I show you the five building blocks, the three levels of sovereignty and how a project runs. Finally, I tell you what we recommend to which audience.
What an AI knowledge base is and what it is not
First, the terms. An AI knowledge base is a system that uses a language model to answer questions from your own documents. The method behind it is called RAG (retrieval augmented generation): for every question, the system retrieves the matching passages and hands them to the model. The model writes the answer from what it has just read.
What it is not: your own AI model trained on your data. Training changes how a model answers, not what it answers from. For company knowledge that is almost always the wrong path, because you lose freshness, source citations and the ability to delete. The full reasoning is in our comparison RAG vs. fine-tuning.
Nor is it a chatbot with uploaded PDFs. The difference shows from the second month on: documents change, employees move departments, questions get more specific. A knowledge base keeps up with that, an upload in a chat window does not.
In short: the model is the speaker, your documents are the memory.
The five building blocks of a knowledge base
Every knowledge base we build consists of the same five building blocks. The order follows the path a document takes through the system.
1. Sources and connectors
It starts with the question of where the knowledge comes from: file shares, SharePoint, Confluence, ticket systems, ERP exports, mailboxes. Each source needs a connector that fetches new and changed documents automatically. A one-off upload is outdated within weeks.
2. Preparation
The text is extracted and split into passages of a few hundred words, along chapters and paragraphs. Scanned PDFs need text recognition, otherwise they stay invisible. In our experience, poor preparation is the most common cause of weak answers, ahead of the choice of model.
3. Permissions
This is the building block most guides leave out, and the most important one. Every passage carries the permissions of its source document. When someone from sales asks a question, the database filters out every passage that person may not open. This happens before the model sees anything.
We implement this with row level security in PostgreSQL: the rule lives in the database, not as a polite request in the prompt. A knowledge base without permissions is a data leak with a search box.
4. Search
Every passage is translated into a vector of numbers that represents its meaning. That way, the search also finds matches that use different words than the question. We store these vectors with the pgvector extension directly in PostgreSQL. That saves a second specialised database, a second backup concept and another data processor. How this works in detail is described in Supabase as an AI backend.
Pure semantic search has one weakness, though: it handles part numbers, standard designations and abbreviations poorly. That is why we combine it with classic keyword search, which PostgreSQL also provides. This is called hybrid search. For questions like «What does EN 1090 say about welds?», exactly this difference makes the answer usable.
5. Model and interface
Finally, a language model writes the answer and cites the sources. The interface can be your own chat window, an integration in Microsoft Teams or a ready-made tool such as Open WebUI.
The instruction to the model matters: it may only answer from the retrieved passages. If it finds nothing suitable, it says «The documents don't cover this» instead of guessing. In daily work, this honesty is worth more than any polished wording.
With Open WebUI, read the licence. Since version 0.6.6 from April 2025, a clause requires the Open WebUI name and logo to stay visible once more than 50 people use the tool within 30 days. For many companies that is not a problem. If you want the interface in your own design, you then need an enterprise licence or an alternative.
Sovereignty: three levels, one ground rule
Sovereign means: you decide where your knowledge lives, who can access it and which vendor processes it. The ground rule is: the index and permissions always stay with you, the language model is replaceable. For the model itself, there are three levels.
| Level | Example | Data processing | Dependency |
|---|---|---|---|
| 1: US vendor with EU data centre | Azure OpenAI, Claude via AWS Frankfurt | EU, but US company | CLOUD Act remains a residual risk |
| 2: EU vendor | Mistral, IONOS AI Model Hub, STACKIT | EU, EU company | contract under EU law |
| 3: Self-hosted | open-weight model on your own GPU server | your server in the EU | hardware only |
Level 1 offers the strongest models today. The catch: under the CLOUD Act, a US company can be obliged to hand over data, even if it is stored in Frankfurt. For general company knowledge, many find that acceptable. For health data or client files, it is not.
Level 2 is the best compromise for most mid-sized companies. Open models such as Llama 3.3 70B are available there via API from German data centres, billed by usage. You need no server of your own and have a contract partner under EU law.
Level 3 pays off when data must not leave the building at all. A GPU server from Hetzner costs from €232.30 net per month (as of August 2026). Purely on numbers, the API almost always wins. Self-hosting is a decision for data sovereignty, not a cost-saving measure. The full calculation is in our article on LLM self-hosting costs.
What your contract with the model vendor must cover
The level alone does not determine sovereignty. What counts is the contract. We check four points with every vendor:
- Processing location: Where are requests processed, and does that also apply during outages and peak loads?
- No training on your data: The vendor must guarantee that your questions and documents do not flow into its models.
- Retention: How long do requests stay in logs, and who can see them?
- Data processing agreement: Without an agreement under Article 28 GDPR, you may not pass personal data to the vendor.
Because the model is replaceable, you don't have to commit forever. Many projects start on level 2 and move individual areas to level 3 later. The index stays unchanged.
How a project runs
A knowledge base for the whole company in one go is the surest path to failure. We work in four steps:
- Choose a department: One with clear documents and many recurring questions, such as customer service, HR or technical documentation.
- Define reference questions: The department writes down twenty to fifty real questions with the correct answer. We measure every change against this list. «Feels better» becomes a percentage.
- Build the pilot: Connectors, preparation, permissions and interface for this one area. Depending on the sources, this takes a few weeks.
- Expand: Only when the pilot answers the reference questions reliably do further areas follow.
An example from customer service
A typical scenario: a machine builder with 40 service technicians has 1,200 manuals, maintenance instructions and service bulletins as PDFs. When technicians have questions, they call the hotline, and the hotline searches the file system.
The knowledge base connects exactly this archive. The technician asks on a tablet: «What torque applies to the spindle nut on the 300 series?» The answer comes with a reference to manual and page. The hotline only gets the questions the system cannot answer. Those questions show exactly where documentation is missing.
What operation costs
In operation, three running costs remain: hosting the database, indexing new documents and model usage per question. The larger block is the one-off setup.
A sample calculation for model usage on level 2: 50 people ask 20 questions a day each, on 20 working days. That is 20,000 questions at around 5,000 tokens each, 100 million tokens in total. At €0.65 per million tokens (Llama 3.3 70B at IONOS, as of August 2026), that costs €65 a month.
One point almost everyone underestimates: maintenance. Outdated documents produce outdated answers. So decide from the start who in the department is responsible for which source. AI makes knowledge findable, people have to keep it current.
Three mistakes we often see in knowledge projects
The technology rarely fails. Projects fail because of decisions made too quickly at the start. We keep running into three of them:
- Connecting everything at once: If you connect SharePoint, wiki, mailboxes and drives on day one, you get a search across twenty years of files, including every outdated version. Answers become unreliable, and trust is gone after two weeks.
- Building without a benchmark: Without reference questions, nobody knows whether a change helps or hurts. Discussions then revolve around individual cases instead of results.
- Adding permissions later: Permissions are hard to retrofit into a running knowledge base, because every passage has to be reassigned. They belong in the first version, even if the pilot only serves one department.
All three mistakes have one thing in common: they save a week at the start and cost months later. A knowledge base is only as good as its smallest cleanly built section.
Our recommendation by audience
The building blocks are the same for everyone. How you combine them depends on your situation. Here is our position, as we present it in initial calls:
| Audience | Typical situation | Our recommendation |
|---|---|---|
| Small teams up to 20 people | few sources, no IT department | ready-made tool such as Open WebUI, run by a service provider, model from an EU vendor |
| Mid-sized companies | several departments, SharePoint, ERP | own knowledge base with pgvector and permissions, model on level 2 |
| Professionals bound by confidentiality | client or patient data | level 3 or an EU vendor with an explicit commitment on confidentiality obligations |
| Industry with technical documentation | manuals, drawings, standards | focus on preparation and text recognition, reference questions from service |
| Public sector and critical infrastructure | strict hosting and procurement rules | level 3 on your own or state-certified infrastructure |
What this means for you as a decision-maker
- Management: Start with one department and a measurable goal, such as fewer follow-up calls in customer service. A company-wide project without a benchmark eats budget without proving it works.
- Head of IT: Insist that the index and permissions live in your own database. Then every model switch is a configuration question, not a new project.
- Data protection: Above all, check whether document permissions are passed on cleanly to every passage. The question «which model?» matters, the question «who sees which answer?» matters more.
What we explicitly advise against
- Uploading all documents to a public chatbot: It is quick to set up and loses all control over permissions and storage location.
- Starting with your own model training: For company knowledge, training is more expensive, harder to update and cannot be deleted. Exhaust RAG first, then talk about training. Our overview training your own AI model shows the decision path.
Our position in one sentence: you recognise a good knowledge base not by its model, but by the fact that every answer has a source and nobody sees more than they should.
Next steps
If you are planning a knowledge base for your company, we clarify three things together: which department suits a pilot, which level of sovereignty fits your data and what the setup costs. For the bigger picture, read our guide to adopting AI in your company.
In 30 minutes you will know what your pilot looks like and on which level it should run. Book a no-obligation video call here.
