Self-hosted LLM
A self-hosted LLM runs on hardware you control rather than through a vendor API. The phrase covers two quite different things — a model in your own server room, and a model on dedicated cards in someone else's data centre — and they behave differently in cost, contract and liability. For most organisations neither is the right first move, and saying so is more useful than selling either.
When does owning hardware pay?
Three conditions, and the economics need all three: sustained and even load, data that contractually cannot leave your premises, and latency requirements that a public network cannot meet. In mid-sized organisations it is rare for all three to hold.
The condition that fails most often is utilisation. A card costs the same whether it is computing or idle, and a typical internal assistant runs below ten per cent. Twenty queries a day on owned hardware means paying for idle time plus operations, patching and on-call — items that appear in no vendor quote and in every invoice afterwards.
What does "self-hosted" actually buy you legally?
Less than the phrase implies. The GDPR does not prohibit third-country transfers; it conditions them on Chapter V — an adequacy decision, standard contractual clauses with a transfer impact assessment, or a derogation. Processing inside the EEA makes that chapter unnecessary, which is a substantial practical simplification but not a statutory requirement, and it does not depend on the hardware standing in your building.
Professional secrecy rules can be stricter than data protection. A German tax practice engaging an external service for a specific client mandate needs that client's consent under § 62a StBerG regardless of where the hardware sits. The operative question is not only "where" but "who can see it, and with whose agreement".
The option most comparisons skip
Open-weight models on dedicated cards inside the EEA, operated by a processor under an Article 28 agreement, with no logging of prompts or completions. That gives you data location and control over the model without purchase, operations or idle capacity.
The honest comparison for most: owned hardware starts to make sense at a load that keeps several cards busy through the working day. Below that, dedicated capacity is cheaper and available sooner. We run both and will tell you when the smaller answer is the correct one.
What it is not to be confused with
On-premise
"On-premise" means in your own facilities. "Self-hosted" is used loosely for dedicated hardware in someone else's data centre too. Worth separating in contract negotiations, because liability and processor obligations differ.
Private cloud at a hyperscaler
An isolated environment run by a provider whose parent company sits outside the EU remains a third-country arrangement even when the servers are in Frankfurt. Whether that holds up is a contracting question, not a geography question.
Frequently asked
How much does it cost to host your own LLM?+
Hardware is the smaller half. Add power, cooling, on-call capacity, model and runtime updates, and the ability to respond when inference stops at four in the afternoon. As a rule of thumb: if you do not already staff that role, the running cost exceeds what the purchase price suggests.
Are open-weight models good enough?+
For general-purpose work the strongest closed models still lead. For the common enterprise task — answering from supplied documents with citations — the gap is small, because the answer comes from the text in front of the model rather than from its memory. That is precisely why the model question is secondary in this design.
Can we start hosted and move in-house later?+
If the build allows it, yes. What matters is that your documents, the ingestion pipeline and the system instructions are not tied to one provider. Separate those from the start and the move takes days; skip that and the move is a rebuild.
Does self-hosting satisfy the EU AI Act?+
It is unrelated. The Act classifies use cases, not deployment locations. A self-hosted system used for CV screening is an Annex III high-risk case; a vendor-hosted document assistant is minimal risk. Where the computation runs affects the GDPR analysis, not the classification.
The arithmetic that belongs before the demo
Tell us the realistic number of queries per day and what data they touch. That is enough to say whether owned hardware pays — and the answer is usually no, which we will say plainly.
Request a test