Private and on-premises LLM deployment for regulated industries

The conversation usually opens with "our data cannot leave our environment", and that sentence turns out to mean four different things depending on who said it. Before choosing an architecture, it is worth finding out which one you are actually bound by — the cost difference between the options is very large.

First, establish what the obligation actually is

1. Is it a legal or regulatory requirement, a contractual commitment to a customer, or an internal policy? All three are real, but only the first is genuinely immovable.
2. Does it constrain where data is processed, where it is stored, or both? Processing and residency are different obligations with different solutions.
3. Does it concern the data itself, or the risk that it trains a third-party model? Those are separable, and the second is usually addressed contractually.
4. Does it apply to all your data, or one classification of it? Segmenting is often far cheaper than lifting the strictest requirement over everything.

Option 1 — Vendor API with contractual and regional controls

Standard commercial API access, with an enterprise agreement covering non-training, retention limits, and processing region, plus a data processing addendum.

• Gives you: the strongest available models, no infrastructure, immediate access to upgrades.
• Does not give you: data that never leaves your perimeter, or independence from a vendor's availability and pricing.
• Suits: the large majority of enterprise use cases, including many with meaningful regulatory exposure.

Get the specific contractual terms in writing — zero-retention or defined-retention, no-training, and the processing region — rather than accepting a marketing page as assurance. If a regulator asks, the clause is the artefact.

Option 2 — Managed model inside your own cloud tenancy

The major clouds offer first-party and third-party models running within your own account and region. Requests do not traverse the model vendor's infrastructure; they stay inside a boundary you already have compliance coverage for.

• Gives you: data staying within an existing audited perimeter, familiar identity and network controls, one fewer vendor in the data path.
• Costs you: a somewhat narrower model selection and a lag behind frontier releases.
• Suits: organisations with an established cloud compliance posture and a residency requirement.

This is the most under-used option, and for a lot of regulated buyers it is the right answer — it resolves the residency question without taking on the operational burden of running inference yourself.

Option 3 — Open-weights model, self-hosted

You run an open-weights model on infrastructure you control, in your data centre or your cloud account. Complete control over the data path, and complete ownership of the operational problem.

• Gives you: full control, no per-token cost, model version stability, and the ability to fine-tune freely on sensitive data.
• Costs you: GPU capacity planning, inference serving, upgrades, evaluation of each model change, and the people who can do all of that.
• Suits: a hard prohibition on external processing, or high sustained volume where per-token pricing has become the dominant cost.

Two things consistently surprise teams here. First, the recurring engineering cost usually exceeds the inference cost saved unless volume is genuinely high. Second, capable open-weights models have narrowed the quality gap considerably for well-scoped tasks — which makes this option much more viable than it was, provided the task is narrow.

Option 4 — Air-gapped

No external network path at all. Models, dependencies, and updates arrive through a controlled physical or one-way process.

This is genuinely necessary in a small number of environments and genuinely miserable everywhere else. Everything you take for granted — pulling a package, checking a hosted dashboard, calling an external API for enrichment — has to be rebuilt or removed. If you are considering it, be certain the obligation demands it rather than that it sounds maximally safe.

Comparing the four

Pick the least controlled option that satisfies the obligation you can actually cite. Every step up the ladder costs real money and buys nothing if the requirement did not demand it.

Things you need regardless of which you choose

The deployment model is only one part of a defensible design. These apply in all four cases:

• Permission-aware retrieval. An index without access-control filtering will surface documents the requesting user was never entitled to see — that is a data breach regardless of where the model runs.
• Redaction before the call where it is feasible. The safest data is the data you did not send.
• Full audit logging: who asked, what was retrieved, what was returned, which model version answered.
• Retention policy for prompts, outputs, and traces — these accumulate sensitive data quickly and are routinely forgotten in the data map.
• A documented human review path for consequential output, with an accountable owner.

In practice the permission-aware retrieval point causes more compliance findings than the deployment model does. It is worth resolving first. It is also one of the questions we suggest putting to any prospective partner in our checklist for choosing an AI development partner.

• compliance
• on-premises
• security
• architecture