Self-Hosted AI: What We Learned After 50+ Deployments

We have now stood up AI systems on customer-controlled infrastructure more than fifty times — in their data centers, their private clouds, and their air-gapped enclaves across three continents. The promise of self-hosting is simple and real: your data stays yours, your costs stop scaling with someone else’s price list, and you control the whole stack. The execution is where it gets interesting. This is what fifty-plus deployments taught us about doing it well.

Why teams self-host in the first place

The motivations cluster into four, and most serious deployments are driven by more than one.

• Data residency. The data physically cannot leave a jurisdiction, a building, or a network. This is non-negotiable in regulated industries, and no contractual promise substitutes for the data simply never crossing the boundary.
• Compliance. Auditors want to see where inference happens, who touched the data, and proof that controls are enforced rather than promised.
• Cost. At sustained volume, per-token pricing becomes the most expensive line in the budget. Owning the hardware flips the math, eventually.
• Control. No surprise deprecations, no rate limits set by someone else, no model swapped underneath you. The stack behaves because you decide how it behaves.

The hard parts

Self-hosting is not “the cloud, but in your building.” The conveniences a managed provider hides become your problems, and they are specific.

GPU scheduling and optimization

GPUs are the most expensive idle resource you will ever own, so utilization is everything. The hard problems are packing multiple models onto the same card without contention, keeping batch sizes high enough to saturate the silicon without blowing latency budgets, and managing memory so a long context request does not evict everything else. Naive scheduling leaves half your compute idle while requests queue. Getting this right is the difference between the cost case working and the cost case collapsing.

Kubernetes

Kubernetes is the lever that makes the rest tractable, and it is also where teams cut themselves. GPU node pools, device plugins, autoscaling that understands accelerators rather than just CPU, rollout strategies that drain in-flight inference gracefully — none of this is default behavior. You earn it.

Networking, including TURN for voice

Networking inside a customer environment is consistently underestimated. Voice is the sharpest edge: real-time audio over WebRTC needs a TURN relay to traverse NAT and firewalls, and that relay has to be reachable by both the browser and the backend on an address that is correct from both sides. We have lost more hours to one misconfigured relay address than to almost anything else. Plan the voice path first, not last.

Secrets and upgrades

Secrets multiply across model registries, storage backends, telephony credentials, and provider keys, and every one of them needs rotation that does not require a maintenance window. Upgrades are the quiet killer — a stack you cannot upgrade safely is a stack that ages into a liability. Design the upgrade path on day one, because retrofitting it later is the hardest work you will do.

Self-hosting does not remove operational complexity. It moves it inside your walls, where you can finally see it, own it, and decide how it behaves. That trade is the entire point — and it is only worth it if you are ready to do the owning.

Patterns that worked across three continents

A handful of patterns showed up in every deployment that went smoothly, regardless of region or industry.

• Treat the environment as the contract. Capture the network topology, the reachable addresses, and the firewall posture up front. Most failures trace back to an assumption about networking that was never written down.
• Make GPU utilization a first-class metric. If you are not watching saturation per card, you are paying for idle silicon and will not notice until the invoice arrives.
• Automate the upgrade from the start. The first deployment is easy. The fifteenth upgrade is what separates a healthy system from an abandoned one.
• Keep the inference interface stable. When the application talks to a consistent endpoint, you can change models, hardware, and providers underneath without touching product code.
• Rehearse failure. Pull a GPU node, kill the relay, expire a secret — in staging, on purpose. The systems that survived production were the ones that had already failed safely in practice.

Data isolation and per-tenant boundaries

When one self-hosted deployment serves multiple tenants, isolation stops being a feature and becomes the foundation. Every request, every cached artifact, and every log line has to be scoped to its tenant, and that scope has to be enforced at the data layer rather than reconstructed in application logic after the fact. A shared model is fine. Shared data is a breach waiting to be discovered. The boundary that holds under audit is the one the database enforces, not the one a developer remembered to add.

When self-hosting is worth it — and when it isn’t

Self-hosting pays off when residency or compliance leaves no alternative, when volume is high and sustained enough for owned hardware to beat per-token pricing, and when control over the stack is itself a requirement. It is the wrong call when traffic is low or spiky, when the team has no appetite for operating GPU infrastructure, or when speed of iteration matters more than ownership. There is no shame in renting until the math and the mandate both point the other way.

Fifty-plus deployments in, the summary is short: self-hosting is not harder than the cloud, it is differently hard. The complexity does not disappear — it moves where you can govern it. For the teams that need that, it is worth every bit of the work.