Self-Hosted LLMs for Enterprise: A Deployment Playbook

Key Takeaways

• Self-hosted LLMs are right for an enterprise when one of four conditions holds: a hard zero-egress or data-residency requirement, sustained high token volume, a fine-tuned model you own, or latency and capacity guarantees a shared API cannot commit to. If none hold, a managed endpoint ships faster and costs less
• The reference stack is five layers — GPU compute, a serving runtime (vLLM for throughput, NVIDIA NIM for supported enterprise inference, Ollama for low-concurrency internal tools), an OpenAI-compatible gateway, evaluation and observability, and a controlled model-ingestion path
• GPU sizing is the first budget decision, not an afterthought: a 70B model at FP16 needs roughly 140 GB of VRAM (2× 80 GB cards); 4-bit quantization brings that to ~40 GB and changes the hardware conversation entirely
• Cost break-even against a frontier API is a range, not a line — published analyses put it anywhere from roughly 100M to 500M+ tokens per month depending on model tier, GPU pricing, and utilisation. Against a cheap open-weight API, self-hosting often never wins on cost alone
• Prodinit delivered a fully air-gapped production platform on private AWS EKS in 4 weeks — 10+ VPC interface endpoints, zero public registry dependencies, zero internet egress confirmed across every component (case study)

Most enterprise self-hosting projects do not fail at inference. They fail around week six, when the model is serving fine on a single GPU node and nobody has decided who gets paged when it OOMs at 2am, how a new model version reaches the private registry, or which team owns the token budget.

The inference part is a solved problem. The operating model is what you are actually buying.

Self-hosted LLMs for enterprise workloads mean running open-weight model inference on infrastructure you control — GPU nodes inside your VPC or datacentre, a serving runtime such as vLLM or NVIDIA NIM, and an internal OpenAI-compatible endpoint your applications call. You gain data control, fixed capacity cost, and model ownership. You take on the operational surface an API vendor was absorbing for you.

When a Self-Hosted LLM Is the Right Call for an Enterprise

A self-hosted LLM is justified in an enterprise when one of four conditions holds: a compliance rule forbids prompts leaving your network, token volume is high enough to beat per-token API pricing, you have fine-tuned a model you own, or you need capacity and latency guarantees no shared API will offer. With none, you are buying operations for nothing.

In practice the first condition drives most enterprise programmes we see. A compliance review lands three weeks before launch, api.openai.com becomes a forbidden destination, and the entire inference layer has to move inside the network boundary. That is the air-gapped deployment pattern, and it has a hard architectural answer.

The weakest reason to self-host is cost intuition without a measurement. Teams assume owning the GPU is cheaper than renting the token. Below a certain volume, at realistic utilisation, it is not — and utilisation is the variable that surprises people. A GPU node reserved for a workload that runs eight hours a day is billed for twenty-four.

Be honest about which condition applies before you write a Terraform module. If it is compliance, the regulated-industry buyer's guide covers what auditors actually ask for. If it is cost, keep reading to the break-even section.

The Reference Architecture for an Enterprise Self-Hosted LLM Stack

An enterprise self-hosted LLM stack has five layers, and skipping any one of them is what turns a working demo into an unsupportable system. The layers are GPU compute, a serving runtime, an OpenAI-compatible gateway, an evaluation and observability layer, and a controlled model-ingestion path. Prodinit builds these on Kubernetes — AWS EKS in cloud environments, bare-metal K8s on-prem.

Layer 1 — GPU compute. Managed node groups with GPU instance types (A10G/L4 for 7–8B models, A100/H100 for 70B-class), the NVIDIA device plugin, node taints so nothing else schedules onto them, and an autoscaler with a floor above zero so you are not cold-starting a 140 GB model on the first request.

Layer 2 — serving runtime. vLLM for high-throughput production serving — PagedAttention and continuous batching are what let a single node hold concurrent load. NVIDIA NIM when you want an optimised, vendor-supported container with an enterprise support contract behind it. Ollama for prototyping and low-concurrency internal tools, not for a customer-facing path. The trade-offs are covered in depth in our on-prem LLM deployment guide.

Layer 3 — gateway. Put an OpenAI-compatible API gateway in front of the runtime, even with one model behind it. It is where you attach authentication, per-team rate limits, token accounting, request logging, and model routing. Without it, every application hardcodes a pod address and you can never move a model.

Layer 4 — evaluation and observability. Latency percentiles, tokens per second, GPU utilisation and KV-cache pressure, plus output-quality scoring on a fixed regression set. A self-hosted model has no vendor status page. You are the status page. Our LLM observability guide covers what to log and why.

Layer 5 — model ingestion. Weights and container images enter through a controlled path — a private S3 bucket and a private registry such as Amazon ECR — never a runtime pull from Hugging Face or Docker Hub. In the air-gapped EKS platform Prodinit built for a regulated fintech, every third-party controller image was pre-mirrored into private ECR before the cluster bootstrapped, and the pipeline ran with zero public registry dependencies at any stage.

Sizing GPUs and Finding Your Cost Break-Even

GPU sizing is arithmetic, and it sets the budget. A model needs roughly two bytes per parameter at FP16, plus KV-cache headroom that scales with concurrency and context length. A 7–8B model fits on a single 24 GB card. A 70B model needs about 140 GB of VRAM at FP16 (two 80 GB GPUs), or roughly 40 GB with 4-bit quantization.

Quantization is the decision that moves the budget most in this exercise. Moving a 70B model from FP16 to 4-bit takes it from a two-GPU node to a single card, with a quality cost you can measure on your own eval set rather than guess at. Run that measurement before you size the cluster, not after.

Break-even against an API is a range, not a threshold. Published cost analyses place it anywhere from roughly 100M to 500M+ tokens per month depending on which API you are comparing against, GPU pricing, and — decisively — utilisation (Cloudzy's cost breakdown walks the math). Compared against a frontier model API, high-volume self-hosting wins. Compared against a cheap open-weight API at a fraction of a cent per thousand tokens, it frequently never wins on cost alone.

Three costs teams leave out of the model: idle GPU hours, the engineer-time cost of running the stack, and the second environment. You need staging. A staging cluster with a GPU node is not free, and running evaluations against production capacity is how you cause the incident you were trying to prevent. If cost control is the goal rather than data control, start with the cheaper interventions in our LLM cost optimization guide before buying hardware.

The Four-Phase Rollout Playbook

A self-hosted LLM rollout that reaches production runs in four phases over roughly 8–12 weeks: a decision spike, a single-node pilot, a hardened platform build, and a progressive cutover. The sequencing matters because each phase produces the evidence that de-risks the next one — and gives you a real exit point if the economics stop working.

Phase 1 — Decision spike (1–2 weeks). Pick the candidate open-weight model. Build a 50–200 case evaluation set from your real traffic, and score the candidate against your current API model on it. Measure tokens per month from actual logs, not estimates. Deliverable: a go/no-go with a quality delta and a cost model. Most of the value of this phase is in the projects it stops.

Phase 2 — Single-node pilot (2–3 weeks). One GPU node, vLLM, the gateway in front, one internal non-critical workload behind it. You are measuring p50/p95 latency under realistic concurrency and finding the throughput ceiling of the node. Deliverable: a sizing number you trust.

Phase 3 — Platform build (3–4 weeks). This is the real engineering: Kubernetes with GPU node groups and autoscaling, private registry and model ingestion, secrets management, network policy, CI/CD, observability, runbooks. Prodinit delivered exactly this scope — networking, compute, data layer, CI/CD, security controls, and AI service integrations — for a regulated fintech in 4 weeks, with the client's team independently deploying new services by day 28. The two failure modes to watch: VPC endpoints discovered missing at runtime, and controller images that were not pre-mirrored before the cluster bootstrapped.

Phase 4 — Progressive cutover (2–3 weeks). Shadow traffic first — mirror production requests to the self-hosted endpoint and compare outputs offline, with no user impact. Then a small percentage of live traffic, then a ramp, with the API provider kept warm as a fallback behind the gateway. Keep that fallback for at least a quarter. A gateway that can fail over to a hosted API is the single cheapest piece of insurance in this architecture.

Who Operates It: The LLMOps Surface You Are Taking On

Self-hosting moves an on-call burden from a vendor to your team, and the enterprises that struggle are the ones that budgeted for GPUs but not for the people. The recurring work is model version upgrades, GPU capacity planning, quantization re-validation, eval-set maintenance, incident response for a service with no vendor status page, and a token-accounting story finance will accept.

Staffing this does not require a large team, but it does require named owners. In practice it is a platform engineer who owns the cluster and the GPU nodes, and an AI engineer who owns the model, the eval set, and the quality bar. Those are two different skill sets, and collapsing them into one person is the most common reason a self-hosted stack quietly rots after launch.

Write the runbooks during Phase 3, while the knowledge is fresh and the person who built it is still on the project. A runbook produced after handoff is a work of fiction. The handoff test is simple: can the internal team deploy a new model version, roll it back, and explain the token bill without calling the people who built it?