Private LLMs for Regulated Industries: A Buyer's Guide

Key Takeaways

• Private LLMs for regulated industries means one thing at the infrastructure layer: model inference that never leaves your network boundary — either via a managed provider's VPC endpoint (Amazon Bedrock) or a self-hosted open-weight model (Ollama, vLLM, NVIDIA NIM) on hardware you control
• Healthcare, financial services, government, and defense each have different compliance drivers, but the same infrastructure requirement: zero internet egress on the inference path and a complete audit trail for every model call
• Two deployment patterns cover the regulated space — Pattern A (Bedrock via VPC interface endpoint, fast, managed) and Pattern B (self-hosted open-weight models on EKS/on-prem, full control, no per-token costs) — most regulated teams start with Pattern A
• Prodinit delivered a full production air-gapped LLM stack for a regulated fintech in 4 weeks: private EKS, 10+ VPC endpoints, Bedrock and Transcribe via endpoint, zero internet egress confirmed, 5 data services live at handoff

Most regulated organisations considering an LLM deployment arrive at the same architectural question within the first week: if we cannot send client data or transaction data to a third-party API, how do we use a production-quality LLM at all? The answer — private LLM deployment inside your network perimeter — is technically straightforward but operationally specific. The infrastructure requirements are non-negotiable, the compliance failure modes are predictable, and the vendor landscape is narrower than it looks from a distance.

Private LLMs for regulated industries means deploying large language model inference inside a network boundary you control — either through a cloud provider's private endpoint that keeps traffic off the public internet, or through self-hosted open-weight models running on infrastructure inside your VPC or data centre. The choice between the two is a function of your model requirements, your operational capacity, and your compliance framework. The decision belongs at the architecture phase, not at the procurement phase.

Which Industries Actually Need a Private LLM?

Four regulated industries consistently hit the infrastructure constraints that make public LLM APIs non-viable. The regulatory drivers differ; the infrastructure requirement is the same.

Healthcare. HIPAA's Privacy Rule requires that protected health information (PHI) — which includes any patient utterance processed by a voice AI, any clinical document processed by a RAG pipeline, and any patient-identifiable data sent to an LLM — be handled under a Business Associate Agreement with every vendor that processes it. OpenAI, Anthropic, and most LLM API providers offer BAAs under enterprise agreements. The compliance gap is not the BAA itself — it is the data lineage question once PHI leaves your network boundary: where was it processed, under what retention policy, and what is the audit trail? For organisations with strict data residency requirements or that are building patient-facing systems, VPC-private inference resolves the lineage question completely. The healthcare AI development guide covers the HIPAA architecture in detail.

Financial services. SOC 2 Type II requires demonstrable data flow controls — auditors ask specifically where model inference data goes and whether it leaves your network boundary without explicit data-handling agreements. FCA and SEC guidance on model risk management creates additional pressure to keep inference within your controlled perimeter, particularly for models that process client financial data. FINRA-regulated firms face examination on AI data handling practices. For most regulated financial services organisations, the compliance question is not "do we need a BAA?" (the answer is yes) but "can we demonstrate complete data residency?" — which only a VPC-private deployment can answer cleanly.

Government and defence. FedRAMP-authorised cloud services are the minimum baseline for US federal deployments; for classified workloads, on-premises self-hosted models are often the only option. Government LLM deployments frequently prohibit any inference traffic leaving an agency-controlled environment. This is the use case where Pattern B (self-hosted on-premises) is mandatory — no cloud endpoint, regardless of how private, satisfies a classified data handling requirement.

Legal and insurance. Attorney-client privilege creates a data handling question for legal AI: when client communications are processed by a third-party API, privilege arguments become complicated. Insurance underwriting data carries PII at scale. Both sectors are increasingly subject to state-level AI governance requirements and to client contract terms that prohibit third-party data processing without explicit consent. Private LLM deployment eliminates the third-party processing question by keeping data inside the firm's own infrastructure.

The Two Deployment Patterns for Private LLMs

Every production private LLM deployment for regulated industries falls into one of two patterns. The right choice depends on model access requirements, operational capacity, and cost structure at your call volume.

Pattern A: Managed endpoint via VPC interface endpoint. Amazon Bedrock serves Claude, Llama, Mistral, Titan, and other foundation models through a VPC interface endpoint. Your inference API calls route through the AWS network directly to the Bedrock control plane — no public internet path, no data leaving the AWS network. All processing is subject to your existing AWS BAA if applicable. Azure OpenAI also supports private endpoints for a subset of models. Google Vertex AI offers VPC Service Controls for similar network isolation.

Pattern A is the right starting point for most regulated organisations because it eliminates the GPU infrastructure problem while satisfying the zero-egress constraint. You get production-quality foundation models, managed serving infrastructure, and compliance-ready network isolation without provisioning a single GPU instance.

Pattern B: Self-hosted open-weight models. Llama 3.x, Qwen2.5, Mistral, DeepSeek, or Phi-4 deployed on GPU instances — either in a private VPC (EKS with GPU node groups) or on-premises hardware — using vLLM, Ollama, or NVIDIA NIM as the serving layer. Model weights are pre-loaded from a private S3 bucket (via S3 gateway endpoint) or from local storage. Inference runs on hardware you control; no inference traffic leaves your environment under any circumstances.

Pattern B is the right call when: model choice requirements exceed Bedrock's catalogue, per-token API costs at scale justify the GPU infrastructure overhead, or the compliance framework prohibits cloud endpoints of any kind (classified workloads, certain government programmes). It also applies when the application requires a custom fine-tuned or quantised model that a managed provider cannot serve.

A document AI client Prodinit worked with used Pattern B for a fully air-gapped document extraction pipeline: Qwen2.5-VL served via Ollama on NVIDIA NIM inside a private on-premises environment, processing document images with PaddleOCR for text extraction and the VLM for structured data interpretation — zero network egress from the inference layer.

What Compliance Actually Requires at the Infrastructure Layer

Compliance frameworks describe data handling requirements in policy language. The infrastructure consequences are specific and non-negotiable.

Zero internet egress on the inference path. Every model call, every document sent for processing, every audio transcript sent to an STT endpoint must stay inside your network boundary. For cloud deployments: private subnets with no internet gateway, and a VPC interface endpoint for every AWS service your workloads call. Missing a single endpoint causes runtime failures. The list for a minimal EKS deployment with Bedrock: ECR API, ECR Docker, Secrets Manager, STS, CloudWatch Logs, ELB, SSM, S3 gateway endpoint, and Bedrock endpoint. For voice AI workloads: add Transcribe.

Immutable per-call audit trail. SOC 2, HIPAA § 164.312(b), and most financial compliance frameworks require an audit trail for every model call: who sent the request, what was processed, when, and what model version handled it. Standard LLM observability tools (those that send telemetry to third-party clouds) do not satisfy this requirement in a zero-egress environment. The audit logging stack must be entirely VPC-internal: CloudWatch Logs, S3 access logging, and a self-hosted or VPC-connected observability layer. Langfuse self-hosted is the option Prodinit uses for LLM call tracing in air-gapped environments.

Private container registry. Every container image — including third-party Kubernetes controllers — must be mirrored into a private registry before any cluster bootstraps. In a zero-egress environment, a container that pulls from Docker Hub or public ECR will fail with ImagePullBackOff. Mirror all dependencies before provisioning. This is the most common cause of deployment day failures for teams that have not built zero-egress environments before.

VPC-internal CI/CD. Standard GitHub Actions hosted runners, GitLab shared runners, and most CI/CD tooling runs on internet-connected infrastructure and cannot reach a private EKS API endpoint. The CI/CD pipeline must be rebuilt for VPC-internal operation: a self-hosted runner deployed as a pod inside the VPC, or AWS CodePipeline with a VPC-connected build agent.

Access controls and BAA/DPA chain. Every vendor that processes data covered by your compliance framework must be under a signed agreement. For a cloud private LLM deployment: the cloud provider (AWS/Azure/GCP BAA or equivalent), the observability platform if self-hosted on covered infrastructure, and any data pipeline tooling that touches covered data. Reviewing the agreement chain before vendor selection — not after architecture is locked — avoids expensive late-stage changes.

Polymath: Production Air-Gapped LLM for Regulated Fintech in 4 Weeks

A regulated financial services AI platform came to Prodinit with a hard requirement: private subnets with zero internet egress, an LLM inference layer for their application, and a complete production deployment in four weeks. Standard cloud deployment tooling did not work. Every component had to be rebuilt for zero-egress operation.

Prodinit delivered the full production infrastructure across five parallel workstreams: networking and VPC endpoints, private EKS cluster configuration, air-gapped CI/CD pipeline, security controls, and AI service integrations — all live on day 28.

What was delivered:

• Multi-AZ VPC with 10+ VPC interface endpoints covering ECR, Secrets Manager, Bedrock, Amazon Transcribe, STS, CloudWatch, ELB, and SSM — zero AWS service call leaves the private network
• Private EKS cluster with managed node groups in private subnets; all controller images pre-mirrored to private ECR before cluster bootstrap to avoid ImagePullBackOff failures
• Self-hosted GitHub Actions runner deployed inside the VPC; deployments via kubectl and Helm through AWS Systems Manager Session Manager with zero open inbound ports
• Amazon Bedrock and Transcribe accessed via VPC interface endpoints — IAM roles scoped to specific model ARNs; inference data never touches the public internet
• RDS PostgreSQL with pgvector, DynamoDB, ElastiCache Redis — all on private subnets, no public endpoints

The CTO's assessment: "Prodinit delivered a production-grade, fully air-gapped EKS environment in four weeks. The infrastructure was solid from day one, and our team was independently deploying new services before the engagement even ended."

The full architecture — network topology, VPC endpoint list, CI/CD configuration, and AI service integration — is documented in the air-gapped EKS fintech case study. For the zero-egress infrastructure layer specifically, the air-gapped AI for fintech guide covers the five infrastructure layers that must be resolved before any application code can deploy.

Procurement Checklist: Evaluating Private LLM Vendors

When evaluating an engineering partner or platform vendor for a private LLM deployment in a regulated environment, the checklist below separates vendors with genuine zero-egress production experience from those who understand the concept but have not shipped it.

Infrastructure experience:

• Has the vendor shipped a production zero-egress LLM deployment before? At what scale and timeline?
• Can they name every VPC endpoint required for your target stack before the project starts?
• Have they rebuilt CI/CD for VPC-internal operation — not just documented that it is required?
• What is their process when a missing VPC endpoint is discovered at runtime?

Compliance depth:

• Can the vendor review your compliance framework (HIPAA, SOC 2, FedRAMP, FCA) and map specific infrastructure requirements before architecture is locked?
• What is their BAA/DPA chain for the components they provide? Is there a gap?
• What does their audit logging architecture look like, and is it queryable for your specific compliance audit requirements?

Model layer:

• For Pattern A: which foundation models are available via private endpoint in your target region? Bedrock's catalogue varies by region.
• For Pattern B: which serving stack (vLLM, Ollama, NVIDIA NIM)? What GPU provisioning and model weight management approach?
• What is the process for updating models inside the zero-egress environment — are new weights pulled from a private registry, and how is that pipeline managed?

Operational handoff:

• What runbook documentation is delivered at handoff — can your team operate and extend the infrastructure independently?
• What does the support model look like for the first 90 days after deployment?
• Has the team done the post-handoff knowledge transfer to make your engineers self-sufficient, or does independence create a dependency on the vendor?