On-Prem LLM Deployment: Ollama, vLLM and NVIDIA NIM in Production

Key Takeaways

• On-prem LLM deployment comes down to picking a serving runtime: Ollama for prototyping and low-concurrency internal tools, vLLM for high-throughput production serving, and NVIDIA NIM for a supported, optimized runtime with enterprise backing
• Throughput separates them: vLLM's PagedAttention and continuous batching deliver multiples of Ollama's throughput under concurrent load — Ollama is convenience-first, vLLM is throughput-first
• GPU sizing is the hard constraint: a 7–8B model fits on a single 24GB card (L4/A10G); a 70B model needs 2× 80GB GPUs at FP16 or ~40GB with 4-bit quantization
• Prodinit built a production on-prem pipeline serving Qwen2.5-VL through Ollama with NVIDIA NIM for optimized inference — no document, prompt, or response left the network

Teams reach for on-prem LLM deployment for one of three reasons: data cannot leave their network, inference costs at API scale no longer make sense, or latency and availability need to be fully under their control. Whatever the driver, the first real decision is not which model to run — it is which serving runtime to run it on. That choice determines your throughput ceiling, your GPU bill, and how much operational work you own.

On-prem LLM deployment means running open-weight models on hardware you control, served through a runtime that handles batching, memory, and the inference API. The three production-standard runtimes — Ollama, vLLM, and NVIDIA NIM — sit at different points on the convenience-versus-throughput spectrum, and picking the wrong one means either over-provisioning GPUs or fighting a runtime that was never built for your load.

What On-Prem LLM Deployment Actually Involves

On-prem LLM deployment has four moving parts: the open-weight model (Llama 3.3, Qwen2.5, Mistral, DeepSeek), the GPU hardware, the serving runtime that turns weights into an inference API, and the operational layer around it — autoscaling, monitoring, and model updates. The runtime is the component teams underestimate most, because it silently determines how many concurrent users a given GPU can serve.

The core technical problem a serving runtime solves is memory management under concurrent load. Every in-flight request consumes GPU memory for its KV cache — the attention state for the tokens generated so far. A naive server allocates a fixed block per request and quickly runs out of memory, forcing requests to queue. The difference between runtimes is largely how cleverly they pack and reuse that memory, which is why the same model on the same GPU can serve very different concurrent loads depending on the runtime.

Everything downstream — GPU count, cost per token, p95 latency under load — flows from that runtime decision. Get it right and a single 80GB GPU serves a team; get it wrong and you provision three to do the same job.

Ollama: Convenience-First for Prototyping and Internal Tools

Ollama is the fastest path from zero to a running local model. It bundles model download, quantization (GGUF format), and an OpenAI-compatible API into a single binary, and it runs comfortably on a workstation GPU or even CPU for smaller models. For prototyping, internal tools, and low-concurrency workloads, it is the lowest-friction option available.

Ollama's design priorities are developer experience and single-user or low-concurrency serving. It defaults to quantized GGUF models (4-bit is common), which shrink memory footprint dramatically — a 7B model runs in roughly 5GB — at a small quality cost. This makes it ideal for running a capable model on modest hardware.

Where Ollama fits:

• Prototyping and evaluating models before committing to a serving architecture
• Internal tools with a handful of concurrent users (a document assistant, an internal chatbot)
• Edge or on-device deployments where a single binary and modest hardware matter
• Vision-language workloads at low concurrency — Prodinit served Qwen2.5-VL through Ollama in a production document-processing pipeline where per-document throughput, not massive concurrency, was the requirement

Where Ollama runs out of room: under genuine concurrent production load, Ollama's throughput falls well short of vLLM. It does not implement the continuous batching and paged KV-cache management that high-concurrency serving needs. If dozens or hundreds of simultaneous requests are the target, Ollama becomes the bottleneck.

vLLM: Throughput-First for Production Serving

vLLM is the production-standard runtime for high-throughput LLM serving. Its two defining features — PagedAttention (KV-cache memory managed like virtual memory pages) and continuous batching (new requests join the batch as soon as slots free up, rather than waiting for the whole batch to finish) — let it serve many multiples of the concurrent requests a naive server or Ollama can handle on the same GPU.

PagedAttention solves the memory-fragmentation problem directly: instead of reserving a large contiguous KV-cache block per request, it allocates small pages on demand and shares them where possible. The practical result is far higher GPU memory utilization and, therefore, far more concurrent requests per card. Continuous batching keeps the GPU saturated instead of idling between batch boundaries.

Where vLLM fits:

• Production inference APIs serving real concurrent traffic
• Cost-sensitive deployments where maximizing tokens-per-GPU directly reduces the hardware bill
• Any workload where p95 latency under concurrent load matters
• Multi-GPU serving of large models via tensor parallelism (70B+ across 2–4 GPUs)

vLLM exposes an OpenAI-compatible API, integrates with Hugging Face model formats, and supports AWQ and GPTQ quantization for fitting larger models onto smaller GPUs. The trade-off relative to Ollama is operational: you manage the deployment, GPU scheduling, and model loading yourself. It is a server component, not a one-command binary.

NVIDIA NIM: Supported and Optimized for Enterprise

NVIDIA NIM (NVIDIA Inference Microservices) packages optimized model-serving containers with TensorRT-LLM engines, an OpenAI-compatible API, and NVIDIA enterprise support. It targets teams that want vLLM-class throughput with a supported, pre-optimized runtime rather than a self-assembled stack — trading some flexibility for operational assurance and vendor backing.

NIM's core value is that NVIDIA has already done the inference optimization. TensorRT-LLM compiles model-specific optimized engines for NVIDIA GPUs, and NIM ships these as ready-to-run containers with tuned configurations. For enterprises standardized on NVIDIA hardware and requiring a supported inference layer — regulated industries, large platform teams — NIM removes the burden of hand-tuning vLLM for each model and GPU combination.

Where NIM fits:

• Enterprises requiring vendor support and SLAs on the inference layer
• Teams standardized on NVIDIA GPUs wanting pre-optimized TensorRT-LLM engines without manual tuning
• Regulated deployments where a supported, auditable runtime simplifies compliance
• Hybrid pipelines pairing a convenience runtime with NIM for the optimized production path — Prodinit used NVIDIA NIM alongside Ollama in an on-prem document pipeline, with NIM providing the optimized inference path

The trade-off: NIM is more opinionated and tied to the NVIDIA ecosystem than raw vLLM, and access runs through NVIDIA AI Enterprise licensing. For teams that value support and pre-optimization over maximal control, that is the point.

On-Prem LLM Runtime Comparison

The decision among the three runtimes comes down to three questions: how much concurrency you need, how much operational work you want to own, and whether you require vendor support. The table below maps each runtime to the workload it fits, based on the same criteria Prodinit weighs when architecting on-prem deployments.

A common and pragmatic pattern is to use more than one: Ollama for local development and evaluation, then vLLM or NIM for the production serving path. Standardizing the OpenAI-compatible API across all three means application code does not change when you switch the runtime underneath it.

GPU Sizing and Quantization for On-Prem Models

GPU memory is the hard constraint in on-prem LLM deployment — it determines which models you can run and how much concurrency you get. The rule of thumb: an FP16 model needs roughly 2GB of GPU memory per billion parameters for the weights alone, plus headroom for the KV cache that scales with concurrent requests and context length.

Practical sizing for common model classes:

• 7–8B models (Llama 3.1 8B, Qwen2.5 7B): ~16GB FP16, fits a single 24GB GPU (NVIDIA L4, A10G) with room for moderate concurrency
• 70B models (Llama 3.3 70B, Qwen2.5 72B): ~140GB FP16, requiring 2× 80GB GPUs (A100/H100) with tensor parallelism — or ~40GB with 4-bit AWQ/GPTQ quantization on a single 48GB card
• Vision-language models (Qwen2.5-VL): budget additional memory for the vision encoder and higher-resolution image tokens beyond the base language model footprint

Quantization is the lever that changes the math. 4-bit quantization (AWQ, GPTQ, or GGUF) cuts memory roughly 4x versus FP16, often letting a large model fit on a single GPU with only a small quality reduction on most tasks. For production, vLLM's AWQ/GPTQ support and NIM's TensorRT quantization preserve throughput better than GGUF, which is optimized more for memory footprint than concurrent-serving speed.

Do not forget KV-cache headroom. A model that exactly fills GPU memory with its weights has no room left to serve requests. Budget 20–40% of GPU memory for the KV cache under production concurrency, and more for long-context workloads. This is precisely where vLLM's PagedAttention earns its keep — it makes that reserved memory serve far more concurrent requests than a naive allocator would.