Key Takeaways
- Ollama vs vLLM is a concurrency decision, not a quality decision — both serve the same open-weight models, but only one of them batches requests
- A peer-reviewed H100 benchmark on identical Qwen3-4B weights measured vLLM at 20–29x higher throughput and 8–19x lower P95 latency than Ollama across four scenarios (Applied Sciences, 2026)
- Ollama's failure mode under load is rejection, not slowness: at 100 concurrent users it served 0.45 req/s and failed 39.29% of requests, while vLLM held 15.20 req/s at a 0% error rate (Applied Sciences, 2026)
OLLAMA_NUM_PARALLELdefaults to 1 and RAM scales withOLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH— you can raise Ollama's concurrency, but you pay for it in GPU memory (Ollama FAQ)- Ollama still wins on time-to-first-deployment and single-user workloads; Prodinit has shipped production pipelines on it where per-document throughput, not concurrency, was the constraint
Most self-hosting decisions get made backwards. A team installs Ollama, gets a model answering prompts in ten minutes, ships it behind an internal API — and discovers at 30 concurrent users that the runtime was never built for the load. The model was never the problem. The serving layer was.
Ollama vs vLLM comes down to one architectural difference: vLLM batches concurrent requests through PagedAttention and continuous batching, while Ollama serves a small number of parallel contexts and queues the rest. Below roughly five concurrent users they are comparable. Above that, vLLM wins by an order of magnitude and Ollama starts rejecting requests.
Ollama vs vLLM: The Short Answer
Pick Ollama when one GPU serves a handful of users and setup time matters more than tokens per second — internal tools, prototypes, batch jobs, vision-language pipelines processing documents one at a time. Pick vLLM the moment a real user-facing product sits in front of the GPU, because sustained concurrency is the specific thing vLLM's engine was designed for.
That boundary is not a matter of taste. It falls out of how each runtime allocates GPU memory, which is the next section.
How Ollama and vLLM Differ Architecturally
The two runtimes make opposite bets on GPU memory. vLLM derives a KV cache budget from --gpu-memory-utilization (0.92 in current stable builds) and manages it as paged blocks shared across in-flight sequences, so new requests slot into a running batch. Ollama allocates a context buffer per parallel request, which makes concurrency a memory-provisioning problem rather than a scheduling one.
vLLM: continuous batching over a paged KV cache. PagedAttention treats the KV cache like virtual memory — fixed-size blocks, allocated on demand, shared between sequences. Continuous batching then swaps a finished sequence out of the batch and a queued one in, instead of waiting for the whole batch to drain. vLLM's own launch benchmarks put this at up to 24x the throughput of HuggingFace Transformers on LLaMA-7B and LLaMA-13B, and 2.2–3.5x that of TGI (vLLM).
Ollama: parallel contexts over a shared model. OLLAMA_NUM_PARALLEL sets how many requests a loaded model handles at once, and the default is 1. Raising it duplicates context buffers, not weights — the FAQ is explicit that required RAM scales by OLLAMA_NUM_PARALLEL × OLLAMA_CONTEXT_LENGTH. Requests beyond the parallel slots queue up to OLLAMA_MAX_QUEUE (default 512) and are then rejected (Ollama FAQ).
That is the whole story. Ollama's concurrency ceiling is set by how much GPU memory you are willing to spend on duplicate context buffers. vLLM's is set by how efficiently it can pack sequences into one cache.
What the Benchmarks Actually Show
The published numbers disagree on magnitude and agree on direction: the gap is near zero at one user and grows with concurrency. The most rigorous comparison is a peer-reviewed study in Applied Sciences that ran both runtimes on a single NVIDIA H100 80GB with identical Qwen3-4B weights and inference configuration across four workload scenarios.
Its headline findings: vLLM delivered 20–29x higher throughput and 8–19x lower P95 latency, and held a 100% request success rate at up to 100 concurrent users. In the simple Q&A scenario at 100 users, vLLM sustained 15.20 req/s with a 0% error rate while Ollama managed 0.45 req/s and failed 39.29% of requests (Ay & Demirdağ, Applied Sciences 16(11), 5435, 2026).
Red Hat's engineering benchmark points the same way on NVIDIA A100 hardware, sweeping 1 to 256 concurrent users: roughly 793 output tokens/sec at peak for vLLM versus 41 for Ollama, and P99 latency of 80ms versus 673ms — with vLLM ahead at every concurrency level tested, even after Ollama was tuned for parallelism (Red Hat Developer).
Two caveats worth carrying into your own sizing work:
- Ratios are not portable. Published multipliers range from about 2x to 29x depending on GPU, model size, quantization, and how Ollama was tuned. Treat them as direction, then benchmark your model on your hardware at your concurrency.
- The error rate matters more than the multiplier. A 20x throughput gap is a capacity planning input. A 39% failure rate is an outage. Ollama past its parallel slots does not degrade gracefully — it queues, then rejects.
Ollama vs vLLM: Decision Table
Runtime selection reduces to four questions: how many concurrent users, how much GPU memory you have, which quantization format your weights ship in, and how much operational work you want to own. The table maps each answer to a runtime.
| Dimension | Ollama | vLLM |
|---|---|---|
| Concurrency sweet spot | 1–5 simultaneous requests | 8–256+ simultaneous requests |
| Batching | Per-request context buffers, OLLAMA_NUM_PARALLEL default 1 | Continuous batching over a paged KV cache |
| Throughput under load | Plateaus, then queues and rejects | Scales with the batch until the cache saturates |
| Time to first deployment | Minutes — single binary, ollama pull | Hours — GPU memory tuning, model format, serving config |
| GPU memory model | Scales with parallel slots × context length | One budget (--gpu-memory-utilization, default 0.92) |
| Model formats | GGUF quantizations, easy local swaps | FP16/BF16, plus AWQ, GPTQ and FP8 quantized weights |
| Multi-GPU | Not the design target | Tensor and pipeline parallelism for models past one card |
| Cold start | Fast | Slower — weight load, memory profiling, CUDA graph capture |
| Best fit | Internal tools, prototypes, low-concurrency document pipelines | User-facing products, high-concurrency APIs, shared GPU fleets |
Where Ollama Still Wins
Ollama is not a worse vLLM — it optimizes for a different constraint, and there are production workloads where that constraint is the real one. It wins wherever time-to-deployment, operational simplicity, or single-stream latency matters more than aggregate tokens per second.
- Low-concurrency document pipelines. Prodinit built a production on-prem pipeline serving Qwen2.5-VL through Ollama, paired with NVIDIA NIM for the optimized inference path, where per-document throughput rather than massive concurrency was the requirement. No document, prompt, or response left the network.
- Internal tools and prototypes. One engineer, one GPU, a working endpoint before lunch. The engineering cost of vLLM's tuning pass is not worth paying for a tool with six users.
- Fast model iteration. Pulling and swapping GGUF quantizations is a one-line operation, which makes Ollama a good evaluation harness even when vLLM will serve production.
- Air-gapped environments with modest load. A single binary and pre-staged weights are far easier to mirror into a zero-egress network than a GPU serving stack. Our air-gapped LLM deployment guide covers the mirroring specifics, and what air-gapped AI means if the term is new.
The failure mode to avoid is the default one: Ollama chosen in week one for its install experience, then left in place when the product grows past it.
The Operational Cost of Running vLLM
vLLM's throughput is not free — it arrives with a tuning surface that Ollama does not have. Budget roughly a day of engineering for a first production deployment, and a recurring cost in weight conversion and capacity planning. Teams that skip this step hit OOM errors on startup and blame the runtime.
Four things actually take the time:
- GPU memory budgeting.
--gpu-memory-utilizationgoverns weights, activations, and KV cache together. Too low and you starve the cache, capping concurrency; too high and the process OOMs. It is a per-instance limit, so co-locating two vLLM servers on one card means splitting it deliberately. - Weight format. vLLM serves FP16/BF16 plus AWQ, GPTQ, and FP8 quantized weights — not Ollama's GGUF files. Moving a model between the two usually means sourcing different weights, which is a supply-chain step in air-gapped builds.
- Cold start. vLLM loads weights, profiles memory, and captures CUDA graphs before serving the first request. That interacts badly with aggressive Kubernetes liveness probes and scale-to-zero autoscaling; readiness probes need headroom.
- Capacity planning. Continuous batching means throughput depends on the mix of sequence lengths in flight, so a single tokens-per-second figure is not a capacity model. Benchmark at your real concurrency and prompt-length distribution.
None of this is exotic, but it is why we treat runtime selection as an infrastructure decision rather than a library choice — the same way self-hosted LLM deployment and inference cost optimization are infrastructure decisions.
How Prodinit Picks a Runtime in Production
Prodinit runs the same three-step check on every self-hosting engagement: establish the real concurrency target, size the GPU against it, then pick the runtime that clears the target with the least operational surface. The order matters — picking the runtime first is what produces the rebuild six months later.
Step 1 — Measure the concurrency target, don't estimate it. Peak simultaneous in-flight requests, not daily request volume. A pipeline processing 50,000 documents a night at a concurrency of 2 is an Ollama workload. A support assistant with 400 daily users and bursty concurrency of 30 is not.
Step 2 — Size the GPU against the model, then the batch. A 7–8B model fits a single 24GB card; a 70B model needs two 80GB GPUs at FP16, or roughly 40GB with 4-bit quantization. Only after the weights fit does KV cache headroom — the thing that sets your real concurrency ceiling — become the binding constraint.
Step 3 — Default to vLLM for anything user-facing. If the endpoint has external users, the concurrency will grow and the rejection behavior is unacceptable. If it is internal and bounded, Ollama's simplicity is a genuine win. For the three-way picture including NVIDIA NIM, our on-prem LLM deployment guide covers GPU sizing and quantization in more depth, and the air-gapped EKS deployment case study shows what the surrounding infrastructure looks like in a regulated environment.
Get Prodinit's AI engineering guides in your inbox
Deep-dives on production LLMs, voice AI, and MLOps — published weekly. No sales emails.
Frequently Asked Questions
No. At a single concurrent request the two are roughly comparable, and some benchmarks put Ollama slightly ahead on single-stream latency. vLLM's advantage comes from batching, so it grows with concurrency — the peer-reviewed H100 comparison measured 20–29x higher throughput across multi-user scenarios, not single-user ones (Applied Sciences, 2026).
Yes, within limits. OLLAMA_NUM_PARALLEL controls how many requests a model serves simultaneously and defaults to 1, and each additional slot costs GPU memory because required RAM scales by parallel slots times context length. Beyond those slots, requests queue to OLLAMA_MAX_QUEUE (default 512) and are then rejected (Ollama FAQ).
Treat roughly five simultaneous requests as the review point and eight as the switch. Below five, tuning Ollama's parallel slots is usually cheaper than operating vLLM. Above eight, you are paying for GPUs that batching would have saved — and benchmarks show Ollama's throughput plateauing well before vLLM's does.
Yes, with preparation. vLLM pulls weights from Hugging Face and its container images from public registries by default, so both must be mirrored into private storage before deployment. Once weights and images are staged locally, vLLM serves entirely offline. The staging work, not the runtime, is the hard part of a zero-egress build.
Technically yes, practically rarely. Both reserve GPU memory up front — vLLM against --gpu-memory-utilization, Ollama per parallel context — so co-location means hand-splitting the card and accepting a lower concurrency ceiling on both. A more common pattern is Ollama on a dev box for model evaluation and vLLM on the production GPU.