Choosing an Inference Server: The Landscape, from Data Centers to Laptops

Every inference server answers the same question, "turn this prompt into tokens, fast", but for very different people. A team serving thousands of users, a developer trying a model on a laptop, and an enthusiast squeezing a 125-billion-parameter model into a gaming PC do not want the same tool. Start by placing yourself, then pick.

Three worlds

• Production engines serve many users from shared GPUs. They batch requests continuously, page the KV cache, reuse shared prefixes, and scale across GPUs and nodes. This is where businesses run models, and where most of this article goes.
• Local apps serve one person on one machine. Most of them run llama.cpp underneath and add a model library, a chat window, and a local API. They are where most people start.
• Specialized engines come from enthusiasts and small teams. Each bets on a few models or one GPU and goes further on that target than the general engines do. They are the best place to see new ideas before they reach the big engines.

The versions in this article are the latest releases on 2026-10-05: vLLM v0.31.0, SGLang v0.5.21, TensorRT-LLM v1.2.1 (1.3 is in release candidates), llama.cpp v0.6.0, Ollama v0.35.1.

Production: pick an engine

Two more names you will meet. LMDeploy, from the InternLM team, is a capable engine with two back ends: its own TurboMind engine and a PyTorch engine. Hugging Face TGI is in maintenance mode and its repository is archived; Hugging Face now points to vLLM and SGLang. Treat TGI guides as history.

All three main engines batch continuously, page the KV cache, and serve OpenAI-compatible /v1/completions and /v1/chat/completions. Prefix reuse granularity is where they differ most: vLLM reuses whole blocks of 16 tokens by default, TensorRT-LLM whole blocks of 32, and SGLang single tokens by default, unless the attention backend needs larger pages (several Blackwell and MLA backends do).

Launch commands

A matched setup for one 80 GB GPU: Qwen3-8B in BF16, 32,768-token maximum context, 256 concurrent requests, an 8,192-token step budget, a 196,608-token KV cache, tool calls and reasoning parsing on. KV sizing: 36 layers × 8 KV heads × 128 dimensions × 2 (key and value) × 2 bytes = 144 KiB per token, so 196,608 tokens take 27 GiB, enough for 128 concurrent requests of 1,280 tokens. Every flag is checked against each engine's source at the versions above; these are commands, not a measured run.

vLLM

$ vllm serve Qwen/Qwen3-8B \
    --max-model-len 32768 \
    --max-num-seqs 256 \
    --max-num-batched-tokens 8192 \
    --kv-cache-memory-bytes 28991029248 \
    --enable-auto-tool-choice --tool-call-parser hermes \
    --reasoning-parser qwen3

SGLang

$ sglang serve --model-path Qwen/Qwen3-8B \
    --context-length 32768 \
    --max-running-requests 256 \
    --max-prefill-tokens 8192 \
    --chunked-prefill-size 8192 \
    --max-total-tokens 196608 \
    --tool-call-parser qwen --reasoning-parser qwen3

Add --schedule-policy lpm if your prompts share prefixes; the default is fcfs.

TensorRT-LLM

$ cat > trtllm.yaml <<'EOF'
kv_cache_config:
  max_tokens: 196608
EOF
$ trtllm-serve Qwen/Qwen3-8B \
    --max_seq_len 32768 \
    --max_batch_size 256 \
    --max_num_tokens 8192 \
    --enable_chunked_prefill \
    --config trtllm.yaml \
    --tool_parser qwen3 --reasoning_parser qwen3 \
    --port 8001

This runs the PyTorch backend, the default since 1.0, with no trtllm-build step. KV block reuse (prefix caching) is on by default.

Ports: vLLM 8000, SGLang 30000, TensorRT-LLM 8001 (its default 8000 clashes with vLLM). None of them authenticate by default, and vLLM listens on all interfaces: keep them local or put them behind a proxy.

Equivalent knobs

Where the same row means different things:

• Step budget: vLLM and TensorRT-LLM count prompt and generated tokens together (TensorRT-LLM only splits long prompts with --enable_chunked_prefill). SGLang's budget covers prompts only: generation runs as its own batch unless you pass --enable-mixed-chunk, and --chunked-prefill-size sets the size of each prompt chunk.
• KV size: vLLM's value is exact, in bytes, and overrides the fraction. SGLang's and TensorRT-LLM's are caps: TensorRT-LLM uses the smaller of max_tokens and the fraction.
• KV fraction: vLLM's is a share of total GPU memory for the whole instance (default 0.92), SGLang's covers weights plus the KV pool, TensorRT-LLM's is a share of the memory left after loading the weights (default 0.9). The same 0.9 gives a different cache in each engine: size the cache with the KV size flags and read the allocated size in each startup log.
• Tools: vLLM also needs --enable-auto-tool-choice.

Benchmark with one client

vllm bench serve speaks the OpenAI API, so one client measures every server the same way. Use the completions endpoint to keep chat templates (and Qwen3's thinking block) out of the input, and sweep concurrency, because the winner at 8 concurrent requests can lose at 128:

$ for c in 1 8 32 128; do
    vllm bench serve --backend openai \
      --base-url http://localhost:30000 --endpoint /v1/completions \
      --model Qwen/Qwen3-8B \
      --dataset-name random --random-input-len 1024 --random-output-len 256 \
      --ignore-eos --seed 0 --num-warmups 10 \
      --max-concurrency $c --num-prompts $((c * 20)) \
      --percentile-metrics ttft,tpot,itl,e2el --save-result
  done

• Change only --base-url between engines. If a server exposes another model name, pass it with --served-model-name and keep --model as the Hugging Face ID for the tokenizer.
• --ignore-eos fixes every output at 256 tokens. Check the total generated tokens against num_prompts × 256: not every server honors it.
• At 128 concurrent requests the 196,608-token cache is about 83% full. If you shrink it below what your top concurrency needs, the engines diverge by design: trtllm-serve queues any request whose full KV it cannot reserve, while vLLM and SGLang admit and then preempt. That row then compares admission policies, not engines.
• Read time to first token and time per output token at p99, at the concurrency you expect in production, not the peak throughput.
• Random tokens never share prefixes, so they hide the prefix-caching differences. Rerun with a sample of your real prompts before deciding.

Gotchas

vLLM's default token budget depends on the detected GPU. Under vllm serve, --max-num-batched-tokens defaults to 16,384 on GPUs with 160 GB or more, 8,192 from 70 GB, and 2,048 below that (source). Set it explicitly so two machines run the same configuration.

SGLang's lpm turns itself off under load. With more than 128 requests waiting, it falls back to first-come-first-served (source), so it helps at moderate queue depth, not at saturation. The radix cache keeps reusing prefixes either way.

Prefix caches match exact tokens. A timestamp or request ID near the top of a system prompt changes every token after it and defeats prefix reuse in every engine. Keep shared prefixes byte-identical and put per-request data at the end.

The TensorRT-engine backend is on its way out. 1.2.1 still ships trtllm-build and --backend tensorrt, but the public tensorrt_llm.LLM is now the PyTorch implementation (source) and the release notes file the backend's removal under 1.2. Guides built on per-shape engines describe the legacy path; start from trtllm-serve <hf-model>.

Above the engine: serving stacks

One engine serves one model on one machine or a few. Once you run many replicas, a layer above the engine decides which replica gets each request, moves KV caches around, and scales the fleet. Four open projects lead here:

• NVIDIA Dynamo (v1.5.0): "the orchestration layer above inference engines", in its own words: it runs SGLang, TensorRT-LLM, or vLLM as workers and adds disaggregated serving (prompt processing and generation on separate GPUs), routing, multi-tier KV caching, and automatic scaling.
• llm-d (v0.10.0): Kubernetes-native serving built around vLLM, with prefix-cache-aware routing, tiered KV offloading, and prefill/decode disaggregation. A CNCF sandbox project founded by Red Hat, Google Cloud, IBM Research, CoreWeave, and NVIDIA.
• vLLM Production Stack: the vLLM project's reference deployment on Kubernetes, with a request router that keeps sessions on the replica holding their KV cache, and KV offloading through LMCache.
• AIBrix (v0.7.0): infrastructure components for vLLM fleets: an LLM gateway and router, high-density LoRA management, and an autoscaler tuned for LLM traffic.

These deserve their own pieces, starting with disaggregated serving. The point for now: if you need more than a few replicas, you choose an engine and a stack, and the stack often decides more of your cost than the engine does.

Getting started: on your own machine

Most people meet local inference through an app, and most of those apps run llama.cpp underneath. llama.cpp is a C/C++ engine that runs GGUF models (quantized from 1.5 to 8 bits) on Apple Silicon, NVIDIA, AMD, and Intel GPUs, Vulkan, or plain CPUs, and can split a model between GPU and CPU memory when it does not fit. It ships its own server, llama-server:

$ llama-server -hf Qwen/Qwen3-8B-GGUF:Q8_0 -np 4 --port 8080

The apps add what most people want first: a model library, a chat window, sensible defaults, a local API.

Because they share llama.cpp, speed differences between these apps come mostly from their default settings (context size, offload, parallel requests), not from the engine. Their strength is getting one person productive in minutes; their limits show up the day several people share them:

• One request at a time by default. Ollama serves one request per model unless you raise OLLAMA_NUM_PARALLEL (source), and picks the context length from your VRAM (4k, 32k, or 256k tokens) unless you set it.
• Slots share the context. In llama-server, -c 32768 -np 8 without --kv-unified gives each request 4,096 tokens (source). Left on auto, -np means 4 slots sharing one unified pool (source).
• Built for one user. They have no admission control, no paged KV cache across dozens of requests, and no multi-node scaling. When a team or an application depends on the endpoint, move to vLLM or SGLang: the OpenAI-compatible API stays the same, so the client code does not change.

The specialists: engines built for one target

The general engines support hundreds of models on many GPUs. A new kind of engine does the opposite: a few models, sometimes one GPU, and every optimization aimed at that target. Most are young, move fast, and are run by one person or a small team, so read their numbers as their authors' claims, on their hardware.

ds4 (DwarfStar 4), by Salvatore Sanfilippo (antirez, the creator of Redis). A self-contained C engine "optimized first for DeepSeek V4 Flash", with DeepSeek V4.1 Flash, GLM 5.x, and Qwen3.8 Flash Next. Its targets are machines people can own: Macs with 96 GB or more (Metal is the primary target), the DGX Spark, AMD Strix Halo, and multi-GPU CUDA boxes. It streams weights from the SSD when the model does not fit in memory, and can join two 128 GB Macs over RDMA with tensor parallelism. It uses its own GGUF files, borrows quantization layouts and some kernels from llama.cpp, and says openly that it is beta and built with strong help from AI coding agents.

Strata, by Niko1221. One model, Qwen3.8-Flash-Next (125 billion parameters, a mixture of experts), on a gaming PC with 12 GB of VRAM or more. Its trick is placement: the experts used most often live on the GPU, all of them in system RAM, a lookup table on the SSD, and the CPU computes the rest in parallel; speculative decoding adds 1.6 to 1.8 times on top. Its authors report 79 tokens per second on an RTX 5070 with 64 GB of RAM at 2-bit (IQ2_XS) quantization. It uses parts of llama.cpp and installs with one script on Windows or Linux.

NInfer, by Neroued. A from-scratch C++/CUDA engine for a few Qwen 27B and 35B checkpoints on exactly one GPU, the RTX 5090: the build rejects any other architecture. It fixes one to eight request lanes at startup and offers NVFP4 or integer weights, an FP8 KV cache, and multi-token-prediction speculative decoding. Its published runs on Qwen3.6-27B in NVFP4, with speculative decoding, reach 202 tokens per second for one request and 1,147 in total with eight. Community forks port it to the RTX 3090 and other cards.

Colibri (colibrì), by JustVugg. The opposite trade from NInfer: not the fastest tokens, but the biggest models on the smallest machines. A pure C engine with no dependencies that treats VRAM, RAM, and NVMe storage as one memory hierarchy for mixture-of-experts models from 744 billion to 2.8 trillion parameters (GLM-5.x, Kimi K3, DeepSeek V4 and V4.1 Flash, and smaller ones). The dense part of the model stays in RAM; the routed experts live on disk and are streamed in as the router asks for them, with the most used ones cached closer. Its README is candid about the cost: GLM-5.2 (744B, int4) decodes at about 1.8 tokens per second on a 128 GB CPU-only desktop, and at 0.05 to 0.1 on a 25 GB machine. Slow, but a frontier-size model runs on a desk, and the project publishes its experiments as a research platform for memory tiering.

llmd, by ZML (not the Kubernetes project llm-d). An inference server written in Zig on MLIR and OpenXLA, compiled ahead of time into one self-contained artifact per hardware target: NVIDIA, AMD, Intel, Google TPU, and Apple. It aims at fast cold starts (its authors cite one to two seconds to load an 8B model in BF16) and small images (1.7 GB for CUDA), with continuous batching and paged attention. It is in alpha and supports a list of model families (Llama, Gemma, Qwen, Mistral, and a few more).

What to take from them, even if you never run one:

• They show where the general engines are going. SSD streaming, expert placement across GPU, RAM, and disk (Strata, Colibri), NVFP4 on consumer cards, and ahead-of-time compiled servers all arrive in specialists first.
• Narrow means fast, and fragile. One model or one GPU lets them skip the abstractions that general engines need. Change the model or the card and the advantage, or the support, is gone.
• They are not production servers yet. Beta status, single maintainers, and small request limits are fine for a personal setup or an experiment, not for a service others depend on.

Wrap-up

Place yourself first: a service for many users means vLLM or SGLang (TensorRT-LLM on an all-NVIDIA fleet) and, past a few replicas, a stack above them; one person on one machine means a llama.cpp-based app; a specialist engine is worth it when it targets exactly your model and your hardware. Then launch with matched settings and run the concurrency sweep on your own prompts.