---
title: "vLLM vs SGLang: A Practical Comparison of Two LLM Inference Servers"
url: "https://romankryvolapov.com/en/vllm-vs-sglang/"
description: "vLLM 0.34 and SGLang 0.5.23 side by side: installation, launching models, GPU memory, formats and quantization, configuration, API, conversation cache, network and security, routing, metrics, logs and tracing — with commands, config files, screenshots and measurements on a 12 GB GPU."
language: en
published: 2026-11-21
updated: 2026-11-21
---
**Hi!**

vLLM and SGLang are the two main open-source servers for running language models on your own GPUs. They do the same job, have the same API and similar speed. They differ in memory handling, configuration, security and monitoring.

Each section of the article starts with the common part, then a subsection on vLLM and a subsection on SGLang. Versions: **vLLM 0.34.0** of 16 November 2026 and **SGLang 0.5.23** of 2 November 2026. Commands, config files and output are taken from a working test bench; what is retold from the documentation without being run is marked.

The main points:

- **A single request** is handled at the same speed by both. The difference appears under load and depends on the model.
- **Memory:** vLLM allocates it by itself, in SGLang you set the share.
- **The conversation cache** is more precise in SGLang: it saves more on long conversations and agents.
- **Security:** neither should be exposed to the internet without a proxy.
- **vLLM** is the default choice, **SGLang** is for agents and large clusters. The selection scheme is [at the end](#What-to-choose).

How inference engines work inside is covered in [a separate article](/en/llm-inference-engines/). Four concepts from it are needed here: [tokens](/en/llm-inference-engines/#Tokens-and-weights), [the prefill and decode phases](/en/llm-inference-engines/#Prefill-and-decode-two-phases-with-different-bottlenecks), [the KV cache](/en/llm-inference-engines/#The-KV-cache) and [batching](/en/llm-inference-engines/#Batching-why-a-server-serves-many-users-at-almost-the-cost-of-one). TTFT is the time to the first token of the answer.

## What vLLM and SGLang are

An inference server keeps the model in GPU memory, accepts requests over HTTP in the OpenAI format and handles many requests at once. The path of a request:

<div style="overflow-x:auto;margin:1.5rem 0" tabindex="0" role="region">
<svg viewBox="0 0 780 290" width="100%" style="min-width:620px;max-width:780px;height:auto;display:block;margin:0 auto" role="img" aria-label="The path of a request: client, API server, queue, scheduler, GPU, answer" xmlns="http://www.w3.org/2000/svg">
  <defs>
    <marker id="vs-request-head" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L10 5 L0 10 z" fill="var(--color-fg-dim)"/>
    </marker>
    <marker id="vs-request-tail" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L10 5 L0 10 z" fill="var(--color-fg-dim)"/>
    </marker>
  </defs>
  <rect x="30" y="24" width="200" height="92" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-link)" stroke-width="1.5"/>
  <text x="130" y="48" text-anchor="middle" fill="var(--color-link)" font-family="var(--font-sans)" font-size="14" font-weight="500">client</text>
  <text x="130" y="68" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">HTTP request</text>
  <text x="130" y="85" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">in the OpenAI format</text>
  <rect x="290" y="24" width="200" height="92" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-border)" stroke-width="1.5"/>
  <text x="390" y="48" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="14" font-weight="500">API server</text>
  <text x="390" y="68" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">chat template, tokens,</text>
  <text x="390" y="85" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">key check</text>
  <rect x="550" y="24" width="200" height="92" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-border)" stroke-width="1.5"/>
  <text x="650" y="48" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="14" font-weight="500">queue</text>
  <text x="650" y="68" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">the request waits</text>
  <text x="650" y="85" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">for memory and a batch slot</text>
  <rect x="550" y="176" width="200" height="92" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-accent)" stroke-width="1.5"/>
  <text x="650" y="200" text-anchor="middle" fill="var(--color-accent)" font-family="var(--font-sans)" font-size="14" font-weight="500">scheduler</text>
  <text x="650" y="220" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">builds a batch every step</text>
  <text x="650" y="237" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">from all active requests</text>
  <rect x="290" y="176" width="200" height="92" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-accent)" stroke-width="1.5"/>
  <text x="390" y="200" text-anchor="middle" fill="var(--color-accent)" font-family="var(--font-sans)" font-size="14" font-weight="500">GPU</text>
  <text x="390" y="220" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">prefill: reading the prompt</text>
  <text x="390" y="237" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">decode: token by token</text>
  <rect x="30" y="176" width="200" height="92" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-link)" stroke-width="1.5"/>
  <text x="130" y="200" text-anchor="middle" fill="var(--color-link)" font-family="var(--font-sans)" font-size="14" font-weight="500">answer</text>
  <text x="130" y="220" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">tokens → text,</text>
  <text x="130" y="237" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">streamed as it is generated</text>
  <path d="M230 70 L290 70" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" marker-end="url(#vs-request-head)"/>
  <path d="M490 70 L550 70" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" marker-end="url(#vs-request-head)"/>
  <path d="M650 116 L650 176" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" marker-end="url(#vs-request-head)"/>
  <path d="M550 222 L490 222" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" marker-end="url(#vs-request-head)" marker-start="url(#vs-request-tail)"/>
  <path d="M290 222 L230 222" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" marker-end="url(#vs-request-head)"/>
  <text x="520" y="212" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12" font-weight="400">step</text>
</svg>
</div>

### vLLM: who builds it

- [Repository](https://github.com/vllm-project/vllm): about 97 thousand stars, Apache 2.0 license, a new version every two weeks.
- Appeared in 2023 in Berkeley. Since 2025 it has been a PyTorch Foundation project. Since January 2026 it is backed by the company Inferact (150 million dollars of investment).
- Who uses it: [LinkedIn](https://www.linkedin.com/blog/engineering/ai/how-we-leveraged-vllm-to-power-our-genai-applications) (more than 50 products), Red Hat (the basis of its products for OpenShift), AWS (SageMaker and Bedrock), NVIDIA (the [NIM 2](https://docs.nvidia.com/nim/large-language-models/2.1.5/about-nim-llm/overview.html) containers contain vLLM or SGLang).

### SGLang: who builds it

- [Repository](https://github.com/sgl-project/sglang): about 39 thousand stars, Apache 2.0 license, a new version every two weeks.
- Appeared in 2024. Led by the non-profit LMSYS, the authors of the Chatbot Arena leaderboard. Since 2026 it is backed by the company RadixArk (100 million dollars of investment).
- Who uses it: xAI (the [Grok-2 model card](https://huggingface.co/xai-org/grok-2) tells you to run the model through SGLang), Meta ([according to its talk](https://www.lmsys.org/blog/2026-11-17-sglang-summit-recap) in November 2026), Cloudflare (Workers AI), LinkedIn (search ranking; LinkedIn uses both servers).

## How they differ from other ways to run a model

vLLM and SGLang are built for many simultaneous requests on your own GPUs. For a single user they are no faster than simple programs.

| Option | What for | Difference from vLLM and SGLang |
|---|---|---|
| **llama.cpp**, Ollama, LM Studio | a model on your own computer | works even without a GPU, compressed GGUF models, starts in seconds; weaker under load from many users |
| **TensorRT-LLM** | maximum speed on NVIDIA | NVIDIA only, fewer models |
| **NVIDIA NIM** | a ready container with paid support | not an engine: vLLM or SGLang inside |
| **TGI** from Hugging Face | the former main rival | archived, its authors recommend vLLM and SGLang |
| **Dynamo, llm-d, AIBrix** | dozens of GPUs in Kubernetes | a layer above the engines: spreads requests across copies of vLLM or SGLang |
| **LiteLLM** | one address for different models, keys, budgets | a gateway in front of the servers, does not run models |
| **Cloud API** | pay per token | nothing to maintain; your own GPU pays off only under constant load |

More in the [map of engines](/en/llm-inference-engines/#Map-of-engines).

### vLLM: its place among other systems

The default choice. The widest list of models and hardware. The [Kubernetes](/en/kubernetes/) layers (llm-d, AIBrix, KServe), Ray Serve and Red Hat products support it first. In the MLPerf tests in April 2026, 20 participants ran on it ([Red Hat's count](https://www.redhat.com/en/blog/red-hat-ai-tops-mlperf-inference-v60-vllm-qwen3-vl-whisper-and-gpt-oss-120b)).

### SGLang: its place among other systems

Stronger on requests with a shared beginning: agents, long conversations, one system prompt for many users. Often ahead on very large models on dozens of GPUs. The same cluster layers support it, but usually later than vLLM.

## Test bench

- RTX 5070 Ti Laptop GPU, 12 GB; Ryzen 9 9955HX; 64 GB of RAM.
- Windows 11, WSL2, Docker Desktop.
- Images `vllm/vllm-openai:v0.34.0` and `lmsysorg/sglang:v0.5.23-runtime`.
- Prometheus 3.16, Grafana 13.3, Jaeger 2.23, nginx 1.30 — in the same Docker network.

The models are the Qwen3.5 family: they understand images, reason before answering, call functions, Apache 2.0 license.

| Model | Weights | On 12 GB |
|---|---|---|
| Qwen3.5-4B, FP8 | 7.5 GiB | the main model of the article |
| Qwen3.5-2B | 4.3 GiB | measurements under load |
| Qwen3.5-0.8B | 1.7 GiB | two copies at once, for routing |
| Qwen3.5-4B uncompressed | 8.7 GiB | does not start |
| Qwen3.5-9B, 4-bit | 8.5 GiB | does not start |

Absolute speeds on a laptop are several times lower than on servers. The settings, the errors and the content of the logs on big hardware are the same.

## Hardware and systems

Both servers need Linux, an accelerator with enough memory and fresh drivers.

### vLLM: hardware and systems

- **OS:** Linux only. Windows through WSL2 or Docker, Mac through the vllm-metal plugin.
- **Python:** 3.11–3.14.
- **NVIDIA:** from the T4 and RTX 20xx. Builds for CUDA 13 only. For old drivers — `VLLM_ENABLE_CUDA_COMPATIBILITY=1`.
- **AMD:** Instinct MI200, MI300, MI350, Radeon RX 7900 and RX 9000 (ROCm).
- **Intel:** Arc and server accelerators.
- **Through plugins:** Google TPU, AWS Trainium, Huawei Ascend, Apple Silicon.
- **Without a GPU:** works, for small models.

### SGLang: hardware and systems

- **OS:** Linux. Windows is not mentioned in the documentation; it works in Docker on top of WSL2.
- **Python:** 3.11 and newer. On 3.10 the installation does not fail but installs an old version of the server.
- **NVIDIA:** from the A100 to the newest server GPUs, RTX 30, 40 and 50, DGX Spark, Jetson. CUDA 13 only.
- **AMD:** Instinct MI300X, MI325X, MI350X, MI355X.
- **Intel:** Arc and Xeon.
- **Other:** Google TPU, Huawei Ascend, Apple Silicon, Moore Threads.
- **RTX 50 and installation through pip:** no model starts without manually setting the paths to the CUDA compiler ([open issue](https://github.com/sgl-project/sglang/issues/49209)). The Docker image does not have this problem.

## Installation

Two ways: a Python package or a Docker image. In production take an image with a version number, not `latest`: both programs are released every two weeks and regularly change their flags.

Models are downloaded from Hugging Face on the first launch into `~/.cache/huggingface`. This folder is mounted into the container so as not to download again.

### vLLM: installation

```bash
uv pip install vllm --torch-backend=auto
vllm download-kernels
```

The second command downloads ready-made kernels for the GPU. Without it the server compiles them on the first launch, which takes several minutes. Run it after every installation and update.

```bash
docker pull vllm/vllm-openai:v0.34.0
```

The image: 10 GB to download, 36 GB on disk.

### SGLang: installation

```bash
uv pip install --prerelease=allow sglang
```

Without `--prerelease=allow` old versions of `uv` silently install SGLang 0.5.9.

```bash
docker pull lmsysorg/sglang:v0.5.23-runtime
```

The `-runtime` image is lightweight: 14.5 GB to download, 50 GB on disk. The full one, `v0.5.23`, also contains build tools.

## First launch

The server starts with one command. vLLM listens on port 8000, SGLang on 30000. The requests to them are the same.

### vLLM: first launch

```bash
docker run --gpus all --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v vllm-cache:/root/.cache/vllm \
  vllm/vllm-openai:v0.34.0 \
  RedHatAI/Qwen3.5-4B-FP8-dynamic \
  --max-model-len 16384 --max-num-seqs 16
```

- `--ipc=host` — the server's processes exchange data through shared memory.
- The `vllm-cache` volume stores what was compiled on the first launch.
- Ready when you see the line `Application startup complete`. On the test bench, after 155 seconds.

What the server writes about memory:

```text
Model loading took 6.23 GiB memory and 11.814035 seconds
Available KV cache memory: 3.99 GiB
Graph capturing finished in 1 secs, took 0.14 GiB
GPU KV cache size: 34,492 tokens,
  Maximum concurrency for 16,384 tokens per request: 2.11x
```

The conversation memory holds 34,492 tokens — two conversations of maximum length.

The request:

```bash
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "RedHatAI/Qwen3.5-4B-FP8-dynamic",
    "messages": [{"role": "user",
      "content": "What is a KV cache in LLM inference? Answer in two sentences."}],
    "max_tokens": 400
  }'
```

The `content` field will contain the model's reasoning, not the answer:

```text
Thinking Process:

1.  **Analyze the Request:**
    *   Topic: KV cache in Large Language Model (LLM) inference.
    *   Constraint: Answer in exactly two sentences.
...
```

The server does not know where the model's reasoning ends until the `--reasoning-parser` flag is set (the [configuration section](#Configuration)).

The list of all API addresses is on the page `http://localhost:8000/docs`, with 24 operations:

![The /docs page of the vLLM server: the list of API addresses](/images/2026/11/vllm-sglang-swagger-vllm.png)

### SGLang: first launch

```bash
docker run --gpus all --ipc=host --shm-size 16g -p 30000:30000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  lmsysorg/sglang:v0.5.23-runtime \
  sglang serve RedHatAI/Qwen3.5-4B-FP8-dynamic \
  --host 0.0.0.0 --port 30000 \
  --context-length 16384 --mem-fraction-static 0.8
```

- `--host 0.0.0.0` is required in a container. By default the server listens on `127.0.0.1`: the log says "ready", but requests from outside do not get through.
- Ready when you see the line `The server is fired up and ready to roll!`. On the test bench, after 85–100 seconds.

What the server writes about memory:

```text
Load weight end. elapsed=2.19 s, quant=compressed-tensors,
  avail mem=4.44 GB, mem usage=6.28 GB.
Mamba Cache is allocated. max_mamba_cache_size: 20,
  conv_state size: 0.02GB, ssm_state size: 0.98GB
KV Cache is allocated. #tokens: 38949, K size: 0.59 GB, V size: 0.59 GB
max_total_num_tokens=38949, max_running_requests=4,
  context_len=16384, available_gpu_mem=0.37 GB
```

The conversation memory is 38,949 tokens, no more than four simultaneous requests.

The request and the answer are the same as in vLLM, only the port changes. The reasoning also lands in `content` until `--reasoning-parser` is set.

On the `/docs` page SGLang has 108 operations:

![The /docs page of the SGLang server: the list of API addresses](/images/2026/11/vllm-sglang-swagger-sglang.png)

Besides the API for clients there is server management here: cache reset, weight updates, adapter loading, profiling. All of this must be closed off from outsiders (the [network section](#Network-and-security)).

## GPU memory

GPU memory decides whether a model starts, how many conversations fit at once and how long they can be. The memory is divided like this:

1. **Model weights.** The size equals the size of the model files and does not change.
2. **Conversation memory** ([KV cache](/en/llm-inference-engines/#The-KV-cache)). For Qwen3.5-4B it is 32 KiB per token: a conversation of 16 thousand tokens takes 512 MiB. It grows with the number of users.
3. **State of hybrid models.** Qwen3.5 has ordinary attention in only 8 layers out of 32. The other 24 store a compressed state of constant size for a conversation. In the logs it is called the Mamba cache.
4. **Working buffers** for the computation of one step. On the test bench — about 1.6 GiB.
5. **Recorded computation graphs** (CUDA graphs): 0.1–0.5 GiB.
6. **Other programs.** On a laptop the desktop and the browser take 1.2 GiB, on a dedicated server — nothing.

The same model in two servers:

<div style="overflow-x:auto;margin:1.5rem 0" tabindex="0" role="region">
<svg viewBox="0 0 810 336" width="100%" style="min-width:620px;max-width:810px;height:auto;display:block;margin:0 auto" role="img" aria-label="Where GPU memory goes in vLLM and SGLang with the same model" xmlns="http://www.w3.org/2000/svg">
  <defs>
    <marker id="vs-memory-head" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L10 5 L0 10 z" fill="var(--color-fg-dim)"/>
    </marker>
  </defs>
  <text x="236" y="28" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="14" font-weight="500">vLLM 0.34.0</text>
  <text x="236" y="46" text-anchor="start" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12" font-weight="400">no memory flags</text>
  <text x="530" y="28" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="14" font-weight="500">SGLang 0.5.23</text>
  <text x="530" y="46" text-anchor="start" fill="var(--color-fg-dim)" font-family="var(--font-mono)" font-size="12" font-weight="400">--mem-fraction-static 0.8</text>
  <text x="16" y="80" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">other programs</text>
  <rect x="236" y="66" width="39" height="20" rx="3" fill="var(--color-border)" fill-opacity="0.9" stroke="var(--color-border)" stroke-width="1"/>
  <text x="283" y="80" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12" font-weight="400">1.22 GiB</text>
  <rect x="530" y="66" width="38" height="20" rx="3" fill="var(--color-border)" fill-opacity="0.9" stroke="var(--color-border)" stroke-width="1"/>
  <text x="576" y="80" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12" font-weight="400">1.19 GiB</text>
  <text x="16" y="114" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">model weights</text>
  <rect x="236" y="100" width="203" height="20" rx="3" fill="var(--color-accent)" fill-opacity="0.55" stroke="var(--color-border)" stroke-width="1"/>
  <text x="447" y="114" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12" font-weight="400">6.33 GiB</text>
  <rect x="530" y="100" width="201" height="20" rx="3" fill="var(--color-accent)" fill-opacity="0.55" stroke="var(--color-border)" stroke-width="1"/>
  <text x="739" y="114" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12" font-weight="400">6.28 GiB</text>
  <text x="16" y="148" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">dialogue memory</text>
  <rect x="236" y="134" width="45" height="20" rx="3" fill="var(--color-accent)" fill-opacity="0.3" stroke="var(--color-border)" stroke-width="1"/>
  <text x="289" y="148" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12" font-weight="400">1.41 GiB</text>
  <rect x="530" y="134" width="38" height="20" rx="3" fill="var(--color-accent)" fill-opacity="0.3" stroke="var(--color-border)" stroke-width="1"/>
  <text x="576" y="148" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12" font-weight="400">1.18 GiB</text>
  <text x="16" y="182" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">hybrid model state</text>
  <text x="236" y="182" text-anchor="start" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12" font-weight="400">part of the dialogue memory</text>
  <rect x="530" y="168" width="32" height="20" rx="3" fill="var(--color-accent)" fill-opacity="0.3" stroke="var(--color-border)" stroke-width="1"/>
  <text x="570" y="182" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12" font-weight="400">1.00 GiB</text>
  <text x="16" y="216" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">working buffers and overhead</text>
  <rect x="236" y="202" width="51" height="20" rx="3" fill="var(--color-link)" fill-opacity="0.45" stroke="var(--color-border)" stroke-width="1"/>
  <text x="295" y="216" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12" font-weight="400">1.59 GiB</text>
  <rect x="530" y="202" width="44" height="20" rx="3" fill="var(--color-link)" fill-opacity="0.45" stroke="var(--color-border)" stroke-width="1"/>
  <text x="582" y="216" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12" font-weight="400">1.37 GiB</text>
  <text x="16" y="250" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">captured graphs</text>
  <rect x="236" y="236" width="4" height="20" rx="3" fill="var(--color-link)" fill-opacity="0.45" stroke="var(--color-border)" stroke-width="1"/>
  <text x="248" y="250" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12" font-weight="400">0.14 GiB</text>
  <rect x="530" y="236" width="17" height="20" rx="3" fill="var(--color-link)" fill-opacity="0.45" stroke="var(--color-border)" stroke-width="1"/>
  <text x="555" y="250" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12" font-weight="400">0.52 GiB</text>
  <text x="16" y="284" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">left free</text>
  <rect x="236" y="270" width="39" height="20" rx="3" fill="var(--color-border)" fill-opacity="0.45" stroke="var(--color-border)" stroke-width="1"/>
  <text x="283" y="284" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12" font-weight="400">1.22 GiB</text>
  <rect x="530" y="270" width="12" height="20" rx="3" fill="var(--color-border)" fill-opacity="0.45" stroke="var(--color-border)" stroke-width="1"/>
  <text x="550" y="284" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12" font-weight="400">0.37 GiB</text>
  <text x="16" y="322" text-anchor="start" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">Each column adds up to 11.91 GiB, the whole GPU. The model is the same: Qwen3.5-4B in FP8.</text>
</svg>
</div>

The order in which the servers divide memory at startup is different:

<div style="overflow-x:auto;margin:1.5rem 0" tabindex="0" role="region">
<svg viewBox="0 0 800 306" width="100%" style="min-width:620px;max-width:800px;height:auto;display:block;margin:0 auto" role="img" aria-label="How vLLM and SGLang allocate GPU memory at startup" xmlns="http://www.w3.org/2000/svg">
  <defs>
    <marker id="vs-memflow-head" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L10 5 L0 10 z" fill="var(--color-fg-dim)"/>
    </marker>
  </defs>
  <text x="200" y="26" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="14" font-weight="500">vLLM</text>
  <rect x="30" y="44" width="340" height="62" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-border)" stroke-width="1.5"/>
  <text x="200" y="68" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="14" font-weight="500">1. Load the weights</text>
  <text x="200" y="88" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">their size is known in advance</text>
  <rect x="30" y="136" width="340" height="62" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-border)" stroke-width="1.5"/>
  <text x="200" y="160" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="14" font-weight="500">2. Trial requests</text>
  <text x="200" y="180" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">measure buffers and graphs</text>
  <path d="M200 106 L200 136" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" marker-end="url(#vs-memflow-head)"/>
  <rect x="30" y="228" width="340" height="62" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-accent)" stroke-width="1.5"/>
  <text x="200" y="252" text-anchor="middle" fill="var(--color-accent)" font-family="var(--font-sans)" font-size="14" font-weight="500">3. The rest goes to dialogue memory</text>
  <text x="200" y="272" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">the server works it out</text>
  <path d="M200 198 L200 228" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" marker-end="url(#vs-memflow-head)"/>
  <text x="600" y="26" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="14" font-weight="500">SGLang</text>
  <rect x="430" y="44" width="340" height="62" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-accent)" stroke-width="1.5"/>
  <text x="600" y="68" text-anchor="middle" fill="var(--color-accent)" font-family="var(--font-sans)" font-size="14" font-weight="500">1. The share is set up front</text>
  <text x="600" y="88" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">--mem-fraction-static</text>
  <rect x="430" y="136" width="340" height="62" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-border)" stroke-width="1.5"/>
  <text x="600" y="160" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="14" font-weight="500">2. Inside the share</text>
  <text x="600" y="180" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">weights, dialogue memory, state</text>
  <path d="M600 106 L600 136" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" marker-end="url(#vs-memflow-head)"/>
  <rect x="430" y="228" width="340" height="62" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-border)" stroke-width="1.5"/>
  <text x="600" y="252" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="14" font-weight="500">3. Outside the share</text>
  <text x="600" y="272" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">buffers and graphs get what is left</text>
  <path d="M600 198 L600 228" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" marker-end="url(#vs-memflow-head)"/>
</svg>
</div>

What follows from this:

- **Besides the weights, another 3–4 GiB is needed.** A model whose files are almost as large as the GPU memory will not start.
- **Conversation memory is what is left** after the weights, buffers and graphs.
- **There are three knobs:** conversation length, number of simultaneous requests, the share of memory for the server.

### vLLM: GPU memory

Before version 0.34 the server grabbed 92% of GPU memory in advance. Since 0.34 it loads the weights, runs trial requests, measures the buffers and graphs and gives everything that is left to conversation memory. It is now better not to set the `--gpu-memory-utilization` flag: any value will only reduce the conversation memory. The old advice "it does not start — lower `--gpu-memory-utilization`" is out of date.

```text
Actual usage is 6.33 GiB for consumed memory (weights + non-torch),
  1.59 GiB for peak activation, and 0.14 GiB for CUDAGraph memory.
Memory after warmup: 4.17 GiB free,
  1.59 GiB kept for transient activations;
  80 KV cache blocks (1.41 GiB) fit within the requested 11.91 GiB.
```

| Flag | What it does |
|---|---|
| `--max-model-len` | maximum conversation length in tokens. By default taken from the model: for Qwen3.5 it is 262,144 |
| `--max-num-seqs` | number of simultaneous requests. 256 by default, 1024 on large server cards |
| `--gpu-memory-utilization` | share of GPU memory for the server. Needed when several programs share the card |
| `--kv-cache-memory-bytes` | exact size of the conversation memory |
| `--kv-cache-dtype fp8` | compressed conversation memory: twice as many conversations |

When the conversation memory runs out, new requests wait in the queue. If it is not enough for those already started, the server preempts the most recently accepted request and later reads its prompt again. The user sees a delay, not an error.

Two copies of the server on one GPU: give each `--gpu-memory-utilization 0.4`, otherwise the first will take everything.

### SGLang: GPU memory

The `--mem-fraction-static` flag sets the share of GPU memory for the weights and conversation memory together. The rest goes to buffers and graphs. Without the flag the server calculates the share itself; on a card with 12 GB it came out to 0.63.

How to choose it, according to the documentation:

- at startup the server writes the amount of free memory left — `available_gpu_mem`;
- on large cards 5–8 GB left over is normal;
- much more left over — the share can be raised;
- the server crashes from lack of memory under load — the share needs to be lowered.

On 12 GB the default value is too cautious. Qwen3.5-4B in FP8 does not start with it:

```text
RuntimeError: Hybrid (mamba/linear-attention) state cache is too small
  to serve any requests. max_mamba_cache_size=3, mamba_ratio=5,
  resulting max_num_reqs=0. Try: (1) reduce --max-running-requests,
  (2) increase --mem-fraction-static, or (3) use GPUs with more memory.
```

With `--mem-fraction-static 0.8` it starts: four simultaneous requests.

| Flag | What it does |
|---|---|
| `--mem-fraction-static` | share of GPU memory for the weights and conversation memory |
| `--context-length` | maximum conversation length in tokens |
| `--max-running-requests` | number of simultaneous requests |
| `--max-total-tokens` | exact size of the conversation memory in tokens |
| `--chunked-prefill-size` | size of the chunks a long prompt is read in. Smaller — smaller buffers |
| `--kv-cache-dtype fp8_e4m3` | compressed conversation memory. For accuracy a scales file is needed: `--quantization-param-path` |
| `--mamba-full-memory-ratio`, `--max-mamba-cache-size` | memory for the state of hybrid models |

When the conversation memory runs out, SGLang takes one of the running requests off and returns it to the queue. In the log: `KV cache pool is full. Retract requests.` If this happens often, raise `--schedule-conservativeness`.

### What does not fit into 12 GB

| Model | Weights in memory | vLLM | SGLang |
|---|---|---|---|
| Qwen3.5-4B uncompressed | 8.6 GiB | no room for conversation memory | 2.57 GB short for the state |
| Qwen3.5-9B, 4-bit | 8.4–9.5 GiB | the same | the same |

Compression to 4 bits shrinks only the main layers. The image-processing block and the token tables stay as they were, so a model with 9.7 billion parameters takes more than 8 GiB.

The vLLM failure:

```text
Memory after warmup: 0.91 GiB free, 1.59 GiB kept for transient activations;
  0 KV cache blocks (0.0 GiB) fit within the requested 11.91 GiB.
ValueError: No available memory for the cache blocks. [...]
  decrease `max_model_len` or `max_num_seqs`,
  or the model needs a device with more memory.
```

The SGLang failure:

```text
Load weight end. avail mem=1.37 GB, mem usage=9.35 GB.
RuntimeError: Not enough GPU memory for hybrid (mamba/linear-attention)
  state cache. Computed max_mamba_cache_size=-27
  (total_rest_memory=-2.57 GB, mamba_cache_per_req=49.12 MB).
```

vLLM reports the lack of memory after 2–4 minutes, at the end of the warmup. SGLang — after 20–30 seconds, right after the weights are loaded.

With a hybrid model vLLM can also give one more error at startup: the default `--max-num-seqs` value is too large.

```text
ValueError: max_num_seqs (256) exceeds available Mamba cache blocks (42).
  Please lower max_num_seqs to at most 42 or increase gpu_memory_utilization.
```

## Model formats and quantization

Both servers work with the safetensors format from Hugging Face. GGUF, familiar from llama.cpp and Ollama, is not their format. What quantization is — in [the section on it](/en/llm-inference-engines/#Quantization).

Both servers recognize a model compressed in advance by themselves, without flags.

### vLLM: formats and quantization

- **Sources:** Hugging Face, a folder on disk, ModelScope (`VLLM_USE_MODELSCOPE=True`), object storage (`vllm serve s3://bucket/model --load-format runai_streamer`).
- **Ready-made compressed models:** FP8, AWQ, GPTQ, INT8, INT4, NVFP4, MXFP4, compressed-tensors, NVIDIA ModelOpt, AMD Quark.
- **Compression at load time:** `--quantization fp8_per_tensor`.
- **GGUF:** a separate plugin `vllm-gguf-plugin`, experimental.
- **bitsandbytes:** a separate plugin `vllm-bnb-plugin`.
- **Compress your own model:** [LLM Compressor](https://github.com/vllm-project/llm-compressor).
- **LoRA adapters:** `--enable-lora --lora-modules sql=path`. The client picks the adapter in the `model` field; requests to different adapters go in one batch.

### SGLang: formats and quantization

- **Sources:** Hugging Face, a folder on disk, ModelScope (`SGLANG_USE_MODELSCOPE=true`), addresses `s3://`, `gs://`, `az://`.
- **Ready-made compressed models:** the same main formats. The `--quantization` flag cannot be added to an already compressed model.
- **Compression at load time:** `--quantization fp8`. Compression through torchao was removed in version 0.5.23.
- **GGUF:** a `gguf` load format exists, the documentation does not give a list of supported architectures.
- **LoRA adapters:** `--enable-lora --lora-paths sql=path`. They are loaded and unloaded on the fly: `/load_lora_adapter`, `/unload_lora_adapter`. In the request: `"model": "base-model:adapter"`.

Speed on the test bench, one user:

| Model | vLLM | SGLang |
|---|---|---|
| Qwen3.5-4B, FP8 | 84 tok/s | 56–60 tok/s |
| Qwen3.5-2B uncompressed | 110 tok/s | 109 tok/s |

FP8 computation for this model is faster in vLLM. On the uncompressed model there is no difference.

## Configuration

Settings are given by flags, a YAML file or environment variables. If a value is set both in a flag and in the file, the flag wins.

Reasoning models and function calling need two parsers, specific to each model family:

- **reasoning parser** separates the reasoning from the answer;
- **tool call parser** turns the model's text into a function call with arguments.

### vLLM: configuration

```yaml title="vllm/config.yaml"
# vLLM 0.34 — all server settings in one file
model: RedHatAI/Qwen3.5-4B-FP8-dynamic
# The name clients see the model under
served-model-name: qwen3.5-4b
host: 0.0.0.0
port: 8000

# Memory: the longest conversation and the number of simultaneous requests
max-model-len: 16384
max-num-seqs: 16

# Separate the model's reasoning from the answer
reasoning-parser: qwen3
# Turn the model's text into function calls
enable-auto-tool-choice: true
tool-call-parser: qwen3_xml

# Show in the response how many prompt tokens came from the cache
enable-prompt-tokens-details: true
# Return the request ID in the X-Request-Id header
enable-request-id-headers: true
```

```bash
docker run --gpus all --ipc=host -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v vllm-cache:/root/.cache/vllm \
  -v ./vllm:/lab/vllm \
  -e VLLM_API_KEY=sk-lab-123 \
  vllm/vllm-openai:v0.34.0 \
  --config /lab/vllm/config.yaml
```

- **Help:** `vllm serve --help=all` — everything, `--help=max-num-seqs` — a single flag.
- **Nested settings** — through a dot: `--logging-config.formatter json`.
- **Short notation for numbers:** `--max-model-len 16K` is 16,384. Capital `K` is 1024, lowercase `k` is 1000.
- **Generation parameters from the model.** The server applies `generation_config.json` from the model folder. Turned off with the `--generation-config vllm` flag.
- **Environment variables:** `HF_TOKEN`, `VLLM_API_KEY`, `VLLM_LOGGING_LEVEL`, `VLLM_NO_USAGE_STATS=1` (opt out of telemetry).
- **Kubernetes:** do not name the service `vllm`. Kubernetes will create variables with the `VLLM_` prefix, and the server will take them for its own settings.

### SGLang: configuration

```yaml title="sglang/config.yaml"
# SGLang 0.5.23 — server settings in one file.
# The model is set on the command line: sglang serve <model> --config config.yaml
# The name clients see the model under
served-model-name: qwen3.5-4b
host: 0.0.0.0
port: 30000

# Memory: the longest conversation and the share of GPU memory for weights and conversation cache
context-length: 16384
mem-fraction-static: 0.8

# Separate the model's reasoning from the answer
reasoning-parser: qwen3
# Turn the model's text into function calls
tool-call-parser: qwen3_coder

# Serve Prometheus metrics at /metrics
enable-metrics: true
# Show in the response how many prompt tokens came from the cache
enable-cache-report: true
```

```bash
docker run --gpus all --ipc=host --shm-size 16g -p 30000:30000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v ./sglang:/lab/sglang \
  lmsysorg/sglang:v0.5.23-runtime \
  sglang serve RedHatAI/Qwen3.5-4B-FP8-dynamic \
  --config /lab/sglang/config.yaml --api-key sk-lab-123
```

- **The model goes on the command line only.** With `model-path` in the file and no model in the command, `sglang serve` fails: `Error: --model-path is required`.
- **The access key goes by flag only:** `--api-key`.
- **The launch command:** `sglang serve`. The old `python -m sglang.launch_server` still works and appears in most guides.
- **Metrics are off** without `--enable-metrics`.
- **The number of cached tokens in the response** — only with `--enable-cache-report`.
- **The parser names are different:** for the same model it is `qwen3_xml` in vLLM and `qwen3_coder` here.
- **The help is more reliable than the documentation:** 91 flags are not described in the documentation table. Look at `sglang serve --help`.
- **Environment variables:** `HF_TOKEN`, `SGLANG_CACHE_DIR` (the folder of compiled kernels, mount it as a volume), `SGLANG_LOGGING_CONFIG_PATH` (your own log format).

## API

Both servers replicate the OpenAI API. The `openai` library works with them unchanged: only the address changes. The examples below were run against both servers without edits.

```bash
# vLLM
export BASE_URL=http://localhost:8000/v1 API_KEY=sk-lab-123
# SGLang
export BASE_URL=http://localhost:30000/v1 API_KEY=sk-lab-123
```

**An ordinary request.** The reasoning comes separately from the answer:

```python title="chat.py"
import os

from openai import OpenAI

# The same code works with both servers: only the address changes.
# vLLM: http://localhost:8000/v1, SGLang: http://localhost:30000/v1
client = OpenAI(
    base_url=os.environ["BASE_URL"],
    api_key=os.environ["API_KEY"],
)

response = client.chat.completions.create(
    model="qwen3.5-4b",
    messages=[
        {"role": "system", "content": "You are a concise assistant."},
        {
            "role": "user",
            "content": "What is a KV cache in LLM inference? Answer in two sentences.",
        },
    ],
    max_tokens=2048,
)

message = response.choices[0].message
# The reasoning comes in a separate field: in vLLM it is called reasoning,
# in SGLang — reasoning_content
reasoning = getattr(message, "reasoning", None) or getattr(message, "reasoning_content", None)
print("REASONING:", (reasoning or "")[:200].replace("\n", " "), "...")
print("ANSWER:", message.content.strip())
print("USAGE:", response.usage.prompt_tokens, "+", response.usage.completion_tokens)
```

```text
REASONING: Thinking Process:  1.  **Analyze the Request:**     *   Topic: KV cache ...
ANSWER: A KV cache stores the attention key-value pairs for all previous tokens,
  allowing the model to retrieve context without re-computing layer activations.
  This reuse significantly reduces the computational cost and latency of
  autoregressive inference, particularly when generating long sequences of text.
USAGE: 36 + 1565
```

An answer of two sentences cost 1565 tokens: almost all of it went on reasoning. With a small `max_tokens` a reasoning model never reaches the answer, and `content` comes back empty.

**Streaming, turning off reasoning, measuring TTFT:**

```python title="stream.py"
import os
import time

from openai import OpenAI

client = OpenAI(base_url=os.environ["BASE_URL"], api_key=os.environ["API_KEY"])

started = time.perf_counter()
first_token_at = None
tokens = 0

stream = client.chat.completions.create(
    model="qwen3.5-4b",
    messages=[
        {
            "role": "user",
            "content": "Explain in three sentences why one GPU can serve many LLM requests at once.",
        }
    ],
    max_tokens=300,
    stream=True,
    # Ask the server to send the token counters at the end of the stream
    stream_options={"include_usage": True},
    # Turn off reasoning: the parameter goes into the model's chat template
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)

for chunk in stream:
    if chunk.usage:
        tokens = chunk.usage.completion_tokens
    if not chunk.choices:
        continue
    piece = chunk.choices[0].delta.content
    if piece:
        if first_token_at is None:
            first_token_at = time.perf_counter()
        print(piece, end="", flush=True)

total = time.perf_counter() - started
ttft = first_token_at - started
speed = tokens / (total - ttft)
print(f"\n\nTTFT: {ttft * 1000:.0f} ms, tokens: {tokens}, speed: {speed:.0f} tokens/s")
```

```text
vLLM:    TTFT: 71 ms, tokens: 93, speed: 84 tokens/s
SGLang:  TTFT: 82 ms, tokens: 93, speed: 56 tokens/s
```

**Function calling.** The model names the function and the arguments, your code calls it:

```python title="tools.py"
import json
import os

from openai import OpenAI

client = OpenAI(base_url=os.environ["BASE_URL"], api_key=os.environ["API_KEY"])

# The description of the function the model is allowed to call
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_gpu_memory",
            "description": "Returns used and total memory of a GPU in megabytes.",
            "parameters": {
                "type": "object",
                "properties": {
                    "gpu_index": {
                        "type": "integer",
                        "description": "Index of the GPU, starting from 0",
                    },
                },
                "required": ["gpu_index"],
            },
        },
    }
]

messages = [{"role": "user", "content": "How much memory is free on GPU 0?"}]

# 1. The model decides a function is needed and returns its name and arguments
first = client.chat.completions.create(
    model="qwen3.5-4b",
    messages=messages,
    tools=tools,
    max_tokens=1024,
)
call = first.choices[0].message.tool_calls[0]
print("CALL:", call.function.name, call.function.arguments)

# 2. The function is run by our code, not by the server; here the result is filled in by hand
result = {"used_mb": 9896, "total_mb": 12227}
messages.append(first.choices[0].message)
messages.append({"role": "tool", "tool_call_id": call.id, "content": json.dumps(result)})

# 3. The model gets the result and answers the person
second = client.chat.completions.create(
    model="qwen3.5-4b",
    messages=messages,
    tools=tools,
    max_tokens=1024,
)
print("ANSWER:", second.choices[0].message.content.strip())
```

```text
CALL: get_gpu_memory {"gpu_index": 0}
ANSWER: GPU 0 has approximately **2.3 GB** free memory.
```

**An answer by schema.** The server forbids the model tokens that violate the schema, and the answer always comes out as valid JSON:

```python title="structured.py"
import json
import os

from openai import OpenAI

client = OpenAI(base_url=os.environ["BASE_URL"], api_key=os.environ["API_KEY"])

# The schema the answer must conform to
schema = {
    "type": "object",
    "properties": {
        "component": {"type": "string", "enum": ["memory", "network", "model", "other"]},
        "severity": {"type": "string", "enum": ["low", "medium", "high"]},
        "summary": {"type": "string"},
    },
    "required": ["component", "severity", "summary"],
    "additionalProperties": False,
}

log_line = "RuntimeError: Not enough GPU memory for hybrid (mamba/linear-attention) state cache."

response = client.chat.completions.create(
    model="qwen3.5-4b",
    messages=[
        {
            "role": "system",
            "content": "Classify the log line of an inference server. Keep the summary under 20 words.",
        },
        {"role": "user", "content": log_line},
    ],
    # The server will not let the model pick a token that violates the schema
    response_format={
        "type": "json_schema",
        "json_schema": {"name": "incident", "schema": schema, "strict": True},
    },
    max_tokens=300,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)

incident = json.loads(response.choices[0].message.content)
print(incident)
```

```text
{'component': 'memory', 'severity': 'high',
 'summary': 'GPU memory insufficient for hybrid attention cache.'}
```

**An image as input:**

```python title="vision.py"
import base64
import os

from openai import OpenAI

client = OpenAI(base_url=os.environ["BASE_URL"], api_key=os.environ["API_KEY"])

# The image goes in the request as a base64 string
with open("dashboard.png", "rb") as file:
    image = base64.b64encode(file.read()).decode()

response = client.chat.completions.create(
    model="qwen3.5-4b",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image}"}},
                {"type": "text", "text": "What does this picture show? Answer in two sentences."},
            ],
        }
    ],
    max_tokens=200,
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)

print(response.choices[0].message.content.strip())
print("prompt tokens:", response.usage.prompt_tokens)
```

A 1200×630 image takes 785 prompt tokens in both servers.

### vLLM: what there is besides the OpenAI API

- **Responses API:** `/v1/responses`.
- **Anthropic Messages API:** `/v1/messages`. This is how Claude Code connects ([guide](https://docs.vllm.ai/en/latest/serving/integrations/claude_code/)).
- **Embeddings, reranking, speech recognition.**
- **Tokenizer:** `/tokenize`, `/detokenize`.
- **gRPC:** `vllm serve --grpc`.
- **The reasoning field:** `reasoning`.
- **The `X-Request-Id` header:** the server accepts the request ID from the client and returns it in the response.
- **The key goes only in `Authorization: Bearer`.** The `x-api-key` header sent by Anthropic clients is not accepted: the answer is 401.

### SGLang: what there is besides the OpenAI API

- **Responses API and Anthropic Messages API** — at the same addresses. `x-api-key` is not accepted here either.
- **Ollama API:** `/api/chat`, `/api/generate`, `/api/tags`.
- **Embeddings, reranking, speech recognition, tokenizer.**
- **Running without HTTP:** `sgl.Engine(model_path=...)` in Python code. vLLM has the `LLM` class for this.
- **The reasoning field:** `reasoning_content`. This is the only place where the client needs to know which server it is talking to.
- **The native API** `/generate` — raw text without a chat template:

```bash
curl http://localhost:30000/generate \
  -H "Authorization: Bearer sk-lab-123" -H "Content-Type: application/json" \
  -d '{"text": "The capital of Bulgaria is",
       "sampling_params": {"max_new_tokens": 8, "temperature": 0}}'
```

```json
{
  "text": " Sofia. Sofia is the largest city in",
  "meta_info": {
    "finish_reason": {"type": "length", "length": 8},
    "prompt_tokens": 5,
    "completion_tokens": 8,
    "cached_tokens": 0
  }
}
```

The response is shortened: the real one has about twenty more service fields.

## Sessions and state

Neither server stores conversations. The client sends the whole history in every request.

So that the history is not re-read on every turn, the [prefix cache](/en/llm-inference-engines/#Prefix-caching) works: after answering, the server keeps the conversation's memory and takes the matching beginning of the next request from it. The cache is on by default and is not tied to a user: a match is looked for by text.

A conversation of four turns with a system prompt of 2000 tokens:

```python title="cache.py"
import os
import time

from openai import OpenAI

client = OpenAI(base_url=os.environ["BASE_URL"], api_key=os.environ["API_KEY"])

# A long system prompt: about 2000 tokens of rules, the same for all requests
rules = "\n".join(
    f"Rule {n}: when the user asks about topic {n}, answer politely and mention the number {n * 7}."
    for n in range(1, 81)
)
messages = [{"role": "system", "content": "You are a support assistant.\n" + rules}]

questions = [
    "What does rule 5 say?",
    "And rule 12?",
    "Which rule mentions the number 140?",
    "Thanks. Summarize our conversation in one sentence.",
]

for turn, question in enumerate(questions, start=1):
    messages.append({"role": "user", "content": question})

    started = time.perf_counter()
    first_token_at = None
    answer = ""
    usage = None
    stream = client.chat.completions.create(
        model="qwen3.5-4b",
        messages=messages,
        max_tokens=120,
        stream=True,
        stream_options={"include_usage": True},
        extra_body={"chat_template_kwargs": {"enable_thinking": False}},
    )
    for chunk in stream:
        if chunk.usage:
            usage = chunk.usage
        if chunk.choices and chunk.choices[0].delta.content:
            if first_token_at is None:
                first_token_at = time.perf_counter()
            answer += chunk.choices[0].delta.content

    # The server does not remember the conversation: the client sends the history every time
    messages.append({"role": "assistant", "content": answer})

    details = usage.prompt_tokens_details
    cached = details.cached_tokens if details else 0
    ttft = (first_token_at - started) * 1000
    prompt = usage.prompt_tokens
    print(f"turn {turn}: prompt {prompt} tokens, from cache {cached}, TTFT {ttft:.0f} ms")
```

vLLM:

```text
turn 1: prompt 2157 tokens, from cache 0, TTFT 339 ms
turn 2: prompt 2202 tokens, from cache 1728, TTFT 84 ms
turn 3: prompt 2253 tokens, from cache 1728, TTFT 155 ms
turn 4: prompt 2320 tokens, from cache 1728, TTFT 103 ms
```

SGLang:

```text
turn 1: prompt 2157 tokens, from cache 0, TTFT 341 ms
turn 2: prompt 2210 tokens, from cache 2112, TTFT 61 ms
turn 3: prompt 2266 tokens, from cache 2176, TTFT 55 ms
turn 4: prompt 2323 tokens, from cache 2240, TTFT 48 ms
```

From the second turn both answer several times faster:

<div style="overflow-x:auto;margin:1.5rem 0" tabindex="0" role="region">
<svg viewBox="0 0 780 232" width="100%" style="min-width:620px;max-width:780px;height:auto;display:block;margin:0 auto" role="img" aria-label="Time to first token by dialogue turn in vLLM and SGLang" xmlns="http://www.w3.org/2000/svg">
  <defs>
    <marker id="vs-turns-head" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L10 5 L0 10 z" fill="var(--color-fg-dim)"/>
    </marker>
  </defs>
  <path d="M80 200 L730 200" stroke="var(--color-border)" stroke-width="1.5"/>
  <rect x="110" y="51" width="54" height="149" rx="2" fill="var(--color-accent)" fill-opacity="0.6" stroke="var(--color-border)" stroke-width="1"/>
  <text x="137" y="45" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12" font-weight="400">339</text>
  <rect x="170" y="50" width="54" height="150" rx="2" fill="var(--color-link)" fill-opacity="0.55" stroke="var(--color-border)" stroke-width="1"/>
  <text x="197" y="44" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12" font-weight="400">341</text>
  <text x="167" y="220" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">turn 1</text>
  <rect x="270" y="163" width="54" height="37" rx="2" fill="var(--color-accent)" fill-opacity="0.6" stroke="var(--color-border)" stroke-width="1"/>
  <text x="297" y="157" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12" font-weight="400">84</text>
  <rect x="330" y="173" width="54" height="27" rx="2" fill="var(--color-link)" fill-opacity="0.55" stroke="var(--color-border)" stroke-width="1"/>
  <text x="357" y="167" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12" font-weight="400">61</text>
  <text x="327" y="220" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">turn 2</text>
  <rect x="430" y="132" width="54" height="68" rx="2" fill="var(--color-accent)" fill-opacity="0.6" stroke="var(--color-border)" stroke-width="1"/>
  <text x="457" y="126" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12" font-weight="400">155</text>
  <rect x="490" y="176" width="54" height="24" rx="2" fill="var(--color-link)" fill-opacity="0.55" stroke="var(--color-border)" stroke-width="1"/>
  <text x="517" y="170" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12" font-weight="400">55</text>
  <text x="487" y="220" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">turn 3</text>
  <rect x="590" y="155" width="54" height="45" rx="2" fill="var(--color-accent)" fill-opacity="0.6" stroke="var(--color-border)" stroke-width="1"/>
  <text x="617" y="149" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12" font-weight="400">103</text>
  <rect x="650" y="179" width="54" height="21" rx="2" fill="var(--color-link)" fill-opacity="0.55" stroke="var(--color-border)" stroke-width="1"/>
  <text x="677" y="173" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12" font-weight="400">48</text>
  <text x="647" y="220" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">turn 4</text>
  <rect x="110" y="14" width="14" height="14" rx="2" fill="var(--color-accent)" fill-opacity="0.6" stroke="var(--color-border)"/>
  <text x="132" y="26" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12.5" font-weight="400">vLLM</text>
  <rect x="200" y="14" width="14" height="14" rx="2" fill="var(--color-link)" fill-opacity="0.55" stroke="var(--color-border)"/>
  <text x="222" y="26" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12.5" font-weight="400">SGLang</text>
  <text x="730" y="26" text-anchor="end" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">time to first token, ms</text>
  <path d="M0 0 L0 0" fill="none" stroke="none"/>
</svg>
</div>

SGLang takes almost everything from the cache, vLLM takes 1728 tokens on every turn:

<div style="overflow-x:auto;margin:1.5rem 0" tabindex="0" role="region">
<svg viewBox="0 0 780 252" width="100%" style="min-width:620px;max-width:780px;height:auto;display:block;margin:0 auto" role="img" aria-label="The second turn of a dialogue: how many prompt tokens each server took from the cache" xmlns="http://www.w3.org/2000/svg">
  <defs>
    <marker id="vs-prefix-head" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L10 5 L0 10 z" fill="var(--color-fg-dim)"/>
    </marker>
  </defs>
  <text x="20" y="30" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="14" font-weight="500">vLLM</text>
  <text x="80" y="30" text-anchor="start" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">the cache is kept in blocks; for this model a block is 576 tokens</text>
  <rect x="20" y="42" width="194" height="46" fill="var(--color-accent)" fill-opacity="0.32" stroke="var(--color-border)" stroke-width="1"/>
  <text x="117" y="69" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12.5" font-weight="400">576</text>
  <rect x="214" y="42" width="194" height="46" fill="var(--color-accent)" fill-opacity="0.32" stroke="var(--color-border)" stroke-width="1"/>
  <text x="311" y="69" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12.5" font-weight="400">576</text>
  <rect x="407" y="42" width="194" height="46" fill="var(--color-accent)" fill-opacity="0.32" stroke="var(--color-border)" stroke-width="1"/>
  <text x="504" y="69" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12.5" font-weight="400">576</text>
  <rect x="601" y="42" width="159" height="46" fill="var(--color-link)" fill-opacity="0.3" stroke="var(--color-border)" stroke-width="1"/>
  <text x="680.5" y="69" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12.5" font-weight="400">474</text>
  <text x="20" y="108" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">from cache: 1728 of 2202 tokens</text>
  <text x="760" y="108" text-anchor="end" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">computed again: 474</text>
  <text x="20" y="158" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="14" font-weight="500">SGLang</text>
  <text x="96" y="158" text-anchor="start" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">the cache is a tree of shared beginnings, matched almost to the token</text>
  <rect x="20" y="170" width="707" height="46" fill="var(--color-accent)" fill-opacity="0.32" stroke="var(--color-border)" stroke-width="1"/>
  <text x="373.5" y="197" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12.5" font-weight="400">2112</text>
  <rect x="727" y="170" width="33" height="46" fill="var(--color-link)" fill-opacity="0.3" stroke="var(--color-border)" stroke-width="1"/>
  <text x="20" y="236" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">from cache: 2112 of 2210 tokens</text>
  <text x="760" y="236" text-anchor="end" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">computed again: 98</text>
</svg>
</div>

### vLLM: cache and state

**How it works.** The conversation memory is cut into blocks. The server computes a hash of every filled block together with everything before it, and compares a new request with the cache block by block. An unfilled block does not go into the cache.

**The block size** is 64 tokens (16 before version 0.34). For hybrid models it is larger: for Qwen3.5 it is 576 tokens. Hence the 1728 in the example: that is three blocks, and the conversation had not grown to a fourth.

**The consequence.** A shared beginning shorter than a block is not cached. With a shared system prompt of 450 tokens, vLLM with Qwen3.5 showed a cache hit of 4%, SGLang on the same load 87%. For ordinary models the block is nine times smaller, and there is no such gap.

**How to see the cache at work:**

- in the response: `usage.prompt_tokens_details.cached_tokens` (the `--enable-prompt-tokens-details` flag);
- in the metrics: `vllm:prefix_cache_hits_total`, `vllm:prefix_cache_queries_total`;
- in the log: `Prefix cache hit rate`, every ten seconds.

**Client isolation.** From the response time you can tell whether someone has sent the same beginning of a request. If clients must know nothing about each other, add a `cache_salt` field to the requests, a different one for each client. Requests with different salts do not share blocks.

**Storing responses on the server** (the `previous_response_id` parameter in the Responses API) is off:

```text
404 invalid_request_error, param: response_id
Response with id 'resp_123' not found.
```

It is turned on by the `VLLM_ENABLE_RESPONSES_API_STORE=1` variable. The responses live in the process memory, are never deleted and disappear on restart. Not suitable for production: the history is stored by the client or by a layer in front of the server.

### SGLang: cache and state

**How it works.** The cache is a tree of shared beginnings (a radix tree). For a new request the longest match is looked up. The match is more precise than with blocks: in the example 2112 tokens out of 2210, and the number grows with every turn. When memory runs out, the branches that have not been touched for the longest time are deleted (`--radix-eviction-policy`).

**How to see the cache at work:**

- in the response: the same `cached_tokens` field (the `--enable-cache-report` flag);
- in the metrics: `sglang:cache_hit_rate`, `sglang:cached_tokens_total`;
- in the log, in the line of every prompt that was read:

```text
Prefill batch, #new-seq: 1, #new-token: 69, #cached-token: 448,
  full token usage: 0.02, mamba usage: 0.80, #running-req: 3, #queue-req: 1
```

Out of 517 prompt tokens, 448 came from the cache.

**Protecting a conversation's cache:** the `--enable-session-radix-cache` flag and the `session_id` field in the request. The branches of this conversation are deleted last. The history is still sent by the client.

**Sessions on the server.** Through the native API you can open a session (`/open_session`) and send only the new text: the server appends it to the stored context. It was made for reinforcement learning and agents, and works only through `/generate`. Not run on the test bench.

**Storing responses on the server** is also off:

```text
400 invalid_request_error, param: previous_response_id
Response store is disabled. Stateful Responses require
  --enable-response-store on a standalone server;
  response storage is unavailable in PD mode.
```

It is turned on with `--enable-response-store`. The limits are the same as in vLLM: process memory, no retention period, empty after a restart.

## Network and security

Neither server is designed to sit on the internet unprotected. Both have an access key, but it does not cover everything. A reverse proxy is put in front of the server and lets only the needed addresses through.

<div style="overflow-x:auto;margin:1.5rem 0" tabindex="0" role="region">
<svg viewBox="0 0 800 386" width="100%" style="min-width:620px;max-width:800px;height:auto;display:block;margin:0 auto" role="img" aria-label="What surrounds a model server: proxy, router, server replicas, metrics and traces" xmlns="http://www.w3.org/2000/svg">
  <defs>
    <marker id="vs-network-head" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L10 5 L0 10 z" fill="var(--color-fg-dim)"/>
    </marker>
  </defs>
  <rect x="20" y="96" width="130" height="84" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-link)" stroke-width="1.5"/>
  <text x="85" y="120" text-anchor="middle" fill="var(--color-link)" font-family="var(--font-sans)" font-size="14" font-weight="500">clients</text>
  <text x="85" y="140" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">applications,</text>
  <text x="85" y="157" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">agents</text>
  <rect x="196" y="96" width="164" height="84" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-border)" stroke-width="1.5"/>
  <text x="278" y="120" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="14" font-weight="500">proxy</text>
  <text x="278" y="140" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">TLS, keys, limits,</text>
  <text x="278" y="157" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">/v1/ only</text>
  <rect x="406" y="96" width="164" height="84" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-border)" stroke-width="1.5"/>
  <text x="488" y="120" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="14" font-weight="500">router</text>
  <text x="488" y="140" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">sends the request where</text>
  <text x="488" y="157" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">its beginning is cached</text>
  <rect x="624" y="30" width="156" height="84" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-accent)" stroke-width="1.5"/>
  <text x="702" y="54" text-anchor="middle" fill="var(--color-accent)" font-family="var(--font-sans)" font-size="14" font-weight="500">model server 1</text>
  <text x="702" y="74" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">vLLM / SGLang</text>
  <text x="702" y="91" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">its own GPU</text>
  <rect x="624" y="162" width="156" height="84" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-accent)" stroke-width="1.5"/>
  <text x="702" y="186" text-anchor="middle" fill="var(--color-accent)" font-family="var(--font-sans)" font-size="14" font-weight="500">model server 2</text>
  <text x="702" y="206" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">vLLM / SGLang</text>
  <text x="702" y="223" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">its own GPU</text>
  <path d="M150 138 L196 138" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" marker-end="url(#vs-network-head)"/>
  <path d="M360 138 L406 138" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" marker-end="url(#vs-network-head)"/>
  <path d="M570 124 L597 124 L597 72 L624 72" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" marker-end="url(#vs-network-head)"/>
  <path d="M570 152 L597 152 L597 204 L624 204" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" marker-end="url(#vs-network-head)"/>
  <rect x="196" y="296" width="164" height="72" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-border)" stroke-width="1.5" stroke-dasharray="4 4"/>
  <text x="278" y="320" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="14" font-weight="500">Prometheus</text>
  <text x="278" y="340" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">scrapes /metrics</text>
  <rect x="20" y="296" width="130" height="72" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-border)" stroke-width="1.5" stroke-dasharray="4 4"/>
  <text x="85" y="320" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="14" font-weight="500">Grafana</text>
  <text x="85" y="340" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">charts</text>
  <rect x="406" y="296" width="164" height="72" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-border)" stroke-width="1.5" stroke-dasharray="4 4"/>
  <text x="488" y="320" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="14" font-weight="500">Jaeger</text>
  <text x="488" y="340" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">request traces</text>
  <path d="M196 332 L150 332" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" stroke-dasharray="5 4" marker-end="url(#vs-network-head)"/>
  <path d="M702 246 L702 272 L278 272 L278 296" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" stroke-dasharray="5 4" marker-end="url(#vs-network-head)"/>
  <path d="M740 246 L740 332 L570 332" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" stroke-dasharray="5 4" marker-end="url(#vs-network-head)"/>
  <text x="600" y="266" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12" font-weight="400">metrics</text>
  <text x="656" y="326" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12" font-weight="400">traces</text>
</svg>
</div>

- **Proxy:** TLS, keys, request rate limiting, a list of allowed addresses.
- **Router:** picks a copy of the server (the next section).
- **Model servers and monitoring** — in the internal network.

The nginx configuration for one server:

```nginx title="nginx/vllm.conf"
upstream llm {
    server vllm:8000;
    # Keep connections to the model server open
    keepalive 32;
}

server {
    listen 80;
    # Requests with images can be large
    client_max_body_size 20m;

    # Only the client API is open to the outside
    location /v1/ {
        proxy_pass http://llm;
        proxy_http_version 1.1;
        proxy_set_header Connection "";
        proxy_set_header Host $host;
        # One request ID in the proxy and model server logs
        proxy_set_header X-Request-Id $request_id;
        # Without this the proxy accumulates the response and sends it in one piece
        proxy_buffering off;
        # A long answer can take minutes
        proxy_read_timeout 600s;
    }

    # Liveness check for the load balancer
    location = /health {
        proxy_pass http://llm;
        access_log off;
    }

    # Everything else, including /metrics, /invocations and /docs, is unreachable from outside
    location / {
        return 404;
    }
}
```

- `proxy_buffering off` — otherwise, instead of text arriving word by word, the user gets a pause and then the whole answer at once.
- `proxy_read_timeout 600s` — by default nginx drops the connection after 60 seconds of silence.
- `location / { return 404; }` — only what is listed explicitly goes outside.
- For production add `listen 443 ssl` and certificates. For SGLang replace the address with `sglang:30000`.

What the server answers to a request **without a key** when a key is set:

| Address | vLLM | SGLang |
|---|---|---|
| `/health` | 200 | 200 |
| `/metrics` | 200 | 200 |
| `/docs`, `/openapi.json` | 200 | 401 |
| `/version` | 200 | — |
| `/v1/models`, `/v1/chat/completions` | 401 | 401 |
| `POST /invocations` (generation) | 200 | — |
| `POST /tokenize` | 200 | 401 |
| server information, cache reset | — | 401 |

### vLLM: network and security

- **Address and port:** `--host`, `--port` (all interfaces and 8000 by default), `--uds` for a Unix socket, `--root-path` for a proxy that is not at the site root.
- **Key:** `--api-key` or `VLLM_API_KEY`. Several are possible: `--api-key k1 k2`.
- **The key covers only `/v1`** and a few service prefixes.
- **`/invocations` generates without a key.** The address was made for AWS SageMaker and accepts the same requests as `/v1/chat/completions`:

```bash
curl -X POST http://localhost:8000/invocations \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3.5-4b",
       "messages": [{"role": "user", "content": "Say OK"}],
       "max_tokens": 5,
       "chat_template_kwargs": {"enable_thinking": false}}'
```

```json
{
  "id": "chatcmpl-ab039d14c5d03cbf",
  "object": "chat.completion",
  "model": "qwen3.5-4b",
  "choices": [{"index": 0, "message": {"role": "assistant", "content": "OK"}}]
}
```

  This is documented behavior: the security page in the vLLM documentation lists `/invocations` among the unprotected addresses. Expose it only through a proxy.

- **TLS:** `--ssl-keyfile`, `--ssl-certfile`, `--enable-ssl-refresh` (re-read the certificates), `--ssl-cert-reqs` (verify the client certificate).
- **CORS:** since version 0.32 requests from other sites are forbidden. Allowed sites — `--allowed-origins`.
- **Communication between nodes** is not encrypted. Keep the nodes in an isolated network.
- **Images by links:** the server downloads the image by the address from the request itself, so it can be pointed at internal network addresses. The limit — `--allowed-media-domains`.
- **Telemetry:** the server sends anonymous statistics to the developers (GPU, model architecture, version, some settings; no request texts). Turned off with `VLLM_NO_USAGE_STATS=1` or `DO_NOT_TRACK=1`.

### SGLang: network and security

- **Address and port:** `--host`, `--port`. By default `127.0.0.1:30000` — only from the same machine.
- **Two keys:** `--api-key` for clients, `--admin-api-key` for management (weight updates, cache reset, server information).
- **Open without a key are only** the health checks and metrics.
- **`/get_server_info` gives out the key.** The response holds all the server settings, including `api_key` in plain text. The request requires a key, but every client with a key sees all the settings. With `--admin-api-key` the address is closed to clients.
- **Without `--api-key` everything is open:** all 108 operations, including adapter loading and weight updates.
- **CORS is open to all sites.** It is hardcoded, there is no flag to restrict it.
- **TLS:** the same `--ssl-keyfile`, `--ssl-certfile`, `--enable-ssl-refresh`. HTTP/2 — `--enable-http2`.
- **Communication between nodes** is not encrypted.
- **Images by links:** `--allowed-media-domains`. Without it the server downloads from any address.
- **Telemetry:** none.

## Multiple copies and routing

When one GPU is not enough, you run several copies of the server and spread requests between them. Each copy has its own prefix cache. If the next turn of a conversation lands on a different copy, it reads the whole history again. That is why model servers have routers that send a request to where its beginning is already in the cache.

<div style="overflow-x:auto;margin:1.5rem 0" tabindex="0" role="region">
<svg viewBox="0 0 810 256" width="100%" style="min-width:620px;max-width:810px;height:auto;display:block;margin:0 auto" role="img" aria-label="Round-robin and cache-aware request routing" xmlns="http://www.w3.org/2000/svg">
  <defs>
    <marker id="vs-routing-head" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L10 5 L0 10 z" fill="var(--color-fg-dim)"/>
    </marker>
  </defs>
  <text x="205" y="24" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="13.5" font-weight="500">Round robin</text>
  <rect x="20" y="96" width="120" height="60" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-link)" stroke-width="1.5"/>
  <text x="80" y="120" text-anchor="middle" fill="var(--color-link)" font-family="var(--font-sans)" font-size="14" font-weight="500">dialogue</text>
  <text x="80" y="140" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">turns 1 and 2</text>
  <rect x="250" y="44" width="140" height="60" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-accent)" stroke-width="1.5"/>
  <text x="320" y="68" text-anchor="middle" fill="var(--color-accent)" font-family="var(--font-sans)" font-size="14" font-weight="500">replica 1</text>
  <text x="320" y="88" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">cache: turn 1</text>
  <rect x="250" y="150" width="140" height="60" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-border)" stroke-width="1.5"/>
  <text x="320" y="174" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="14" font-weight="500">replica 2</text>
  <text x="320" y="194" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">cache empty</text>
  <path d="M140 112 L195 112 L195 66 L250 66" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" marker-end="url(#vs-routing-head)"/>
  <text x="170" y="104" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12" font-weight="400">1</text>
  <path d="M140 140 L195 140 L195 180 L250 180" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" stroke-dasharray="5 4" marker-end="url(#vs-routing-head)"/>
  <text x="170" y="158" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12" font-weight="400">2</text>
  <text x="205" y="240" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">turn 2 lands on replica 2: the history is read again</text>
  <text x="605" y="24" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="13.5" font-weight="500">Cache-aware</text>
  <rect x="420" y="96" width="120" height="60" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-link)" stroke-width="1.5"/>
  <text x="480" y="120" text-anchor="middle" fill="var(--color-link)" font-family="var(--font-sans)" font-size="14" font-weight="500">dialogue</text>
  <text x="480" y="140" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">turns 1 and 2</text>
  <rect x="650" y="44" width="140" height="60" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-accent)" stroke-width="1.5"/>
  <text x="720" y="68" text-anchor="middle" fill="var(--color-accent)" font-family="var(--font-sans)" font-size="14" font-weight="500">replica 1</text>
  <text x="720" y="88" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">cache: turn 1</text>
  <rect x="650" y="150" width="140" height="60" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-border)" stroke-width="1.5"/>
  <text x="720" y="174" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="14" font-weight="500">replica 2</text>
  <text x="720" y="194" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">cache empty</text>
  <path d="M540 112 L595 112 L595 66 L650 66" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" marker-end="url(#vs-routing-head)"/>
  <text x="570" y="104" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12" font-weight="400">1</text>
  <path d="M540 130 L615 130 L615 86 L650 86" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" stroke-dasharray="5 4" marker-end="url(#vs-routing-head)"/>
  <text x="570" y="146" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12" font-weight="400">2</text>
  <text x="605" y="240" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">turn 2 lands on replica 1: the history is cached</text>
  <path d="M400 14 L400 246" stroke="var(--color-border)" stroke-width="1" stroke-dasharray="4 4"/>
</svg>
</div>

A measurement on the test bench: two copies of Qwen3.5-0.8B on one GPU, six conversations of five turns each with a long system prompt.

| Server | Request distribution | Prompt tokens from cache |
|---|---|---|
| vLLM | one copy | 59% |
| vLLM | nginx, to the less busy one | 34% |
| vLLM | vLLM Router, cache-aware | 52% |
| SGLang | router, round robin | 53% |
| SGLang | router, cache-aware | 73% |

Distribution that ignores the cache takes almost half of the hits away from vLLM. The test is small (30 requests), and the shares change a little from run to run.

### vLLM: copies and routing

**nginx** is the option from the vLLM documentation. The beginning of the configuration from the network section changes:

```nginx title="nginx/balance.conf"
upstream llm {
    # A new request goes to the server with fewer open connections
    least_conn;
    server vllm-a:8000 max_fails=3 fail_timeout=30s;
    server vllm-b:8000 max_fails=3 fail_timeout=30s;
    keepalive 32;
}
```

**[vLLM Router](https://github.com/vllm-project/router)** is a router in Rust from the vLLM team. It grew out of the SGLang router ([announcement](https://vllm.ai/blog/2025-12-13-vllm-router-release)).

```bash
pip install vllm-router==0.1.17
vllm-router \
  --worker-urls http://vllm-a:8000 http://vllm-b:8000 \
  --policy cache_aware \
  --host 0.0.0.0 --port 8090
```

Distribution rules (`--policy`):

| Value | How it distributes |
|---|---|
| `round_robin` | round robin |
| `random` | at random |
| `power_of_two` | to the less busy of two random ones |
| `consistent_hash` | by a hash of the request |
| `sticky_least_loaded` | to the less busy one, pinning the conversation |
| `cache_aware` | to where the beginning of the request is in the cache |
| `kv_aware` | by cache information from the servers themselves |

It can also: retry failed requests, exclude a failing copy, discover copies in Kubernetes.

**[Production Stack](https://github.com/vllm-project/production-stack)** is a kit for Kubernetes from the same team: a router, servers, monitoring, installed through Helm. Not run on the test bench.

**Cluster layers:** llm-d, AIBrix, NVIDIA Dynamo, KServe — each has a page in the vLLM documentation.

### SGLang: copies and routing

The router is in the server image:

```bash
docker run -d --name sglang-router --network lab -p 30010:30000 \
  --entrypoint python lmsysorg/sglang:v0.5.23-runtime \
  -m sglang_router.launch_router \
  --worker-urls http://sglang-a:30000 http://sglang-b:30000 \
  --policy cache_aware \
  --host 0.0.0.0 --port 30000 --prometheus-port 29000
```

Distribution rules: `round_robin`, `random`, `power_of_two`, `cache_aware`, `consistent_hashing`, `prefix_hash`, `bucket`, `manual`. It can also: retry requests, exclude failing copies, limit the number of simultaneous requests, discover copies in Kubernetes. It serves its own metrics on a separate port.

SGLang currently has three routers:

| Project | Package | State |
|---|---|---|
| SGLang Model Gateway | `sglang-router` | in the image and the documentation; the package has not been updated since January 2026 |
| [SMG](https://github.com/smg-project/smg) | `smg` | the continuation of the first one in a separate repository; works with vLLM too |
| sgl-router | `sgl-router` | a new lightweight one, first version on 18 November 2026 |

Before choosing, check which of them is being developed.

**Copies inside one server:** `--dp-size N`. For this case the documentation itself recommends a router.

**Cluster layers:** llm-d, AIBrix, Dynamo and its own Kubernetes operator — OME.

## Observability

- **Metrics** — counters and timings in the Prometheus format at `/metrics` on the main port.
- **Logs** — what the server is doing, every request if you wish.
- **Tracing** — the path of one request step by step, following the OpenTelemetry standard.

The timings that metrics and traces show:

<div style="overflow-x:auto;margin:1.5rem 0" tabindex="0" role="region">
<svg viewBox="0 0 780 200" width="100%" style="min-width:620px;max-width:780px;height:auto;display:block;margin:0 auto" role="img" aria-label="The timings of one request: queue, prompt reading, first token, generation" xmlns="http://www.w3.org/2000/svg">
  <defs>
    <marker id="vs-timeline-head" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L10 5 L0 10 z" fill="var(--color-fg-dim)"/>
    </marker>
    <marker id="vs-timeline-tail" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L10 5 L0 10 z" fill="var(--color-fg-dim)"/>
    </marker>
  </defs>
  <rect x="30" y="92" width="110" height="44" fill="var(--color-border)" fill-opacity="0.9" stroke="var(--color-border)" stroke-width="1"/>
  <text x="85" y="118" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">queue</text>
  <rect x="140" y="92" width="190" height="44" fill="var(--color-link)" fill-opacity="0.45" stroke="var(--color-border)" stroke-width="1"/>
  <text x="235" y="118" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">prefill: reading the prompt</text>
  <rect x="330" y="92" width="420" height="44" fill="var(--color-accent)" fill-opacity="0.4" stroke="var(--color-border)" stroke-width="1"/>
  <text x="540" y="118" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">decode: answer tokens, one by one</text>
  <path d="M360 127 L360 136" stroke="var(--color-fg-dim)" stroke-width="1"/>
  <path d="M390 127 L390 136" stroke="var(--color-fg-dim)" stroke-width="1"/>
  <path d="M420 127 L420 136" stroke="var(--color-fg-dim)" stroke-width="1"/>
  <path d="M450 127 L450 136" stroke="var(--color-fg-dim)" stroke-width="1"/>
  <path d="M480 127 L480 136" stroke="var(--color-fg-dim)" stroke-width="1"/>
  <path d="M510 127 L510 136" stroke="var(--color-fg-dim)" stroke-width="1"/>
  <path d="M540 127 L540 136" stroke="var(--color-fg-dim)" stroke-width="1"/>
  <path d="M570 127 L570 136" stroke="var(--color-fg-dim)" stroke-width="1"/>
  <path d="M600 127 L600 136" stroke="var(--color-fg-dim)" stroke-width="1"/>
  <path d="M630 127 L630 136" stroke="var(--color-fg-dim)" stroke-width="1"/>
  <path d="M660 127 L660 136" stroke="var(--color-fg-dim)" stroke-width="1"/>
  <path d="M690 127 L690 136" stroke="var(--color-fg-dim)" stroke-width="1"/>
  <path d="M720 127 L720 136" stroke="var(--color-fg-dim)" stroke-width="1"/>
  <path d="M30 62 L330 62" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" marker-end="url(#vs-timeline-head)" marker-start="url(#vs-timeline-tail)"/>
  <text x="180" y="52" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">time to first token (TTFT)</text>
  <path d="M540 62 L570 62" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" marker-end="url(#vs-timeline-head)" marker-start="url(#vs-timeline-tail)"/>
  <text x="585" y="52" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">gap between tokens</text>
  <path d="M30 166 L750 166" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" marker-end="url(#vs-timeline-head)" marker-start="url(#vs-timeline-tail)"/>
  <text x="390" y="186" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">end-to-end request time</text>
  <text x="30" y="24" text-anchor="start" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12" font-weight="400">request arrives</text>
  <text x="750" y="24" text-anchor="end" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12" font-weight="400">answer finished</text>
</svg>
</div>

Prometheus, Grafana and Jaeger next to the servers:

```yaml title="monitoring/compose.yaml"
# Prometheus, Grafana and Jaeger next to the model servers.
# The lab network is created beforehand: docker network create lab
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    volumes:
      - ./prometheus.yaml:/etc/prometheus/prometheus.yml:ro
    ports:
      - "9090:9090"
    networks: [lab]

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    environment:
      # Sign-in without a password — only for a test bench on your own computer
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
    volumes:
      # The data source and dashboards are connected automatically at startup
      - ./provisioning:/etc/grafana/provisioning:ro
      - ./dashboards:/var/lib/grafana/dashboards:ro
    ports:
      - "3000:3000"
    networks: [lab]
    depends_on: [prometheus]

  jaeger:
    image: jaegertracing/jaeger:latest
    container_name: jaeger
    ports:
      - "16686:16686" # web interface
      - "4317:4317"   # receiving traces over OTLP gRPC
      - "4318:4318"   # receiving traces over OTLP HTTP
    networks: [lab]

networks:
  lab:
    external: true
```

```yaml title="monitoring/prometheus.yaml"
global:
  scrape_interval: 5s

scrape_configs:
  # Both servers serve metrics at /metrics on their main port
  - job_name: vllm
    static_configs:
      - targets: ["vllm:8000"]

  - job_name: sglang
    static_configs:
      - targets: ["sglang:30000"]
```

Two files so that Grafana connects Prometheus and the dashboards from the folder by itself:

```yaml title="monitoring/provisioning/datasources/prometheus.yaml"
apiVersion: 1
datasources:
  - name: Prometheus
    uid: prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
```

```yaml title="monitoring/provisioning/dashboards/dashboards.yaml"
apiVersion: 1
providers:
  - name: lab
    type: file
    options:
      path: /var/lib/grafana/dashboards
```

The charts below were taken under the same load: eight clients without pauses, a shared system prompt, Qwen3.5-4B in FP8, four minutes.

### vLLM: metrics, logs, tracing

**Metrics** are always on.

| Metric | What it shows |
|---|---|
| `vllm:num_requests_running` | requests running |
| `vllm:num_requests_waiting` | requests in the queue. The main signal for adding copies |
| `vllm:kv_cache_usage_perc` | how full the conversation memory is, 0–1 |
| `vllm:time_to_first_token_seconds` | time to first token |
| `vllm:inter_token_latency_seconds` | pause between tokens |
| `vllm:e2e_request_latency_seconds` | time of the whole request |
| `vllm:prompt_tokens_total`, `vllm:generation_tokens_total` | tokens read and written |
| `vllm:prefix_cache_hits_total`, `vllm:prefix_cache_queries_total` | prefix cache hits |
| `vllm:num_preemptions_total` | requests preempted for lack of memory |
| `vllm:request_success_total` | finished requests with the reason for finishing |

**The dashboard** is in the repository: `examples/observability/prometheus_grafana/grafana.json`. It is meant for manual import; for automatic provisioning replace the data source variable:

```bash
sed -i 's/${DS_PROMETHEUS}/prometheus/g' monitoring/dashboards/vllm.json
```

![The vLLM dashboard in Grafana under load: latencies, tokens per second, queue, cache fill](/images/2026/11/vllm-sglang-grafana-vllm.png)

On the charts: eight requests are running, no queue; about 400 output tokens per second; the first token comes after 0.1 seconds on average; the conversation memory is half full.

**The log.** Every ten seconds there is a line with totals:

```text
Engine 000: Avg prompt throughput: 1913.3 tokens/s,
  Avg generation throughput: 371.4 tokens/s,
  Running: 9 reqs, Waiting: 0 reqs, GPU KV cache usage: 49.4%,
  Prefix cache hit rate: 3.6%
```

| Flag | What it does |
|---|---|
| `--enable-log-requests` | a line for every request: ID and parameters. Texts only at the DEBUG level |
| `--enable-log-outputs` | log the responses too |
| `--max-log-len` | truncate long texts |
| `--disable-access-log-for-endpoints /health,/metrics` | do not log health checks |
| `--logging-config.formatter json` | log in JSON |

A log line in JSON:

```json
{
  "asctime": "2026-11-21 06:22:43,840",
  "levelname": "INFO",
  "name": "vllm.v1.core.kv_cache_utils",
  "processName": "EngineCore",
  "process": 99,
  "message": "GPU KV cache size: 171,008 tokens, ..."
}
```

**Tracing.** The libraries are already in the image. You need the receiver's address and two variables:

```bash
docker run --gpus all --ipc=host -p 8000:8000 --network lab --name vllm \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v ./vllm:/lab/vllm \
  -e VLLM_API_KEY=sk-lab-123 \
  -e OTEL_SERVICE_NAME=vllm \
  -e OTEL_EXPORTER_OTLP_TRACES_INSECURE=true \
  vllm/vllm-openai:v0.34.0 \
  --config /lab/vllm/config.yaml \
  --otlp-traces-endpoint grpc://jaeger:4317 --enable-log-requests
```

For every request vLLM writes one record with ready-made numbers:

![A vLLM request trace in Jaeger: one record with the queue, prompt reading and generation times](/images/2026/11/vllm-sglang-jaeger-vllm.png)

A prompt of 517 tokens was read in 0.12 seconds, an answer of 93 tokens was written in 1.28 seconds, the first token arrived after 0.15 seconds. The field names come from the OpenTelemetry convention for generative AI.

### SGLang: metrics, logs, tracing

**Metrics** are turned on with the `--enable-metrics` flag.

| Metric | What it shows |
|---|---|
| `sglang:num_running_reqs` | requests running |
| `sglang:num_queue_reqs` | requests in the queue |
| `sglang:token_usage` | how full the conversation memory is |
| `sglang:mamba_usage` | how full the state memory of hybrid models is |
| `sglang:time_to_first_token_seconds` | time to first token |
| `sglang:inter_token_latency_seconds` | pause between tokens |
| `sglang:e2e_request_latency_seconds` | time of the whole request |
| `sglang:gen_throughput` | generation speed, tokens per second |
| `sglang:cache_hit_rate` | share of prompt tokens from the cache |
| `sglang:num_retracted_requests_total` | requests taken off for lack of memory |

**The dashboard** is in the repository: `examples/monitoring/grafana/dashboards/json/sglang-dashboard.json`. In version 0.5.23 it shows no data:

![The ready-made SGLang dashboard in Grafana: every chart says No data](/images/2026/11/vllm-sglang-grafana-sglang-empty.png)

The server publishes metrics with a colon (`sglang:num_running_reqs`), while the queries in the dashboard are written with an underscore (`sglang_num_running_reqs`). The fix:

```bash
sed -i 's/sglang_/sglang:/g' monitoring/dashboards/sglang.json
```

![The SGLang dashboard in Grafana under load: latencies, tokens per second, queue, cache hits](/images/2026/11/vllm-sglang-grafana-sglang.png)

On the charts: four requests are running, four are in the queue — no more fit on this GPU with this model; about 215 tokens per second; the first token after 2–4 seconds because of the queue; a cache hit of 0.87.

**The log.** A line for every prompt read and once every 40 generation steps:

```text
Prefill batch, #new-seq: 1, #new-token: 69, #cached-token: 448,
  full token usage: 0.02, mamba usage: 0.80,
  #running-req: 3, #queue-req: 1, input throughput (token/s): 137.62
Decode batch, #running-req: 4, #full token: 1011,
  full token usage: 0.03, mamba num: 16, mamba usage: 0.80,
  gen throughput (token/s): 217.87, #queue-req: 4
```

The conversation memory is almost empty (`full token usage: 0.03`), the state memory is 80% full (`mamba usage: 0.80`). It is the bottleneck.

| Flag | What it does |
|---|---|
| `--log-requests` | log every request |
| `--log-requests-level` | verbosity from 0 (metadata) to 3 (full texts) |
| `--log-requests-format json` | request records in JSON |
| `--log-requests-target` | where to write: console, a folder or both |
| `--log-level`, `--log-level-http` | verbosity of the server log and of HTTP |

A request record in JSON, shortened:

```json
{
  "timestamp": "2026-11-21T06:09:40.003283",
  "event": "request.finished",
  "rid": "369fb465a2ce4d5294e441b2e8fdba7f",
  "obj": {
    "sampling_params": {"temperature": 1.0, "max_new_tokens": 120, "top_p": 1.0},
    "stream": true
  }
}
```

Only request records are written in JSON. The rest of the log is text; for JSON you need your own log configuration file (`SGLANG_LOGGING_CONFIG_PATH`).

**Tracing:**

```bash
docker run --gpus all --ipc=host --shm-size 16g -p 30000:30000 --network lab --name sglang \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -v ./sglang:/lab/sglang \
  lmsysorg/sglang:v0.5.23-runtime \
  sglang serve RedHatAI/Qwen3.5-4B-FP8-dynamic \
  --config /lab/sglang/config.yaml --api-key sk-lab-123 \
  --enable-trace --otlp-traces-endpoint jaeger:4317 \
  --log-requests --log-requests-level 1 --log-requests-format json
```

SGLang writes a tree of steps:

![An SGLang request trace in Jaeger: a tree of steps from tokenization to the generation of each batch of tokens](/images/2026/11/vllm-sglang-jaeger-sglang.png)

Tokenization takes 3.6 ms, reading a prompt of 2157 tokens takes 769 ms, then generation in batches of 16 ms.

The vLLM record is lighter: one event per request. The SGLang tree is more detailed, but it is dozens of records per request; the detail is reduced with the `SGLANG_TRACE_LEVEL` variable.

### Health checks

| | vLLM | SGLang |
|---|---|---|
| The process is alive | `/health`: 200 or 503 | `/health` |
| The server actually generates | — | `/health_generate`: generates one token |
| Ready to accept requests | — | `/ready`: 503 during startup, pause and shutdown |

While the model is loading, the vLLM port is closed. In Kubernetes you need a startup probe (`startupProbe`) with a margin of several minutes, otherwise the pod will restart endlessly.

## Performance

There is no general answer to "which is faster". Independent 2026 measurements disagree:

- [Winder.AI](https://winder.ai/vllm-vs-ollama-vs-sglang-llm-inference-comparison/), September, H100, Qwen3.8-27B: SGLang ahead by 7%.
- [Jarvislabs](https://jarvislabs.ai/blog/vllm-sglang-trtllm-comparison), May, H100, Qwen models of earlier generations: vLLM ahead.
- Which public tests to trust — in the [section on benchmarks](/en/llm-inference-engines/#Benchmarks-which-numbers-to-trust).

The measurement on the test bench:

- **Model:** Qwen3.5-2B uncompressed.
- **Client:** `vllm bench serve` for both servers. The prompt is 512 random tokens, the answer is 256 tokens. New prompts on every level.
- **vLLM:** `--max-model-len 8192 --max-num-seqs 64`.
- **SGLang:** with default settings and tuned.

```bash
docker run --rm --network lab --gpus all --entrypoint vllm \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:v0.34.0 bench serve \
  --backend openai --base-url http://vllm:8000 --endpoint /v1/completions \
  --model qwen3.5-2b --tokenizer Qwen/Qwen3.5-2B \
  --dataset-name random --random-input-len 512 --random-output-len 256 \
  --num-prompts 96 --max-concurrency 16 --ignore-eos --seed 42185
```

<div style="overflow-x:auto;margin:1.5rem 0" tabindex="0" role="region">
<svg viewBox="0 0 800 304" width="100%" style="min-width:620px;max-width:800px;height:auto;display:block;margin:0 auto" role="img" aria-label="Generation speed of vLLM and SGLang at different numbers of concurrent requests" xmlns="http://www.w3.org/2000/svg">
  <defs>
    <marker id="vs-bench-head" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L10 5 L0 10 z" fill="var(--color-fg-dim)"/>
    </marker>
  </defs>
  <path d="M40 250 L774 250" stroke="var(--color-border)" stroke-width="1.5"/>
  <rect x="60" y="241" width="38" height="9" rx="2" fill="var(--color-accent)" fill-opacity="0.6" stroke="var(--color-border)" stroke-width="1"/>
  <text x="79" y="235" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="11.5" font-weight="400">110</text>
  <rect x="102" y="241" width="38" height="9" rx="2" fill="var(--color-border)" fill-opacity="0.95" stroke="var(--color-border)" stroke-width="1"/>
  <text x="121" y="235" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="11.5" font-weight="400">109</text>
  <rect x="144" y="242" width="38" height="8" rx="2" fill="var(--color-link)" fill-opacity="0.55" stroke="var(--color-border)" stroke-width="1"/>
  <text x="163" y="236" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="11.5" font-weight="400">107</text>
  <text x="121" y="270" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12.5" font-weight="400">1</text>
  <rect x="206" y="213" width="38" height="37" rx="2" fill="var(--color-accent)" fill-opacity="0.6" stroke="var(--color-border)" stroke-width="1"/>
  <text x="225" y="207" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="11.5" font-weight="400">468</text>
  <rect x="248" y="214" width="38" height="36" rx="2" fill="var(--color-border)" fill-opacity="0.95" stroke="var(--color-border)" stroke-width="1"/>
  <text x="267" y="208" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="11.5" font-weight="400">454</text>
  <rect x="290" y="217" width="38" height="33" rx="2" fill="var(--color-link)" fill-opacity="0.55" stroke="var(--color-border)" stroke-width="1"/>
  <text x="309" y="211" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="11.5" font-weight="400">416</text>
  <text x="267" y="270" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12.5" font-weight="400">4</text>
  <rect x="352" y="141" width="38" height="109" rx="2" fill="var(--color-accent)" fill-opacity="0.6" stroke="var(--color-border)" stroke-width="1"/>
  <text x="371" y="135" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="11.5" font-weight="400">1382</text>
  <rect x="394" y="191" width="38" height="59" rx="2" fill="var(--color-border)" fill-opacity="0.95" stroke="var(--color-border)" stroke-width="1"/>
  <text x="413" y="185" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="11.5" font-weight="400">753</text>
  <rect x="436" y="160" width="38" height="90" rx="2" fill="var(--color-link)" fill-opacity="0.55" stroke="var(--color-border)" stroke-width="1"/>
  <text x="455" y="154" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="11.5" font-weight="400">1150</text>
  <text x="413" y="270" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12.5" font-weight="400">16</text>
  <rect x="498" y="112" width="38" height="138" rx="2" fill="var(--color-accent)" fill-opacity="0.6" stroke="var(--color-border)" stroke-width="1"/>
  <text x="517" y="106" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="11.5" font-weight="400">1756</text>
  <rect x="540" y="188" width="38" height="62" rx="2" fill="var(--color-border)" fill-opacity="0.95" stroke="var(--color-border)" stroke-width="1"/>
  <text x="559" y="182" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="11.5" font-weight="400">783</text>
  <rect x="582" y="138" width="38" height="112" rx="2" fill="var(--color-link)" fill-opacity="0.55" stroke="var(--color-border)" stroke-width="1"/>
  <text x="601" y="132" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="11.5" font-weight="400">1427</text>
  <text x="559" y="270" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12.5" font-weight="400">32</text>
  <rect x="644" y="60" width="38" height="190" rx="2" fill="var(--color-accent)" fill-opacity="0.6" stroke="var(--color-border)" stroke-width="1"/>
  <text x="663" y="54" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="11.5" font-weight="400">2415</text>
  <rect x="686" y="191" width="38" height="59" rx="2" fill="var(--color-border)" fill-opacity="0.95" stroke="var(--color-border)" stroke-width="1"/>
  <text x="705" y="185" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="11.5" font-weight="400">756</text>
  <rect x="728" y="131" width="38" height="119" rx="2" fill="var(--color-link)" fill-opacity="0.55" stroke="var(--color-border)" stroke-width="1"/>
  <text x="747" y="125" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="11.5" font-weight="400">1518</text>
  <text x="705" y="270" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-mono)" font-size="12.5" font-weight="400">64</text>
  <text x="407" y="292" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">concurrent requests</text>
  <rect x="60" y="12" width="14" height="14" rx="2" fill="var(--color-accent)" fill-opacity="0.6" stroke="var(--color-border)"/>
  <text x="82" y="24" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">vLLM</text>
  <rect x="170" y="12" width="14" height="14" rx="2" fill="var(--color-border)" fill-opacity="0.95" stroke="var(--color-border)"/>
  <text x="192" y="24" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">SGLang, defaults</text>
  <rect x="390" y="12" width="14" height="14" rx="2" fill="var(--color-link)" fill-opacity="0.55" stroke="var(--color-border)"/>
  <text x="412" y="24" text-anchor="start" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">SGLang, tuned</text>
  <text x="774" y="24" text-anchor="end" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5" font-weight="400">tokens per second</text>
</svg>
</div>

Generation speed, tokens per second:

| Simultaneous requests | vLLM | SGLang default | SGLang tuned |
|---|---|---|---|
| 1 | 110 | 109 | 107 |
| 4 | 468 | 454 | 416 |
| 16 | 1382 | 753 | 1150 |
| 32 | 1756 | 783 | 1427 |
| 64 | 2415 | 756 | 1518 |

Median time to first token, milliseconds:

| Simultaneous requests | vLLM | SGLang default | SGLang tuned |
|---|---|---|---|
| 1 | 43 | 45 | 46 |
| 4 | 134 | 154 | 157 |
| 16 | 293 | 400 | 361 |
| 32 | 306 | 7796 | 698 |
| 64 | 330 | 16,667 | 5529 |

Conclusions:

1. **For a single user the servers are equal.**
2. **Batching works in both.** 64 simultaneous requests give vLLM 22 times more text than one.
3. **SGLang by default limited itself to 12 requests:** little memory for the state of the hybrid model. The other requests waited in the queue.
4. **After tuning SGLang is twice as fast** (29 simultaneous requests), but it did not catch up with vLLM.

The tuned SGLang:

```bash
sglang serve Qwen/Qwen3.5-2B --host 0.0.0.0 \
  --served-model-name qwen3.5-2b --context-length 8192 --enable-metrics \
  --max-running-requests 64 --mamba-full-memory-ratio 4 \
  --cuda-graph-max-bs-decode 64 --mem-fraction-static 0.72
```

**Requests with a shared beginning:** four beginnings of 3000 tokens each, 24 requests per beginning, eight at a time.

```bash
docker run --rm --network lab --gpus all --entrypoint vllm \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:v0.34.0 bench serve \
  --backend openai --base-url http://vllm:8000 --endpoint /v1/completions \
  --model qwen3.5-2b --tokenizer Qwen/Qwen3.5-2B \
  --dataset-name prefix_repetition \
  --prefix-repetition-num-prefixes 4 --prefix-repetition-prefix-len 3000 \
  --prefix-repetition-suffix-len 200 --prefix-repetition-output-len 100 \
  --num-prompts 96 --max-concurrency 8 --ignore-eos --seed 43995
```

| | vLLM | SGLang |
|---|---|---|
| Generation speed, tokens per second | 689 | 653 |
| Median time to first token | 141 ms | 172 ms |
| Prompt tokens from cache | 86% | 92% |

On a long shared beginning the servers are equal. The difference shows on short matches and in a conversation that grows bit by bit (the section [on sessions](#Sessions-and-state)).

Limits of the measurement:

- **A laptop.** On a server card with 80 GB the state memory will not be the bottleneck.
- **A hybrid model.** SGLang's lag is related to how it lays out the model's state. On ordinary models it can be the other way round.
- **Memory at the limit.** The tuned SGLang took 11.3 of 12 GB.
- **Random prompts** are the worst case for a cache.

Measure both servers on your own model, your own GPU and your own load.

### vLLM: what to tune for speed

| Setting | Effect |
|---|---|
| `--max-num-seqs` | ceiling of simultaneous requests. Higher — higher total speed, longer pause between tokens for each user |
| `--max-num-batched-tokens` | tokens per step. Lower — smoother pauses, higher — faster first token |
| `--performance-mode` | ready-made sets: `balanced`, `interactivity`, `throughput` |
| `-O0` … `-O3` | optimization level. `-O0` — fast startup and slow work |
| `--spec-method`, `--spec-model`, `--spec-tokens` | [speculative decoding](/en/llm-inference-engines/#Speculative-decoding) |
| the `/root/.cache/vllm` volume | compilation cache; without it every new container compiles again |
| `VLLM_WSL2_ENABLE_PIN_MEMORY=1` | under WSL2 about 12% of generation speed, from a [measurement in the project discussions](https://github.com/vllm-project/vllm/issues/58849). Not checked on the test bench |

### SGLang: what to tune for speed

| Setting | Effect |
|---|---|
| `--max-running-requests` | ceiling of simultaneous requests |
| `--cuda-graph-max-bs-decode` | the largest batch a graph is recorded for. On cards under 20 GB — 8; above this the server runs without a graph and slower |
| `--schedule-policy` | queue order. `fcfs` — by arrival time, `lpm` — first the one with the longest match in the cache |
| `--schedule-conservativeness` | caution in accepting requests. Below 1 — bolder |
| `--chunked-prefill-size` | size of the chunks a prompt is read in |
| `--mamba-full-memory-ratio`, `--max-mamba-cache-size` | memory for the state of hybrid models |
| `--speculative-algorithm` | speculative decoding |
| `SGLANG_CACHE_DIR` and a volume | cache of compiled kernels |

A rule from the documentation: the conversation memory is less than 90% full and the queue is not empty — lower `--schedule-conservativeness`.

## Multiple GPUs

This section is written from the documentation, without being run. The ways to spread a model across cards are described in the [section on scaling](/en/llm-inference-engines/#Scaling-from-one-GPU-to-a-rack); here are the flags.

| Method | vLLM | SGLang |
|---|---|---|
| Layers split across cards (TP) | `--tensor-parallel-size` | `--tp-size` |
| Layers in turn on different cards (PP) | `--pipeline-parallel-size` | `--pp-size` |
| Copies of the model (DP) | `--data-parallel-size` | `--dp-size` |
| MoE experts across cards (EP) | `--enable-expert-parallel` | `--ep-size` |

### vLLM: multiple GPUs

Four cards in one server:

```bash
vllm serve <model> --tensor-parallel-size 4
```

Two servers with eight cards each, the main node and the second one:

```bash
vllm serve <model> --tensor-parallel-size 8 --pipeline-parallel-size 2 \
  --nnodes 2 --node-rank 0 --master-addr <main node address>

vllm serve <model> --tensor-parallel-size 8 --pipeline-parallel-size 2 \
  --nnodes 2 --node-rank 1 --master-addr <main node address> --headless
```

- **Separating prefill and decode** onto different servers exists, but is marked experimental. It evens out latencies rather than raising total speed. In production it is done through llm-d, Dynamo or AIBrix.
- **Offloading conversation memory** to RAM: `--kv-offloading-size`.
- **Autoscaling in Kubernetes** — by the `vllm:num_requests_waiting` metric. CPU load tells you nothing here.

### SGLang: multiple GPUs

Four cards in one server:

```bash
sglang serve <model> --tp-size 4
```

Two servers with four cards each:

```bash
sglang serve <model> --tp-size 4 --nnodes 2 --node-rank 0 \
  --dist-init-addr <first node address>:50000

sglang serve <model> --tp-size 4 --nnodes 2 --node-rank 1 \
  --dist-init-addr <first node address>:50000
```

- **Separating prefill and decode** is SGLang's strong side: servers with `--disaggregation-mode prefill` and `--disaggregation-mode decode`, with a router in front of them using `--pd-disaggregation`.
- **The multi-level HiCache cache:** conversation memory is offloaded to RAM and to disk.
- **Large models with experts:** `--attn-dp-size`.

## Updates

Both programs are released every two weeks, and every version renames or removes something.

1. Pin the image version.
2. Read the breaking changes section in the release notes.
3. Test the update on a copy with the same model.

### vLLM: updates

| Version | What broke |
|---|---|
| 0.34 | conversation memory is counted in a new way; cache block of 64 tokens instead of 16 |
| 0.33 | CUDA 12 builds and two speculative decoding methods removed |
| 0.32 | Python 3.10 removed; requests from other sites in the browser forbidden |
| 0.31 | `fp8` compression renamed to `fp8_per_tensor` |

- **The documentation lags behind.** In 0.34 the installation page names CUDA 12.9 as the default build, although the packages are built for 13.0. Trust `vllm serve --help=all`.
- **Check function calls on your own data.** At the hosting provider Ubicloud they matched the schema in 92% of cases, until strict checking was turned on at the server ([write-up](https://www.ubicloud.com/blog/fixing-tool-call-accuracy-in-vllm)).

### SGLang: updates

| Version | What broke |
|---|---|
| 0.5.23 | the router moved, its flags were renamed |
| 0.5.22 | Python 3.11 required; `--enable-dp-attention` replaced by `--attn-dp-size` |
| 0.5.21 | several API addresses removed; a chat template from the request is rejected |
| 0.5.20 | CUDA 12 builds and 17 flags removed; the Responses API store turned off |

- **Deprecated things are removed after two releases,** that is, after a month.
- **Install security updates.** From 27 August to 9 September 2026 a Cloudflare service running on SGLang returned other customers' data in 155 responses: an SGLang bug in the phase-separation mode plus a wrong memory configuration at Cloudflare ([write-up](https://blog.cloudflare.com/kv-cache-exposure-remediation/)). Fixed in 0.5.20–0.5.22.
- **Version 1.0 was announced** in November 2026, but has not been released yet.

## Common errors

| What happens | Cause | What to do |
|---|---|---|
| The server does not start although the model is smaller than the GPU memory | besides the weights, 3–4 GiB are needed | a smaller or compressed model; the section [on GPU memory](#GPU-memory) |
| Reasoning arrives instead of the answer | the reasoning parser is not set | `--reasoning-parser` |
| A function call arrives as text | the tool call parser is not set | `--tool-call-parser` |
| SGLang in a container does not answer from outside | listens on `127.0.0.1` | `--host 0.0.0.0` |
| SGLang: `--model-path is required` | the model is given only in the config file | the model on the command line |
| SGLang: `state cache is too small` | the default memory share is too small | `--mem-fraction-static 0.8` |
| vLLM: `max_num_seqs exceeds available Mamba cache blocks` | too many simultaneous requests for a hybrid model | lower `--max-num-seqs` |
| The SGLang dashboard is empty | the queries have `sglang_` instead of `sglang:` | replace the metric names |
| vLLM answers without a key | `/invocations` is not protected by the key | a proxy with a list of addresses |
| An SGLang client sees the server key | `/get_server_info` gives out the settings | `--admin-api-key` |
| A shared system prompt is not cached in vLLM | it is shorter than a cache block | for hybrid models — a prompt longer than 576 tokens |
| The answer through nginx arrives in one piece | the proxy buffers | `proxy_buffering off` |

## What to choose

<div style="overflow-x:auto;margin:1.5rem 0" tabindex="0" role="region">
<svg viewBox="0 0 800 340" width="100%" style="min-width:620px;max-width:800px;height:auto;display:block;margin:0 auto" role="img" aria-label="How to choose between llama.cpp, a cloud API, SGLang and vLLM" xmlns="http://www.w3.org/2000/svg">
  <defs>
    <marker id="vs-choose-head" viewBox="0 0 10 10" refX="9" refY="5" markerWidth="7" markerHeight="7" orient="auto-start-reverse">
      <path d="M0 0 L10 5 L0 10 z" fill="var(--color-fg-dim)"/>
    </marker>
  </defs>
  <rect x="20" y="16" width="380" height="56" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-border)" stroke-width="1.5"/>
  <text x="210" y="49" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="14" font-weight="500">A single user?</text>
  <rect x="500" y="16" width="280" height="56" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-link)" stroke-width="1.5"/>
  <text x="640" y="40" text-anchor="middle" fill="var(--color-link)" font-family="var(--font-sans)" font-size="14" font-weight="500">llama.cpp / Ollama</text>
  <text x="640" y="60" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">simpler and no slower</text>
  <path d="M400 44 L500 44" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" marker-end="url(#vs-choose-head)"/>
  <text x="450" y="36" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12" font-weight="400">yes</text>
  <path d="M210 72 L210 100" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" marker-end="url(#vs-choose-head)"/>
  <text x="222" y="90" text-anchor="start" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12" font-weight="400">no</text>
  <rect x="20" y="100" width="380" height="56" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-border)" stroke-width="1.5"/>
  <text x="210" y="133" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="14" font-weight="500">Low or bursty load?</text>
  <rect x="500" y="100" width="280" height="56" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-link)" stroke-width="1.5"/>
  <text x="640" y="124" text-anchor="middle" fill="var(--color-link)" font-family="var(--font-sans)" font-size="14" font-weight="500">a cloud API</text>
  <text x="640" y="144" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">a GPU will not pay off</text>
  <path d="M400 128 L500 128" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" marker-end="url(#vs-choose-head)"/>
  <text x="450" y="120" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12" font-weight="400">yes</text>
  <path d="M210 156 L210 184" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" marker-end="url(#vs-choose-head)"/>
  <text x="222" y="174" text-anchor="start" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12" font-weight="400">no</text>
  <rect x="20" y="184" width="380" height="56" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-border)" stroke-width="1.5"/>
  <text x="210" y="217" text-anchor="middle" fill="var(--color-fg)" font-family="var(--font-sans)" font-size="14" font-weight="500">Agents, long dialogues, shared beginnings?</text>
  <rect x="500" y="184" width="280" height="56" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-accent)" stroke-width="1.5"/>
  <text x="640" y="208" text-anchor="middle" fill="var(--color-accent)" font-family="var(--font-sans)" font-size="14" font-weight="500">SGLang</text>
  <text x="640" y="228" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">and measure against vLLM</text>
  <path d="M400 212 L500 212" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" marker-end="url(#vs-choose-head)"/>
  <text x="450" y="204" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12" font-weight="400">yes</text>
  <path d="M210 240 L210 268" fill="none" stroke="var(--color-fg-dim)" stroke-width="1.5" marker-end="url(#vs-choose-head)"/>
  <text x="222" y="258" text-anchor="start" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12" font-weight="400">no</text>
  <rect x="20" y="268" width="380" height="56" rx="8" fill="var(--color-bg-elevated)" stroke="var(--color-accent)" stroke-width="1.5"/>
  <text x="210" y="292" text-anchor="middle" fill="var(--color-accent)" font-family="var(--font-sans)" font-size="14" font-weight="500">vLLM</text>
  <text x="210" y="312" text-anchor="middle" fill="var(--color-fg-dim)" font-family="var(--font-sans)" font-size="12.5">the default choice</text>
</svg>
</div>

| | vLLM 0.34 | SGLang 0.5.23 |
|---|---|---|
| Built by | PyTorch Foundation, Inferact | LMSYS, RadixArk |
| License | Apache 2.0 | Apache 2.0 |
| Models and hardware | the widest choice | slightly narrower |
| Startup on the test bench | 155 seconds | 85–100 seconds |
| Failure on lack of memory | after 2–4 minutes | after 20–30 seconds |
| Memory | measures and allocates by itself | you set the share |
| Config file | everything, including the model | everything except the model |
| API | OpenAI, Anthropic | OpenAI, Anthropic, Ollama, its own |
| Prefix cache | by blocks | a tree, more precise on short matches |
| State on the server | no | sessions in the native API |
| Open without a key | checks, metrics, documentation, `/invocations` | checks and metrics |
| Metrics | always on | turned on by a flag |
| Ready-made dashboard | works | the metric names need fixing |
| Tracing | one record per request | a tree of steps |
| Telemetry to the developers | yes, can be turned off | no |
| Router | a separate package | in the image; three projects |
| Speed, one request | the same | the same |
| Speed, 64 requests | 2415 tok/s | 1518 tok/s after tuning |

- **vLLM** is the default choice: more often works without tuning, more models and hardware, all the surrounding tooling supports it first.
- **SGLang** is for agents and long conversations with shared beginnings, for sessions on the server and phase separation on a large cluster. It requires memory tuning.
- **Both** are worth measuring if the service is large. Client code does not depend on the choice; the launch command, the config file and the dashboard change.
- **Neither,** if there is one user: [llama.cpp](/en/llama-cpp/) is simpler. With a small or uneven load a cloud API is cheaper.
