LLM inference engines: vLLM, llama.cpp, LiteRT-LM and others — speed, hardware, scaling

Hi!

Downloading a language model is not enough: you need a program to run it. It loads the weights into memory, turns text into tokens, pushes them through the neural network, picks the next token and makes sure there is enough memory for every user. Such a program is called an inference engine. Inference is the work of a model that has already been trained, as opposed to training it.

The engine decides how many answers per second one GPU produces, how many users it can hold, and whether the model runs at all — on a server, on a laptop or on a phone. There are dozens of engines, and they solve different problems:

• server engines — vLLM, SGLang, TensorRT-LLM — serve hundreds and thousands of users on datacenter GPUs;
• local engines — llama.cpp and MLX — run a model on your own computer;
• embedded engines — LiteRT-LM, ExecuTorch, ONNX Runtime GenAI — run inside a mobile app, on a laptop or in a browser.

Do not confuse an engine with the programs around it. Ollama and LM Studio are not engines: they download models, show a chat and answer over an API, while llama.cpp and MLX do the computing inside. NVIDIA Dynamo and llm-d are not engines either but a layer for a cluster that hands requests out between copies of an engine. Both get their own sections at the end of the article.

This article covers what each of them does, what makes it fast, what hardware it runs on and how it scales. Most of the space goes to three engines, one per class: vLLM, llama.cpp and LiteRT-LM. Everything below is the state as of early November 2026, according to documentation, repositories and public benchmarks; dates without a year are in 2026.

The commands in the article were checked on my laptop: an RTX 5070 Ti Laptop GPU with 12 GB of memory, a Ryzen 9 9955HX CPU, 64 GB of RAM, Windows 11. The measurements were made on it too: vLLM and llama.cpp under load, and LiteRT-LM on the GPU and the CPU.

How inference works: five concepts

Without these concepts a comparison of engines turns into a pile of meaningless numbers. If you know them already, skip straight to the map of engines.

Tokens and weights

A model works not with letters but with tokens — pieces of text from one character to a whole word long. It also produces its answer one token at a time.

Weights are the numbers obtained in training. A model "with 1.7 billion parameters" stores 1.7 billion such numbers. In BF16 format each number takes 2 bytes, so the whole model weighs about 3.4 GB. During inference the weights do not change: the engine only reads them and multiplies them with the data.

Prefill and decode: two phases with different bottlenecks

A model's answer is born in two phases, and each has its own bottleneck.

1. Prefill — the model reads the request. All the prompt tokens are processed at once, in one large matrix multiplication. The GPU is fully busy computing here, and the speed is limited by its compute power. This phase decides how quickly the first word of the answer appears.
2. Decode — the model writes the answer one token at a time. For every next token all the model's weights have to be read from memory, while there is little computation per token. The GPU waits for data more than it computes. Speed here is limited by memory bandwidth — how many gigabytes per second can be read.

This gives a simple estimate: generation speed for one user ≈ memory bandwidth ÷ size of the weights. You can see it on my laptop. llama.cpp generates Qwen3-1.7B in BF16 (3.2 GiB) at 131 tokens per second. The same model compressed to 4 bits (1.03 GiB) runs at 285 tokens per second: the weights are three times smaller, and generation is more than twice as fast. Reading the prompt barely changed: 13,000 and 15,700 tokens per second.

The same effect shows up clearly in Apple's measurements. The M5 chip processes the prompt 3.3–4.1 times faster than the M4: its GPU gained neural accelerators for matrix multiplication. But it generates only 19–27% faster — exactly as much as the memory bandwidth grew: 153 GB/s versus 120.

The KV cache

So as not to recompute the whole text at every step, the model saves intermediate results of the attention mechanism for every token it has already processed — the keys and values. That is the KV cache. It grows with every token, and for long conversations it is bigger than the model itself.

Let's count for the same Qwen3-1.7B. It has 28 layers, each with 8 key-value heads of 128 numbers, and each number takes 2 bytes. One token is 28 × 8 × 128 × 2 (key and value) × 2 bytes = 112 KiB. A conversation of 32 thousand tokens takes 3.5 GiB — more than the weights. A hundred users with such conversations take 350 GiB.

That is why the main thing that sets server engines apart is how they manage memory for the KV cache: how many conversations fit into the GPU at once, and how many of them do not have to be recomputed.

Batching: why a server serves many users at almost the cost of one

In the decode phase the GPU waits for data more than it computes. If at every step you generate a token not for one user but for 16 at once, the weights are read from memory once for all of them, while the computation grows sixteenfold — and finally keeps the GPU busy. Each user gets the answer a little slower, but in total the server produces almost 16 times more tokens. This is called batching.

Simple batching waits until a batch of requests has gathered and holds it until the longest answer is finished. Continuous batching adds and removes requests at every generation step: one answer is done, and the next request from the queue takes its place right away. All engines built for many users work this way today.

Metrics: what "a fast engine" means

Speed has several dimensions, and different engines optimise different ones:

• TTFT (time to first token) — how long you wait for the first token of the answer. It matters in chats and for agents.
• TPOT (time per output token) — the pause between tokens of one answer. 10 ms means 100 tokens per second per user. The inverse of TPOT is called interactivity: how many tokens per second one user sees.
• Throughput — how many tokens per second the server produces in total across all users, usually per GPU. It determines the price of a million tokens.
• Goodput — the same throughput, but counting only the requests that met the promised latencies (the SLA). A server that produces lots of tokens but makes you wait 10 seconds for the first one is of no use to anyone.

Throughput and interactivity pull in opposite directions: the more users in one batch, the more tokens in total and the slower the answer for each of them. That is why an honest benchmark shows not a single number but a curve: how many tokens per second per GPU the engine gives if every user is promised 30, 50 or 100 tokens per second.

Quantization

Quantization stores the weights not in 16 bits but in 8, 4 or even fewer. The model gets smaller, decode speeds up because there is less to read, and quality drops a little. Each class of engines has its own formats:

• GGUF — in llama.cpp and everything built on it;
• FP8, NVFP4, MXFP4, AWQ, GPTQ — in server engines;
• .litertlm — in LiteRT-LM.

FP8, NVFP4 and MXFP4 are 8- and 4-bit floating-point number formats that new GPUs compute in hardware: FP8 starting with NVIDIA Ada and Hopper, FP4 on NVIDIA Blackwell and AMD MI350.

Map of engines

The table lists the engines covered here and their versions as of early November. Scaling in this table means how much hardware the engine can put to work on one model.

Programs and the cluster layer are not engines: they use engines inside.

Engines that shut down or stalled are left out of the review:

• TGI (Text Generation Inference) from Hugging Face. It has been in maintenance mode since 11 December 2025, and now its repository is archived. Hugging Face recommends vLLM or SGLang instead, and llama.cpp and MLX for running locally.
• LoRAX from Predibase and IPEX-LLM from Intel — the repositories are archived.
• DeepSpeed-MII from the DeepSpeed project — the last commit was in June 2025.
• MediaPipe LLM Inference API from Google. Its documentation page now opens the LiteRT-LM documentation: Google is moving developers there.

How to choose in a minute

• A service for many users on your own GPUs → vLLM. SGLang is an equal alternative, and either can win on a particular model and hardware, so test both. NVIDIA only and you need the last percent of speed → TensorRT-LLM, usually together with Dynamo.
• Tens and hundreds of GPUs in Kubernetes → an engine plus a cluster layer: llm-d or NVIDIA Dynamo.
• A model on your own computer, for yourself → the llama.cpp engine directly or a program on top of it — Ollama or LM Studio. On a Mac — the MLX engine, including inside LM Studio and Ollama.
• A model that does not fit into the GPU → llama.cpp: it keeps part of the model in system RAM. For huge MoE models such as DeepSeek and Kimi — KTransformers.
• A model inside an Android or iOS app → LiteRT-LM. If the team lives in PyTorch — ExecuTorch. On iOS there is also Apple's system model through Foundation Models.
• A model in the browser → the Prompt API built into Chrome, WebLLM or transformers.js.

vLLM

vLLM is a Python server and library for language model inference; everything that needs speed is written in C++, CUDA and Triton. It can do more than anyone else: it supports more than two hundred model architectures, runs on GPUs from three vendors and on a dozen other accelerators, and scales from one GPU to a rack. When people say "serve a model on our own GPUs", they most often mean vLLM.

Where it came from and who builds it

vLLM appeared in 2023 in the Sky Computing Lab at the University of California, Berkeley, together with the paper on PagedAttention — a way to store the KV cache in pages, covered below. Even then the authors showed up to 24 times the throughput of the Hugging Face Transformers library and up to 3.5 times that of TGI.

Since May 2025 vLLM has been a PyTorch Foundation project under the Linux Foundation. In January 2026 its authors founded Inferact and raised 150 million dollars at an 800 million valuation; the round was led by Andreessen Horowitz and Lightspeed. The engine itself stays open, under the Apache 2.0 license.

The scale of the project: more than 2,000 code contributors and about 97 thousand stars on GitHub. In June 2026 alone 1,918 commits were merged into the main branch — 64 a day. A release comes out every two weeks, and every candidate goes through the full test suite, nightly performance runs and a model accuracy check. The latest version at the time of writing is 0.33.0 of 2 November: 927 commits from 389 authors in two weeks.

How it is built

The vllm serve command starts not one program but several processes:

• The API server accepts HTTP requests, turns text into tokens, applies the chat template, loads images and streams the answer back to the client as it is generated.
• The engine core is the scheduler and the KV cache manager. It spins in an endless loop and at every step decides which requests, and how many of their tokens, go into the next model run.
• Worker processes — one per GPU. Each holds its part of the weights and does the computation.

The processes exchange messages over ZeroMQ. That has a practical consequence the documentation warns about: every process needs its own physical CPU core. On a server with four GPUs that is at least six cores. With fewer cores the GPUs sit idle waiting for the scheduler, especially in virtual machines.

The scheduler does not split work into prefill and decode phases. It has a token budget per step (max_num_batched_tokens) and fills it like this:

1. First it takes one token from every request that is already writing its answer.
2. It gives the remaining budget to the prompts of new requests. A long prompt is cut into chunks and processed over several steps — this is called chunked prefill.

This way a new user's long prompt does not stop generation for everyone else. The default budget depends on the GPU: for a server on cards with less than 70 GB it is 2,048 tokens, on the H100 and H200 — 8,192, on the B200 and B300 — 16,384. A smaller budget gives more even pauses between tokens, a bigger one gives a faster first token.

If there is not enough KV cache for everyone, the scheduler preempts the most recently admitted request: it frees its memory and, once there is room again, computes its prompt from scratch.

PagedAttention: the KV cache in pages

Before vLLM, servers gave each request one contiguous chunk of KV-cache memory for the maximum possible answer length. A request might write 50 tokens while memory was reserved for 2,000. The vLLM authors measured that 60–80% of KV-cache memory was wasted this way.

PagedAttention solves this the way an operating system solves the same problem with RAM — with pages. The comparison from the authors' paper: blocks are pages, tokens are bytes, requests are processes.

1. KV-cache memory is cut in advance into blocks of a fixed size, for example 16 tokens each.
2. Every request has a block table — which physical blocks hold its tokens, and in what order.
3. Blocks are handed out as the answer grows. At most one partly filled block per request is wasted; by the authors' measurements, less than 4% of memory.

Several times more concurrent requests fit into the same GPU, and throughput grows with them.

The second consequence is shared blocks. If two requests start the same way — a shared system prompt, one conversation history, several answer variants to one question — their tables point to the same physical blocks. The KV cache for the common beginning is stored and computed once.

The idea became the standard: a paged KV cache exists in SGLang, in TensorRT-LLM and even in the vLLM plugin for the Mac. Meanwhile the original PagedAttention kernel that vLLM started with was removed from the code in July 2026 (version 0.25.0). The paged cache is now read by general attention kernels — FlashAttention, FlashInfer, Triton and others — and what remains of the original implementation is the memory layout.

Prefix caching

vLLM computes a hash of every filled block together with everything that came before it, and does not throw finished blocks away when a request ends. A new request with the same beginning gets them right away and skips their prefill. This is called automatic prefix caching, and it is on by default.

For agents this is the main source of speed. An agent sends the whole history to the model again at every step, and almost all of the prompt matches the previous one. In the AgentX benchmark, which replays real sessions of coding agents, more than 96% of the prompt can theoretically be reused.

The cache has a flip side worth remembering when you measure: run the same test twice, and the second run gets ready-made blocks and shows unrealistically good numbers. I ran into it myself: in a repeated run with the same prompts, vLLM's time to first token dropped by half.

CUDA graphs, torch.compile and Model Runner V2

Models in vLLM are written in PyTorch. At startup vLLM prepares them for fast work in two stages:

1. Compilation with torch.compile. Chains of small operations are fused into large GPU kernels, and data goes to memory less often.
2. Capturing CUDA graphs. A CUDA graph is a pre-recorded sequence of kernel launches for a typical batch size. The GPU then replays it in one go, without Python at every step. On small batches this gives a noticeable share of the speed.

The price is a long startup: from tens of seconds to several minutes. Optimization levels from -O0 to -O3 shift the balance between startup time and speed, and the --enforce-eager flag turns off both compilation and graphs for quick debugging. The compiled artefacts are saved to disk, in ~/.cache/vllm, and repeated starts are faster.

On my laptop the first start of a 1.7-billion-parameter model took three minutes: 13 seconds of compilation, 39 seconds of graph capture, the rest was loading weights and preparation. A repeated start took 79 seconds, of which compilation took 0.16 seconds. Most of the remaining time went into loading the weights: in Docker on Windows they were read from the Windows disk in 34–38 seconds, and from the WSL file system in 0.4 seconds.

In 2026 the heart of the engine was rewritten. Model Runner V2 is a new implementation of the part that prepares the inputs and runs the model on the GPU. Input preparation moved to the GPU, and scheduling became asynchronous: while the GPU computes one step, the CPU is already planning the next. Since version 0.29.0 (September) Model Runner V2 is the default for all models. In parallel, work is under way on an API server in Rust that should take receiving and parsing requests off Python; for now it is experimental.

For fast restarts there is now the vllm preload daemon, which keeps already-quantized weights in GPU memory while the engine restarts, and snapshots of an already initialized engine — still experimental.

Speculative decoding

In the decode phase the GPU is underused: it waits for memory. Speculative decoding puts that idle time to work.

1. A small, fast "draft" guesses several next tokens.
2. The big model checks all the guessed tokens in one pass — the same way it reads a prompt.
3. The matching tokens are accepted; at the first mismatch the draft is discarded, and the big model's token is taken instead.

The answer is the same as the big model would give on its own, but each step yields several tokens. The ways to guess in vLLM are:

• EAGLE (EAGLE-3, EAGLE 3.1, P-EAGLE) — a small add-on that predicts tokens from the model's own internal states;
• MTP (multi-token prediction) — heads built into the model that predict several tokens ahead at once; DeepSeek, Qwen3.5, Gemma 4 and others have them;
• a draft model — a separate small model from the same family;
• n-gram and suffix decoding — no extra model: the continuation is looked up in the prompt itself and in past answers. This works well when the model edits code or retells a document;
• DFlash and DSpark — drafts based on block diffusion that propose a whole block of tokens in one pass. They appeared in 2026.

The documentation honestly warns: the gain is large under light load and smaller under heavy load. When the GPU is already busy with a large batch, checking someone else's guesses stops being free. That is why adaptive modes appeared that decide by themselves how many tokens to guess. How big the gain can be shows in my LiteRT-LM measurement below: MTP sped up Gemma 4 E2B generation on the laptop GPU by 1.7 times.

Quantization

vLLM understands almost all server formats: FP8, NVFP4, MXFP4 and MXFP8, INT8 and INT4, GPTQ and AWQ checkpoints, the compressed-tensors format, NVIDIA ModelOpt, AMD Quark and TorchAO. GGUF is supported experimentally, and bitsandbytes moved into a separate plugin in version 0.28.0. To quantize your own models for vLLM, the same developers offer the LLM Compressor library.

The KV cache is quantized separately. FP8 halves it, which doubles the number of concurrent conversations. In 2026 4-bit variants were added: NVFP4 for NVIDIA and formats of their own for AMD. What exactly gets faster depends on the GPU generation: FP8 is computed in hardware starting with NVIDIA Ada and Hopper, FP4 on Blackwell and AMD MI350.

API and features

The vLLM server mirrors the OpenAI API: completions, chat completions, the Responses API, embeddings, speech recognition and translation. Besides that, it understands the Anthropic Messages API — the documentation has instructions for connecting Claude Code to it — and gRPC.

Features you need in production:

• structured output — an answer that strictly follows a JSON schema or a grammar, through the xgrammar or guidance libraries;
• tool calls and reasoning — for dozens of model families there are parsers that turn the model's text into function calls and separate the reasoning from the answer;
• many LoRA adapters on top of one base model in one batch — this way one GPU serves dozens of fine-tuned variants;
• multimodality — images, video and audio as input;
• embeddings and reranking — models for search run on the same server;
• sleep mode — the weights are temporarily offloaded from the GPU to switch quickly to another model.

New large open models are usually supported on release day: model developers prepare support together with the vLLM team in advance.

Scaling: from one GPU to a rack

When a model does not fit into one GPU, or one card is not fast enough, vLLM spreads the work across several. There are five ways to do it, and they are combined.

• Tensor parallelism (TP) cuts every weight matrix into parts. Each GPU computes its part of every layer, and after each layer the cards exchange results. This needs a very fast link between the cards — NVLink inside one server. It is the most common way to run a 70-billion-parameter model on several cards.
• Pipeline parallelism (PP) gives the cards whole layers: the first computes layers 1 to 20, the second 21 to 40. Data is exchanged only at the boundaries, so this lets you stretch a model across several servers with a slower network. The price is that the cards wait for each other.
• Data parallelism (DP) means several copies of the model, each serving its own requests. Throughput grows, not the maximum model size.
• Expert parallelism (EP) is for Mixture of Experts (MoE) models. Each layer of such a model has tens or hundreds of "experts", and only a few of them are switched on for each token. The experts are spread across the cards, and tokens are sent to the ones they need. Combining DP for the attention layers with EP for the experts across tens of cards is called wide-EP; the records for DeepSeek, Kimi and similar models rest on it.
• Context parallelism (CP) cuts one very long request of hundreds of thousands of tokens across cards. It works differently for prefill and decode.

By default vLLM joins several servers through Ray, but it can do without it. The KV cache travels between machines through pluggable connectors: NIXL from NVIDIA, Mooncake, LMCache, MoRI-IO from AMD.

Splitting prefill and decode (disaggregated serving, P/D). Prefill loads the compute, decode loads the memory. When both phases run on the same cards, a new user's long prompt briefly slows generation down for everyone else. The solution is separate vLLM instances: some only read prompts, others only generate answers, and the KV cache is passed between them over the network. Each group can be tuned and scaled separately. The vLLM documentation stresses that this is a way to control latency, especially the worst-case latency, not to raise throughput, and the feature is still marked experimental. Since late 2025 image encoders of multimodal models are separated the same way, and in 2026 a plugin appeared that separates even the attention layers from the expert layers.

A multi-tier KV cache. A KV cache that does not fit into the GPU can be offloaded to RAM, to disk and to object storage, and then loaded back instead of being recomputed. For agents with long histories this is a direct saving on prefill.

Elastic expert parallelism — GPUs can be added to and removed from an MoE deployment without restarting the server.

How fast all of this is shows in the developers' own numbers — these are their measurements, not independent ones:

• DeepSeek on the H200. With wide-EP and prefill/decode separation — 2,200 tokens per second per GPU in a multi-node deployment, December 2025.
• Kimi K3 on AMD versus NVIDIA. On eight AMD MI355X — 7,567 tokens per second per GPU versus 6,083 on eight NVIDIA B300 at the same speed of 50 tokens per second for each user. The post was written by AMD and Embedded LLM; the numbers come from the InferenceX benchmark.
• DeepSeek-V4.1-Flash. On the GB200 decode speed is within 6% of DeepSeek's own engine, DeepInfer, October 2026.

How to hand out requests between dozens of vLLM instances is covered in the section on the layer above engines for clusters.

Hardware

Plugins live in separate repositories and follow their own schedule, usually trailing the main vLLM by a few releases.

vLLM does not support Windows — only through WSL2 or Docker. That is exactly how it runs on my laptop.

Weak spots

• Linux only. On Windows — WSL2 or Docker. The CUDA image weighs 35 GB.
• Slow startup. Loading weights, compiling, capturing CUDA graphs — minutes, not seconds.
• It grabs GPU memory up front. By default vLLM takes 92% of the GPU's memory, assuming the card belongs to it alone. On my laptop the browser and the desktop held about a gigabyte, and vLLM simply refused to start: there was less than 92% free. On a shared GPU you have to set the share by hand with --gpu-memory-utilization.
• No faster for a single user. On a home GPU llama.cpp is just as fast for one stream, and compressed GGUF models are smaller and faster still.
• Fast change. A release every two weeks changes dependencies and removes old things: the latest release, for example, dropped the CUDA 12.9 builds and two speculative decoding methods.
• Large deployments take engineering work. Choosing the combination of TP, DP and EP, the token budgets and the KV-cache connectors is a task of its own.

How to run it

On Linux with an NVIDIA GPU and Python 3.11 to 3.14, vLLM installs with one command. The second one downloads prebuilt FlashInfer kernels so the engine does not build them on first start. The model is downloaded from Hugging Face when the server first starts. The --gpu-memory-utilization 0.6 flag is needed if other programs use the GPU too; on a dedicated server you can drop it:

uv pip install vllm
vllm download-kernels
vllm serve Qwen/Qwen3-1.7B --max-model-len 8192 --gpu-memory-utilization 0.6

The same in Docker. The --ipc=host flag is needed because vLLM's processes pass data to each other through shared memory:

docker run --gpus all -p 8000:8000 --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:v0.33.0 Qwen/Qwen3-1.7B \
  --gpu-memory-utilization 0.6 --max-model-len 8192

The server speaks the OpenAI API on port 8000:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "Qwen/Qwen3-1.7B", "messages": [{"role": "user", "content": "What is a KV cache? Answer in two sentences."}]}'

You can also use vLLM without a server, straight from Python. This is handy for batch processing: the engine gathers requests into batches by itself and keeps the GPU fully busy. Note the if __name__ == "__main__" check at the end: without it the script fails when the engine starts.

from vllm import LLM, SamplingParams


def main():
    # The model is downloaded from Hugging Face on first run
    llm = LLM(model="Qwen/Qwen3-1.7B", max_model_len=4096, gpu_memory_utilization=0.6)
    params = SamplingParams(temperature=0.7, max_tokens=512)

    # A hundred questions go to the model in shared batches, not one by one
    conversations = [
        [{"role": "user", "content": f"Give one fact about the number {n}."}]
        for n in range(100)
    ]
    # Qwen3 reasons first by default; reasoning is turned off here
    outputs = llm.chat(conversations, params, chat_template_kwargs={"enable_thinking": False})

    for output in outputs[:3]:
        print(output.outputs[0].text)


# vLLM starts the engine in a separate process, and that process imports this file again.
# Without the check it would reach the creation of LLM once more and fail with an error
if __name__ == "__main__":
    main()

On my laptop a hundred such requests were processed in 5 seconds — about 1,950 answer tokens per second. One request at a time, the same GPU generates about 117 tokens per second: that is batching in action.

SGLang

SGLang is vLLM's main rival in both purpose and features. It is built by LMSYS, a non-profit organisation of researchers known for the Chatbot Arena leaderboard. In 2026 a company called RadixArk formed around the project, according to TechCrunch at a 400 million dollar valuation. Apache 2.0 license, version 0.5.23 of 2 November, about 39 thousand stars on GitHub.

How it differs from vLLM:

• RadixAttention. The prefix cache is kept in a radix tree — a tree where every branch corresponds to the common beginning of several requests. The engine itself finds the longest match among everything in the cache and reuses it. In vLLM block hashing gives a similar result. From the start the SGLang authors focused on programs made of many model calls with shared prefixes — agents and branching conversations.
• A scheduler without idle time. The CPU prepares the next step while the GPU runs the current one, and the GPU does not wait for Python. In vLLM asynchronous scheduling does the same.
• A cache-aware router. The SGLang Model Gateway, written in Rust, spreads requests across instances so that each request lands where its prefix has already been computed.
• Large MoE deployments. The SGLang team published detailed open recipes for DeepSeek on tens of GPUs: on 96 H100s with prefill/decode separation and wide-EP, later on a GB200 NVL72 rack. On the GB200 they got 26 thousand input and 13 thousand output tokens per second per GPU for DeepSeek-R1. These are their own 2025 measurements; I did not find newer ones.

Hardware: NVIDIA from Ampere to Blackwell, including RTX and Jetson; AMD MI300X–MI355X; Intel Arc and Xeon; Google TPU; Huawei Ascend; Apple Silicon; China's Moore Threads. In practice SGLang and vLLM now catch up with each other every two weeks, and the choice between them is a matter of the specific model and hardware. For example, in MLPerf Inference 6.1 Crusoe ran DeepSeek-R1 on 512 AMD MI355X GPUs on SGLang. Before choosing, run both on your own workload.

TensorRT-LLM

TensorRT-LLM is NVIDIA's own engine, and it runs only on NVIDIA GPUs. On them, though, it is often the fastest: NVIDIA ships kernels for new hardware in it before they reach the open engines — FP4 on Blackwell, the links between the 72 GPUs of an NVL72 rack, prefill/decode separation. Apache 2.0 license. The stable version is 1.2.1 from April; the 1.4 branch comes out as pre-releases.

Working with it used to start with a long "engine build": the model was compiled ahead of time for a specific GPU and specific batch sizes. Version 1.2 removed that path, and TensorRT-LLM now runs on top of PyTorch, like vLLM and SGLang. The server is started with trtllm-serve and speaks the OpenAI API. Inside are continuous batching, a paged KV cache, EAGLE-3 and MTP speculative decoding, and wide-EP.

The downsides are NVIDIA only, frequent breaking changes, fewer supported models than vLLM, and a long first start: for example, the kernels for DeepSeek-V4 on Blackwell take 5–10 minutes to compile. TensorRT-LLM is usually chosen by those who squeeze the last percent out of a large NVIDIA fleet and are ready to pay for it in engineering time. For clusters NVIDIA ships the Dynamo layer to go with it — see the section on the layer above engines.

Other server engines

• LMDeploy from the Shanghai AI Laboratory (the authors of the InternLM models). Two engines — TurboMind in C++ and CUDA and an engine on PyTorch; it runs on NVIDIA and Huawei Ascend. Version 0.18.0 from September.
• KTransformers — for huge MoE models on one server with few GPUs. The attention layers run on the GPU and the experts on the CPU with Intel AMX and AVX-512 instructions, so a model with hundreds of billions of parameters fits into the server's RAM. It integrates with SGLang. Version 0.7.2.
• OpenVINO Model Server from Intel — a server for Intel CPUs, integrated and discrete GPUs and NPUs.
• xLLM from JD and MindIE from Huawei — engines for Chinese accelerators, Huawei Ascend first of all. MindIE's features are gradually moving into the vllm-ascend plugin.

Benchmarks: which numbers to trust

You cannot compare engines "in general": the result depends on the model, the hardware, the request lengths, the number of users, the promised latencies and how much time engineers spent on tuning. There are two sources you can trust more than the rest.

InferenceX (formerly InferenceMAX) from the analyst firm SemiAnalysis. Every night it runs vLLM, SGLang and TensorRT-LLM (including under Dynamo) on NVIDIA GPUs from the H100 to the Vera Rubin NVL72, on AMD MI300X, MI325X and MI355X, and on TPU v7x. The results are "throughput per GPU versus speed for each user" curves and the price of a million tokens. Since August it has included the AgentX test — real sessions of coding agents, with long histories and shared prefixes.

The conclusions of the February InferenceX v2 release on DeepSeek-R1 are these. In configurations with prefill/decode separation, wide-EP and FP4, NVIDIA leads with SGLang and TensorRT-LLM. The AMD MI355X with SGLang is on par with NVIDIA on a single server in FP8. But on FP4 with split phases AMD lags behind: the kernels and communication libraries for ROCm are still immature. The authors themselves note the limitations: in v2 the prompts consisted of random tokens and the prefix cache was off, so the optimization that matters most for agents did not take part in the test.

MLPerf Inference from the MLCommons consortium — rounds every six months, with results checked by the other participants. In round 6.0 in April NVIDIA submitted results on TensorRT-LLM and Dynamo; in round 6.1 in September Crusoe ran DeepSeek-R1 on 512 AMD MI355X GPUs on SGLang. MLPerf shows the ceiling of the hardware well, but vendor teams tune it for weeks, and your configuration will not run that fast.

Read the other numbers with three questions in mind:

1. Who measured? Almost all publications come from engine developers or hardware vendors. Even an honest vendor measurement shows the configuration that got the most work.
2. What was counted? For example, in one of vLLM's October posts the throughput on the AgentX test is 954 thousand tokens per second per GPU. But that number includes cached input tokens, which cost almost nothing, and agent sessions consist almost entirely of them. The number is correct, but you cannot compare it with "answer tokens per second".
3. What workload? Single-H100 comparisons from hosting companies' blogs contradict each other by tens of percent, and in places repeat the same numbers.

The most reliable thing is to measure yourself, on your model and your request profile. There is vllm bench serve for that — it can load any server with the OpenAI API, not just vLLM — and GuideLLM. That is what I did.

A laptop measurement: vLLM and llama.cpp under load

The question asked most often: if llama.cpp is just as fast, why do you need vLLM? I ran both on the same GPU with the same model.

The setup:

• Hardware — an RTX 5070 Ti Laptop with 12 GB, Windows 11. About a gigabyte of GPU memory is held by the desktop and the browser.
• Model — Qwen3-1.7B in BF16, unquantized: the same weights in both engines. For llama.cpp, a GGUF file in BF16.
• vLLM 0.33.0 — in Docker, since there is none for Windows. KV-cache memory is exactly 16,384 tokens, up to 16 requests at once.
• llama.cpp b12233 — the regular Windows build with CUDA 13: llama-server with 16 slots and a shared KV cache of the same 16,384 tokens.
• Load — the same vllm bench serve client for both: a prompt of 512 random tokens, an answer of exactly 256 tokens, from 1 to 16 concurrent requests. Each level gets new prompts so that the prefix cache does not help.

Here is the command that loaded the server; it works with any OpenAI-compatible server:

vllm bench serve --backend openai --base-url http://localhost:8000 \
  --endpoint /v1/completions --model Qwen/Qwen3-1.7B \
  --dataset-name random --random-input-len 512 --random-output-len 256 \
  --num-prompts 96 --max-concurrency 16 --ignore-eos --seed 5016

The results — total generation speed, median time to first token and median pause between tokens:

What follows from this:

1. For one user there is no difference. 116 and 120 tokens per second — both programs hit the same GPU memory.
2. As the load grows, vLLM pulls ahead. At 16 concurrent requests it produces 1.4 times more tokens, and the pause between tokens for each user grows more slowly: from 8.4 to 12.5 ms versus 8.1 → 17.6 ms for llama.cpp. A step for many requests at once is vLLM's main scenario, and it specialises in it; for llama.cpp it is a secondary one.
3. This is not the limit. On my laptop memory limited the test to 16 requests. On a server GPU with 80 GB vLLM can hold hundreds of concurrent conversations, and the bigger the batch, the more the efficiency of the step matters.

Two findings along the way worth knowing about.

llama-server has a setting that gets in the way under load. In recent versions it saves every finished conversation to RAM by default (the --cache-ram parameter, 8 GiB by default) so it can resume it instantly if the user comes back. The saving happens at the moment a new request arrives — and the new request waits. In my first run with the default settings the time to first token grew with the number of users up to half a second, and in one run with 8 users some requests waited more than a minute for their first token. In Docker it was even more visible — up to 3.5 seconds. With the -cram 0 flag everything evened out; the table above is with it. If you have many different users rather than one who keeps returning to their chat, turn this cache off.

The prefix cache distorts repeated measurements. My first vLLM run used the same prompts at every load level, and the time to first token with two users came out half of that with one: vLLM simply took from the cache what it had already computed. When measuring, change --seed or restart the server.

llama.cpp

llama.cpp is an engine in C and C++ that started running language models on ordinary computers in 2023. Georgi Gerganov wrote it to run LLaMA on a MacBook, and it has been the most portable engine ever since: it runs on Windows, Linux, macOS, Android and iOS, on a CPU without a GPU and on almost any GPU. Under the hood is the author's own tensor library, ggml. Ollama, LM Studio, Jan, LocalAI, llamafile and dozens of other programs are built on llama.cpp.

MIT license, about 134 thousand stars on GitHub. In February 2026 the ggml.ai team moved to Hugging Face and keeps working on the project; the project itself stays open and community-driven. There are no fixed releases: every merged pull request becomes a new build, and more than ten of them come out a day. As of 9 November the current build is b12233.

Hardware

llama.cpp's main strength is its backends. The same model runs on:

Prebuilt binaries come with every version: for Windows — CPU, CUDA 12 and 13, Vulkan, SYCL, ROCm, OpenVINO and Adreno builds; for Linux, macOS and Android — their own sets. There are Docker images for CUDA, ROCm, Vulkan, Intel and other backends, and packages: winget install llama.cpp on Windows and brew install llama.cpp on macOS.

Enthusiasts collect speed on different hardware in open GitHub discussions, all on the same model, Llama 2 7B in the 4-bit Q4_0 format. A few generation numbers from those tables: RTX 5090 — 290 tokens per second, AMD MI300X — 233, Radeon RX 7900 XTX — 167, Apple M5 Max — 120. The numbers come from different llama.cpp builds, so they compare only roughly.

GGUF and quantization

Models for llama.cpp are stored in the GGUF format: one file holds the weights, the tokenizer, the chat template and a description of the architecture. Popular open models usually appear on Hugging Face in GGUF within the first days after release, right away in a dozen quantization variants.

The variants differ in bits per weight. Here is how much Llama 3.1 8B weighs in different formats, according to the table in the repository:

Q4_K_M is a sensible default: the model is three times smaller, and the quality loss is usually small. Q8_0 is almost indistinguishable from the original. The IQ formats with two or three bits are for when the model would not fit otherwise; quality on them already drops noticeably. For the most careful results models are quantized with an importance matrix (imatrix): llama.cpp runs a sample of text through the model and compresses more gently the weights that affect the answer more. In 2026 4-bit floating-point formats were added: MXFP4, in which the gpt-oss models were released, and NVFP4 — an NVIDIA format that Blackwell GPUs compute in hardware.

llama-server

llama-server is the server that turns llama.cpp into a replacement for a cloud API. Recent builds added a short form of it: llama serve. What it can do:

• APIs. OpenAI — chat completions, completions, the Responses API, embeddings; the Anthropic Messages API; reranking; streaming speech recognition over WebSocket. A web chat UI is built in.
• Several users. The -np parameter sets the number of slots — concurrent conversations. Continuous batching is on by default. With the -kvu flag the slots share one KV-cache pool; if the number of slots is not set, it is on automatically.
• Prompt cache. The server reuses the computed KV cache if a new request starts the same way as the previous one, and keeps finished conversations in RAM. Under load from different users the latter gets in the way — see the measurement above.
• Speculative decoding. A draft model, EAGLE-3, MTP since May 2026, DFlash, DSpark, and looking up the continuation in text already written (n-gram).
• Images, audio and video as input, if the model has a multimodal projector.
• Tools. Function calls through chat templates, a built-in MCP client, reasoning control: on, off, or with a budget.
• Several models. In router mode the server loads a model from a folder on the first request to it and unloads extra ones when memory runs out.

Downloading a model from Hugging Face and starting the server is one command; the suffix after the colon picks the quantization variant:

llama-server -hf unsloth/Qwen3-1.7B-GGUF:Q4_K_M --port 8080

A server for several users needs more flags. Here is the configuration from my measurement: all layers on the GPU (-ngl all), FlashAttention (-fa on), 16 slots (-np 16) with a shared KV-cache pool of 16,384 tokens (-c 16384 -kvu), and no saving of conversations to RAM (-cram 0):

llama-server -m Qwen3-1.7B-BF16.gguf -ngl all -fa on -np 16 -c 16384 -kvu -cram 0

Without -kvu the context is split between the slots equally, and each gets only 1,024 tokens.

A model bigger than GPU memory

Here llama.cpp has no competition. The -ngl parameter sets how many layers of the model go onto the GPU; the CPU computes the rest from system RAM. By default llama.cpp picks as many layers as will fit. A model that does not fit into the GPU still runs, just slower.

For MoE models there is a better trick. In such models only a small share of the experts works for each token, while the attention layers always work. If you keep attention and the shared parts on the GPU and move the experts to RAM, the CPU gets little work per token and the speed drops much less. The flags --cpu-moe (all experts in RAM) and --n-cpu-moe N (experts of the first N layers) do this.

An example from my laptop. Qwen3.6-35B-A3B is an MoE model with 35 billion parameters, of which about 3 billion work for each token. In Q4_K_M quantization it weighs 20.6 GiB — almost twice the GPU memory. Here is how to run it, keeping the experts of 28 layers in RAM:

llama-server -m Qwen3.6-35B-A3B-UD-Q4_K_M.gguf --n-cpu-moe 28 -c 16384

Speed with different ways of splitting the model, measured with llama-bench:

Almost 50 tokens per second of generation from a 35-billion-parameter model on a laptop is faster than a person reads. The weak spot is reading the prompt: 140–230 tokens per second, so the model will read a 10-thousand-token document for about a minute. The first request to the server is also slower than the rest — 31 tokens per second versus 44–49: the weights are paged in from disk on first access. You have to pick --n-cpu-moe by hand: the smaller the number, the more experts on the GPU and the faster it runs, as long as memory lasts.

Several GPUs and several machines

On several GPUs in one computer llama.cpp by default hands them layers in turn. That lets you run a large model but does not speed it up: the cards work one after another. In April 2026 experimental tensor parallelism (-sm tensor) appeared, in which the cards compute each layer together. In the author's measurements on two RTX 4090s, Gemma 4 31B went from 44 to 59 tokens per second of generation, while the MoE model Gemma 4 26B, on the contrary, got slower. The speed-up exists only on CUDA so far, and the author says plainly that the feature is not ready for production.

Several machines are joined through the RPC backend: the main node hands parts of the model to the others over the network. Besides TCP, transfer over RDMA is now supported. The documentation calls RPC an experiment and warns that running it on an open network is not safe.

What llama.cpp cannot do

• Hundreds of users on one GPU. Slots and continuous batching exist, but under load llama.cpp falls noticeably behind vLLM, and it lacks server mechanisms such as prefill/decode separation, cache-aware routing between instances and wide-EP.
• Scaling to a cluster. Tensor parallelism and RPC are experiments.
• Day-one support for models — not always. A new architecture has to be implemented in ggml by hand, and sometimes this takes weeks, especially for multimodal and hybrid models.

But for one user on their own hardware llama.cpp is the simplest and often the fastest option: one model file, one program, start-up in seconds.

MLX

MLX is Apple's machine learning library for its own chips, and mlx-lm runs language models on it, with an OpenAI-compatible server. Its main advantage is unified memory: on Apple Silicon the CPU and the GPU work with the same RAM, and MLX does not copy data between them. A Mac with 128 or 512 GB of unified memory runs models that would need several GPUs on a PC.

Several Macs can be joined: MLX spreads the model across machines connected with Thunderbolt 5 cables, and starting with macOS 26.2 passes data between them over RDMA, bypassing the CPU. In Apple's measurements from October 2026, four Mac Studios with the M5 Ultra generated Qwen 3.8 27B in BF16 at 51 tokens per second versus 20 on one. MLX also has a CUDA backend for Linux, but its main platform is the Mac.

I did not find independent comparisons of MLX and llama.cpp on the same Mac with the same methodology: blogs give different results, usually with MLX ahead on generation.

You can run vLLM on a Mac too: the vllm-metal plugin puts vLLM's scheduler and paged KV cache on top of MLX. It is useful if the Mac serves several users: on a batch of eight requests it finished faster than mlx-lm and llama.cpp.

Other local engines

• ExLlamaV3 with the TabbyAPI server — its own EXL3 quantization format and tensor parallelism for home GPUs; NVIDIA only.
• ik_llama.cpp — a fork of llama.cpp by the author of many of its quantization formats: formats and CPU and CUDA optimizations of its own.
• mistral.rs — an engine in Rust with continuous batching and a paged KV cache; CUDA, Metal and CPU.
• OpenVINO GenAI from Intel — for Intel CPUs, GPUs and NPUs, including in laptops.

LiteRT-LM

LiteRT-LM is Google's engine for running language models right on the user's device: in a phone, a laptop, a browser, on a Raspberry Pi. The model works without a network, data goes nowhere, and you do not pay for tokens. Google itself calls it a production-ready layer that runs LLMs on top of LiteRT, and uses it in its own products: the Gemini Nano model built into Chrome, features of Chromebook Plus and Pixel Watch run on it. Apache 2.0 license, version 0.19.1 of 5 November.

Where it came from

Google has three layers with similar names, and they are easy to mix up:

1. LiteRT is the former TensorFlow Lite, renamed in 2024. It is a general on-device neural network engine: it executes a model's graph on the CPU, GPU or NPU.
2. LiteRT-LM is a layer on top of LiteRT made specifically for language models. It adds what a plain neural network engine lacks: a tokenizer, a KV cache, a generation loop, sampling, conversation history, function calls.
3. MediaPipe LLM Inference API is the old high-level API through which Gemma was run on Android, iOS and in the browser in 2024–2025. Google is now moving developers from it to LiteRT-LM, and its documentation page opens the LiteRT-LM documentation. Do not start a new project on it.

Platforms and hardware

On the CPU LiteRT-LM computes through the XNNPACK library. On the GPU — through ML Drift, Google's own kernels: OpenCL on Android, Metal on Apple devices, WebGPU in the browser. On Windows, judging by the log of my run, the GPU is used through WebGPU too, so CUDA is not needed. On the NPU it works with Qualcomm and MediaTek chips on Android, and on Windows with Intel NPUs through OpenVINO, still in early preview. A model for an NPU is usually compiled ahead of time for a specific chip, which is why Hugging Face has separate files of one model for different phone processors.

Languages and APIs:

• stable — Kotlin (Android and the regular JVM), Python, C++ and the command-line tool;
• early preview — Swift for iOS and macOS and JavaScript for the browser;
• community — Flutter.

Models and format

A model for LiteRT-LM is one .litertlm file holding the weights, the tokenizer and metadata. Ready models are published by litert-community on Hugging Face: Gemma 4 E2B and E4B (E stands for effective: part of the weights does not have to be kept in fast memory, and in memory use the model behaves like one with 2 or 4 billion parameters, although it has more) and Gemma 4 12B, Gemma 3n, Qwen3 and Qwen3.5, Phi-4-mini, FunctionGemma for function calling, EmbeddingGemma 2 for embeddings, and speech recognition and synthesis models. You can convert your own model from Hugging Face with the litert-torch library.

Features:

• images and audio as input for multimodal models;
• function calls and answers that follow a JSON schema or a grammar; since version 0.19 the constraint is applied right on the GPU;
• speculative decoding through MTP for Gemma 4 — Google promises up to 2.2 times faster on phone GPUs;
• a conversation API — the engine keeps the conversation history itself, and a conversation's KV cache can be saved and restored so that the history is not read again;
• LoRA adapters, embeddings, a server with the OpenAI API through litert-lm serve.

Speed

Google's measurements for Gemma 4 E2B (a 2.58 GB file) are vendor numbers; the prompt length is not stated:

On my laptop I ran the same model with the built-in benchmark — 256 prompt tokens and 256 answer tokens:

Both tables show the same thing as the beginning of the article. The GPU reads the prompt 5–10 times faster — that is compute. But on the Samsung it generates almost as fast as the CPU, 52 and 47 tokens per second: generation is limited by memory, and the phone's CPU and GPU share it. On the iPhone the GPU is twice as fast here too. MTP on the laptop GPU sped up generation by 1.7 times. The benchmark feeds synthetic text, and on real text the share of correctly guessed tokens and the gain may differ.

Phones come with two caveats that tables do not mention. The first is heat: during long generation a phone lowers its clocks, and speed drops within minutes — on an iPhone by almost half after just a couple of runs. The second is memory: the systems kill apps that take too much RAM, which is why phones run models with 1–4 billion parameters in 4-bit quantization.

How to try it

The quickest way is the command-line tool. It downloads the model from Hugging Face and runs on Windows, macOS, Linux and Raspberry Pi:

uv tool install litert-lm
litert-lm run --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
  gemma-4-E2B-it.litertlm --backend=gpu \
  --prompt="What is a KV cache? Answer in two sentences."

Measure speed on your own hardware with the same tool:

litert-lm benchmark --from-huggingface-repo litert-community/gemma-4-E2B-it-litert-lm \
  gemma-4-E2B-it.litertlm --backend gpu --speculative-decoding on

From Python — through the litert-lm-api package. The model here is the same file, downloaded with hf download litert-community/gemma-4-E2B-it-litert-lm gemma-4-E2B-it.litertlm --local-dir .:

import litert_lm

# Backend.CPU() works everywhere, Backend.NPU() on Android with a supported chip
with litert_lm.Engine("gemma-4-E2B-it.litertlm", backend=litert_lm.Backend.GPU()) as engine:
    with engine.create_conversation() as conversation:
        # The answer arrives in pieces; the conversation keeps its history itself
        stream = conversation.send_message_async("What is a KV cache? Answer in two sentences.")
        for chunk in stream:
            for item in chunk.get("content", []):
                if item.get("type") == "text":
                    print(item["text"], end="", flush=True)

In a Kotlin app the dependency for Android is com.google.ai.edge.litertlm:litertlm-android, and for the regular JVM litertlm-jvm. The code is the same. The engine is created once for the whole app: loading the model takes seconds. For the GPU on Android you need to add the libOpenCL.so and libvndksupport.so libraries to the manifest with android:required="false", and a 2–4 GB model should be downloaded after the app is installed rather than packed into the APK.

import com.google.ai.edge.litertlm.Backend
import com.google.ai.edge.litertlm.Engine
import com.google.ai.edge.litertlm.EngineConfig
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.flow.Flow
import kotlinx.coroutines.flow.flow
import kotlinx.coroutines.flow.flowOn
import kotlinx.coroutines.withContext

class LocalModel(modelPath: String, cacheDir: String) : AutoCloseable {
    private val engine = Engine(
        EngineConfig(
            modelPath = modelPath,
            // Backend.CPU() works everywhere, Backend.NPU(...) only on Android
            backend = Backend.GPU(),
            // Kernels built for this GPU go here: the second start is faster
            cacheDir = cacheDir,
        )
    )

    // Loading the weights and preparing the GPU take seconds: not on the main thread
    suspend fun load() = withContext(Dispatchers.Default) { engine.initialize() }

    // The answer arrives in pieces, as in a chat; the conversation keeps its history itself
    fun ask(question: String): Flow<String> = flow {
        engine.createConversation().use { conversation ->
            conversation.sendMessageAsync(question).collect { emit(it.toString()) }
        }
    }.flowOn(Dispatchers.Default)

    override fun close() = engine.close()
}

I ran this class on the regular JVM under Windows with Gemma 4 E2B on the GPU: the model loaded in 5.9 seconds and answered correctly. One trap: the litertlm-jvm build for Windows uses the GPU through WebGPU but does not include the DirectX shader compiler — the dxcompiler.dll and dxil.dll files. Without them the engine is not created, failing with "Device could not be created". What helped was putting both libraries in one folder with litertlm_jni.dll from the same jar file and pointing -Djava.library.path at that folder; the libraries themselves ship, for example, in the litert-lm-api Python package. Android does not have this problem.

Other engines for apps and the browser

ExecuTorch

ExecuTorch is Meta's engine for running PyTorch models on devices: from phones and glasses to microcontrollers. Version 1.0 came out in October 2025, 1.5.1 is current, BSD license. A model is exported from PyTorch into a .pte file for the backend you need, and for language models there is a ready export and generation loop.

ExecuTorch's strength is its backends. Through them the engine uses almost any accelerator: XNNPACK on the CPU, Core ML, Metal and MLX on Apple devices, Vulkan on GPUs, Qualcomm, MediaTek and Samsung NPUs, Arm Ethos-U and Cortex-M in microcontrollers, OpenVINO on Intel. Meta uses it in Instagram, WhatsApp, Messenger and Facebook, in Quest headsets and Ray-Ban Meta glasses. Choose ExecuTorch if you train the model yourself in PyTorch or need an accelerator LiteRT-LM does not have. If you take a ready open model for Android, LiteRT-LM is simpler.

ONNX Runtime GenAI and Windows ML

ONNX Runtime GenAI is Microsoft's generation loop on top of ONNX Runtime; version 0.19.0, MIT license. Foundry Local, Windows ML and the AI Toolkit for VS Code run on it. Accelerators plug in through execution providers: CUDA and TensorRT RTX for NVIDIA, OpenVINO for Intel, QNN for Snapdragon, DirectML and WebGPU.

Windows 11, starting with version 24H2, has Windows ML — a copy of ONNX Runtime built into the system that downloads the provider for the user's hardware by itself: NVIDIA, Intel, AMD, including AMD NPUs through VitisAI, and Snapdragon. The old DirectML is marked as legacy in it. If your Windows app must work on any laptop — with NVIDIA, Intel, AMD or Snapdragon — this is the most direct path.

Apple

• Foundation Models — a Swift API to the Apple Intelligence system model. The model is already in the system, so there is nothing to download; in Apple's 2025 technical report it is a model of about 3 billion parameters. The API supports structured answers into Swift types and tool calls. iOS 27 added connecting other models through a common protocol, a model on Apple's Private Cloud Compute servers and images as input. You cannot replace or fine-tune the system model itself.
• Core AI — a new framework in iOS and macOS 27 for your own neural networks on the CPU, GPU and Neural Engine.
• MLX Swift — the same MLX as on the Mac, but for Swift apps, including on the iPhone.

Others

• MLC LLM compiles a model through Apache TVM for specific hardware: Vulkan, Metal, OpenCL, CUDA, WebGPU. The project is alive but ships only as nightly builds, without releases.
• MNN from Alibaba — a phone engine with support for Snapdragon NPUs and speculative decoding; the MNN Chat app runs on it.
• Qualcomm GenieX — an open version of Qualcomm's engine for Snapdragon NPUs on Android, Windows on ARM and Linux; still a preview.

In the browser

• Chrome's built-in AI. This is not an engine but a browser API: Chrome downloads the Gemini Nano model by itself and offers the Prompt API to it from JavaScript, while LiteRT-LM does the computing, according to Google. For Chrome extensions the API has been stable since version 138; for regular pages the status table says stable since version 148, though the text on the same page still describes trial access — check before relying on it. Requirements: Windows 10 or 11, macOS 13 or newer, Linux or a Chromebook Plus; 22 GB of free space; a GPU with more than 4 GB of memory, or 16 GB of RAM and 4 cores. It does not work on Android and iOS.
• transformers.js from Hugging Face — a library that runs ONNX models right in the browser through WebGPU; ONNX Runtime does the computing in it. Since version 4 of March 2026 it has its own WebGPU runtime built on ONNX Runtime, which also works in Node.js, Bun and Deno.
• WebLLM — the browser version of MLC LLM with an OpenAI-compatible API.
• LiteRT-LM for JavaScript — in early preview, also through WebGPU.

Summary table: what runs where

Programs built on engines

These programs compute nothing themselves. They take a ready engine — most often llama.cpp or MLX — and add what a person needs: a model catalogue and downloads, a chat window, a local server with an API, memory management. Their speed is set by the engine inside; you choose them for convenience.

Ollama

Ollama is the simplest way to run a model locally: ollama run and a model name from its catalogue. It is a wrapper with its own model registry, download manager and an API compatible with OpenAI and, partly, with Anthropic. It runs on Windows, macOS and Linux, on NVIDIA GPUs, on AMD through ROCm or Vulkan, and on Apple. MIT license, version 0.40.15.

There are two engines inside. On a Mac with Apple Silicon — Apple's MLX. For GGUF models, since version 0.30 of May 2026, Ollama uses llama.cpp again: before that the team was writing its own engine on top of ggml, but returned to llama.cpp for compatibility and speed. There are also cloud models: those that do not fit locally run on Ollama's servers.

Ollama is not meant for many users: by default each model handles one request at a time (OLLAMA_NUM_PARALLEL=1), and the rest wait in the queue.

LM Studio

LM Studio is a desktop app with a convenient UI: a catalogue of models from Hugging Face, a chat, a local server with the OpenAI API. Inside there are three engines to choose from: llama.cpp, MLX and, on Apple Silicon, Splash. Since version 0.4.0 of January 2026 it handles several requests in parallel (four slots by default) and runs without a UI as the llmster background service with the lms command-line tool. The app is proprietary; its terms of use allow personal and internal work use. Version 0.4.28.

Other programs

• Lemonade — a local server backed by AMD: on the GPU it runs llama.cpp, and on the NPU of Ryzen AI processors the FastFlowLM engine.
• Foundry Local from Microsoft — local inference on ONNX Runtime with automatic choice of NPU, GPU or CPU.
• Jan, LocalAI, llamafile — apps and servers on top of llama.cpp.

A layer above engines for clusters

An engine is one server, even if it spans many GPUs. To spread requests across dozens of its instances you need a layer above it, and that layer is not an engine: vLLM, SGLang or TensorRT-LLM still do the computing. The main job of that layer is routing. An ordinary load balancer hands out requests round-robin. Then a request with a long history lands on an instance that does not have that history in its cache, and it is computed again. A smart router knows where each prefix lives and how long each instance's queue is.

NVIDIA Dynamo

NVIDIA Dynamo is not an engine but a layer above engines for a cluster. Version 1.0 came out in March at the GTC conference; 1.6.2 is current. Underneath runs your choice of vLLM, SGLang or TensorRT-LLM, and Dynamo adds:

• a router that knows which instance holds the KV cache for a given prefix;
• a KV-cache manager that offloads it to RAM, to SSD and to network storage;
• NIXL, a library for fast KV-cache transfer between GPUs and machines; both vLLM and SGLang use it;
• a planner that adds and removes instances to meet the promised latencies;
• a Kubernetes operator that takes the topology of an NVL72 rack into account.

The well-known Triton Inference Server was renamed Dynamo-Triton in 2025 and formally lives on, but everything new for language models NVIDIA does in Dynamo.

llm-d and others

• llm-d — distributed inference on Kubernetes on top of vLLM (and SGLang). It routes requests with the prefix cache in mind through the Gateway API Inference Extension, splits prefill and decode, and supports wide-EP, a multi-tier KV cache and autoscaling by latency. It was founded by Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA, has been a CNCF sandbox project since March 2026, and version 1.0 came out on 20 October. KServe's resource for language models is built on it too.
• AIBrix — a Kubernetes control plane from the same vllm-project organisation: routing, autoscaling, packing LoRA adapters. Version 0.8.0 came out in October.
• production-stack — the reference Helm chart for vLLM from its developers; it does less than llm-d.
• vLLM Router — a load balancer in Rust that knows about the cache and about prefill/decode separation, and Kinference from Buoyant, introduced on 4 November — a control plane with routing policies and per-customer cost tracking.

Conclusions

• vLLM is the default choice for a server: the widest support for models and hardware, scaling from one GPU to a rack, an ecosystem around it (llm-d, Dynamo, AIBrix). Its price is Linux, a slow start and the time it takes to tune large deployments. SGLang is an equal rival, TensorRT-LLM is for those who have only NVIDIA and care about every percent.
• llama.cpp is the choice for one machine: it runs everywhere, works with models bigger than GPU memory, and for one user is no slower than the server engines. Under load from many users it falls behind: in my measurement at 16 concurrent requests vLLM produced 1.4 times more tokens.
• LiteRT-LM is the choice for an app on the user's device, especially on Android: Google battle-tests it in its own products, and phone GPUs and NPUs work through it without separate code for every chip.
• Ollama and LM Studio are not engines but convenient programs on top of llama.cpp and MLX: for one person on their own computer they are the simplest way. Dynamo and llm-d are a cluster layer, needed once there are many engine instances.

All three engines ship new versions every few days or weeks, and much of what is described here appeared or changed in 2026. So before choosing, check the latest releases and run your own model on your own workload — vllm bench serve for servers, and llama-bench or litert-lm benchmark for a single device.