Define the service before choosing the engine
A local assistant used by one person and a shared API can run on the same GPU while demanding different behavior. The assistant needs a responsive stream. The shared API must also absorb arrivals, divide memory among active requests and control its queue.
Write down the model revision, weight format, prompt length, output length and simultaneous users. Include weights, runtime buffers and KV cache for the active sequences in the memory budget. Successfully loading the weights is only the first capacity check.
vLLM for shared serving
vLLM combines scheduling, model execution and cache management. Paged allocation lets a sequence obtain cache blocks as needed instead of reserving one large contiguous region upfront. Its scheduler can admit new requests while others finish, making a continuously changing batch possible.
That organization can improve aggregate throughput when there is enough traffic and memory to keep requests active. Measure latency alongside throughput: processing more tokens in total is not sufficient if individual users wait too long.
Compatibility is specific to the model, accelerator and quantization method. A downloadable checkpoint does not establish an efficient execution path. The distinction between a container and a quantization algorithm, explained in GGUF vs GPTQ vs AWQ, matters before selecting the server.
llama.cpp for portable inference and GGUF serving
llama.cpp supports a range of CPU and GPU backends and models packaged as GGUF. It is a practical starting point for local inference and constrained deployments. Splitting work between CPU and GPU can help a model fit, but moving data changes the performance problem.
Its server is not limited to one active request. The current documentation describes parallel decoding and continuous batching. The number of slots, configured through —parallel, has to match the context and memory available.
This feature does not prove parity with another engine on a particular workload. It means concurrency must be tested rather than dismissed as unsupported. Compare the per-user stream, first-token delay, memory use and error rate as load grows.
TensorRT-LLM: identify the backend
TensorRT-LLM targets inference on NVIDIA GPUs. A precompiled engine is one deployment path, with compatibility constraints that need checking against the serving stack. It is not the only path: NVIDIA also documents a PyTorch backend and a high-level LLM API without a preliminary TensorRT-engine compilation step in that workflow.
Record the backend, model architecture and precision when publishing a result. A benchmark obtained through a specialized path does not automatically describe another configuration of the same project.
Prepare comparable candidates
| Question | vLLM | llama.cpp | TensorRT-LLM |
|---|---|---|---|
| Starting workload | Shared API and scheduling | Local inference or GGUF server | NVIDIA inference stack |
| Concurrent requests | Integrated scheduling | Server slots and continuous batching | Depends on backend and deployment |
| Model and format | Check supported combinations | Check GGUF model and operators | Check model, backend and precision |
| Preparation | Configure the service | Choose GGUF and CPU/GPU placement | PyTorch API or engine path |
Separate engine timing from service timing
First measure one request at a time to establish basic model behavior. Then raise concurrency while keeping prompt and output lengths fixed. Run one series with new prefixes and another with reused prefixes; otherwise caching can silently favor a candidate.
The llama.cpp benchmark tool separates prompt processing from generation and can repeat measurements with machine-readable output. Its timings exclude tokenization and sampling. A microbenchmark therefore needs a second test of the complete server, not a direct comparison against an end-to-end API result.
For that second test, report time to first token, inter-token latency, full request latency, successful output throughput and failures. Include a median and a high percentile, such as p95. Choose the acceptable latency beforehand so the criterion is independent of which candidate wins.
The vLLM serving benchmark distinguishes request arrival rate from maximum concurrency. These are separate constraints: a high arrival rate can create a queue even if only a few sequences execute at once. Keep raw results, configuration and exact revisions with the report.
Quality belongs in the same comparison. A smaller quantized checkpoint that returns worse answers has changed the product being measured. Evaluate the actual tasks that motivated the service, then compare performance among acceptable candidates.
Make the deployment decision
For a shared API, begin with the runtimes that support the model and hardware you already have. For a local GGUF workload, a simple llama.cpp deployment provides a useful starting point. On NVIDIA infrastructure, test the appropriate TensorRT-LLM path alongside alternatives instead of assuming that an engine build is compulsory.
Include operational work when translating throughput into inference cost. A faster isolated generation is valuable only when it improves the service you can actually operate.
Sources and method
Reviewed on November 3, 2026: llama.cpp server, llama-bench, vLLM serving benchmark, NVIDIA’s TensorRT-LLM user guide and the PagedAttention paper.
Documented capabilities are verifiable facts for the cited software. The matrix is editorial guidance. The proposed protocol has not been executed here on a GPU test bench, and no numerical performance ranking is asserted.
