Runtimes & Inference

Runtimes: serving LLMs efficiently

vLLM, llama.cpp, TensorRT-LLM, KV cache and quantization: how to serve language models efficiently, from the datacenter to a local machine.

13 articles , page 2 of 2