Speculative decoding: when a small model makes your LLM faster

The model fits, but every sentence takes time

You run a local LLM to edit a function or summarize a document. It loads successfully, and memory capacity appears sufficient. Yet the answer builds slowly, one piece at a time. Replacing the GPU is not the only option worth examining.

Speculative decoding tries to advance generation in groups. A fast mechanism proposes several tokens, the pieces of text handled by the model. The target model verifies those proposals before they become part of the answer.

LM Studio and llama.cpp document the feature. This article examines their documentation, llama.cpp code as of September 10, 2026, and the research behind the method. The example timings below are hypothetical. LeCompute has not run a hardware inference benchmark for this article.

Draft quickly, then check several positions together

An autoregressive model uses earlier tokens to generate the next one. Ordinary decoding advances through dependent steps: producing four new tokens takes four successive target-model steps.

In conventional speculative decoding, a smaller draft model proposes a sequence. The target can evaluate its positions together because the proposed preceding tokens are already available. At the first rejection, the runtime discards the continuation that depends on that proposal, corrects the rejected position and restarts from the accepted prefix.

Leviathan, Kalman and Matias formalize this mechanism. It can spread a target-model pass across several useful tokens, but drafting and verification also cost time. Fewer sequential passes do not automatically mean a faster answer.

The opportunity is clear when generation leaves compute resources available while waiting for data. Batched verification can use more of those resources. If other requests already keep the GPU busy, the extra work changes the tradeoff. A single-chat speedup therefore cannot predict the result on a loaded server.

What lossless decoding actually guarantees

The draft does not replace the target as the authority over the answer. The exact algorithm uses both models’ probabilities in its acceptance rule and corrects the distribution at the first rejection. Chen and colleagues also establish distribution preservation under their algorithm’s conditions.

Accepting a token because it merely looks plausible is not equivalent: an approximate rule may change the distribution. Distribution preservation also does not guarantee identical text from the same random seed. Numerical operations and the sequence of random draws can differ.

The vLLM losslessness documentation separates the theoretical property, algorithmic tests and numerical stability. Keep the target checkpoint, quantization and sampling settings fixed in your comparison. Accelerating a quantized target does not recover any quality lost through quantization.

Acceptance rate is only part of the calculation

Take an intentionally simple example. The target alone takes 40 ms per token. The draft proposes four tokens in 12 ms, verification takes 52 ms, and coordination adds 4 ms. One speculative iteration costs 68 ms. These numbers explain the arithmetic; they are not measurements from a particular GPU.

If the first three proposals are accepted and the fourth is corrected, the iteration produces four useful tokens. Ordinary decoding would take 4 × 40 = 160 ms. The ratio is 160 / 68, about 2.35 times the reference speed in this example.

The useful formula is useful tokens × reference time per token / speculative iteration time. It includes the entire iteration. Real timings change with context and proposal length. Two methods with the same acceptance rate can have very different costs, and the position of the first rejection matters.

Pick a pair the runtime can actually use

A smaller model is not compatible merely because it shares a product-family name. Tokenization must align: a token identifier must represent the expected text. For conventional drafting, the llama.cpp compatibility code checks vocabulary type, special-token behavior and vocabulary entries.

Specialized methods change that contract. EAGLE-3 reads target-model hidden states and needs a draft trained for its target. Token-repetition methods can propose continuations without loading a second model. Select the method using the documentation for your revision, rather than treating every speculator as interchangeable.

Start with a documented pair accepted by the engine. LM Studio’s documentation lists Qwen 2.5 14B Instruct with Qwen 2.5 0.5B Instruct, for example. That example does not establish compatibility with every format and backend; check the pair against your engine before measuring it.

Leave room for the extra model

A separate draft needs its weights and generation state in memory. The target’s KV cache KV cache The stored key and value vectors an LLM has already computed for every token it has processed. It avoids recomputing attention over the whole history, at the cost of a memory footprint that grows with the context length. , which stores attention keys and values, still occupies space, alongside verification buffers and context allocations.

If the target nearly fills GPU memory, adding the draft may move data to system RAM or reduce the context margin. Transfer overhead can then consume a compute-side improvement. Inspect actual allocations and layer placement before judging the draft by parameter count alone.

Memory requirements depend on the method and backend. A speculator trained to use target hidden states and a method without a second model do not have the same budget as two independent LLMs. Our KV cache guide explains why context management remains relevant even when model weights fit.

Try LM Studio or llama.cpp

In LM Studio, first load the target and measure representative tasks without a draft. Then select a compatible Draft Model in the speculative-decoding settings. The feature documentation explains this path and the possibility of slowdowns. Panel names depend on the version: LM Studio 0.4.0 reorganized earlier interface modes. Record the loaded engine version as well as the application version.

For llama.cpp, these commands follow the options documented at revision 72797e89198a. Replace target.gguf and draft.gguf with compatible local files. The 8,192-token context and four-token draft are starting points for the experiment, not claimed optimal settings.

llama-server --model target.gguf --ctx-size 8192 --parallel 1 --host 127.0.0.1 --port 8080 --spec-type none

Stop that server after measuring, then start the draft configuration under the same conditions. These commands leave hardware placement to the engine settings. Inspect its logs to establish which layers actually run on the GPU.

llama-server --model target.gguf --model-draft draft.gguf --ctx-size 8192 --parallel 1 --host 127.0.0.1 --port 8080 --spec-type draft-simple --spec-draft-n-max 4

The old --draft and --draft-max arguments are marked as removed in the server README at this revision. Check llama-server --help for a different binary. Successful startup is not proof that speculation is active: inspect drafting and acceptance statistics too.

Measure completed work, not the best counter

The downloadable measurement client uses Python’s standard library and llama.cpp’s native /completion endpoint. It replays three prompts, warms up and then performs five passes, disables prefix reuse between requests and writes JSON. It preserves client-observed total time, server timings and outputs so differences can be inspected. It does not measure streaming time to first token or GPU memory.

python speculative-benchmark.py --label baseline --output baseline.json
python speculative-benchmark.py --label draft --output draft.json

Run the first command against the baseline server, then the second after replacing it. The client uses greedy sampling, selecting the highest-probability token, to make comparison easier. It sends raw prompts without a chat template: to evaluate an assistant, follow up with the conversation format your model expects in both configurations. Add your real prompts and usual sampling parameters in that separate experiment. Its output cap can truncate answers, so inspect output lengths and content before interpreting elapsed time.

Compare each task with itself, examine medians and spread, then check the outputs. If answers have substantially different lengths or fail to finish the same work, a time ratio no longer measures equal-work acceleration. For an agent, also measure first-output delay, tool calls and complete task success.

Keep speculation if it improves your tasks with acceptable memory headroom. A slowdown or no gain is useful evidence for trying a lighter draft, changing proposal length or retaining the target alone. If several computers handle independent requests, NVIDIA PAIR addresses that separate problem: routing requests rather than speeding up one generation.

Sources and method

Foundations. Yaniv Leviathan, Matan Kalman and Yossi Matias, Fast Inference from Transformers via Speculative Decoding, May 18, 2023 version; Charlie Chen and colleagues, Accelerating Large Language Model Decoding with Speculative Sampling, 2023. Their proofs apply to the stated algorithms, without guaranteeing every software variant.

Implementations examined. llama.cpp commit 72797e89198ab564fd0e6baa54ab196e8dd1d884, September 10, 2026: method documentation, compatibility code and server options linked above. LM Studio, speculative decoding and interface changes. vLLM, lossless guarantees. Reviewed September 10, 2026. Embedded engine versions may differ from upstream.

Calculations and verification. The timing table uses explicit teaching assumptions, without associated hardware. Commands were checked against the cited options, not used for a LeCompute hardware benchmark. The Python client is a reproduction aid whose logic can be checked without a model. Its results must come from your server, not the illustrative values in this article.