GGUF vs GPTQ vs AWQ: a format, two algorithms, one problem

The blind spot

When you compare runtimes, you measure tokens per second. When you size a machine, you count gigabytes. We have devoted a dossier to each of those two questions, the comparison of serving engines and the VRAM budget, and both take the quantized model as a given: it fits in so much memory, it puts out so many tokens. There remains a question that comes before both and reads neither in tokens per second nor in gigabytes.

You have a model in 16-bit weights and want to reduce its memory footprint. Four-bit quantization is one option when supported, but it is not required for every local model or every machine. Its effective size includes scales, metadata and any tensors kept at higher precision. The method also changes numerical error, so compare quality and runtime compatibility alongside file size. Outliers, values much larger than their neighbors, are one important challenge that the methods below address.

A format is not an algorithm

Before the mechanisms, a distinction that serves as a reading grid for everything else, and that is almost always confused because the three names circulate at the same level.

GGUF GGUF GPT-Generated Unified Format. llama.cpp's file format that stores the tensors, their quantization types, the vocabulary and the model metadata in a single file. Its aligned layout lets it be read with mmap, so the model starts in a fraction of a second. is a container. A single file where the quantization type is metadata written tensor by tensor, next to the weights, the vocabulary and the chat template. Nothing forces a single scheme on it: one file can store one layer in 4 bits and its neighbor in 6, exactly as a video container lets each track declare its own codec. The runtime selects implementations compatible with each tensor type and backend. Those implementations perform the required decoding or quantized computation; one generic kernel does not automatically support every new type.

GPTQ GPTQ A post-training weight quantization algorithm (3 to 4 bits). It quantizes the weights one by one and compensates for the error on the remaining weights through the inverse of the layer's Hessian, to preserve the output. It needs a calibration set; its output is served with kernels like Marlin. and AWQ AWQ Activation-aware Weight Quantization. A 4-bit quantization that protects the ~1% most important channels, spotted by the magnitude of their activations rather than their weights, by scaling them up before rounding. No reconstruction and no backpropagation, so it stays robust outside the calibration domain. are algorithms. Their output is not a format of their own: it is quantized weights stored in an ordinary safetensors file, produced by a procedure that decided, weight by weight, which integer value to write. The storage layout is dictated downstream by the CUDA kernel that will run those weights.

This asymmetry is not a vocabulary detail. It explains why GGUF proliferates into variants and why GPTQ and AWQ converge, why you do not load a GGUF into vLLM without paying dearly, and why the first rules over local inference while the others rule over the server. We will come back to it, but keep the formula: the format describes a layout, the algorithm describes a decision.

The common ground: quantizing, and the real problem

Quantizing a weight means bringing it from a float down to a small integer paired with a scale factor: w ≈ scale × q, where q fits in 4 bits. All three families do this by group of weights, typically 128 for GPTQ and AWQ, 32 or 256 for GGUF, and never with a single scale for the whole matrix. The reason is mechanical: one global scale has to cover the entire dynamic range of the matrix, so it stretches to accommodate a few extreme weights, which crushes the resolution of all the ordinary ones. A local scale per group re-fits the grid as closely as possible to each packet of values, for an overhead of a few bits.

A point that matters for what follows: the conversion back to float does not happen ahead of time, it happens inside the compute kernel, just before the multiplication. The autoregressive decoding of an LLM is bound by memory bandwidth, not by raw compute: on an Ampere card, the silicon can do a hundred to two hundred floating-point operations in the time it takes to read one byte. Reading weights on 4 bits instead of 16, then dequantizing them on-chip, therefore means reading four times less memory for barely more compute. That is what makes quantization worthwhile, and it is also what welds a file to a kernel, we will get there.

That leaves the central problem, the one each method solves in its own way. In an LLM, from a few billion parameters onward, outliers appear: a handful of channels whose activations have a disproportionate magnitude, up to 150,000 per sequence on a 13-billion-parameter model, but concentrated on fewer than ten dimensions. Those channels carry an outsized share of the quality. Crushing them onto a 4-bit grid calibrated for ordinary values destroys the model. All of post-training quantization boils down to one question: how not to sacrifice those few values that count.

GPTQ: redistributing the error through curvature

The first answer goes back to a 1990s idea on network pruning, Optimal Brain Surgeon, reused for quantization under the name OBQ and then made tractable at LLM scale by GPTQ. The intuition is this: when you force a weight onto the grid, you introduce an error, but you can compensate for that error by adjusting the weights not yet quantized, so that the output of the layer stays as close as possible to the original. You are not trying to preserve the weights one by one, you are trying to preserve what they produce.

Formally, you minimize the gap between the full-precision output and the quantized output, ||WX − ŴX||², where X is a set of calibration activations. This loss is quadratic in the weights, and the quantity that governs its geometry is its Hessian Hessian The matrix of a function's second derivatives, here the curvature of a layer's output error with respect to its weights. It encodes how sensitive the output is to each weight and the correlations between weights; its inverse tells you how to redistribute the quantization error. , its curvature matrix, which here equals H = 2·X·Xᵀ. It is worth pausing on this object, because it is the one doing all the work. The curvature measures how sensitive the output is to a perturbation of each weight, and above all how the weights are coupled to one another through the activations they share: two input channels correlated in the calibration data show up as linked in H. Its inverse, H⁻¹, therefore holds the redistribution recipe: when you quantize a weight, H⁻¹ tells you in which direction, and by how much, to correct the remaining weights to best cancel the effect on the output.

The update rule says it in one line: after quantizing the weight q, you propagate to the others a correction δ = −(error on q) / [H⁻¹]_qq × H⁻¹_{:,q}. The denominator [H⁻¹]_qq normalizes by the local curvature of the weight you just froze; the vector H⁻¹_{:,q} gives the direction of the carry-over. You can see in passing why calibration is indispensable: without a set of real activations X, the Hessian H = X·Xᵀ does not exist, and you know neither which weights are sensitive nor how they compensate for one another.

The price of exactness would be prohibitive: quantizing a 175-billion-parameter model by the exact method would take months. GPTQ brings it down to a few hours on a single card through three compromises. It quantizes the columns in a fixed order shared by all rows of the matrix, which lets it factor H only once instead of once per row, and gains three orders of magnitude. It precomputes H⁻¹ through a Cholesky decomposition, more numerically stable. And it adds damping on the diagonal, H⁻¹ = (2·X·Xᵀ + λI)⁻¹ with λ at about 1% of the mean diagonal, to keep the matrix invertible. The calibration corpus fits in 128 segments of 2048 tokens drawn from generic data, chosen precisely to contain nothing task-specific. That choice has a flip side, which the next approach takes as its target: GPTQ fits its reconstruction on this corpus, and can therefore overfit to it.

AWQ: protecting the channels their activations betray

AWQ starts from a blunt empirical observation. If you keep only 1% of a model’s weights in full precision, well chosen, the degradation at 3 bits collapses: on a 6.7-billion-parameter model, perplexity drops from 43.2 back to 13.0. Everything is in “well chosen”. AWQ’s counter-intuitive move is that those salient channels are not spotted by the magnitude of the weights, but by that of the activations that pass through them. The logic is direct: a weight counts only through what it produces, and a weight that systematically multiplies high-magnitude activations weighs heavily in the output, so its quantization error is amplified accordingly. Selecting channels by weight magnitude, or at random, barely helps; selecting them by activations does.

Keeping 1% of the weights in float would pose a hardware problem: a mix of precisions within one matrix is painful to serve efficiently. AWQ sidesteps the obstacle without ever leaving uniform 4-bit, through a simple identity. Multiplying a salient weight channel by a factor s, and dividing the corresponding activation by the same s, leaves the product WX unchanged: WX = (W·diag(s)) · (diag(s)⁻¹·X). But a weight scaled up before quantization falls more finely onto the grid, so its relative error decreases. You have protected the salient channel without taking it out of the format, by shifting the difficulty onto the activation, which absorbs it better. The factor s is derived from the per-channel activation magnitude, tuned by a single exponent α that a grid search over about twenty values is enough to fix; beyond a certain point, scaling the salient channels up further degrades the others, and the optimum sits around s = 2.

What sets AWQ apart is what it does not do. No backpropagation, no sequential reconstruction, no Hessian: a closed-form scaling followed by a plain round-to-nearest. It draws two properties from this. It is cheap to produce. And because it reconstructs nothing on the calibration corpus, it does not overfit to it: when you change the calibration distribution, its perplexity moves by only 0.5 to 0.6 points, where the error-compensation method drifts by 2.3 to 4.9. That out-of-domain robustness is its real advantage, and it comes from its simplicity. The price of that simplicity is that AWQ exposes almost no knobs: one operating point, 4 bits, group of 128, and the exponent α. Remember it, it answers half of the variants question.

GGUF: a menu of block recipes

On to the container, and its profusion. A GGUF file stores the weights in blocks, and it is in the structure of those blocks that everything plays out. The historical types take blocks of 32 weights with a shared float scale: Q4_0 is symmetric, w = scale × q; Q4_1 adds a per-block offset for asymmetric distributions, w = scale × q + minimum. Simple, but the scale is expensive relative to 32 weights.

The invention that made GGUF take off for local inference is the k-quants, which appeared in mid-2023. The principle lies in a quantization of the scales themselves. You group the weights into super-blocks of 256, subdivided into sub-blocks, and the scale of each sub-block is no longer stored as a float: it is itself quantized on 4 to 8 bits, then rescaled by a single float factor valid for the whole super-block. The arithmetic that justifies this double stage is worth doing. A 16-bit float scale for a 16-weight sub-block would on its own cost one bit per weight, in metadata alone. By quantizing those scales and keeping only one float per super-block of 256, that overhead falls to a fraction of a bit. That is exactly what lets Q2_K hold at 2.56 bits per weight without collapsing, or Q4_K show 4.5 effective bits where a naive 4-bit plus its scales would cost more.

Then comes the real heart of the proliferation: per-tensor mixed precision. A file labeled Q4_K_M is not uniformly in Q4_K. A llama.cpp function picks a type per tensor according to its role and its position in the network. The tensors judged sensitive, the attention value projections and the feed-forward down layers, are promoted to Q6_K; the others stay in Q4_K. The selection of which layers to promote follows an explicit heuristic, readable in the code: the first and last layers, plus one layer in three in the middle, which promotes about half of the tensors concerned. The _S, _M, _L suffixes are nothing more than three settings for the aggressiveness of that promotion, from the thriftiest to the most generous. Each combination of block type and promotion policy is one more point on the size-quality curve, and that is the whole answer: one more “Q” is one more recipe, not one more algorithm.

A last stage goes lower still, the i-quants. Below 3 bits, the per-block scale no longer suffices, and GGUF switches to a codebook quantization: instead of storing one value per weight, you store an index into a precomputed vector of several weights, drawn from a regular lattice of well-spread points, in the lineage of the QuIP# work on the E8 lattice. There, calibration becomes necessary again, and GGUF introduces it through an importance matrix. Rather than minimizing a uniform reconstruction error, you weight it by the activation statistics collected over a calibration text, <activation²> per column, which amounts to telling the quantizer which weights it is allowed to mishandle and which it must preserve. Below 3 bits, this matrix is no longer optional: without it, the IQ2 types produce, in their author’s own words, noise.

Criterion	GGUF	GPTQ	AWQ
Nature	container (type per tensor)	algorithm (weights in safetensors)	algorithm (weights in safetensors)
Base unit	super-block of 256 (k-quants)	group of 128	group of 128
Attack on outliers	per-tensor mixed precision	error carry-over (Hessian)	activation scaling
Calibration	optional (imatrix, required below 3 bits)	required (128 × 2048 tokens)	required (activation magnitudes)
Kernel, runtime	type-specific ggml implementations; llama.cpp, Ollama, LM Studio	Marlin, Machete, exllama; vLLM, TGI, SGLang	Marlin, exllama; vLLM, SGLang
Target	consumer, CPU and GPU, memory-bound	server GPU	server GPU, on-device

Why a zoo on one side, a handful on the other

GGUF records tensor types and metadata within one container. A mixed-precision recipe can reuse existing types while choosing which tensors receive more bits. Introducing a new tensor type is more work: its block layout and encode, decode and compute implementations must be supported on the intended backends. The container does not make a new kernel free.

GPTQ and AWQ implementations also depend on a precise layout, integer packing and scale arrangement. Optimized kernels such as Marlin Marlin A CUDA kernel for mixed 4-bit by 16-bit matrix multiplication (IST Austria), tuned to approach the memory-bandwidth limit on Ampere GPUs and beyond. It runs GPTQ/AWQ quantized weights; its successor Machete targets Hopper GPUs. support particular combinations of hardware and quantization settings. Four bits in groups of 128 is a configuration to verify, not a universal definition of either algorithm or an investment limited to datacenter GPUs. Choose among the combinations the actual backend implements.

A quantized artifact needs a backend that understands its layout and implements suitable kernels. The current vLLM documentation routes GGUF loading through the external vllm-gguf-plugin and labels the path experimental. That is more precise than the earlier blanket statement that support was being deprecated. A fixed tokens/s ratio from an older setup cannot rank all present GGUF and native checkpoints.

What calibration does to your model, and to your comparisons

That leaves the question everyone asks: which one keeps the model best? The honest answer is that there is no universal 4-bit winner, and the way we long believed there was is instructive. The received idea that “AWQ is better than GPTQ” was refuted head-on by a study with more than 500,000 evaluations on the Llama-3.1 family: at equal tuning, the two hold even on academic tests, and GPTQ even pulls ahead on real-world tasks like code, provided it is tuned correctly. The belief came from comparisons where GPTQ was poorly configured. What AWQ keeps is what its construction gives it and nothing more: robustness when the usage distribution drifts away from the calibration set. GGUF, for its part, draws from its importance matrix a net gain below 3 bits and a marginal one above Q4_K_M. Three constructions, three regimes where each shines, none that dominates everywhere.

The horizon: below the native-float frontier

Everything we have just described quantizes the weights and leaves the activations in 16 bits. That is a choice, and it is being challenged from above. The datacenter is migrating toward low-precision float formats that the silicon can now compute natively, the NVFP4 and MXFP4 of the Blackwell cards, whose mechanics we detailed in our FP8 and FP4 dossier. Where GPTQ, AWQ and GGUF are a software stage that arranges integers for hardware that only knows 16-bit float, native FP4 FP4 A 4-bit floating-point format, the 2026 frontier of high-throughput inference. Four times less memory than FP16, but a very narrow dynamic range: it only holds up with fine-grained scaling through block formats (MXFP4, NVFP4). computes directly in the compressed format. On that ground, the trio already looks like the old world.

There is, however, a precise reason it will not disappear from local inference any time soon, and it is hardware. Native FP4 is only fully usable on the datacenter Blackwell. The consumer Blackwell, an RTX 5090 for instance, falls back onto a path that decompresses the weights toward a Marlin-style integer kernel: it gains the memory, not the throughput. As long as the silicon under desks does not accelerate FP4 for real, post-training INT4 will remain the format of local inference, carried by two advantages the datacenter disdains: a single file to download, and no heavy calibration to run.

Around the trio, finally, a generation of methods is moving the lines on axes it does not occupy. Hadamard rotations, QuaRot and SpinQuant, scatter the outliers before quantization and make 4 bits possible on the activations too, not only the weights. Lattice quantizations, QTIP and its EXL3 implementation, push the frontier below 4 bits, down to models still coherent at 1.6 bits per weight. HQQ removes calibration. QAT QAT Quantization-Aware Training. Rather than compressing an already-trained model (and taking the accuracy hit), you briefly retrain it while simulating quantization, so it learns to live in low precision (often int4). It sharply reduces the degradation compared with naive post-training quantization. acts upstream, at training time, and Google already ships GGUFs trained this way for Gemma. None of them replaces the trio; they graft onto it, reusing its building blocks, the rotation before GPTQ, the error carry-over in EXL3.

The real question this dossier leaves open is therefore not “which one to choose today”, but what will survive when consumer hardware finally learns to compute 4-bit float natively. The operating point of the server algorithms could then melt into the native format. But the distinction that has structured this whole text, a container that arranges bits on one side, an algorithm welded to its kernel on the other, does not depend on any generation of silicon. It will still describe the split between the one you download as a file and the one you serve behind an API, long after today’s 4 bits has aged.

Choose a checkpoint for the server you will actually run

Start with a supported model architecture and a runtime available on your hardware. Verify the exact quantization variant and checkpoint metadata, then compare answer quality on representative tasks. Four-bit files can differ in scales, protected tensors and effective storage, so equal parameter counts and nominal bits are not a complete control.

Next, test prompt processing and generation at the intended context and concurrency. The runtime comparison and serving protocol provides that checklist. The KV cache memory calculation explains why a smaller weight file does not necessarily leave enough room for long prompts or several users.

For reproducibility, keep the model and quantizer revisions, calibration corpus, runtime version, hardware, context distribution and quality checks with the result. Repacking a checkpoint is a separate operation from choosing its quantization algorithm. A successful load proves compatibility for that attempt, not production throughput or unchanged accuracy.

Sources and method

Original research freeze: July 11, 2026. Revision November 3, 2026: tensor-type implementation costs, GGUF plugin status and practical selection method updated using the current vLLM documentation. No new hardware benchmark was performed. Labels: verified fact (primary source), credible estimate (consistent third-party analysis), assumption (stated reasoning). This dossier documents how the formats are built, not a score comparison: every quality figure is a snapshot that depends on the model, the bench and the kernel.

Container and tensor implementations

GGUF specification and GGML quantization implementation distinguish container metadata from the code for individual tensor types.

GPTQ and Hessian-based compensation

• Fact: GPTQ, Frantar et al., arXiv 2210.17323 (ICLR 2023): objective ||WX−ŴX||², Hessian H = 2XXᵀ (verbatim Section 3), OBS/OBQ update rule, shared fixed order (“three orders of magnitude”), Cholesky, damping (2XXᵀ+λI)⁻¹, calibration 128 × 2048 tokens of C4, 175B in ~4 GPU-hours. OBS lineage: Hassibi & Stork, NeurIPS 1992.
• Fact: damping λ ≈ 1% of the mean diagonal, act-order/desc_act, implementation defaults (GPTQModel, HF wrapper): Transformers/GPTQModel docs, accessed July 11, 2026. 2026 tooling: AutoGPTQ archived April 11, 2025, successor GPTQModel.

AWQ and activation scaling

• Fact: AWQ, Lin et al., arXiv 2306.00978 (MLSys 2024): 0.1 to 1% of salient channels, selection by activation magnitude (PPL 43.2 → 13.0), invariance WX=(W·diag(s))(diag(s)⁻¹X), grid search of 20, optimum s≈2. Out-of-domain robustness +0.5 to 0.6 PPL against +2.3 to 4.9 for GPTQ (Figure 6). No backpropagation and no reconstruction (verbatim). AutoAWQ archived May 11, 2025, taken over by llm-compressor.

GGUF, k-quants, i-quants

• Fact: container format, type per tensor, 32-byte alignment for mmap: ggml spec (github.com/ggml-org/ggml, gguf.md).
• Fact: legacy quants (blocks of 32, Q4_0 symmetric / Q4_1 affine) and k-quants (super-blocks of 256, two-level scales, Q2_K 2.56 bpw to Q6_K 6.56 bpw): ikawrakow’s PR #1684 (June 5, 2023), structures and static_assert in ggml-common.h.
• Fact: per-tensor mixed precision, use_more_bits heuristic (first and last layers, one in three), promotion of attn_v and ffn_down to Q6_K for Q4_K_M: src/llama-quant.cpp (verbatim).
• Fact: i-quants (codebook, E8 lattice, QuIP# lineage arXiv 2402.04396) and importance matrix weighting the error by <activation²>, made mandatory below 3 bits: PR #4773, #4861, #4897. Rationale for the proliferation (“finer control over the size-quality trade-off”, edge target): Discussion #5063.

Fidelity, kernels, lock-in

• Fact: no universal winner, refutation of “AWQ better than GPTQ”: Kurtic et al., “Give Me BF16 or Give Me Death”, arXiv 2411.02355 (ACL 2025), more than 500,000 evaluations on Llama-3.1.
• Fact: kernel implementations and supported layouts matter; see the Marlin paper. The historical vLLM maintenance discussion does not establish current GGUF plugin performance.
• Method: old isolated throughput ratios were removed from the argument because they do not establish a present-day ranking across runtimes, models and versions.

Outliers and horizon

• Fact: emergent outliers (magnitude ≥ 6, from ~6.7B parameters): LLM.int8(), arXiv 2208.07339. SmoothQuant, arXiv 2211.10438.
• Fact: native Blackwell FP (NVFP4 blocks of 16, MXFP4 blocks of 32): see our FP8/FP4 dossier. Supported assumption: consumer Blackwell (SM120) in Marlin weight-only fallback, vLLM issues #30135 and #31085 (December 2025).
• Fact: QuaRot (arXiv 2404.00456), SpinQuant (arXiv 2405.16406), QTIP (arXiv 2406.11235) and EXL3 (exllamav3), HQQ (Mobius Labs), Gemma 3 QAT (Google, April 2025).