GGUF vs GPTQ vs AWQ: a format, two algorithms, one problem

The blind spot

When you compare runtimes, you measure tokens per second. When you size a machine, you count gigabytes. We have devoted a dossier to each of those two questions, the comparison of serving engines and the VRAM budget, and both take the quantized model as a given: it fits in so much memory, it puts out so many tokens. There remains a question that comes before both and reads neither in tokens per second nor in gigabytes.

Take a model in 16-bit floating-point weights. To run it anywhere other than a cluster, you first have to bring it down to 4 bits. There, three big families present themselves, and the choice is not decided by file size, which will be almost identical, nor by speed, which will depend mostly on the kernel. It is decided by what is left of the model once compressed. Two 4-bit quantizations of the same model, same footprint, same throughput class, do not keep the same amount of intelligence, because they do not attack in the same way the only problem that matters in quantization quantization Reducing the number of bits that encode each weight of a model (from 16 bits down to 8, 4, or fewer). It shrinks the memory footprint by the same factor, at the cost of a controlled accuracy loss, without changing the parameter count. : outliers outliers In an LLM, a handful of channels whose activations reach a disproportionate magnitude, concentrated on a few dimensions. They carry an outsized share of the quality: crushing them onto a low-precision grid badly degrades the model, which any quantization must avoid. . This dossier takes apart how each one goes about it, because it is that construction, not a table of scores, that decides what you lose.

A format is not an algorithm

Before the mechanisms, a distinction that serves as a reading grid for everything else, and that is almost always confused because the three names circulate at the same level.

GGUF GGUF GPT-Generated Unified Format. llama.cpp's file format that stores the tensors, their quantization types, the vocabulary and the model metadata in a single file. Its aligned layout lets it be read with mmap, so the model starts in a fraction of a second. is a container. A single file where the quantization type is metadata written tensor by tensor, next to the weights, the vocabulary and the chat template. Nothing forces a single scheme on it: one file can store one layer in 4 bits and its neighbor in 6, exactly as a video container lets each track declare its own codec. Dequantization is handled by a generic llama.cpp kernel that reads this metadata and knows how to decode each type on the fly.

GPTQ GPTQ A post-training weight quantization algorithm (3 to 4 bits). It quantizes the weights one by one and compensates for the error on the remaining weights through the inverse of the layer's Hessian, to preserve the output. It needs a calibration set; its output is served with kernels like Marlin. and AWQ AWQ Activation-aware Weight Quantization. A 4-bit quantization that protects the ~1% most important channels, spotted by the magnitude of their activations rather than their weights, by scaling them up before rounding. No reconstruction and no backpropagation, so it stays robust outside the calibration domain. are algorithms. Their output is not a format of their own: it is quantized weights stored in an ordinary safetensors file, produced by a procedure that decided, weight by weight, which integer value to write. The storage layout is dictated downstream by the CUDA kernel that will run those weights.

This asymmetry is not a vocabulary detail. It explains why GGUF proliferates into variants and why GPTQ and AWQ converge, why you do not load a GGUF into vLLM without paying dearly, and why the first rules over local inference while the others rule over the server. We will come back to it, but keep the formula: the format describes a layout, the algorithm describes a decision.

The common ground: quantizing, and the real problem

Quantizing a weight means bringing it from a float down to a small integer paired with a scale factor: w ≈ scale × q, where q fits in 4 bits. All three families do this by group of weights, typically 128 for GPTQ and AWQ, 32 or 256 for GGUF, and never with a single scale for the whole matrix. The reason is mechanical: one global scale has to cover the entire dynamic range of the matrix, so it stretches to accommodate a few extreme weights, which crushes the resolution of all the ordinary ones. A local scale per group re-fits the grid as closely as possible to each packet of values, for an overhead of a few bits.

A point that matters for what follows: the conversion back to float does not happen ahead of time, it happens inside the compute kernel, just before the multiplication. The autoregressive decoding of an LLM is bound by memory bandwidth, not by raw compute: on an Ampere card, the silicon can do a hundred to two hundred floating-point operations in the time it takes to read one byte. Reading weights on 4 bits instead of 16, then dequantizing them on-chip, therefore means reading four times less memory for barely more compute. That is what makes quantization worthwhile, and it is also what welds a file to a kernel, we will get there.

That leaves the central problem, the one each method solves in its own way. In an LLM, from a few billion parameters onward, outliers appear: a handful of channels whose activations have a disproportionate magnitude, up to 150,000 per sequence on a 13-billion-parameter model, but concentrated on fewer than ten dimensions. Those channels carry an outsized share of the quality. Crushing them onto a 4-bit grid calibrated for ordinary values destroys the model. All of post-training quantization boils down to one question: how not to sacrifice those few values that count.

GPTQ: redistributing the error through curvature

The first answer goes back to a 1990s idea on network pruning, Optimal Brain Surgeon, reused for quantization under the name OBQ and then made tractable at LLM scale by GPTQ. The intuition is this: when you force a weight onto the grid, you introduce an error, but you can compensate for that error by adjusting the weights not yet quantized, so that the output of the layer stays as close as possible to the original. You are not trying to preserve the weights one by one, you are trying to preserve what they produce.

Formally, you minimize the gap between the full-precision output and the quantized output, ||WX − ŴX||², where X is a set of calibration activations. This loss is quadratic in the weights, and the quantity that governs its geometry is its Hessian Hessian The matrix of a function's second derivatives, here the curvature of a layer's output error with respect to its weights. It encodes how sensitive the output is to each weight and the correlations between weights; its inverse tells you how to redistribute the quantization error. , its curvature matrix, which here equals H = 2·X·Xᵀ. It is worth pausing on this object, because it is the one doing all the work. The curvature measures how sensitive the output is to a perturbation of each weight, and above all how the weights are coupled to one another through the activations they share: two input channels correlated in the calibration data show up as linked in H. Its inverse, H⁻¹, therefore holds the redistribution recipe: when you quantize a weight, H⁻¹ tells you in which direction, and by how much, to correct the remaining weights to best cancel the effect on the output.

The update rule says it in one line: after quantizing the weight q, you propagate to the others a correction δ = −(error on q) / [H⁻¹]_qq × H⁻¹_{:,q}. The denominator [H⁻¹]_qq normalizes by the local curvature of the weight you just froze; the vector H⁻¹_{:,q} gives the direction of the carry-over. You can see in passing why calibration is indispensable: without a set of real activations X, the Hessian H = X·Xᵀ does not exist, and you know neither which weights are sensitive nor how they compensate for one another.

The price of exactness would be prohibitive: quantizing a 175-billion-parameter model by the exact method would take months. GPTQ brings it down to a few hours on a single card through three compromises. It quantizes the columns in a fixed order shared by all rows of the matrix, which lets it factor H only once instead of once per row, and gains three orders of magnitude. It precomputes H⁻¹ through a Cholesky decomposition, more numerically stable. And it adds damping on the diagonal, H⁻¹ = (2·X·Xᵀ + λI)⁻¹ with λ at about 1% of the mean diagonal, to keep the matrix invertible. The calibration corpus fits in 128 segments of 2048 tokens drawn from generic data, chosen precisely to contain nothing task-specific. That choice has a flip side, which the next approach takes as its target: GPTQ fits its reconstruction on this corpus, and can therefore overfit to it.

AWQ: protecting the channels their activations betray

AWQ starts from a blunt empirical observation. If you keep only 1% of a model’s weights in full precision, well chosen, the degradation at 3 bits collapses: on a 6.7-billion-parameter model, perplexity drops from 43.2 back to 13.0. Everything is in “well chosen”. AWQ’s counter-intuitive move is that those salient channels are not spotted by the magnitude of the weights, but by that of the activations that pass through them. The logic is direct: a weight counts only through what it produces, and a weight that systematically multiplies high-magnitude activations weighs heavily in the output, so its quantization error is amplified accordingly. Selecting channels by weight magnitude, or at random, barely helps; selecting them by activations does.

Keeping 1% of the weights in float would pose a hardware problem: a mix of precisions within one matrix is painful to serve efficiently. AWQ sidesteps the obstacle without ever leaving uniform 4-bit, through a simple identity. Multiplying a salient weight channel by a factor s, and dividing the corresponding activation by the same s, leaves the product WX unchanged: WX = (W·diag(s)) · (diag(s)⁻¹·X). But a weight scaled up before quantization falls more finely onto the grid, so its relative error decreases. You have protected the salient channel without taking it out of the format, by shifting the difficulty onto the activation, which absorbs it better. The factor s is derived from the per-channel activation magnitude, tuned by a single exponent α that a grid search over about twenty values is enough to fix; beyond a certain point, scaling the salient channels up further degrades the others, and the optimum sits around s = 2.

What sets AWQ apart is what it does not do. No backpropagation, no sequential reconstruction, no Hessian: a closed-form scaling followed by a plain round-to-nearest. It draws two properties from this. It is cheap to produce. And because it reconstructs nothing on the calibration corpus, it does not overfit to it: when you change the calibration distribution, its perplexity moves by only 0.5 to 0.6 points, where the error-compensation method drifts by 2.3 to 4.9. That out-of-domain robustness is its real advantage, and it comes from its simplicity. The price of that simplicity is that AWQ exposes almost no knobs: one operating point, 4 bits, group of 128, and the exponent α. Remember it, it answers half of the variants question.

GGUF: a menu of block recipes

On to the container, and its profusion. A GGUF file stores the weights in blocks, and it is in the structure of those blocks that everything plays out. The historical types take blocks of 32 weights with a shared float scale: Q4_0 is symmetric, w = scale × q; Q4_1 adds a per-block offset for asymmetric distributions, w = scale × q + minimum. Simple, but the scale is expensive relative to 32 weights.

The invention that made GGUF take off for local inference is the k-quants, which appeared in mid-2023. The principle lies in a quantization of the scales themselves. You group the weights into super-blocks of 256, subdivided into sub-blocks, and the scale of each sub-block is no longer stored as a float: it is itself quantized on 4 to 8 bits, then rescaled by a single float factor valid for the whole super-block. The arithmetic that justifies this double stage is worth doing. A 16-bit float scale for a 16-weight sub-block would on its own cost one bit per weight, in metadata alone. By quantizing those scales and keeping only one float per super-block of 256, that overhead falls to a fraction of a bit. That is exactly what lets Q2_K hold at 2.56 bits per weight without collapsing, or Q4_K show 4.5 effective bits where a naive 4-bit plus its scales would cost more.

Then comes the real heart of the proliferation: per-tensor mixed precision. A file labeled Q4_K_M is not uniformly in Q4_K. A llama.cpp function picks a type per tensor according to its role and its position in the network. The tensors judged sensitive, the attention value projections and the feed-forward down layers, are promoted to Q6_K; the others stay in Q4_K. The selection of which layers to promote follows an explicit heuristic, readable in the code: the first and last layers, plus one layer in three in the middle, which promotes about half of the tensors concerned. The _S, _M, _L suffixes are nothing more than three settings for the aggressiveness of that promotion, from the thriftiest to the most generous. Each combination of block type and promotion policy is one more point on the size-quality curve, and that is the whole answer: one more “Q” is one more recipe, not one more algorithm.

A last stage goes lower still, the i-quants. Below 3 bits, the per-block scale no longer suffices, and GGUF switches to a codebook quantization: instead of storing one value per weight, you store an index into a precomputed vector of several weights, drawn from a regular lattice of well-spread points, in the lineage of the QuIP# work on the E8 lattice. There, calibration becomes necessary again, and GGUF introduces it through an importance matrix. Rather than minimizing a uniform reconstruction error, you weight it by the activation statistics collected over a calibration text, <activation²> per column, which amounts to telling the quantizer which weights it is allowed to mishandle and which it must preserve. Below 3 bits, this matrix is no longer optional: without it, the IQ2 types produce, in their author’s own words, noise.

Criterion	GGUF	GPTQ	AWQ
Nature	container (type per tensor)	algorithm (weights in safetensors)	algorithm (weights in safetensors)
Base unit	super-block of 256 (k-quants)	group of 128	group of 128
Attack on outliers	per-tensor mixed precision	error carry-over (Hessian)	activation scaling
Calibration	optional (imatrix, required below 3 bits)	required (128 × 2048 tokens)	required (activation magnitudes)
Kernel, runtime	generic ggml; llama.cpp, Ollama, LM Studio	Marlin, Machete, exllama; vLLM, TGI, SGLang	Marlin, exllama; vLLM, SGLang
Target	consumer, CPU and GPU, memory-bound	server GPU	server GPU, on-device

Why a zoo on one side, a handful on the other

Now the opening question can be answered in full. GGUF proliferates because adding a variant there costs almost nothing: since the type is per-tensor metadata decoded by a generic kernel, a new recipe is just a new block-and-promotion policy, which a contributor proposes through a pull request. The project’s culture, turned toward inference at the margin on heterogeneous, memory-bound consumer hardware, values precisely this granularity: every half-bit saved counts when you are trying to fit a model into a laptop’s RAM, and multiplying the points on the size-quality frontier is a service rendered, not a debt. Its author puts it plainly: more quants means finer control over the size-versus-quality trade-off, at the cost of code to maintain.

GPTQ and AWQ live in the opposite economy. Their speed comes not from the algorithm but from the kernel that runs the weights: Marlin Marlin A CUDA kernel for mixed 4-bit by 16-bit matrix multiplication (IST Austria), tuned to approach the memory-bandwidth limit on Ampere GPUs and beyond. It runs GPTQ/AWQ quantized weights; its successor Machete targets Hopper GPUs. on Ampere and Ada cards, Machete on Hopper, built to approach the bandwidth limit. Those kernels are hand-written for a precise memory layout, a given integer packing and scale arrangement. Adding a variant is not adding a recipe, it is writing a new, highly optimized CUDA kernel, an investment that only pays off on datacenter GPUs. The ecosystem therefore converges on one dominant operating point, 4 bits in groups of 128, served fast. AWQ pushes the logic to the extreme: its value lies in a single robust 4-bit, it has no reason to expose any other.

That same weld between format and kernel explains why you do not mix the two worlds. A quantized file is laid out for the kernel that expects it; another kernel has to repack it first. vLLM can technically load GGUF, but it has to translate the layout at runtime, and the result is clear-cut: on the same bench, a GGUF runs at 93 tokens per second there when the native format delivers 461. The vLLM maintainer prices that gap another way: six thousand lines of GGUF-specific CUDA kernels, for a usage on the order of 0.1% of its base. Conversely, the same AWQ model goes from 68 to 741 tokens per second depending on whether you serve it with the naive kernel or with Marlin. The format does not only decide quality: it decides who will be able to make it run.

What calibration does to your model, and to your comparisons

That leaves the question everyone asks: which one keeps the model best? The honest answer is that there is no universal 4-bit winner, and the way we long believed there was is instructive. The received idea that “AWQ is better than GPTQ” was refuted head-on by a study with more than 500,000 evaluations on the Llama-3.1 family: at equal tuning, the two hold even on academic tests, and GPTQ even pulls ahead on real-world tasks like code, provided it is tuned correctly. The belief came from comparisons where GPTQ was poorly configured. What AWQ keeps is what its construction gives it and nothing more: robustness when the usage distribution drifts away from the calibration set. GGUF, for its part, draws from its importance matrix a net gain below 3 bits and a marginal one above Q4_K_M. Three constructions, three regimes where each shines, none that dominates everywhere.

The horizon: below the native-float frontier

Everything we have just described quantizes the weights and leaves the activations in 16 bits. That is a choice, and it is being challenged from above. The datacenter is migrating toward low-precision float formats that the silicon can now compute natively, the NVFP4 and MXFP4 of the Blackwell cards, whose mechanics we detailed in our FP8 and FP4 dossier. Where GPTQ, AWQ and GGUF are a software stage that arranges integers for hardware that only knows 16-bit float, native FP4 FP4 A 4-bit floating-point format, the 2026 frontier of high-throughput inference. Four times less memory than FP16, but a very narrow dynamic range: it only holds up with fine-grained scaling through block formats (MXFP4, NVFP4). computes directly in the compressed format. On that ground, the trio already looks like the old world.

There is, however, a precise reason it will not disappear from local inference any time soon, and it is hardware. Native FP4 is only fully usable on the datacenter Blackwell. The consumer Blackwell, an RTX 5090 for instance, falls back onto a path that decompresses the weights toward a Marlin-style integer kernel: it gains the memory, not the throughput. As long as the silicon under desks does not accelerate FP4 for real, post-training INT4 will remain the format of local inference, carried by two advantages the datacenter disdains: a single file to download, and no heavy calibration to run.

Around the trio, finally, a generation of methods is moving the lines on axes it does not occupy. Hadamard rotations, QuaRot and SpinQuant, scatter the outliers before quantization and make 4 bits possible on the activations too, not only the weights. Lattice quantizations, QTIP and its EXL3 implementation, push the frontier below 4 bits, down to models still coherent at 1.6 bits per weight. HQQ removes calibration. QAT QAT Quantization-Aware Training. Rather than compressing an already-trained model (and taking the accuracy hit), you briefly retrain it while simulating quantization, so it learns to live in low precision (often int4). It sharply reduces the degradation compared with naive post-training quantization. acts upstream, at training time, and Google already ships GGUFs trained this way for Gemma. None of them replaces the trio; they graft onto it, reusing its building blocks, the rotation before GPTQ, the error carry-over in EXL3.

The real question this dossier leaves open is therefore not “which one to choose today”, but what will survive when consumer hardware finally learns to compute 4-bit float natively. The operating point of the server algorithms could then melt into the native format. But the distinction that has structured this whole text, a container that arranges bits on one side, an algorithm welded to its kernel on the other, does not depend on any generation of silicon. It will still describe the split between the one you download as a file and the one you serve behind an API, long after today’s 4 bits has aged.

Sources and method

Editorial freeze: July 11, 2026. Labels: verified fact (primary source), credible estimate (consistent third-party analysis), assumption (stated reasoning). This dossier documents how the formats are built, not a score comparison: every quality figure is a snapshot that depends on the model, the bench and the kernel.

GPTQ and Hessian-based compensation

• Fact: GPTQ, Frantar et al., arXiv 2210.17323 (ICLR 2023): objective ||WX−ŴX||², Hessian H = 2XXᵀ (verbatim Section 3), OBS/OBQ update rule, shared fixed order (“three orders of magnitude”), Cholesky, damping (2XXᵀ+λI)⁻¹, calibration 128 × 2048 tokens of C4, 175B in ~4 GPU-hours. OBS lineage: Hassibi & Stork, NeurIPS 1992.
• Fact: damping λ ≈ 1% of the mean diagonal, act-order/desc_act, implementation defaults (GPTQModel, HF wrapper): Transformers/GPTQModel docs, accessed July 11, 2026. 2026 tooling: AutoGPTQ archived April 11, 2025, successor GPTQModel.

AWQ and activation scaling

• Fact: AWQ, Lin et al., arXiv 2306.00978 (MLSys 2024): 0.1 to 1% of salient channels, selection by activation magnitude (PPL 43.2 → 13.0), invariance WX=(W·diag(s))(diag(s)⁻¹X), grid search of 20, optimum s≈2. Out-of-domain robustness +0.5 to 0.6 PPL against +2.3 to 4.9 for GPTQ (Figure 6). No backpropagation and no reconstruction (verbatim). AutoAWQ archived May 11, 2025, taken over by llm-compressor.

GGUF, k-quants, i-quants

• Fact: container format, type per tensor, 32-byte alignment for mmap: ggml spec (github.com/ggml-org/ggml, gguf.md).
• Fact: legacy quants (blocks of 32, Q4_0 symmetric / Q4_1 affine) and k-quants (super-blocks of 256, two-level scales, Q2_K 2.56 bpw to Q6_K 6.56 bpw): ikawrakow’s PR #1684 (June 5, 2023), structures and static_assert in ggml-common.h.
• Fact: per-tensor mixed precision, use_more_bits heuristic (first and last layers, one in three), promotion of attn_v and ffn_down to Q6_K for Q4_K_M: src/llama-quant.cpp (verbatim).
• Fact: i-quants (codebook, E8 lattice, QuIP# lineage arXiv 2402.04396) and importance matrix weighting the error by <activation²>, made mandatory below 3 bits: PR #4773, #4861, #4897. Rationale for the proliferation (“finer control over the size-quality trade-off”, edge target): Discussion #5063.

Fidelity, kernels, lock-in

• Fact: no universal winner, refutation of “AWQ better than GPTQ”: Kurtic et al., “Give Me BF16 or Give Me Death”, arXiv 2411.02355 (ACL 2025), more than 500,000 evaluations on Llama-3.1.
• Fact: Marlin kernels (arXiv 2408.11743, ~4× up to batch 16-32) and Machete (Hopper); GGUF maintenance cost in vLLM (RFC #39583, ~6000 lines of CUDA, ~0.1% of usage).
• Estimate: AWQ 68 → 741 tok/s depending on the kernel, GGUF 93 vs 461 tok/s in vLLM: third-party bench (Qwen2.5-32B, unreviewed blog post), cited with its scope.

Outliers and horizon

• Fact: emergent outliers (magnitude ≥ 6, from ~6.7B parameters): LLM.int8(), arXiv 2208.07339. SmoothQuant, arXiv 2211.10438.
• Fact: native Blackwell FP (NVFP4 blocks of 16, MXFP4 blocks of 32): see our FP8/FP4 dossier. Supported assumption: consumer Blackwell (SM120) in Marlin weight-only fallback, vLLM issues #30135 and #31085 (December 2025).
• Fact: QuaRot (arXiv 2404.00456), SpinQuant (arXiv 2405.16406), QTIP (arXiv 2406.11235) and EXL3 (exllamav3), HQQ (Mobius Labs), Gemma 3 QAT (Google, April 2025).