Every GGUF file you download is a lossy compression choice someone already made for you. The name tells you what you’re trading away.

What quantization actually does

A model’s weights start as 16-bit or 32-bit floats. Quantization rounds each weight down to fewer bits — 8, 5, 4, even 2. Fewer bits means a smaller file and faster inference, because the GPU or CPU moves less data per token. It also means each weight is a coarser approximation of the original, and that shows up as worse output: more repetition, more wrong facts, more broken JSON.

The question is never “does quantization hurt quality” — it always does, some amount. The question is whether you can tell the difference at the level you picked.

Reading the filename

A file named llama-3-8b.Q4_K_M.gguf breaks down like this:

• Q4 — 4 bits per weight, roughly. This is the ballpark number.
• K — uses the newer “k-quant” scheme, which groups weights into blocks and quantizes each block with its own scale factor, instead of one scale for the whole tensor. Almost always better than the legacy scheme at the same bit count. Prefer K over non-K every time it’s available.
• M (or S, L) — the size tier within that k-quant level. S is smaller and rougher, M is the usual pick, L keeps a few sensitive layers at higher precision.

So Q4_K_M reads as: 4-bit, k-quant, medium tier. That’s the default most people should reach for.

What you actually lose at each level

Level	Bits/weight	Size vs FP16	What you notice
Q8_0	~8.5	~50%	Essentially indistinguishable from the source model
Q6_K	~6.6	~40%	Safe for anything, including code and math
Q5_K_M	~5.7	~35%	Very close to Q6, good default for larger models
Q4_K_M	~4.8	~28%	Occasional word choice drift, still reliable for chat and RAG
Q3_K_M	~3.9	~22%	Noticeable degradation on reasoning and long context
Q2_K	~2.6	~16%	Coherent short answers, breaks down on anything complex

These percentages are relative to FP16 for a given model, not absolute file sizes — an 8B model at Q4_K_M lands around 4.5-5GB, a 70B model at the same level around 40GB.

The rule that actually matters

A bigger model at a lower quant usually beats a smaller model at a higher quant, for the same file size. A 14B model at Q4_K_M will generally outperform an 8B model at Q8_0, even though the file sizes are close. Quantization damage is a percentage hit on top of the model’s capability — it doesn’t invent capability. If you’re choosing between “biggest model that fits” and “highest precision on a smaller model,” pick the bigger model almost every time, down to about Q4. Below Q3, the damage gets steep enough that it’s worth reconsidering.

What to pick

• You have the VRAM to spare and want the safest output: Q6_K or Q8_0.
• Normal case, running on a single consumer GPU or Apple Silicon: Q4_K_M. This is the level llama.cpp and Ollama default to for a reason.
• You’re squeezing a model onto something small and accept some quality loss: Q3_K_M, and test it against your actual prompts before committing.
• Avoid: anything below Q2, and the legacy non-K quants (Q4_0, Q4_1) unless a tool specifically requires them for compatibility.

Don’t trust a vibes-based “feels fine” test. If quality matters for your use case, run the same 20-30 prompts through two quant levels and diff the outputs. The gap is often smaller than people assume at Q4 and larger than people assume at Q3.

---