Quantization Explained: GGUF, Q4, Q8, and Why It Matters

You’ve Seen the Numbers

You’re browsing models on Ollama or HuggingFace and you see things like:

• qwen3.8:27b-q4_K_M
• llama3.1:8b-q8_0
• mistral:7b-fp16

What does Q4 mean? What’s Q8? Why is the same model 4GB at one setting and 16GB at another? And what the hell is GGUF?

This guide explains all of it — in plain English, with real numbers from actual models.

What Is Quantization?

Quantization is compression for AI models. That’s it.

A neural network is basically billions of numbers (weights) stored as 16-bit or 32-bit floating point numbers. Each number takes up memory. When you quantize a model, you reduce the precision of those numbers — storing them in fewer bits — so the model takes up less space.

Think of it like image compression. A raw PNG photo might be 25MB. Save it as a JPEG at 85% quality and it’s 3MB. Most people can’t tell the difference. The image is “good enough” and it’s 8x smaller.

Quantization does the same thing for language models. You trade a small amount of quality for a massive reduction in file size and memory usage.

Why It Exists

Without quantization, running a decent language model locally would be impossible for most people.

Here’s the raw size of a 27-billion parameter model at full precision (FP16):

• 27 billion parameters × 2 bytes each = 54 GB

That’s 54GB just to load the model — before you’ve even started generating text. You’d need a GPU with 64GB+ of VRAM, which costs thousands of dollars.

Now quantize that same model to 4-bit:

• 27 billion parameters × ~0.5 bytes each = ~14 GB

Suddenly it fits on a single 16GB graphics card. That’s the entire point.

The Numbers: Q4, Q5, Q8, F16

Here’s what each label actually means:

The tradeoff is always the same: lower bits = smaller file = less VRAM needed = some quality loss.

Real-World Example: Qwen 3.8 27B

Let’s look at the same model at different quantization levels. Qwen 3.8 27B is a popular model for local AI:

Most people running this model on a 16GB GPU use Q4_K_M. It’s the sweet spot — the model fits comfortably, quality is good, and there’s room left for the context window (the memory your conversation takes up as it gets longer).

What Is GGUF?

GGUF (GPT-Generated Unified Format) is the file format that stores quantized models. It was created by the llama.cpp project and has become the standard for local AI.

Before GGUF, there was GGML (the older format). If you see GGML mentioned, it’s the predecessor — same idea, older format. GGUF replaced it because it’s more flexible and extensible.

Why GGUF matters:

• Single file: Everything (weights, tokenizer, metadata) in one .gguf file
• Flexible quantization: Different layers can use different bit depths (that’s what the _K variants do)
• Universal: Works with Ollama, LM Studio, llama.cpp, and most local AI tools
• Efficient: Designed for fast loading and memory mapping

When you see q4_K_M, the _K_M part means it uses a “K-quant” method with “Medium” mixing — different layers of the model use different precision levels to optimize the quality/size tradeoff. The important layers get more bits, the less important ones get fewer.

Which One Should You Pick?

For most people, Q4_K_M is the answer. It’s the default in Ollama for a reason — it’s the best balance of size and quality. Here’s a quick guide:

8GB VRAM (RTX 3060 Ti, 4060)

• Q4 for 7B-8B models (fits comfortably)
• Q3 for 12B-14B models (tight but workable)
• Stick to smaller models for best experience

12GB VRAM (RTX 3060, 4070)

• Q4 for 12B-14B models (sweet spot)
• Q3 for 27B models (usable but degraded)
• Q8 for 7B-8B models (if you want max quality)

16GB VRAM (RTX 4080, 5060 Ti)

• Q4_K_M for 27B models (most popular choice)
• Q5 or Q6 for 12B-14B models (excellent quality)
• Q8 for 7B-8B models (near-perfect quality)

24GB VRAM (RTX 3090, 4090)

• Q8 for 27B models (excellent quality)
• Q5 or Q6 for 30B+ models
• Q4 for 70B models (if you want to try)

No GPU (CPU only)

• Q4 for 3B-7B models (runs on CPU, just slower)
• Q2 for larger models (not recommended — quality too low)
• CPU inference is 5-10x slower than GPU

Does Quantization Actually Hurt Quality?

Yes, but less than you’d think.

Studies and community testing consistently show:

• Q8: Indistinguishable from FP16 for most tasks. Quality loss is in the noise.
• Q6: Near-identical for chat and general use. Minor differences in edge cases.
• Q5: Slight degradation on complex reasoning and coding. Fine for most use.
• Q4: Noticeable on careful comparison, especially in coding and math. Still good for general chat, writing, and Q&A.
• Q3 and below: Degradation becomes obvious. Models may hallucinate more, lose nuance, or produce lower-quality code.

For most people doing general chat, writing, and Q&A, Q4 is completely fine. If you’re doing coding or complex reasoning, bump up to Q5 or Q6 if your VRAM allows.

The KV Cache: The Hidden Memory Cost

Here’s something most guides don’t mention: the model file size isn’t your total memory usage.

When you’re having a conversation with a model, it has to remember everything you’ve said so far. This is stored in something called the KV cache (Key-Value cache), and it grows as your conversation gets longer.

A 27B model at Q4 might be 16.5GB on disk, but with a long conversation, the KV cache can add 2-4GB more. That’s why a “16GB” model doesn’t actually fit on a 16GB GPU with a long conversation — you need headroom.

Rule of thumb: Leave 20% of your VRAM for the KV cache and runtime overhead. On a 16GB card, aim for models that are 12-13GB or smaller at your chosen quantization.

How to Specify Quantization in Ollama

Ollama makes this easy. When you pull a model, you can specify the quantization:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

# Default (usually Q4_K_M)
ollama pull qwen3.8:27b

# Specific quantization
ollama pull qwen3.8:27b-q8_0
ollama pull qwen3.8:27b-q5_K_M
ollama pull qwen3.8:27b-q4_K_M

# Full precision (if you have the VRAM)
ollama pull qwen3.8:27b-fp16

If you don’t specify, Ollama picks Q4_K_M by default — which is the right choice for most people.

The Takeaway

Quantization is the reason local AI works at all. Without it, you’d need $10,000+ worth of GPUs to run anything decent. With it, a used $200 graphics card can run capable models that would have seemed impossible just two years ago.

The short version:

• Use Q4_K_M as your default
• Bump to Q5 or Q6 if you have VRAM headroom and want better quality
• Use Q8 for small models where quality matters most
• Leave 20% VRAM headroom for the KV cache
• Don’t go below Q3 — the quality drop isn’t worth the savings

That’s it. You now understand quantization better than most people running local AI. Go pull a model.