Quantization Explained: Making LLMs Fit Your GPU
If you’ve spent any time in the homelab or local AI space recently, you’ve probably run into the same frustrating wall we all do. You read about an amazing new open-source Large Language Model (LLM) like Llama 3 or Mistral. You excitedly fire up your terminal, download the model, hit “Run,” and then… boom.
CUDA out of memory. Tried to allocate 2.00 GiB...
It’s the classic homelabber’s heartbreak. You spent your hard-earned money on a GPU with 8GB or 12GB of VRAM, and it’s still not enough to run a “small” 7-billion parameter model.
But wait. How are people on Reddit and YouTube running these massive models on consumer hardware? How is someone running a 70-billion parameter model on a single RTX 3090?
The answer is a mathematical magic trick called quantization.
In this article, we’re going to break down exactly what quantization is, how it works, why it’s the only reason local AI is viable for homelabs, and how you can start using it today.
What the Heck is Quantization?
To understand quantization, we first need to understand how neural networks store their “knowledge.”
An LLM is essentially a massive web of mathematical weights (parameters). These weights determine how the model processes the text you feed it. By default, these weights are stored as high-precision floating-point numbers—specifically, 32-bit floating-point numbers (FP32).
A single 32-bit number takes up 4 bytes of memory.
Let’s do the math on a relatively small 7-billion parameter model (like Llama 2 or 3):
- 7,000,000,000 parameters × 4 bytes = 28,000,000,000 bytes
- 28,000,000,000 bytes ÷ 1024 = ~27.3 GB of VRAM
To run a 7B model in its default, uncompressed FP32 format, you need almost 28GB of VRAM just to load the weights. That doesn’t even account for the memory needed to process your prompt (the context window) or generate the response (the KV cache). You’d need an expensive enterprise GPU or multiple consumer cards just to get it running.
Quantization is the process of reducing the precision of these numbers to save space.
Think of it like saving an image. A raw, uncompressed BMP image preserves every single pixel perfectly, but the file size is massive. If you save that same image as a JPEG, you compress it. You lose a tiny bit of visual fidelity, but the file size shrinks by 90%. To the naked eye, the JPEG looks identical to the BMP.
Quantization does the exact same thing to LLMs.
The Levels of Precision
Instead of storing weights in FP32 (4 bytes per parameter), we can step down the precision:
- FP16 / BF16 (16-bit): 2 bytes per parameter. Our 7B model now takes ~14GB. This is the standard for most cloud inference, but still too big for many consumer GPUs.
- INT8 (8-bit): 1 byte per parameter. Our 7B model now takes ~7GB. Now we’re talking! This fits comfortably on an 8GB RTX 3060 or 4060.
- INT4 (4-bit): 0.5 bytes per parameter. Our 7B model now takes ~3.5GB. This will run on practically any modern GPU, including older 6GB cards, with room left over for a decent context window.
By dropping from 32-bit to 4-bit, we’ve reduced the memory requirement by 87.5%.
Does It Make the Model Dumber?
This is the most common question. If we’re essentially “rounding off” the numbers in the AI’s brain, doesn’t it get worse at its job?
The short answer is: slightly, but not as much as you’d think.
Neural networks are surprisingly resilient. Research has shown that the vast majority of the “knowledge” in an LLM is preserved even at 4-bit precision. An 8-bit quantized model is practically indistinguishable from the original 16-bit model in everyday use.
A 4-bit model might lose a tiny fraction of a percentage point on complex reasoning benchmarks, but for 95% of use cases—chatting, summarizing text, writing code, role-playing—you will never notice the difference.
The trade-off is overwhelmingly in your favor: you sacrifice a microscopic amount of accuracy to gain the ability to run the model on hardware you already own.
The Flavors of Quantization
When you start downloading local models, you’ll notice they don’t just say “4-bit.” You’ll see a confusing alphabet soup of formats and terms. Here is the homelab cheat sheet for what you’re looking at.
GGUF (and the K-Quants)
GGUF (GPT-Generated Unified Format) is currently the undisputed king of local AI. Created by the team behind llama.cpp, GGUF is designed to run on CPUs, Apple Silicon (Macs), and GPUs.
If your GPU runs out of VRAM, GGUF allows you to “offload” the rest of the model to your system RAM and CPU. It’s slower, but it means you can run massive models even if your GPU is too small.
When browsing GGUF models (usually on Hugging Face), you’ll see different quantization levels denoted by a Q followed by a number. The most common are the K-quants:
- Q4_K_M: The absolute sweet spot. It stands for 4-bit K-medium. It offers an excellent balance of size, speed, and quality. If you don’t know what to pick, pick this.
- Q5_K_M: Slightly larger and slightly smarter than Q4. Great if you have a little extra VRAM to spare.
- Q8_0: 8-bit. Almost perfect parity with the original model, but takes twice the VRAM of a Q4.
GPTQ and AWQ
If GGUF is the king of flexibility, GPTQ and AWQ (Activation-aware Weight Quantization) are the kings of pure GPU speed.
These formats are designed specifically to run on Nvidia and AMD GPUs. They don’t support CPU offloading like GGUF does, but because they are highly optimized for pure GPU execution, they generate text incredibly fast.
AWQ is generally considered the modern successor to GPTQ, offering slightly better accuracy at the 4-bit level and faster inference speeds on modern GPUs.
EXL2 (ExLlamaV2)
If you are an enthusiast chasing maximum tokens-per-second, EXL2 is where it’s at. It allows for fractional quantization. Instead of being locked into 4-bit or 8-bit, you can download an EXL2 model quantized to, say, 4.5 bits per weight or 6.2 bits per weight. This lets you perfectly size a model to fit exactly into your available VRAM, leaving just enough room for a massive context window.
Calculating Your VRAM Needs
Before you download a model, it helps to know if it will actually fit on your GPU. Here is a simple formula to estimate the VRAM required to load a quantized model:
(Parameters in billions) × (Bytes per parameter) = VRAM needed in GB
Let’s look at a 4-bit (0.5 bytes) quantized model:
- 7B model: 7 × 0.5 = 3.5 GB of VRAM for weights.
- 13B model: 13 × 0.5 = 6.5 GB of VRAM for weights.
- 70B model: 70 × 0.5 = 35 GB of VRAM for weights.
Don’t forget the Context Window! Loading the weights is only half the battle. As you chat with the model, it has to remember the conversation. This memory is stored in the “KV Cache.”
A good rule of thumb is to take your total GPU VRAM, subtract the model weight size, and whatever is left over is what you can use for context. For example, if you have a 12GB RTX 3060, and you load a 7B Q4 model (3.5GB), you have about 8.5GB of VRAM left over. That is plenty of room for a massive context window, allowing you to paste in entire documents for the model to read.
Practical Walkthrough: Running Your First Quantized Model
Let’s get hands-on. The easiest way to dip your toes into the local AI waters is using Ollama. Ollama is a lightweight, open-source tool that runs in the background and handles all the complex llama.cpp configurations for you. It automatically downloads highly optimized Q4_K_M GGUF models.
Step 1: Install Ollama
If you’re on Linux (as many homelabbers are), you can install Ollama with a single curl script:
| |
If you are on Windows or macOS, you can download the installer directly from the Ollama website. It includes a nice desktop app.
Step 2: Run a Model
Open your terminal and tell Ollama to run Llama 3 (an 8-billion parameter model by Meta):
| |
The first time you run this command, Ollama will reach out to its repository and pull the model. Notice the file size it downloads—it will be around 4.7 GB. That is your clue that it is downloading a 4-bit quantized GGUF file. An uncompressed 8B model would be over 16GB!
Step 3: Verify VRAM Usage
Once the model finishes downloading, you’ll be dropped into a chat prompt. You can ask it a question:
| |
While it is generating the response, open your system monitor or run nvtop (an amazing task manager for Nvidia GPUs) in another terminal window.
You will see your GPU usage spike to 100%, and you will see your VRAM fill up. If you have an 8GB card, you’ll see roughly 5GB of VRAM being used.
Congratulations! You are running an 8-billion parameter neural network entirely locally on your own hardware, without sending your data to OpenAI or Google.
Step 4: Pushing the Limits
Let’s try something bigger. Let’s run a 70-billion parameter model. Exit the current chat by typing /bye.
Now, let’s run a heavily quantized version of a 70B model:
| |
If you only have an 8GB GPU, Ollama will load 8GB worth of the model into your VRAM, and it will push the remaining ~27GB into your system RAM. It will be slow—probably generating only 1 or 2 tokens (words) per second—but it will run.
If you have a Mac with unified memory (like an M2 or M3 Max with 64GB of RAM), this will run beautifully, as Mac architecture allows the GPU to access system RAM at incredibly high speeds.
Choosing the Right Tool for the Job
While Ollama is fantastic for quick CLI access and API integrations, you might want a graphical interface.
- LM Studio: A great desktop app for Windows, Mac, and Linux. It has a built-in model browser that lets you search Hugging Face directly, filter by GGUF, and see exactly how much RAM/VRAM a specific quantization will take before you download it.
- Text-generation-webui (Oobabooga): The power-user choice. If you want to use AWQ, GPTQ, or EXL2 formats for maximum speed, this is the backend you want. It’s a bit more complex to set up via Docker or Conda, but it unlocks the absolute maximum performance of your GPU.
- Open WebUI: If you want a ChatGPT-like interface that connects to your Ollama backend, deploy Open WebUI in a Docker container. It’s a staple in the homelab community.
Summary
Quantization is the great equalizer of the AI world. By compressing 16-bit or 32-bit floating-point numbers down to 4-bit or 8-bit integers, we shrink massive, enterprise-grade neural networks down to sizes that fit on consumer graphics cards.
While there is a minor trade-off in accuracy, the ability to run private, uncensored, and highly capable LLMs in your own home—without paying API fees—is a massive win for the self-hosted community.
Call to Action: What GPU are you running in your homelab right now, and what’s the largest model you’ve managed to squeeze into it using quantization? Drop your specs and your favorite models in the comments below!
If you found this guide helpful, make sure to subscribe to the Self-Hosted Tech YouTube channel and hit the bell so you don’t miss our upcoming tutorials on building your own local AI RAG (Retrieval-Augmented Generation) pipelines. Happy hosting!