Run Local AI with No GPU: Your CPU Is All You Need

If you’ve been following the explosion of AI over the last couple of years, you might be under the impression that to run your own large language models (LLMs) locally, you need a massive, $2,000+ graphics card with 24GB of VRAM.

Well, I’m here to let you in on a little secret: the GPU cartels want you to think that, but it’s simply not true.

Welcome to the world of CPU inference. With recent breakthroughs in model optimization and inference engines, you can run surprisingly capable AI models entirely on your CPU and system RAM. No Nvidia, no AMD ROCm, no fancy Apple Silicon unified memory—just your trusty old processor doing the heavy lifting.

In this tutorial, we’re going to break down exactly how local AI works without a GPU, what hardware you actually need, and I’ll walk you through setting up a completely local, private ChatGPT-style interface using Docker.

Why Run Local AI in the First Place?

Before we get our hands dirty with terminal commands, let’s talk about why you’d want to do this. If OpenAI, Anthropic, and Google offer amazing APIs, why bother hosting your own?

Total Privacy: When you run a model locally, your prompts never leave your network. You can feed it your proprietary code, personal journals, or sensitive financial data without worrying about it being used to train someone else’s corporate model.
Zero Recurring Costs: API costs add up fast. Once you have the hardware (which you already own), running local models is completely free.
Offline Capability: Whether your internet goes down or you’re working in a remote cabin, your AI assistant is still online.
The Homelab Itch: Let’s be honest, we’re homelabbers. We run our own media servers, DNS blockers, and smart home hubs. Running our own AI is just the next logical step.

The Magic Trick: Quantization and GGUF

So, how do we run a model that was originally designed to run on massive data center GPUs on a standard desktop CPU? The answer is a process called quantization.

Large language models are essentially giant webs of numbers (weights) stored in high precision—usually 16-bit floating-point (FP16). An 8-billion parameter model in FP16 takes up about 16GB of memory. That’s too much for most system RAM setups to handle efficiently alongside an operating system.

Quantization is a compression technique. It rounds off those highly precise numbers into lower precision formats, like 8-bit integers (INT8) or 4-bit integers (INT4). By dropping from 16-bit to 4-bit, we shrink the model size by roughly 75%. That same 8-billion parameter model suddenly only takes up about 5GB of memory.

The file format that makes this possible for CPU inference is GGUF (GPT-Generated Unified Format). GGUF is designed specifically to run efficiently on CPUs using a framework called llama.cpp. It allows the model to load into your standard system RAM and utilizes your CPU’s instruction sets (like AVX2) to crunch the math as fast as possible.

Hardware Requirements: What Do You Actually Need?

You don’t need a GPU, but you do need a reasonably decent machine. Here is the holy trinity of CPU AI:

System RAM (The most important part): Because the model is loaded directly into your system memory, RAM capacity is king. To run an 8B parameter model comfortably, you want at least 16GB of RAM. For smaller models (like a 3B parameter model), 8GB will scrape by, but 16GB is the sweet spot.
A Modern CPU: Any CPU from the last 5–7 years will work, but the more cores and threads you have, the better. Modern Intel Core i5/i7s and AMD Ryzen 5/7s are fantastic for this. AVX2 instruction set support is highly recommended for speed.
An SSD: Loading a 5GB model from a spinning hard drive will take forever. An NVMe SSD or at least a decent SATA SSD is required to load the model into RAM in a reasonable timeframe.

Note on performance: Manage your expectations. A high-end GPU can output 50-100 tokens (words/pieces of words) per second. A good CPU running a 4-bit quantized model will likely output between 5 and 15 tokens per second. It’s slower, but perfectly usable for most text-based tasks.

The Walkthrough: Running Ollama and Open WebUI

Let’s get to the fun part. For this walkthrough, we are going to use Ollama, an incredibly popular, lightweight framework for running LLMs, and Open WebUI, a beautiful, self-hosted frontend that looks and acts a lot like ChatGPT.

Since we’re homelabbers, we’re going to do this entirely in Docker.

Step 1: Set Up Ollama in Docker

First, let’s get the Ollama backend running. SSH into your Linux server (or open your terminal if you’re doing this on a desktop).

Create a directory for your AI stack and navigate into it:

1
2
mkdir ~/local-ai-stack
cd ~/local-ai-stack

Create a docker-compose.yml file:

1
nano docker-compose.yml

Paste the following configuration into the file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
version: '3.8'

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ./ollama-data:/root/.ollama
    restart: unless-stopped

Save and exit (Ctrl+O, Enter, Ctrl+X in nano).

Bring the container online:

1
docker compose up -d

Ollama is now running. It will automatically detect that you don’t have a GPU available and fall back to CPU mode using llama.cpp under the hood.

Step 2: Download and Run a Model

Now we need an actual AI model. Ollama makes this incredibly easy. We are going to pull Llama 3 (8B), Meta’s incredibly capable open-source model.

Because we’re running in Docker, we need to execute the pull command inside the container. Run this:

1
docker exec -it ollama ollama run llama3

This command does two things:

It downloads the 4-bit quantized GGUF version of Llama 3 8B (about 4.7GB).
Once the download finishes, it drops you into an interactive chat prompt.

Try typing a question: > Explain quantum computing to a five-year-old.

You should see the text slowly stream onto your screen. Congratulations! You are running a cutting-edge LLM entirely on your CPU.

Type /bye to exit the chat.

Step 3: Add a Web UI

Chatting in the terminal is cool, but it’s not exactly a great daily driver. Let’s bolt on Open WebUI.

Open your docker-compose.yml file again:

1
nano docker-compose.yml

We are going to add the Open WebUI service and link it to Ollama. Update your file to look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
version: '3.8'

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ./ollama-data:/root/.ollama
    restart: unless-stopped

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - ./open-webui-data:/app/backend/data
    depends_on:
      - ollama
    restart: unless-stopped

Save and exit. Then, spin up the new container:

1
docker compose up -d

Step 4: Chatting in the Browser

Open your web browser and navigate to http://<your-server-ip>:3000.

You’ll be greeted by the Open WebUI setup screen. Create an admin account (this is stored locally in the Docker volume we created; it never leaves your machine).

Once logged in, you’ll see a familiar chat interface. At the top, you can select your model—llama3 should be there since we pulled it earlier.

Click “New Chat” and start asking questions. You now have a completely private, self-hosted, GPU-free alternative to ChatGPT running in your homelab!

Tips for Squeezing Out More Performance

If you’re finding the generation speed a bit sluggish, here are a few tips to optimize CPU inference:

Tweak the Thread Count: By default, Ollama will try to guess how many CPU threads to use. Sometimes it guesses wrong. You can explicitly set the thread count when running a model. For example: docker exec -it ollama ollama run llama3 --num-thread 4 (replace 4 with the number of physical cores you have, not logical threads).
Try Smaller Models: If Llama 3 8B is too slow, try a smaller model. Microsoft’s Phi-3 Mini is a 3.8B parameter model that punches way above its weight class. You can download it by running docker exec -it ollama ollama run phi3. It will run significantly faster on older CPUs.
Close Background Apps: Since you are using system RAM and CPU, heavy background tasks (like transcoding a 4K movie on Plex) will severely impact your tokens-per-second. Give the AI some breathing room.
RAM Speed Matters: Unlike GPU VRAM (which is standardized and incredibly fast), system RAM speed varies. If you’re building a dedicated CPU AI box, investing in faster DDR5 RAM will yield noticeable improvements in generation speed.

What Can You Actually Do With This?

Now that you have a local AI running, what is it good for?

Summarization: Paste in long articles, documentation, or meeting transcripts and ask it for a bulleted summary.
Coding Assistance: Llama 3 is surprisingly good at Python, Bash, and YAML. Ask it to write you a Docker Compose file or a quick Python script to sort your downloads folder.
Brainstorming and Drafting: Use it to draft emails, blog posts, or creative writing without sending your ideas to a third party.
Local RAG (Retrieval-Augmented Generation): Open WebUI actually has a built-in document feature. You can upload PDFs or text files directly into the UI, and the AI will read them and answer questions based on your documents. It’s like having a private chatbot trained on your own data.

Summary

The barrier to entry for local AI has never been lower. You don’t need to empty your wallet on a high-end graphics card to experience the power of self-hosted LLMs. By leveraging quantization (GGUF), the Ollama framework, and a bit of Docker magic, you can turn any old desktop or homelab server into a private, offline AI assistant.

It might not generate text at lightning speed, but the privacy, control, and zero-cost operation make CPU inference an absolute must-try for any self-hoster.

Over to you: What model are you going to run first? Have you tried CPU inference before, or is this your first time stepping into local AI? Drop a comment below and let us know how your setup goes! And if you found this tutorial helpful, make sure to subscribe to the Self-Hosted Tech YouTube channel and blog for more guides on homelabs, Docker, and local AI.

Run Local AI with No GPU: Your CPU Is All You Need#

Why Run Local AI in the First Place?#

The Magic Trick: Quantization and GGUF#

Hardware Requirements: What Do You Actually Need?#

The Walkthrough: Running Ollama and Open WebUI#

Step 1: Set Up Ollama in Docker#

Step 2: Download and Run a Model#

Step 3: Add a Web UI#

Step 4: Chatting in the Browser#

Tips for Squeezing Out More Performance#

What Can You Actually Do With This?#

Summary#