Complete Homelab AI Stack Guide: Run Your Own Local AI

Welcome back to Self-Hosted Tech! If you’ve been hanging around the homelab community for the last year, you’ve probably noticed that local AI has completely taken over the conversation. We went from running Pi-hole and Plex to suddenly hosting large language models (LLMs) in our closets.

But let’s be honest: diving into local AI can feel overwhelming. Between Hugging Face repositories, quantization formats like GGUF, Python environments, and vector databases, it’s easy to get lost in the weeds.

That’s why we’re putting together the complete homelab AI stack guide. We’re going to cut through the noise, explain exactly what you need, and walk you through setting up a fully private, self-hosted AI environment using tools you already know and love—specifically, Docker.

Grab a coffee, fire up your terminal, and let’s get your homelab thinking.

Why Self-Host Your AI?

Before we get our hands dirty with YAML files, why bother running AI locally when ChatGPT and Claude are just a browser tab away?

Total Privacy: Your data never leaves your network. Whether you’re summarizing proprietary company documents, writing personal journal entries, or generating code with sensitive API keys, it stays in your house.
Zero API Costs: Once you’ve bought the hardware, inference is free. No monthly subscriptions, no per-token billing surprises.
Customization and Control: You choose the model. You choose the system prompt. You can fine-tune it, restrict it, or let it run wild. It’s your sandbox.
Offline Capability: Internet goes down? Your AI still works.

Hardware Requirements: The Elephant in the Room

AI is resource-heavy. There’s no way around it. But you don’t need a $20,000 NVIDIA H100 cluster to have fun. Here is the practical breakdown of what you need.

The GPU: Your Best Friend

If you want fast, responsive AI, you need a GPU. Specifically, you want an NVIDIA GPU because their CUDA ecosystem is the gold standard for AI workloads. AMD is making strides with ROCm, but NVIDIA is still the path of least resistance for homelabbers.

The Sweet Spot: An NVIDIA RTX 3060 with 12GB of VRAM. It’s relatively cheap on the used market, and 12GB of VRAM is enough to comfortably run excellent 8-billion parameter models (like Llama 3 8B) with plenty of context room to spare.
The Upgrade: An RTX 4060 Ti 16GB or a used RTX 3090 (24GB VRAM) if you want to run larger 14B to 30B models.

CPU and RAM: The Fallback

Can you run AI without a GPU? Yes. Tools like llama.cpp are designed to run on your CPU. However, it will be significantly slower. If you’re going the CPU route, you want a modern multi-core processor and at least 32GB of system RAM.

Note for Apple users: Apple Silicon (M1/M2/M3/M4) is shockingly good at local AI because it uses unified memory. A Mac Mini with 32GB of RAM can easily act as a dedicated, low-power AI server.

Storage: Speed Matters

AI models are massive files (often 4GB to 20GB each). You want to store these on an NVMe SSD. Loading a 10GB model into VRAM from a spinning hard drive will take forever; from an NVMe drive, it takes seconds.

Understanding the Homelab AI Stack

To build a functional AI stack, you need a few distinct components. Think of it like a traditional web app: you need a backend, a frontend, and a database.

The Inference Engine (Backend): This is the software that actually loads the model into your GPU/CPU and generates text. We will use Ollama. It’s incredibly easy to use, manages model downloads automatically, and exposes a clean API.
The User Interface (Frontend): You don’t want to chat with your AI in a terminal window (well, maybe you do, but it’s not great for sharing with your family). We will use Open WebUI, a beautiful, ChatGPT-like interface that connects to Ollama.
The Vector Database (For RAG): If you want your AI to read your documents and answer questions about them (Retrieval-Augmented Generation, or RAG), you need a vector database. The good news? Open WebUI has a vector database (ChromaDB) built right in, so we don’t need a separate container for this beginner stack!

The Practical Walkthrough: Deploying Your AI Stack

Let’s build the stack. We’re going to use Docker Compose, which is the standard way we deploy things in the homelab.

Step 1: Prepare Your Environment

Make sure you have Docker and Docker Compose installed on your Linux server (or Windows/Mac if that’s what you’re using). If you have an NVIDIA GPU, you also need the NVIDIA Container Toolkit installed on your host machine so Docker can talk to your GPU.

Create a new directory for your AI stack and navigate into it:

1
2
mkdir ~/ai-stack
cd ~/ai-stack

Step 2: The Docker Compose File

Create a docker-compose.yml file in your ai-stack directory:

1
nano docker-compose.yml

Paste the following configuration into the file. This will spin up both Ollama and Open WebUI, link them together on a private Docker network, and set up persistent storage so your models and chat history don’t disappear when you restart the containers.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
version: '3.8'

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"
    volumes:
      - ./ollama_data:/root/.ollama
    # If you have an NVIDIA GPU, uncomment the deploy block below.
    # If you are using CPU only, leave it commented out.
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: 1
    #           capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - ./open-webui_data:/app/backend/data
    depends_on:
      - ollama

Step 3: Fire It Up

Save the file and exit the text editor. Now, simply run:

1
docker compose up -d

Docker will pull the images and start the containers. You can check the status by running docker compose ps.

Step 4: Accessing the Interface

Open your web browser and navigate to http://<your-server-ip>:3000.

The first time you visit Open WebUI, it will ask you to create an admin account. This account is stored locally in the open-webui_data volume. Go ahead and create your username and password. You are now looking at your very own, completely private ChatGPT clone!

Running Your First Model

Right now, your frontend is up, but your backend has no “brains” loaded. We need to download a model. Ollama makes this incredibly easy.

Option 1: Via the Command Line

Open a terminal on your server and exec into the Ollama container to pull a model. Let’s use Meta’s Llama 3, which is a fantastic, highly capable open-weight model.

1
docker exec -it ollama ollama run llama3

This command will download the model (it’s about 4.7GB) and drop you into a chat prompt. You can chat with it right there in the terminal to test it out. Type /bye to exit.

Option 2: Via the Web UI

If you prefer a GUI, go to your Open WebUI in the browser.

Click on your profile picture in the bottom left.
Go to Admin Settings -> Models.
You’ll see a section to download a model. Type llama3 into the box and click the download button.

Once the model is downloaded, go back to the main chat screen, select llama3 from the model dropdown at the top, and say hello! You now have a fully functioning local AI.

Taking It Further: Talking to Your Documents (RAG)

A standalone chatbot is fun, but the real power of a homelab AI stack is making it work with your data. This is where Retrieval-Augmented Generation (RAG) comes in.

Instead of trying to stuff a 500-page PDF into the AI’s prompt window (which won’t work due to context limits), RAG works like this:

Your document is chopped into small chunks.
Those chunks are converted into numbers (vectors) that represent their meaning.
The vectors are stored in a database.
When you ask a question, the AI searches the database for the most relevant chunks, reads them, and generates an answer based only on those chunks.

Open WebUI has this entire process built-in. You don’t need to deploy a separate Langchain or ChromaDB setup just to read a few PDFs.

How to use it:

In Open WebUI, click the + button next to the chat bar (or click the paperclip icon).
Upload a PDF, a text file, or even a CSV.
Ask a question about the document in the chat bar. For example, “Can you summarize the first chapter of this manual?”

Open WebUI will automatically process the document in the background using its built-in embedding model and vector database, retrieve the right context, and feed it to Llama 3 to answer your question. It feels like magic, but it’s all happening on your hardware.

Expanding Your Stack

Once you have Ollama and Open WebUI running smoothly, you might want to expand. Here are a few next steps to consider for your homelab:

Try Different Models: Don’t stop at Llama 3. Try mistral for a great general-purpose model, or codellama if you want help writing code. Just use the ollama run <model-name> command.
Add Automation with n8n: Since Ollama exposes a standard API on port 11434, you can point automation tools like n8n or Home Assistant to it. Imagine a webhook that triggers an AI to summarize your security camera footage logs and send you a Telegram message!
Dedicated RAG Tools: If you outgrow Open WebUI’s built-in RAG capabilities, look into deploying AnythingLLM or Dify. These are dedicated AI workflow tools that give you much more control over how documents are chunked and retrieved.

Summary

Building a homelab AI stack is no longer a complex, multi-day project requiring a degree in data science. With Ollama handling the heavy lifting of model inference and Open WebUI providing a polished, feature-rich interface, you can have a completely private, offline AI running in about ten minutes.

By leveraging Docker Compose, you ensure your setup is portable, easy to back up, and simple to update. Whether you’re running it on a beefy server with dual 3090s or a humble Mac Mini, local AI puts the power of generative artificial intelligence right back in your hands.

What’s next? We want to hear from you! What models are you running in your homelab? Are you using your AI for coding, writing, or analyzing your homelab logs? Drop a comment below or join our community Discord to share your setups.

If you found this guide helpful, make sure to subscribe to the Self-Hosted Tech YouTube channel and bookmark the blog for more practical homelab tutorials. Until next time, keep your networks private and your containers running!

Complete Homelab AI Stack Guide: Run Your Own Local AI#

Why Self-Host Your AI?#

Hardware Requirements: The Elephant in the Room#

The GPU: Your Best Friend#

CPU and RAM: The Fallback#

Storage: Speed Matters#

Understanding the Homelab AI Stack#

The Practical Walkthrough: Deploying Your AI Stack#

Step 1: Prepare Your Environment#

Step 2: The Docker Compose File#

Step 3: Fire It Up#

Step 4: Accessing the Interface#

Running Your First Model#

Option 1: Via the Command Line#

Option 2: Via the Web UI#

Taking It Further: Talking to Your Documents (RAG)#

How to use it:#

Expanding Your Stack#

Summary#