What is MoE (Mixture of Experts) in LLMs?

If you’ve been hanging out in the local AI and self-hosting communities for the last few months, you’ve probably heard the term “MoE” thrown around. Maybe you saw someone on Reddit raving about Mixtral, or you saw a new model release on Hugging Face touting “Mixture of Experts” architecture.

But what exactly is MoE? And more importantly, why should you, as a homelabber with a limited budget for GPUs, care about it?

In this article, we’re going to break down Mixture of Experts in plain English. We’ll look at how it works, why it’s a massive game-changer for self-hosted AI, and then we’ll do a practical walkthrough on how to actually run an MoE model right on your own hardware.

The Problem with Massive LLMs

To understand why MoE is so cool, we first have to understand the problem it solves.

Traditional Large Language Models (LLMs) like Llama-2 70B or Llama-3 70B are known as “dense” models. In a dense model, every single time the model generates a new word (a token), every single parameter in the entire network is activated and used in the calculation.

If you have a 70-billion-parameter model, generating one token requires running data through all 70 billion parameters. For us self-hosters, that means two things:

  1. You need a massive amount of VRAM just to hold the model in memory (often 40GB+ for an unquantized 70B model).
  2. Generating text is relatively slow because the GPU has to churn through all 70 billion parameters for every single word.

MoE flips this paradigm on its head.

What is MoE? The Corporate Analogy

Let’s use an analogy. Imagine you run a massive corporation with 100,000 employees. If your company operates like a “dense” LLM, every single time a customer asks a question—no matter how simple—you route that question to all 100,000 employees. Every single person stops what they are doing, reads the question, and contributes a tiny bit to the final answer. It’s incredibly thorough, but highly inefficient and slow.

Now, imagine you restructure your company into a “Mixture of Experts.” You divide your 100,000 employees into 10 specialized departments (Experts) of 10,000 people each: Legal, Engineering, HR, Sales, etc.

You also hire a receptionist (the Router). When a customer asks a question, the receptionist quickly reads it and says, “This is a legal question. I’m only going to send this to the Legal department.”

Instead of bothering all 100,000 employees, only 10,000 people work on the answer. You get the same high-quality answer (because the Legal department is specialized), but you get it much faster, and you didn’t burn out the rest of your company.

Translating the Analogy to Neural Networks

In an MoE LLM, the “corporation” is the total parameter count. The “departments” are smaller neural network sub-components called Experts. The “receptionist” is a tiny neural network called the Router or Gating Network.

Here is the technical magic: Instead of passing a token through one massive Feed-Forward Network (FFN) layer, an MoE layer contains multiple smaller FFNs (the experts). When a token enters this layer, the router looks at the token and decides which experts are best suited to process it.

Usually, the router selects the top 2 experts (this is called “top-2 routing”). The token is processed by only those two experts, their outputs are combined, and the token moves on to the next layer.

Sparse vs. Dense: The Technical Breakdown

Because only a fraction of the experts are activated for any given token, MoE models are considered sparse models. Dense models are dense because 100% of their parameters are used for every token.

Let’s look at the most famous open-source MoE model: Mixtral 8x7B.

  • It has 8 experts in each of its transformer layers.
  • Each expert is roughly the size of a 7B or 8B parameter model.
  • The total parameter count of the entire model is about 47 billion parameters.

Here is the kicker: Even though Mixtral 8x7B has 47 billion parameters, it only uses about 13 billion parameters per token.

Why 13 billion and not just 7 billion? Because the “attention” mechanisms of the model (which look at the context of your prompt) are shared across all experts, and those take up a few billion parameters. But the heavy lifting—the FFN layers—is only done by the 2 active experts.

The result? You get the intelligence and reasoning capability of a 47B-class model, but with the inference speed and compute requirements of a 13B model.

Why MoE Matters for Self-Hosters

For the self-hosting and homelab community, MoE is nothing short of revolutionary.

When you rely on cloud APIs like OpenAI or Anthropic, you pay per token. You don’t care how their servers are architected. But when you run models locally, you care deeply about two metrics: VRAM and Tokens Per Second (TPS).

  1. Speed: Because an MoE model only computes a fraction of its parameters per token, your GPU can spit out tokens much faster. If you try to run a dense 47B model on your hardware, it might crawl at 2 or 3 tokens per second. A 47B MoE model might fly at 15 or 20 tokens per second.
  2. Intelligence per Compute: If you have a setup that can comfortably run a dense 13B or 14B model (like Llama-2 13B), you can run Mixtral 8x7B and get a massive leap in reasoning, coding, and instruction-following ability for the exact same compute cost per token.

Note on VRAM: It is important to understand that while the compute per token is lower, you still have to load all 47 billion parameters into memory. You can’t just load the two active experts into VRAM, because different tokens will require different experts. So, MoE saves you compute and time, but you still need the memory capacity to hold the whole model (though quantization helps a lot here, which we’ll get to).

The Big Players in Open Source MoE

If you’re looking to experiment with MoE models on your homelab, you have some fantastic options right now:

  • Mixtral 8x7B: The model that put MoE on the map for self-hosters. Created by Mistral AI, it matches or beats GPT-3.5 in benchmarks and is an incredible general-purpose assistant.
  • Mixtral 8x22B: The bigger brother. 8 experts, each roughly 22B in size. Total parameters are around 141B, but active parameters are about 39B. This thing is a beast and rivals GPT-4 in many tasks.
  • DeepSeek MoE: DeepSeek has released some incredibly efficient MoE models, pushing the boundaries of how granular experts can be (using more, smaller experts instead of fewer, larger ones).
  • Qwen MoE: Alibaba’s Qwen team has also released highly capable MoE variants, especially good for multilingual tasks.

Practical Walkthrough: Running Mixtral 8x7B Locally

Let’s get our hands dirty. The easiest way to run an MoE model on your homelab server or local PC is using Ollama. Ollama handles the model downloading, quantization, and API serving for you.

Prerequisites

  • A machine running Linux, macOS, or Windows.
  • A decent GPU. To run a quantized Mixtral 8x7B comfortably, you’ll want around 24GB to 32GB of VRAM (e.g., an RTX 3090/4090, or a Mac with 32GB+ Unified Memory).
  • If you have less VRAM, you can still run it by offloading to system RAM, but it will be slower.

Step 1: Install Ollama

If you haven’t installed Ollama yet, it’s just a single script. Open your terminal and run:

1

curl -fsSL https://ollama.com/install.sh | sh

This will install the Ollama service and command-line tools. You can verify it’s running by checking the version:

1

ollama --version

Step 2: Pull and Run the MoE Model

Ollama makes it incredibly easy to pull models. By default, Ollama will pull a 4-bit quantized version of models, which is perfect for homelabbers trying to save VRAM.

Let’s pull Mixtral 8x7B. Run this command:

1

ollama run mixtral

Note: The first time you run this, it will download the model. A 4-bit quantized Mixtral model is roughly 26GB, so go grab a coffee while it downloads.

Step 3: Interact with the Model

Once the download finishes, you’ll be dropped directly into an interactive prompt. Let’s test its reasoning capabilities. Try asking it a question that requires some logic:

1

>>> I have 3 apples. I eat 1 apple and give 1 to my friend. How many apples do I have left? Explain your reasoning step-by-step.

Mixtral should easily handle this and give you a quick, accurate response. Notice how fast the text appears. That is the MoE magic at work—your GPU is only churning through ~13B parameters per token, even though the model is holding 47B parameters in memory.

Step 4: Serve the Model via API

For most homelabbers, the goal isn’t just to chat in the terminal; it’s to expose the model to other apps. Ollama runs a local API on port 11434 by default.

You can test it with a simple curl command from another terminal:

1
2
3
4
5

curl http://localhost:11434/api/generate -d '{
  "model": "mixtral",
  "prompt": "Write a python script to reverse a string.",
  "stream": false
}'

You’ll get a JSON response back with the generated code. You can now point your favorite self-hosted AI frontends (like Open WebUI, LibreChat, or AnythingLLM) to http://localhost:11434 and start using Mixtral as your local, private AI assistant.

Hardware Considerations for MoE in the Homelab

When building or tuning a server for MoE models, keep a few things in mind:

1. VRAM is Still King

As mentioned, you have to load the whole model into memory. For a 4-bit quantized Mixtral 8x7B (usually in GGUF format), you need about 26-28GB of VRAM to fit the model and have room for the context window (your prompt).

  • 24GB GPU (e.g., RTX 3090): You can fit almost all of it in VRAM, but you might need to offload 1 or 2 layers to your system RAM.
  • 16GB GPU: You’ll have to offload a significant chunk to system RAM. This will bottleneck your speed because system RAM is much slower than GPU VRAM.
  • Mac Studio / Macbook Pro with 64GB Unified Memory: These are actually incredible MoE machines because the unified memory is very fast and shared between the CPU and GPU.

2. Quantization Formats

If you are using Ollama, it uses GGUF format by default. If you are using other backends like text-generation-webui or vLLM, you might want to look into EXL2 (ExLlamaV2) formats. EXL2 allows for custom quantization ratios (like 3.5 bits per weight), which can help you squeeze a massive MoE model into a smaller VRAM footprint while maintaining excellent speed.

3. CPU Offloading

If you don’t have enough VRAM, MoE models actually handle CPU offloading a bit better than dense models in some scenarios. Because the compute per token is lower, the CPU isn’t asked to do as much heavy math, meaning a fast CPU with fast DDR5 RAM can still generate text at a readable pace (around 5-8 tokens per second) even if a lot of the model is offloaded.

Summary

Mixture of Experts (MoE) is arguably the most important architectural shift in open-source AI in the last year. By splitting a massive model into specialized “experts” and using a router to only activate a few of them per token, MoE gives us the intelligence of massive models with the compute footprint of much smaller ones.

For self-hosters, this means we can run GPT-3.5-level (and sometimes GPT-4-level) intelligence right in our own basements, keeping our data private and our hardware fully utilized. With tools like Ollama, getting an MoE model like Mixtral up and running takes literally minutes.

Call to Action: Have you tried running Mixtral or other MoE models in your homelab? What hardware are you using, and what kind of tokens-per-second are you seeing? Drop a comment below and let us know your setup!

If you found this tutorial helpful, make sure to subscribe to the Self-Hosted Tech YouTube channel and bookmark the blog for more deep dives into homelabs, local AI, and self-hosting. Until next time, keep those servers humming!