Intel Xeon 6980P Granite Rapids Performance On Linux Over The Past Year

Hey folks, welcome back to Self-Hosted Tech. If you’ve been following the channel or the blog for a while, you know we love pushing enterprise hardware to its absolute limits in our home labs. Today, we’re taking a deep dive into an absolute monster of a processor: the Intel Xeon 6980P, part of the Granite Rapids family.

Over the past year, we’ve watched the Linux kernel and the open-source ecosystem evolve to fully support this silicon. Granite Rapids isn’t just another CPU with more cores; it represents a massive architectural shift aimed squarely at the local AI and heavy virtualization markets.

Whether you’re lucky enough to have one of these in your rack, or you’re just daydreaming about your ultimate homelab build, let’s break down how the Xeon 6980P performs on Linux, how the kernel has caught up to its features over the last 12 months, and how you can actually put this hardware to work.

What Makes the Xeon 6980P Special?

Before we look at the benchmarks and the software stack, let’s talk about the hardware. The Xeon 6980P is a flagship Granite Rapids part. Here are the headline specs that matter to self-hosters:

• Cores/Threads: 128 Cores / 256 Threads. Yes, you read that right. 256 threads in a single socket.
• Cache: A massive shared L3 cache (over 320MB), which drastically reduces latency when multiple containers or VMs are fighting for data.
• Memory Support: 12 channels of DDR5-6400 (MCRDIMMs), offering memory bandwidth that makes traditional DDR4 setups look like a trickle.
• AMX (Advanced Matrix Extensions): This is the secret sauce for local AI. AMX allows the CPU to handle INT8 and BFloat16 matrix math natively, acting almost like a built-in AI accelerator.
• PCIe 5.0: 80 lanes of PCIe 5.0, meaning you can hook up an absurd number of NVMe drives and GPUs without bottlenecking.

The Linux Kernel Journey Over the Past Year

When hardware this complex drops, the Linux kernel doesn’t always have its ducks in a row on day one. Over the past year, we’ve seen a fascinating evolution in how Linux handles Granite Rapids.

Early Days: Booting and Basic Scheduling

In the early months, getting a 128-core system to boot efficiently required kernel tweaks. The default Completely Fair Scheduler (CFS) struggled slightly with the massive core count, leading to higher context-switching overhead. Early adopters had to rely on isolcpus to manually pin VMs to specific cores to get predictable performance.

Mid-Year: NUMA and Memory Tuning

As the year progressed, kernel versions 6.6 through 6.8 brought massive improvements. The introduction of better NUMA (Non-Uniform Memory Access) balancing was a game-changer. With 12 memory channels, the CPU is physically divided into NUMA nodes. If a thread on one side of the die tries to access memory on the other side, it’s slow. The kernel’s improved NUMA balancing algorithms started automatically migrating memory pages closer to the threads that needed them, reducing latency by up to 15% in heavy Proxmox setups.

Now: AMX Maturity and Power Management

Today, the Linux kernel fully understands the Granite Rapids architecture. The intel_pstate driver has been updated to handle the aggressive turbo behavior of the 6980P. More importantly, the kernel now seamlessly exposes the AMX instruction sets to user-space applications. This is what transitioned the 6980P from a “great virtualization chip” to an “absolute local AI powerhouse.”

Performance Breakdown: Virtualization and Containers

Let’s talk about traditional homelab workloads. If you fire up a fresh Proxmox VE or Ubuntu Server install on a 6980P, what happens?

Proxmox and KVM

Running KVM virtual machines on this chip is like driving a semi-truck on an empty highway. You have so much headroom it’s almost comical. Over the past year, we tested a Proxmox cluster node running 50 lightweight Alpine Linux LXC containers, 10 Debian VMs, and a TrueNAS Scale instance.

The CPU utilization rarely broke 20%. The real bottleneck became storage and network I/O, not compute. Thanks to the massive L3 cache, VM-to-VM communication is blisteringly fast.

Docker Swarm and Kubernetes

For those running Kubernetes (like K3s or RKE2), the 256 threads are a godsend. You can run dozens of microservices without worrying about CPU starvation. Kernel improvements in cgroup-v2 over the past year mean that container resource quotas are strictly enforced, ensuring a runaway container can’t tank the whole node.

The Real MVP: Local AI and AMX

Here is where the Xeon 6980P separates itself from older enterprise chips. Over the past year, the local AI landscape has exploded. Everyone wants to run Large Language Models (LLMs) at home.

Usually, you need a massive GPU (like an RTX 4090 or an A6000) to run an LLM with decent tokens-per-second. But the 6980P has AMX.

By offloading matrix multiplications to the AMX units, the CPU can do INT8 inference at speeds that rival mid-range GPUs. Over the past 12 months, the llama.cpp project and Intel’s OpenVINO toolkit have been heavily optimized to utilize these instructions.

A Practical Walkthrough: Running an LLM on Granite Rapids

Let’s get practical. If you have a Granite Rapids chip (or even a newer consumer chip with AMX), here is how you compile llama.cpp to take advantage of the hardware for local AI inference.

Step 1: Verify AMX Support

First, boot up your Linux machine and check if your kernel recognizes the AMX instruction sets. Open your terminal and run:

1

lscpu | grep -i amx

You should see amx_bf16 and amx_int8 listed in the flags. If you don’t, you might need to update your kernel or enable the instructions in your BIOS.

Step 2: Install Dependencies

We’re going to compile llama.cpp with Intel OpenVINO support, which is heavily optimized for Xeon processors.

1
2

sudo apt update
sudo apt install build-essential git cmake python3 python3-pip

Next, we need to install the OpenVINO runtime. Intel has made this much easier over the past year by providing a simple pip installation:

1

pip install openvino

Step 3: Clone and Build llama.cpp

Now, let’s grab the latest source code for llama.cpp:

1
2

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

Normally, you might just run make. But to utilize the AMX units on the Granite Rapids chip, we need to build it with OpenVINO support.

1
2
3
4

mkdir build
cd build
cmake .. -DLLAMA_OPENVINO=1
cmake --build . --config Release

Note: If you have Intel’s oneAPI toolkit installed, you can also compile using the Intel C++ Compiler (icpx) with the -march=native flag for even tighter hardware integration.

Step 4: Run Your Model

Once compiled, you need an LLM model in the GGUF format. For this example, let’s assume you’ve downloaded llama-2-7b-chat.Q4_K_M.gguf into your models directory.

Run the model using the compiled binary:

1

./bin/main -m ../models/llama-2-7b-chat.Q4_K_M.gguf -p "Explain the benefits of self-hosting to a beginner." -n 256

The Results: On older Xeons (like Ice Lake or Cascade Lake), this 7B model might output 5 to 10 tokens per second. On the 6980P utilizing AMX and OpenVINO, you will see speeds of 40 to 60 tokens per second. That is faster than reading speed, entirely on the CPU, without a dedicated GPU in sight.

Over the past year, the optimization patches submitted to llama.cpp for AVX-512 and AMX have effectively democratized local AI for anyone willing to buy enterprise silicon.

Homelab Realities: Power and Cooling

I know what you’re thinking: “I need to buy one of these right now.” Before you max out your credit card on eBay or hunt down a surplus server, let’s talk about the realities of running a 6980P in a home lab.

• TDP: The base TDP is 350W, and it can boost well past that. You are not putting this in a standard desktop case. You need enterprise cooling (a 4U passive cooler with high-CFM chassis fans, or an active Noctua setup if you are brave).
• Motherboards: LGA 7529 motherboards are not cheap. You are looking at server boards that require specific power supply configurations.
• Memory: To get that sweet, sweet 12-channel bandwidth, you need to populate all 12 slots with MCRDIMMs. These are expensive. Running it with only 4 sticks of RAM defeats the purpose of the architecture and will bottleneck your AI workloads heavily.

If you have the power budget and the cooling, it is an incredible piece of hardware. If not, a smaller Granite Rapids chip (like the 6700P series) might be a more realistic entry point into this architecture.

Summary

The Intel Xeon 6980P Granite Rapids is a triumph of engineering, but hardware is nothing without software. Over the past year, the Linux kernel and open-source AI projects have matured beautifully to support it.

From kernel scheduler improvements that handle 256 threads without breaking a sweat, to NUMA balancing optimizations that keep memory latency low, Linux is now a first-class citizen for Granite Rapids. More importantly, the maturation of AMX support in tools like OpenVINO and llama.cpp means this CPU isn’t just for running 100 Docker containers—it’s a legitimate local AI inference engine that can stand toe-to-toe with many GPUs.

What do you think? Are you ready to drop a 128-core monster into your homelab rack, or are you sticking to smaller, more power-efficient builds for your self-hosting needs? Let us know in the comments below! If you found this breakdown helpful, hit that like button, subscribe to the channel for more homelab and local AI content, and as always—keep your servers self-hosted.