Show HN: LLMspeed – Feel How Fast a Local LLM Runs Before You Buy the Hardware
If you’ve spent any time in the self-hosting and homelab communities lately, you already know that local AI is the hottest topic on the block. We all want to run our own LLMs (Large Language Models) for privacy, for learning, or just for the sheer joy of having a supercomputer humming away in our closet. But there’s a massive roadblock standing in the way of almost everyone: hardware uncertainty.
Dropping $1,600 on an RTX 4090, or even $800 on a used Mac Studio, is a tough pill to swallow when you don’t actually know what you’re getting. You’ve read the forum posts. You’ve seen the benchmarks. Someone says, “I’m getting 45 tokens per second on Llama 3 8B with my RTX 4070 Ti.” But what does “45 tokens per second” actually feel like? Is that fast enough to use as a daily-driver chat interface, or will you be pulling your hair out waiting for the AI to finish its sentences?
Recently, a brilliant project popped up on Hacker News called LLMspeed. It solves this exact problem. It’s a simple, elegant tool that lets you feel how fast a local LLM runs before you ever swipe your credit card.
Let’s break down why this tool is a game-changer for the homelab community, how local AI hardware actually works, and how you can use LLMspeed to plan your next build.
The Problem with LLM Benchmarks
When you look at standard AI benchmarks, you’re usually staring at a spreadsheet of numbers. You see metrics like:
- Tokens per second (t/s): How fast the model generates text.
- Time to first token (TTFT): How long it takes for the AI to start responding after you hit enter.
- Prompt processing speed: How fast the AI reads your input.
For a beginner, these numbers are completely abstract. If I tell you that a base-model M2 Mac Mini with 16GB of RAM can output roughly 18 tokens per second running Mistral 7B, you might think, “Okay, 18 sounds decent.” But is it?
To put it in perspective, the average human reads at about 200 to 250 words per minute, which translates to roughly 5 to 8 tokens per second. So, 18 t/s is actually quite comfortable to read along with. But if you’re trying to use an LLM to generate code or summarize massive documents, you might want 50 t/s or more.
The core issue is that reading a number on a screen doesn’t tell you if the latency will annoy you. You don’t know if the “Time to First Token” is going to make the interface feel sluggish. You need to experience the speed. That’s where LLMspeed comes in.
What is LLMspeed?
LLMspeed is an open-source utility designed to simulate the generation speed of various LLMs on different hardware configurations.
Instead of requiring you to buy the hardware and run the model locally, LLMspeed uses a database of community-sourced and verified benchmarks. You tell the tool what hardware you are considering (e.g., an Nvidia RTX 3060, an Apple M3 Max, or an AMD Radeon 7900 XTX) and what model you want to run (e.g., Llama 3 8B at 4-bit quantization).
The tool then streams a text response to your terminal or web interface at the exact speed that the specified hardware would generate it. It essentially acts as a “try before you buy” simulator for AI hardware. You get to watch the text crawl or fly across your screen, giving you an immediate, visceral understanding of what that tokens-per-second metric actually means in practice.
A Quick Primer on Local AI Hardware
Before we dive into the walkthrough, it’s worth understanding why different hardware yields such wildly different speeds. If you’re used to traditional homelab computing—where CPU cores and raw clock speeds are king—local AI flips that script entirely.
For LLMs, the most critical component is memory bandwidth, followed closely by VRAM (Video RAM) capacity.
- VRAM Capacity: This dictates if a model can run. An 8B parameter model at 4-bit quantization takes up about 5-6GB of VRAM. If your GPU only has 8GB, you can run it, but you won’t have much room left for context. If you try to run a 70B model on an 8GB card, it will either fail or spill over into your system RAM (which is disastrously slow).
- Memory Bandwidth: This dictates how fast a model runs. Once the model is loaded into memory, generating each token requires shuffling the entire model through the processor. An RTX 3060 has 360 GB/s of bandwidth. An RTX 4090 has 1,008 GB/s. An Apple M2 Ultra has 800 GB/s. The higher the bandwidth, the faster the tokens fly out.
LLMspeed factors all of this in. It knows that running a model partially in system RAM on a consumer CPU (DDR4/DDR5) will cap you at a painful 3-5 t/s, while keeping it entirely in an RTX 4090’s VRAM will blast you up to 80+ t/s.
Practical Walkthrough: Simulating Your Next AI Rig
Let’s get hands-on. In this walkthrough, we’re going to simulate the experience of running Llama 3 8B on three different hardware tiers to help us decide what to buy.
Step 1: Install LLMspeed
LLMspeed is typically distributed as a Python package or a standalone binary. Assuming you have Python 3.10+ installed on your current machine, you can usually install it via pip. Open up your terminal and run:
| |
Once installed, you can verify it’s working by checking the help menu:
| |
You should see a list of commands, including options to list supported hardware, list supported models, and run a simulation.
Step 2: Check Supported Hardware and Models
Before we run the simulation, let’s see what hardware profiles are available in the tool’s database. Run:
| |
You might see an output that looks something like this:
| |
Next, let’s check the available models:
| |
| |
Note: The q4_k_m suffix indicates 4-bit quantization, which is the most popular format for home users as it balances performance and accuracy perfectly.
Step 3: Run the Simulation
Now for the fun part. Let’s say you have a budget of around $300 and you’re looking at a used RTX 3060 12GB. You want to know if it’s fast enough to run Llama 3 8B as a local chat assistant.
Run the following command:
| |
Your terminal will go quiet for a fraction of a second (simulating the Time to First Token), and then text will begin to stream. But it won’t stream instantly—it will be artificially throttled to exactly the speed of an RTX 3060.
You’ll watch the text appear on your screen, word by word. For an RTX 3060 running an 8B model, you’ll likely see around 30 to 35 tokens per second.
As you watch it, ask yourself: Is this too slow for me? For many people, 35 t/s feels incredibly snappy and perfectly readable.
Step 4: Compare Against a High-End Setup
Now, let’s simulate the dream setup. You want to see what an $1,800 RTX 4090 feels like with the exact same model and prompt.
| |
Hit enter. This time, the text will blast onto the screen so fast that you won’t be able to read it as it generates. At 80+ tokens per second, the text will appear almost instantly in large chunks.
Step 5: The “Spillover” Test
Finally, let’s test a scenario that catches a lot of beginners off guard. You have an 8GB GPU (like an RTX 3060 Ti), and you want to run a larger model, like Mixtral 8x7B. The model requires roughly 24GB of VRAM at 4-bit. Your GPU can’t hold it, so it spills over into your system RAM.
| |
Watch what happens. The text will trickle onto the screen. You might see a word appear every second or two. This simulates the agonizing 2 to 4 tokens per second you get when relying on system RAM and a CPU to process an LLM.
By using LLMspeed, you instantly realize that buying an 8GB GPU to run large models is a bad idea, and you clearly see the value of buying a 24GB card or a high-unified-memory Mac.
How to Use This Data for Your Homelab
Once you’ve “felt” the speeds, you can make much more informed purchasing decisions. Here is a quick framework for building a local AI homelab based on what LLMspeed can show you:
The Budget Tier ($200 - $400)
- Hardware: Used RTX 3060 12GB or Mac Mini M1 with 16GB RAM.
- What you can run: 7B to 8B models (Llama 3, Mistral) very comfortably.
- The LLMspeed experience: Snappy and interactive. Great for basic chat, writing emails, and simple coding tasks.
The Mid-Range Tier ($600 - $1,200)
- Hardware: Used RTX 3090 24GB or Mac Studio M2 Max with 64GB RAM.
- What you can run: 8B models blindingly fast. You can also run 30B-34B models (like Command R) or Mixtral 8x7B at respectable speeds.
- The LLMspeed experience: 8B models feel instant. Larger models feel like the budget tier. This is the sweet spot for homelabbers who want to get serious about local AI without going broke.
The Enthusiast Tier ($1,600+)
- Hardware: RTX 4090 24GB, dual RTX 3090s, or Mac Studio M2 Ultra with 128GB+ RAM.
- What you can run: Anything up to 70B models (Llama 3 70B) at highly usable speeds.
- The LLMspeed experience: Everything feels fast. You can use 70B models as daily drivers, which rival GPT-3.5 and sometimes GPT-4 in raw capability.
Summary
Buying hardware for local AI doesn’t have to be a blind gamble. While spreadsheets and forum posts full of benchmarks are helpful, nothing beats actually experiencing the latency and generation speed firsthand.
Tools like LLMspeed bridge the gap between abstract metrics and real-world usability. By simulating the exact tokens-per-second output of specific GPUs and Apple Silicon chips, you can definitively answer the question: “Will this hardware be fast enough for my patience?”
Before you buy your next graphics card or Mac for your homelab, take ten minutes to run a few simulations. You might find out that a budget RTX 3060 is all you need, or you might justify the leap to a 3090. Either way, you’ll be making your purchasing decision with your own eyes.
What hardware are you currently running your local LLMs on? Have you used simulation tools like this before, or do you just rely on raw benchmark data? Drop a comment below and let us know what your setup is, and what model you’re running on it! If you found this guide helpful, make sure to check out our other posts on self-hosting and local AI. Happy homelabbing!