Choosing Your First Local LLM Model: A Beginner’s Guide

Welcome back to Self-Hosted Tech! If you’ve been following the explosion of AI over the last couple of years, you might be feeling a mix of awe and mild annoyance. On one hand, tools like ChatGPT and Claude are incredible. On the other hand, relying on the cloud means you are at the mercy of subscription fees, internet outages, and corporate privacy policies.

As homelabbers and self-hosters, our natural reaction to a cool cloud service is to figure out how to run it on our own hardware. The good news? Local AI has matured at a staggering pace. You no longer need a server farm to run a capable Large Language Model (LLM). In fact, you can run a surprisingly intelligent AI right on your daily driver laptop or a modest home server.

But if you head over to Hugging Face (the GitHub of the AI world), you’ll be greeted by thousands of models with confusing names, bizarre version numbers, and cryptic file extensions. How do you choose your first local LLM?

Don’t worry. In this guide, we’re going to demystify the jargon, look at your hardware options, pick out the best beginner-friendly models, and walk through exactly how to get one running on your machine.

Demystifying the Jargon: Parameters and Quants

Before we pick a model, we need to understand two fundamental concepts: parameters and quantization.

What are Parameters?

When you hear a model referred to as “7B” or “8B”, that “B” stands for billion parameters. You can think of parameters as the synapses in the AI’s brain. They are the numerical values the model uses to process input and generate text.

Generally speaking, more parameters mean a smarter, more capable model. However, more parameters also mean a larger file size and a higher demand on your computer’s memory. A 7B model in its raw, uncompressed form is about 14 gigabytes in size (using 16-bit precision).

What is Quantization (and why is it a lifesaver)?

If we had to run models at their raw 16-bit precision, local AI would be completely inaccessible to the average homelabber. Enter quantization (often shortened to “quant”).

Quantization is a compression technique that reduces the precision of the model’s parameters. Instead of using 16 bits to store a number, we might use 4 bits. This drastically reduces the file size and the amount of memory required to run the model, with only a very slight—and often imperceptible—decrease in the model’s intelligence.

When you download a local model, you will usually download a GGUF file. GGUF (GPT-Generated Unified Format) is the file format designed specifically for running quantized models on standard CPUs and GPUs. A 7B model quantized to 4-bit (usually labeled Q4_K_M) shrinks down to about 4 to 5 gigabytes. Suddenly, running an LLM on a laptop with 16GB of RAM is entirely feasible.

Hardware Realities: What Do You Actually Need?

Let’s be real: AI is resource-intensive. But the landscape has changed. Thanks to the GGUF format and software like llama.cpp (which we’ll use indirectly in our walkthrough), you don’t strictly need a massive, expensive NVIDIA GPU.

Here is the basic breakdown of hardware requirements:

System RAM (The CPU Route)

If you have a standard desktop, laptop, or home server with a decent amount of system RAM (16GB to 32GB), you can run local LLMs entirely on your CPU.

  • Pros: Cheap and accessible. You can run very large models if you have enough RAM.
  • Cons: Slower generation speed. System RAM bandwidth is much lower than GPU memory bandwidth. You might get 5 to 15 tokens (words/pieces of words) per second.

GPU VRAM (The Fast Route)

If you have a dedicated GPU (NVIDIA, AMD, or even Apple Silicon’s unified memory), the model can be loaded directly into Video RAM (VRAM).

  • Pros: Incredibly fast. A decent GPU can output 30 to 80+ tokens per second, making the AI feel instantaneous.
  • Cons: VRAM is expensive and limited. Most consumer GPUs have 8GB to 12GB of VRAM, which limits you to running smaller models (like 7B or 8B parameters) entirely on the GPU.

The Apple Silicon Sweet Spot

If you are a Mac user with an M1, M2, or M3 chip, you are in luck. Apple Silicon uses “unified memory,” meaning your CPU and GPU share the same pool of RAM. A Mac Studio or MacBook Pro with 32GB or 64GB of RAM can run large local models incredibly fast, bridging the gap between the CPU and dedicated GPU routes.

The Golden Rule of Local AI: To run a model, you need RAM/VRAM equal to the file size of the model, plus a little extra overhead for the “context window” (the memory of your current conversation). If you have 16GB of total system memory, stick to models that take up 6-8GB or less.

Top Contenders for Your First Local LLM

Now that you know what the numbers mean, which model should you actually download? The AI space moves fast, but as of right now, these are the undisputed champions for beginners.

1. Meta Llama 3 (8B)

If you only try one model, make it this one. Released by Meta, Llama 3 8B punches way above its weight class. It feels almost as smart as ChatGPT was a year or two ago, but it fits comfortably into 8GB of RAM when quantized. It’s great at general conversation, coding help, and creative writing. It is the best default choice for anyone starting out.

2. Microsoft Phi-3 Mini (3.8B)

If your hardware is a bit older, or if you want a model that loads instantly and generates text lightning-fast, Phi-3 Mini is a marvel. At only 3.8 billion parameters, the quantized file is around 2.5GB. Despite its tiny size, Microsoft trained it on incredibly high-quality data, making it surprisingly capable at logic, coding, and instruction following. It’s the perfect model for a Raspberry Pi 5 or an old laptop.

3. Mistral (7B) and Mixtral (8x7B)

Mistral is a European AI company that has put out some fantastic open-weight models. The standard Mistral 7B is a great alternative to Llama, known for being highly uncensored and very creative. If you have a beefier machine (32GB+ RAM), their Mixtral model uses an architecture called “Mixture of Experts,” which gives you the intelligence of a much larger model while keeping the speed of a smaller one.

Practical Walkthrough: Running Llama 3 with Ollama

Reading about AI is fun, but running it is better. For our walkthrough, we are going to use Ollama.

Ollama is a brilliantly simple piece of software that acts as a wrapper around llama.cpp. It handles the downloading, quantization, and serving of the model, so you don’t have to manually hunt for GGUF files on Hugging Face. It runs on macOS, Linux, and Windows.

Step 1: Install Ollama

Head over to ollama.com and download the installer for your operating system.

If you are running Linux and prefer the terminal, you can install it with a simple curl script:

1

curl -fsSL https://ollama.com/install.sh | sh

Once installed, Ollama runs as a background service on your machine.

Step 2: Download and Run Your First Model

Open up your terminal (or Command Prompt/PowerShell on Windows). To run Llama 3 8B, simply type:

1

ollama run llama3

That’s it. The first time you run this command, Ollama will automatically download the 4-bit quantized GGUF file (which is about 4.7GB). Depending on your internet speed, this might take a few minutes.

Once the download finishes, you will be dropped into a chat prompt:

1

>>> Send a message (/? for help)

Step 3: Have a Conversation

Go ahead and test it out. Type a prompt just like you would in ChatGPT:

1

>>> Write a python script that renames all files in a directory to lowercase.

The model will immediately start generating the code. You’ll notice that if you have a modern CPU or a GPU, the text streams in quite rapidly.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

import os

def rename_files_to_lowercase(directory):
    for filename in os.listdir(directory):
        if filename != filename.lower():
            old_path = os.path.join(directory, filename)
            new_path = os.path.join(directory, filename.lower())
            os.rename(old_path, new_path)
            print(f"Renamed: {filename} -> {filename.lower()}")

# Example usage:
# rename_files_to_lowercase('/path/to/your/directory")

To exit the chat, just type /bye or press Ctrl+D.

Step 4: Try Another Model

Want to see how a smaller, faster model compares? Just ask Ollama to run it:

1

ollama run phi3

Ollama will download Microsoft’s Phi-3 and drop you into a new chat. You can see the list of models you have downloaded by running ollama list.

Next Steps: Adding a Graphical Interface

Running an LLM in the terminal is cool, but if you want to share your local AI with your family or just want a ChatGPT-like web interface, you’ll want to add a frontend.

Because Ollama runs as a local API (defaulting to http://localhost:11434), you can easily plug it into self-hosted web UIs.

One of the most popular options is Open WebUI. If you have Docker installed on your homelab server or local machine, you can spin it up in seconds:

1

docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:main

Once that container is running, open your browser and navigate to http://localhost:3000. You’ll create a quick admin account, and just like that, you have a beautiful, fully featured web interface. In the model dropdown at the top of the screen, you’ll see llama3 and phi3—the exact models you pulled down via the Ollama CLI.

You now have a completely free, entirely private, offline alternative to ChatGPT running on your own hardware.

Summary and Call to Action

Choosing your first local LLM doesn’t have to be overwhelming. By understanding that parameters dictate intelligence and size, and that quantization (GGUF) makes it possible to run these models on everyday hardware, you’re already ahead of the curve.

For most beginners, installing Ollama and running Llama 3 8B is the absolute best starting point. It offers a fantastic balance of intelligence, speed, and memory footprint. If your machine struggles with it, drop down to Phi-3 Mini. Once you have the models running locally, hooking them up to a self-hosted UI like Open WebUI gives you the polished experience of a cloud service with the privacy of a local homelab.

What’s next? Take ten minutes today to install Ollama and pull down your first model. Play around with it. Ask it to summarize documents, write bash scripts, or just have a casual chat.

Have you started playing with local AI in your homelab yet? What hardware are you running it on? Drop a comment below and let us know what models you’re loving right now. And if you found this guide helpful, make sure to subscribe to the Self-Hosted Tech YouTube channel and blog for more practical tutorials on taking control of your tech stack. Happy self-hosting!