Choosing Your First Local LLM Model: A Beginner's Guide

The Problem: Too Many Models

You installed Ollama. You ran ollama list and saw… hundreds of models. Llama 3.1, GLM 4.7, Qwen 3, Mistral, Phi, Gemma, DeepSeek — and that’s just the famous ones. Each comes in multiple sizes (7B, 13B, 70B…) and multiple quantizations (Q4, Q5, Q8, F16…).

Which one do you actually run?

This guide cuts through the noise. We’ll cover the major model families, what they’re good at, how to match them to your hardware, and which one to pull first.

The Major Model Families (2026)

Llama (Meta)

The most recognizable name in local AI. Meta’s Llama series essentially launched the open-weight LLM movement.

• Current: Llama 3.1 / 3.2
• Sizes: 1B, 3B, 8B, 70B, 405B
• Strengths: Well-rounded, huge community, tons of fine-tunes and variants, tool calling support
• Weaknesses: License restrictions on the larger models (not fully open-source for commercial use)
• Best for: General chat, coding, creative writing, getting started

Run it: ollama pull llama3.1:8b

GLM (Zhipu AI)

Originally from China, GLM models have become favorites in the local AI community for being punchy above their weight class.

• Current: GLM 4.7 Flash (30B MoE), GLM 4.5, GLM 5.2
• Sizes: 9B, 30B (MoE), 70B+
• Strengths: Excellent reasoning, strong coding ability, Mixture-of-Experts architecture means the 30B only uses ~6-7GB VRAM per token while delivering near-70B quality. Uncensored/abliterated variants available.
• Weaknesses: Less community ecosystem than Llama, fewer fine-tunes
• Best for: Coding, reasoning tasks, running a capable model on limited hardware, agents/automation

Run it: ollama pull glm4.7:flash-q4_K_M

Qwen (Alibaba)

Alibaba’s open-weight models are consistently top-tier on benchmarks. Qwen 3 was a major leap.

• Current: Qwen 3, Qwen 3.5
• Sizes: 0.6B, 1.7B, 7B, 14B, 32B, 72B, 110B
• Strengths: Exceptional multilingual support (120+ languages), strong math and reasoning, long context windows (up to 256K tokens), excellent coding
• Weaknesses: Some variants are heavily safety-tuned (refuses certain prompts), abliterated versions exist but require finding them
• Best for: Multilingual use, math, long documents, coding, research

Run it: ollama pull qwen3:8b

Mistral

French AI lab that was early to the open-weight game. Mistral models are efficient and well-optimized.

• Current: Mistral Large, Mixtral 8x7B (MoE), Mistral NeMo
• Sizes: 7B, 12B (NeMo), 8x7B (MoE), 123B (Large)
• Strengths: Efficient, great instruction-following, MoE variants deliver large-model quality at smaller-model cost, permissive Apache 2.0 license on some models
• Weaknesses: Smaller models (7B) are less impressive vs Llama 3.1 8B in 2026
• Best for: Efficient inference, European language support, commercial use (Apache license)

Run it: ollama pull mistral:7b

Phi (Microsoft)

Microsoft’s “small but mighty” models. Designed to punch above their parameter count.

• Current: Phi-4
• Sizes: 3.8B, 14B
• Strengths: Surprisingly capable for the size, runs on almost anything, great for edge devices and weak hardware
• Weaknesses: Limited context window, not as versatile as larger models, training data biases
• Best for: Low-resource hardware, quick tasks, edge deployment, Raspberry Pi

Run it: ollama pull phi4:14b

Gemma (Google)

Google’s open-weight offering, derived from the same tech as Gemini.

• Current: Gemma 3
• Sizes: 1B, 4B, 9B, 27B
• Strengths: Strong reasoning, multimodal (vision) support in some variants, good efficiency
• Weaknesses: Restrictive license, smaller community than Llama
• Best for: Vision-language tasks, lightweight reasoning, Google ecosystem users

Run it: ollama pull gemma3:9b

DeepSeek

Chinese lab that made waves with cost-efficient training. Their models are strong at reasoning and math.

• Current: DeepSeek V3, DeepSeek R1 (reasoning-focused)
• Sizes: 7B, 67B, 671B (MoE)
• Strengths: Exceptional at math and logical reasoning, R1 variant “thinks” before answering (chain-of-thought), very cost-effective
• Weaknesses: Large models need significant hardware, some safety tuning
• Best for: Math, logical reasoning, complex problem-solving, research

Run it: ollama pull deepseek-r1:7b

How to Choose: Match Model to Hardware

Your hardware determines what you can run. Here’s the practical breakdown:

8GB VRAM (RTX 3060, 4060, 3060 Ti)

Our pick: GLM 4.7 Flash 30B. The MoE architecture means only ~6-7GB is active per token, but you get near-70B quality. Best bang for the buck on 8GB cards.

12GB VRAM (RTX 3080, 4070 Ti)

Our pick: GLM 4.7 Flash 30B at Q5_K_M for better quality, or Qwen 3 14B if you want a traditional dense model with more consistent performance.

16GB+ VRAM (RTX 4080, 5080)

You can run 14B models at Q8, or start looking at 32B models at Q4. Mixtral 8x7B fits comfortably.

Our pick: Qwen 3 32B at Q4_K_M (~20GB VRAM) or Mixtral 8x7B at Q4 (~24GB).

24GB VRAM (RTX 3090, 4090)

The sweet spot for serious local AI. You can run most models at high quality.

Our pick: Qwen 3 32B at Q5_K_M, or GLM 4.7 Flash 30B at Q8. You can also explore 70B models at Q4 (~42GB — needs system RAM offloading, slower but works).

No GPU (CPU Only)

You’re limited to small models, but it’s not hopeless:

Our pick: Llama 3.2 3B for general use, or Qwen 3 1.7B if your machine is really weak. See our Run Local AI with No GPU guide for detailed setup.

What About Quantization?

If you see Q4, Q5, Q8 and don’t know what that means — it’s how much the model is compressed. Lower number = smaller file = less VRAM, but slightly lower quality.

Quick version:

• Q4_K_M: Best balance. Start here. ~4 bits per parameter.
• Q5_K_M: Noticeably better quality, ~25% more VRAM.
• Q8_0: Near-original quality, 2x the size of Q4.
• F16: Full precision. Rarely needed locally.

For a deep dive, read our upcoming Quantization Explained guide.

The First Model You Should Run

If you want one recommendation, right now, today:

For most people (8-12GB GPU):

1
2

ollama pull glm4.7:flash-q4_K_M
ollama run glm4.7:flash-q4_K_M

GLM 4.7 Flash 30B is the best quality-to-VRAM ratio available. It’s a Mixture-of-Experts model — 30B total parameters, but only ~6-7B active per token. You get near-70B reasoning quality while only using ~7GB VRAM. It codes well, follows instructions well, and abliterated (uncensored) variants exist if you need them.

If you want the safest, most community-supported choice:

1
2

ollama pull llama3.1:8b
ollama run llama3.1:8b

Llama 3.1 8B is the Toyota Corolla of local LLMs. It’s not the flashiest, but it works, has the biggest ecosystem, and you’ll find help for any problem online.

If you’re on a weak machine or no GPU:

1
2

ollama pull llama3.2:3b
ollama run llama3.2:3b

Llama 3.2 3B runs on almost anything with 8GB RAM. It’s surprisingly capable for its size.

Red Flags: Models to Avoid as a Beginner

• Anything 70B+ on 8GB VRAM: It’ll technically run with offloading, but at 1-2 tokens/second. Painful.
• F16 or FP8 models: Massive files, no noticeable quality improvement over Q8 for local use. Waste of disk space.
• Obscure fine-tunes from HuggingFace: Some are great, many are broken. Stick to base models from major labs until you know what you’re doing.
• “Uncensored” models from random repos: Quality varies wildly. If you want an abliterated model, get one from a reputable source like huihui-ai on HuggingFace.

How to Actually Try Them

The beauty of local AI is that switching models is free. You’re not paying per request:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# Pull a model
ollama pull llama3.1:8b

# Chat with it
ollama run llama3.1:8b

# Exit
/bye

# Try a different one
ollama pull qwen3:8b
ollama run qwen3:8b

# List what you have
ollama list

# Delete one you don't want
ollama rm llama3.1:8b

Spend an afternoon pulling 3-4 models and asking each the same questions. You’ll quickly develop a feel for which one you prefer. The “best” model is subjective — it’s whichever one gives you answers you find most useful.

What We Run

On our homelab, we run GLM 4.7 Flash 30B (Q4_K_M) on a single RTX 3080 12GB as our daily driver for agents and automation. It’s been the most capable model we’ve found that fits comfortably on 12GB VRAM. We also keep Llama 3.1 8B around for tasks that need the bigger ecosystem (tool calling integrations, specific fine-tunes).

For heavier work, we’re building toward a multi-GPU rig that can run 70B+ models. But for day-to-day local AI? The 8-12GB tier is where it’s at in 2026.

Next Steps

• Run Your First Local AI Model in 15 Minutes — Get Ollama installed and running
• Best GPUs for Local LLMs in 2026 — Hardware buying guide
• Quantization Explained (Coming Soon) — Deep dive on Q4, Q8, and why it matters
• Ollama vs LM Studio — Which tool to use for running your models

Don’t overthink the first model. Pull one, try it, pull another, compare. The best model is the one that’s running.