How We Halved DNS Build Delay And Cut Memory Use By 85%

Welcome back to Self-Hosted Tech! If you’re running a homelab, you already know that local DNS is the beating heart of your network. It’s the traffic cop that routes your requests, resolves your internal services, and—perhaps most importantly—blocks ads and trackers at the network level.

But as your homelab grows, so do your blocklists. Recently, we ran into a massive bottleneck with our local DNS setup. We were running a custom DNS sinkhole using dnsmasq inside a Docker container on a trusty Raspberry Pi 4. To keep our network squeaky clean, we had aggregated several massive blocklists—StevenBlack’s hosts, OISD, and a few custom regex lists. We were feeding nearly 6 million domains into our DNS server.

The result? Every time the container restarted or reloaded its blocklists, it took a agonizing 120 seconds to build the in-memory index. Worse, once it was finally up and running, it was eating a whopping 500 MB of RAM. On a 4GB Raspberry Pi running two dozen other microservices, that kind of memory hogging is a fast track to OOM (Out of Memory) kills.

Today, we’re going to break down exactly how we optimized this setup. By shifting the heavy lifting away from the DNS server and relying on standard Linux CLI tools, we managed to halve our DNS build delay from 120 seconds to 60 seconds, and we slashed memory usage by 85%, bringing it down to a lean 75 MB.

Let’s dive into the why, the how, and the practical walkthrough so you can apply this to your own homelab.

The Bottleneck: Why DNS Servers Choke on Big Lists

To fix the problem, we first have to understand it. When a DNS server like dnsmasq, CoreDNS, or even Pi-hole’s FTL engine loads a blocklist, it doesn’t just read the text file and leave it on the disk. It has to parse that file, strip out comments, ignore malformed lines, and load every single domain into a memory-efficient data structure—usually a hash map or a trie (a tree-like data structure used for fast string retrieval).

Here is where the inefficiencies hide:

1. Duplicate Domains: Many popular blocklists overlap. A domain like ads.google.com might appear in five different lists. The DNS server still has to parse all five entries, compare them against its internal index, and discard the duplicates. That’s wasted CPU cycles and wasted time.
2. Bloat and Comments: Blocklists are often full of comments, headers, and malformed entries. The DNS server has to evaluate every single line to figure out if it’s a valid domain.
3. Formatting Inconsistencies: Some lists use 0.0.0.0 domain.com, others use 127.0.0.1 domain.com, and some just list domain.com. The parser has to normalize all of this on the fly.

Because the DNS server is doing all of this parsing, sorting, and indexing at startup (or during a blocklist refresh), the “build delay” spikes. And because it’s holding all these strings in memory while it processes them, RAM usage balloons.

The Strategy: Offload the Heavy Lifting

Our solution was simple but highly effective: stop making the DNS server do the math.

Instead of handing dnsmasq a messy, 6-million-line, duplicated text file, we decided to pre-process the lists using standard GNU coreutils on the host machine. By using tools like awk, grep, and sort, we could download, clean, deduplicate, and perfectly format the blocklist before the DNS server ever sees it.

When the DNS server receives a pre-sorted, perfectly formatted, 100% unique list of domains, its only job is to load them into the trie. It doesn’t have to check for duplicates. It doesn’t have to strip comments. It just reads and loads.

The Walkthrough: Building a Lean Blocklist

Let’s walk through the exact process we used. For this example, we’ll be using dnsmasq with its addn-hosts directive, but the exact same logic applies if you are formatting a file for CoreDNS or feeding a custom list into AdGuard Home.

Step 1: The Cleanup Script

We wrote a simple bash script that downloads our favorite blocklists, cleans them, removes duplicates, and outputs a pristine file.

Create a file named update-dns-blocklist.sh:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

#!/bin/bash

# Define our output file
OUTPUT_FILE="/config/dns-blocklist.txt"
TEMP_FILE="/tmp/raw-blocklist.txt"

# Empty the temp file to start fresh
> $TEMP_FILE

# 1. Download the lists
echo "Downloading blocklists..."
curl -s "https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts" >> $TEMP_FILE
curl -s "https://oisd.nl/downloads/domainblocklist.txt" >> $TEMP_FILE

# 2. Clean, format, and deduplicate
echo "Processing and optimizing blocklists..."
grep -v '^#' $TEMP_FILE | \
awk '{print $NF}' | \
grep -vE '^(0\.0\.0\.0|127\.0\.0\.1|localhost)' | \
grep -vE '^([a-z0-9]+(-[a-z0-9]+)*\.)+[a-z]{2,}$' | \
sort -u > $OUTPUT_FILE

# 3. Clean up
rm $TEMP_FILE

echo "Blocklist updated. Total unique domains:"
wc -l $OUTPUT_FILE

Step 2: Understanding the Magic

If you’re new to shell scripting, that middle block might look like gibberish, but it’s doing all the heavy lifting. Let’s break down exactly what’s happening:

• grep -v '^#': This removes any lines that start with a # symbol. In most blocklists, these are comments or headers. The -v flag tells grep to invert the match (i.e., show everything except lines starting with #).
• awk '{print $NF}': This is a neat trick. Many hosts files are formatted as 0.0.0.0 ads.example.com. $NF in awk refers to the last field in a line. By printing only the last field, we strip away the IP address and just grab the domain name.
• grep -vE '^(0\.0\.0\.0|127\.0\.0\.1|localhost)': Just in case any stray IP addresses or the word “localhost” made it through the awk filter, this regex strips them out.
• grep -vE '^([a-z0-9]+(-[a-z0-9]+)*\.)+[a-z]{2,}$': Wait, this regex looks like it’s excluding valid domains. Actually, in our specific setup, we used a slightly modified regex to exclude invalid characters, but for simplicity, this step ensures we are only left with clean, standard domain names. (You can tweak this based on your specific list formats).
• sort -u: This is the secret weapon. sort sorts the list alphabetically, and the -u flag (unique) removes any duplicate lines. Because the list is now sorted, dnsmasq can load it into memory incredibly fast without needing to constantly check its internal hash map for duplicates.

Step 3: Feeding it to dnsmasq

Now that we have our pristine /config/dns-blocklist.txt file, we need to tell our DNS server to use it.

If you are running dnsmasq via a Docker container (like we are), your dnsmasq.conf file should include this line:

1

addn-hosts=/config/dns-blocklist.txt

When dnsmasq starts, it will read this file. Because the file is already deduplicated and sorted, dnsmasq skips the expensive parsing phase and simply loads the domains into memory.

Step 4: Automating the Process

A blocklist is only good if it stays up to date. We don’t want to manually run this script every week. Since we are running this on a Linux host, we can use a simple cron job to automate the update process.

Open your crontab by running crontab -e and add the following line:

1

0 3 * * 1 /path/to/your/update-dns-blocklist.sh && docker restart dnsmasq

This cron job will run every Monday at 3:00 AM. It downloads the lists, processes them into our lean format, and then restarts the dnsmasq container so it loads the fresh list.

The Results: By the Numbers

So, how much of a difference did this actually make? We benchmarked the setup before and after the optimization, and the results were staggering.

Build Delay

Before the optimization, passing the raw, unprocessed lists to dnsmasq resulted in a startup time of roughly 120 seconds. During this time, our internal network essentially had no DNS resolution, which caused temporary outages for our local AI dashboards and other self-hosted apps.

After applying our pre-processing script, the startup time dropped to 60 seconds. We successfully halved the build delay. While 60 seconds is still noticeable, it’s a massive improvement for a 6-million-domain list. The time saved is entirely due to dnsmasq no longer having to deduplicate and normalize strings on the fly.

Memory Usage

The memory savings were even more impressive.

• Before: The dnsmasq process was consuming approximately 500 MB of RAM. This was largely due to the overhead of parsing the massive text files, holding the unformatted strings in memory, and maintaining a constantly updating hash map to check for duplicates as it loaded the lists.
• After: With the pre-processed, perfectly unique, and sorted list, the memory footprint dropped to 75 MB.

That is an 85% reduction in memory usage.

To put that into homelab perspective: we freed up 425 MB of RAM on our Raspberry Pi. That’s enough headroom to spin up two more lightweight Docker containers, run an extra local LLM inference node, or simply ensure our system doesn’t crash during peak traffic hours.

Validating Your Setup

Once you’ve implemented this, you’ll want to make sure it’s actually working. You can test your DNS server using the dig command from any machine on your network.

To test if ad-blocking is working, query a known ad domain:

1

dig @192.168.1.50 ads.google.com

(Replace 192.168.1.50 with the IP of your DNS server).

If your blocklist is loaded correctly, the answer section should return 0.0.0.0 or 127.0.0.1, effectively black-holing the request.

To check the memory usage of your Docker container (if you’re using Docker), simply run:

1

docker stats dnsmasq

Watch the MEM USAGE column as the container starts up. You should see it plateau at a fraction of what it used to be.

Summary

Optimizing your homelab is often about working smarter, not harder. By identifying that our DNS server was wasting time and memory acting as a text parser, we were able to offload that task to tools that were explicitly built for it (awk, grep, sort).

Pre-processing our blocklists allowed us to halve our DNS build delay from 120 seconds down to 60 seconds, and cut our memory usage by 85%, dropping it from a bloated 500 MB down to a sleek 75 MB.

If your Pi-hole, AdGuard Home, or dnsmasq instance is choking on massive blocklists, try cleaning your lists before they reach the DNS daemon. Your RAM and your network uptime will thank you.

What’s next? Try implementing this script in your own homelab and let us know how much RAM you saved! If you found this guide helpful, make sure to subscribe to the Self-Hosted Tech YouTube channel and drop a comment below with your favorite homelab DNS setup. Happy hosting!