Cloudflare And OpenAI: Making AI Discovery Better For Everyone

Hey folks, welcome back to Self-Hosted Tech. If you’ve been following the channel or the blog, you know we love cramming oversized GPUs into tiny mini-PCs, stacking 2U servers in our closets, and running our own local AI models. We do this because we love the hardware, but also because we want control over our data and our compute.

But even if we run our own local LLMs on our homelab rigs, we still live in a broader internet ecosystem. Recently, a fascinating partnership and series of updates between Cloudflare and OpenAI have started changing how AI bots discover, crawl, and interact with websites. For self-hosters, this is a massive deal.

Today, we’re going to break down what Cloudflare and OpenAI are doing to make AI discovery better, why it matters for your homelab hardware, and how you can use Cloudflare to safely expose your local AI endpoints to the internet while keeping unwanted AI crawlers away from your private data.

The Backstory: The AI Scraping Wild West

For the last couple of years, the internet has been under siege by AI crawlers. Companies training Large Language Models (LLMs) sent out bots to scrape every piece of text, code, and image they could find. Traditional robots.txt files were largely ignored, and self-hosters found their bandwidth being eaten alive by aggressive crawlers hitting their self-hosted wikis, blogs, and Nextcloud instances.

If you’re running a homelab, bandwidth is often a precious commodity. Maybe you have a symmetrical gigabit connection, but if an AI scraper decides to download your entire 50GB self-hosted media wiki every week, it’s going to impact your network’s performance—and the performance of the services you actually want to use.

Enter Cloudflare. As the internet’s largest reverse proxy and CDN, Cloudflare sits in front of a massive percentage of the web. They recently introduced a suite of tools (often referred to under the “AI Audit” banner) to give website owners granular control over AI bots.

Simultaneously, OpenAI realized that the “scrape everything and apologize later” strategy wasn’t sustainable. They started working with Cloudflare to standardize how their crawlers identify themselves and how publishers can block or allow them.

Why Should Homelabbers Care?

You might be thinking, “I just run a local Ollama instance on an old RTX 3060 in my basement. Why do I care about OpenAI and Cloudflare?”

Here’s why: As your homelab grows, you inevitably start hosting public-facing services. You might host a personal blog on a Raspberry Pi, a public-facing documentation site on an Intel N100 mini-PC, or even a web UI for your local AI models so your friends can try them out.

When you expose hardware from your living room to the public internet, you need a bouncer. Cloudflare Tunnels (formerly Argo Tunnels) have become the gold standard for homelabbers to expose services without opening router ports. Cloudflare handles the DNS, the SSL, and the security.

With the new Cloudflare and OpenAI collaboration, you can now:

1. Protect your hardware’s bandwidth: Stop AI training bots from hammering your self-hosted sites.
2. Control AI Search indexing: Allow OpenAI’s search bots to index your public blog so you show up in SearchGPT, while blocking the bots that train the models.
3. Safely expose local AI: Use Cloudflare to put your local AI hardware on the web securely, ensuring only authorized users can access your GPU compute.

Understanding OpenAI’s New User Agents

To make AI discovery better, OpenAI has split their bots into distinct user agents. This means you no longer have to block “OpenAI” entirely if you just don’t want them training on your data.

Here are the main ones you need to know:

• GPTBot: This is the bot used to crawl the web for training data. If you don’t want your self-hosted blog contributing to GPT-5, you block this.
• OAI-SearchBot: This is the bot used for OpenAI’s search products (like SearchGPT). It crawls to index pages for retrieval, not for training. If you want traffic from people using OpenAI’s search, you should allow this bot.
• ChatGPT-User: This user agent represents actions taken by a human user asking ChatGPT to fetch a specific URL. You generally want to allow this so ChatGPT can read a page a user explicitly asked it to look at.

Cloudflare’s firewall now has built-in, verified integrations for these specific OpenAI bots. Cloudflare verifies that a bot claiming to be GPTBot actually belongs to OpenAI through reverse DNS checks. This stops spoofing dead in its tracks.

Practical Walkthrough: Exposing Local AI and Managing Bots

Let’s get practical. We are going to set up a local AI stack on our homelab hardware, expose it to the internet using a Cloudflare Tunnel, and then set up Cloudflare Web Application Firewall (WAF) rules to manage how OpenAI interacts with our network.

Step 1: The Hardware and Docker Stack

For this example, let’s assume you have a homelab server with an NVIDIA GPU (like an RTX 3060 12GB, which is the current darling of budget local AI). We’re going to run Ollama for the model backend and Open WebUI as our frontend interface.

Create a docker-compose.yml file on your server:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

version: '3.8'

services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    ports:
      - "3000:8080"
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
    volumes:
      - open_webui_data:/app/backend/data
    depends_on:
      - ollama

volumes:
  ollama_data:
  open_webui_data:

Run docker-compose up -d. You now have a local AI interface running on port 3000 of your homelab server.

Note: If you are exposing Open WebUI to the public, make sure you disable “New Signups” in the admin settings so random people on the internet don’t burn your GPU compute!

Step 2: Setting Up the Cloudflare Tunnel

Instead of opening port 443 on your router and dealing with port forwarding (which exposes your home IP to the world), we are going to use Cloudflare Tunnels.

First, install cloudflared on your homelab server. If you’re on Ubuntu/Debian:

1
2
3
4

sudo mkdir -p --mode=0755 /usr/share/keyrings
curl -fsSL https://pkg.cloudflare.com/cloudflare-main.gpg | sudo tee /usr/share/keyrings/cloudflare-main.gpg >/dev/null
echo "deb [signed-by=/usr/share/keyrings/cloudflare-main.gpg] https://pkg.cloudflare.com/cloudflared $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/cloudflared.list
sudo apt update && sudo apt install cloudflared -y

Next, authenticate cloudflared with your Cloudflare account:

1

cloudflared tunnel login

This will give you a URL to click in your browser. Once authenticated, create a tunnel:

1

cloudflared tunnel create local-ai-tunnel

Now, create a configuration file at ~/.cloudflared/config.yml:

1
2
3
4
5
6
7

tunnel: <your-tunnel-id-here>
credentials-file: /root/.cloudflared/<your-tunnel-id-here>.json

ingress:
  - hostname: ai.yourdomain.com
    service: http://localhost:3000
  - service: http_status:404

Finally, go to your Cloudflare dashboard and add a DNS CNAME record for ai.yourdomain.com pointing to your tunnel ID, and start the tunnel:

1

cloudflared tunnel run local-ai-tunnel

Boom. Your local AI hardware is now accessible from the internet via https://ai.yourdomain.com, completely bypassing your router’s firewall and hiding your home IP address.

Step 3: Crafting the WAF Rules for OpenAI

Now that your hardware is online, let’s use Cloudflare to manage AI discovery. We want to allow OAI-SearchBot and ChatGPT-User (so people can find and read your site), but block GPTBot (so they don’t train on your data).

Head over to your Cloudflare Dashboard, select your domain, and navigate to Security > WAF.

We are going to create a custom rule. Click Create rule and name it “Block AI Training Bots”.

Instead of using the visual editor, we can use the Expression Editor. Cloudflare has built-in fields for verified bots. Paste the following expression:

1

(http.user_agent contains "GPTBot") and (cf.verified_bot eq false)

Wait, why cf.verified_bot eq false? Because if someone tries to spoof the GPTBot user agent to bypass other security rules, Cloudflare will flag it as an unverified bot and block it. If it is a verified GPTBot, this specific rule won’t catch it.

Let’s refine this. We actually want to block the verified GPTBot entirely, but we want to make sure spoofers are blocked too. Let’s make two rules, or one comprehensive one.

Let’s block all GPTBot traffic (verified or not) on our public-facing sites, but leave the search bot alone.

Rule 1: Block GPTBot Training

• Expression: (cf.bot_management.verified_bot eq true and http.user_agent contains "GPTBot") or (http.user_agent contains "GPTBot" and cf.bot_management.verified_bot eq false)
• Action: Block

Actually, Cloudflare makes this even easier now. In the WAF custom rules, you can simply use the “Verified Bot” category. But to be incredibly specific about OpenAI’s new split user agents, the expression above is bulletproof.

If you also host a public blog on the same domain (e.g., blog.yourdomain.com) and you want to be indexed by SearchGPT, you don’t need to do anything. Cloudflare allows verified good bots by default. OAI-SearchBot will be allowed through, while GPTBot hits the brick wall you just built.

Step 4: Protecting Your Local AI Endpoint

Remember the ai.yourdomain.com subdomain we set up for Open WebUI? You probably don’t want any bots crawling that. It’s a private application.

Let’s create another WAF rule to protect our local AI compute from being indexed or scraped by anything.

Rule 2: Protect AI Endpoint

• Expression: (http.host eq "ai.yourdomain.com" and not http.request.uri.path contains "/api/v1/auth") and (cf.bot_management.verified_bot eq true)
• Action: Block

This rule ensures that no verified bot (Google, Bing, OpenAI, Anthropic, etc.) can crawl your local AI interface. You can also add IP address restrictions to this rule if you only want to allow your own devices to access the endpoint.

The Bigger Picture: Edge Compute and AI Discovery

What Cloudflare and OpenAI are doing here is laying the groundwork for a new internet standard. As AI becomes more integrated into how we search and consume information, the line between “search engine” and “AI model” is blurring.

For us hardware nerds and self-hosters, this is a massive win. We are running enterprise-grade infrastructure out of our homes. We have servers generating heat, spinning fans, and crunching vectors. We need tools that respect our hardware limits and our data privacy.

By splitting their crawlers into distinct, verifiable user agents, OpenAI is giving us the granular control we need. And by providing the WAF infrastructure to enforce those choices, Cloudflare is acting as the ultimate homelab enabler.

You can now confidently expose your self-hosted projects to the web, knowing that you can participate in AI search discovery without donating your server’s bandwidth to a multi-billion dollar model training run.

Summary

The partnership between Cloudflare and OpenAI is a huge step forward for web discovery and self-hosted privacy. By standardizing AI crawler user agents and integrating them into Cloudflare’s bot management tools, self-hosters finally have the power to choose how AI interacts with their hardware.

We walked through how to deploy a local AI stack using Docker and an NVIDIA GPU, expose it securely using Cloudflare Tunnels, and implement WAF rules to block AI training bots while allowing AI search bots.

Call to Action

What do you think about OpenAI’s split user agents? Are you blocking all AI bots from your homelab, or are you embracing the new era of AI search? Let us know in the comments below!

If you found this guide helpful, make sure to smash that like button and subscribe to the channel for more deep dives into homelab hardware, local AI, and self-hosting. Until next time, keep those servers cool and your tunnels secure!