Introducing Web Search API Via AI Gateway

If you’ve been hanging around the homelab community for any amount of time, you know that our setups evolve. We start with a simple Raspberry Pi running Pi-hole, move on to a NAS for our media, and eventually, we find ourselves racking servers in our closets to run our own AI models. Welcome to the modern homelab.

As local AI has exploded, thanks to projects like Ollama, text-generation-webui, and vLLM, we’ve run into a very familiar problem: our local Large Language Models (LLMs) are trapped in time. They know everything up to their training cutoff date, but ask them who won the game last night or what the current weather is in your city, and they’ll confidently hallucinate an answer.

To fix this, we need to give our local models eyes on the outside world. Today, we are going to talk about introducing a Web Search API via an AI Gateway. It sounds like a mouthful, but it’s actually a beautifully elegant architecture that you can deploy in your homelab this weekend. Let’s break down what it is, why you need it, the hardware required to run it, and a practical walkthrough to get it working.

What is an AI Gateway?

Before we dive into the web search aspect, let’s talk about the AI Gateway.

If you’ve ever worked with web applications, you’re probably familiar with a reverse proxy like Nginx or Traefik. An AI Gateway is essentially a reverse proxy, but specifically designed for AI models. Instead of routing HTTP traffic to different web servers, it routes AI prompts to different models, APIs, or endpoints.

Tools like LiteLLM, OneAPI, or Portkey act as AI Gateways. You point all your local applications, scripts, and chat interfaces to a single endpoint (the gateway). The gateway then translates your requests into the specific format required by whatever backend model you want to use—whether that’s a local Ollama instance running Llama 3, a cloud API like OpenAI, or a specialized vision model.

Why use an AI Gateway in a Homelab?

Standardization: Different local models and APIs use slightly different request formats. An AI gateway standardizes everything to the OpenAI API format. If your app supports OpenAI, it now supports your entire homelab.
Load Balancing: If you have multiple GPUs or multiple servers, you can load balance requests across them.
Fallbacks: If your local model server crashes, the gateway can automatically fall back to a cloud API so your workflow isn’t interrupted.
Centralized Management: You can manage API keys, rate limits, and logging all in one place.

The Missing Link: Web Search API

Even with a perfect AI Gateway routing to a powerful local model, the knowledge cutoff problem remains. This is where the Web Search API comes in.

By integrating a web search capability into your AI pipeline, you are essentially building an automated version of Retrieval-Augmented Generation (RAG). Instead of searching your local documents, the system searches the live internet, scrapes the top results, stuffs that text into the prompt context, and asks the LLM to synthesize an answer based on that fresh data.

You have two main routes for the Web Search API:

Hosted APIs (Tavily, Serper, Brave Search API): These are incredibly fast, optimized for AI, and return clean markdown instead of messy HTML. However, they cost money or have rate limits.
Self-Hosted Search (SearXNG): The holy grail for homelabbers. SearXNG is a privacy-respecting metasearch engine. You can run it in Docker, expose its JSON API, and use it as your own free, unlimited Web Search API.

For this article, we are going the true self-hosted route. We will use SearXNG to fetch live data, pass that data through an AI Gateway (LiteLLM), and have our local model (via Ollama) generate the final answer.

Hardware Considerations for Your AI Homelab

Since this is a hardware-focused deep dive, let’s talk about what you actually need to run this stack. The beautiful part about this architecture is that it separates the workload. You don’t need a massive GPU on your gateway server; you just need smart networking and enough compute to handle the routing and web scraping.

The AI Compute Node (The Heavy Lifter)

This is the server that actually runs the LLM. Inference is heavily dependent on your GPU.

GPU: You’ll want at least 12GB of VRAM to comfortably run an 8B parameter model (like Llama 3) with a decent context window. An NVIDIA RTX 3060 12GB is the current budget king of homelab AI. If you have deeper pockets, an RTX 4070 Ti or a used RTX 3090 with 24GB of VRAM will let you run much larger models.
CPU & RAM: A modern 6-core CPU (like a Ryzen 5 5600X) and 32GB of system RAM are plenty. The RAM is important because the model weights are loaded into system memory before being transferred to the GPU.

The Gateway and Search Node (The Router)

This is where SearXNG and LiteLLM will live. Because these tasks are primarily CPU and network-bound, you don’t need a massive GPU here.

Hardware: An Intel N100 Mini PC is perfect for this. It sips power (around 6 watts), has plenty of single-core performance for Python-based routing, and can easily handle Docker containers.
Alternative: An older 8th Gen Intel Core i5 mini PC or a thin client works great too.
Networking: Since the Gateway will be scraping the internet and sending large text payloads to your GPU server, a Gigabit ethernet connection between your homelab machines is highly recommended over Wi-Fi.

If you want to run everything on a single machine, that’s totally fine too! Just make sure your primary server has the GPU for inference and enough RAM to handle the Docker overhead.

Walkthrough: Setting Up the Stack

Let’s get our hands dirty. We are going to set up three components: SearXNG (for search), Ollama (for the LLM), and LiteLLM (as the AI Gateway).

Step 1: Deploy Ollama

Assuming you are on your AI Compute Node, the easiest way to get Ollama running is via their official install script or Docker. Let’s use Docker Compose to keep things homelab-standard.

Create a docker-compose.yml file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
version: '3.8'
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama
    # If you are using an NVIDIA GPU, uncomment the following lines
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: 1
    #           capabilities: [gpu]

volumes:
  ollama_data:

Spin it up with docker compose up -d. Then, pull a model. We’ll use Llama 3 8B, which is fast and highly capable:

1
docker exec -it ollama ollama run llama3

Step 2: Deploy SearXNG (Your Web Search API)

On your Gateway node (or the same machine, if that’s your setup), let’s spin up SearXNG. We need to make sure the JSON format is enabled in its settings.

Create a directory for SearXNG and a settings.yml file inside a searxng folder:

1
2
3
4
5
6
7
8
9
# searxng/settings.yml
use_default_settings: true
server:
  bind_address: "0.0.0.0"
  port: 8080
search:
  formats:
    - html
    - json

Now, create your docker-compose.yml for SearXNG:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
version: '3.8'
services:
  searxng:
    image: searxng/searxng:latest
    container_name: searxng
    ports:
      - "8080:8080"
    volumes:
      - ./searxng:/etc/searxng
    environment:
      - SEARXNG_BASE_URL=http://localhost:8080/
    restart: unless-stopped

Run docker compose up -d. You can test that your Web Search API is working by hitting the JSON endpoint:

1
curl "http://localhost:8080/search?q=latest+homelab+news&format=json"

If you get a massive JSON payload back, congratulations! You have a self-hosted Web Search API.

Step 3: Deploy LiteLLM (The AI Gateway)

Now we need the AI Gateway to tie everything together. LiteLLM makes it easy to proxy our Ollama instance.

Create a docker-compose.yml for LiteLLM:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
version: '3.8'
services:
  litellm:
    image: ghcr.io/berriai/litellm:main-latest
    container_name: litellm
    ports:
      - "4000:4000"
    environment:
      - LITELLM_MASTER_KEY=sk-1234 # Change this in production!
    volumes:
      - ./litellm_config.yaml:/app/config.yaml
    command: --config /app/config.yaml --port 4000 --num_workers 4

Next to that compose file, create litellm_config.yaml:

1
2
3
4
5
model_list:
  - model_name: local-llama3
    litellm_params:
      model: ollama/llama3
      api_base: http://<YOUR_OLLAMA_IP>:11434

Note: Replace <YOUR_OLLAMA_IP> with the IP address of your AI Compute Node.

Bring up the gateway: docker compose up -d. You can test the gateway by sending an OpenAI-formatted request to port 4000:

1
2
3
4
5
6
7
curl -X POST "http://localhost:4000/chat/completions" \
     -H "Authorization: Bearer sk-1234" \
     -H "Content-Type: application/json" \
     -d '{
           "model": "local-llama3",
           "messages": [{"role": "user", "content": "Say hello!"}]
         }'

Step 4: The Python Integration

Now we have our Web Search API (SearXNG) and our AI Gateway (LiteLLM routing to Ollama). We need a script that connects them. This script will:

Take a user query.
Send the query to SearXNG.
Parse the JSON results and extract the top snippets.
Send those snippets + the original query to LiteLLM.
Print the LLM’s synthesized answer.

Create a file named ai_search.py:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
import requests
import json

# Configuration
SEARXNG_URL = "http://localhost:8080/search"
LITELLM_URL = "http://localhost:4000/chat/completions"
LITELLM_API_KEY = "sk-1234"
MODEL_NAME = "local-llama3"

def search_web(query):
    """Query SearXNG and return top results."""
    params = {
        "q": query,
        "format": "json"
    }
    response = requests.get(SEARXNG_URL, params=params)
    if response.status_code == 200:
        results = response.json().get("results", [])
        # Grab the top 5 results' content
        snippets = []
        for res in results[:5]:
            title = res.get("title", "")
            content = res.get("content", "")
            snippets.append(f"Title: {title}\nSnippet: {content}")
        return "\n\n".join(snippets)
    else:
        return "Error fetching search results."

def ask_llm(query, context):
    """Send the query and context to the AI Gateway."""
    system_prompt = (
        "You are a helpful AI assistant. Use the following web search results "
        "to answer the user's question. If the answer is not in the results, "
        "say you don't know. Always cite your sources by referencing the titles.\n\n"
        f"Web Search Results:\n{context}"
    )
    
    payload = {
        "model": MODEL_NAME,
        "messages": [
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": query}
        ],
        "temperature": 0.3
    }
    
    headers = {
        "Authorization": f"Bearer {LITELLM_API_KEY}",
        "Content-Type": "application/json"
    }
    
    response = requests.post(LITELLM_URL, json=payload, headers=headers)
    if response.status_code == 200:
        return response.json()["choices"][0]["message"]["content"]
    else:
        return f"Error from LLM: {response.text}"

if __name__ == "__main__":
    user_query = input("Ask a question about recent events: ")
    
    print("\n[1/3] Searching the web via SearXNG...")
    search_context = search_web(user_query)
    
    print("[2/3] Sending context to AI Gateway (LiteLLM)...")
    answer = ask_llm(user_query, search_context)
    
    print("\n[3/3] AI Response:\n")
    print(answer)

Testing it Out

Make sure you have the requests library installed (pip install requests), and run your script:

1
python ai_search.py

Try asking it something that requires real-time data, like: “What is the latest news regarding the Raspberry Pi 5?”

You will see the script pause, fetch the live data from SearXNG, pass it through LiteLLM, and your local Llama 3 model will spit out a highly accurate, up-to-date response based on the live web search—all running entirely on your own hardware, using your own APIs, without a single penny going to a cloud provider.

Summary and Call to Action

Introducing a Web Search API via an AI Gateway completely changes the game for local, self-hosted AI. By combining the routing power of an AI Gateway like LiteLLM, the real-time fetching capability of a self-hosted SearXNG instance, and the raw inference power of a local GPU running Ollama, you’ve built a private, secure, and highly capable alternative to ChatGPT with web browsing.

Not only does this architecture keep your data entirely within your homelab, but it also standardizes your AI interactions. As you add more models, more GPUs, or even external APIs to your setup, your AI Gateway will seamlessly route traffic where it needs to go, while your Web Search API keeps your models grounded in reality.

What’s next? Try containerizing this Python script and adding a simple web UI (like Streamlit or Gradio) so you can access your new local search engine from any device on your network.

If you found this walkthrough helpful, be sure to check out the rest of the blog for more homelab, self-hosting, and local AI tutorials. Have you set up an AI Gateway in your homelab yet? What hardware are you running it on? Drop a comment below or join the conversation on our community Discord—we’d love to see what you’re building!

Introducing Web Search API Via AI Gateway#

What is an AI Gateway?#

Why use an AI Gateway in a Homelab?#

The Missing Link: Web Search API#

Hardware Considerations for Your AI Homelab#

The AI Compute Node (The Heavy Lifter)#

The Gateway and Search Node (The Router)#

Walkthrough: Setting Up the Stack#

Step 1: Deploy Ollama#

Step 2: Deploy SearXNG (Your Web Search API)#

Step 3: Deploy LiteLLM (The AI Gateway)#

Step 4: The Python Integration#

Testing it Out#

Summary and Call to Action#