Show HN: Giving Opus 5.5 A Simulated Paint Canvas

Welcome back to Self-Hosted Tech! If you’ve been hanging around the homelab and local AI communities lately, you’ve probably seen the explosion of agentic workflows. We’ve been moving past simple chatbots and into territory where LLMs write code, execute it, evaluate the results, and try again.

Recently, a fascinating project popped up on Hacker News titled “Giving Opus 5.5 A Simulated Paint Canvas.” The premise was brilliantly simple: what if you gave a state-of-the-art language model a blank digital canvas, a set of virtual paintbrushes, and told it to paint a picture?

Language models don’t inherently “see” spatial relationships the way human artists do. They predict tokens. But by giving an LLM a set of Python functions that manipulate pixels, and then feeding the results back to it, we can force it to develop spatial reasoning.

Today, we are going to build our own version of this agentic canvas. Because this is Self-Hosted Tech, we aren’t just going to use a cloud API. We’re going to build an autonomous loop where Anthropic’s Opus 5.5 acts as the “artist brain” in the cloud, while our local homelab hardware acts as the “studio”—executing the code, rendering the image, and using a local vision model to critique the artwork and send feedback back to Opus.

Let’s grab our brushes and dive in.

The Concept: How an LLM Paints

Before we look at the hardware and code, let’s understand the architecture of this project. Opus 5.5 is a massive, proprietary model, so we’ll be calling it via API. However, to make this a true agentic loop, the model needs feedback. If Opus just spits out Python code and we run it blindly, it’s a one-shot generator. We want an artist.

Here is the loop we are going to build:

The Prompt: We tell Opus 5.5 it is a painter. We give it a Python API (functions like draw_circle(), set_background(), draw_line()).
The Generation: Opus writes a Python script to paint a specific subject (e.g., “A cat sitting on a windowsill”).
The Execution: Our local homelab server runs the Python script, generating a .png file.
The Local Critique: A self-hosted vision-language model (like LLaVA) running on our local GPU looks at the generated image and describes what it sees.
The Feedback: We send Opus 5.5 the original prompt, its code, and the local vision model’s description of the resulting image. We ask Opus to adjust its painting based on the critique.
Repeat: The loop continues until the local vision model confirms the image matches the prompt.

Hardware Requirements: Building the Local Studio

Since this is a hardware-focused blog, let’s talk about the rig you need to run this locally. The heavy lifting for the LLM text generation is done by Anthropic’s servers, but your local hardware has two crucial jobs: executing the Python rendering script quickly and running a local vision model for the feedback loop.

Here is the ideal homelab setup for this project:

CPU: Any modern multi-core processor (e.g., Intel i5/i7 or AMD Ryzen 5/7). The Python execution and image handling are lightweight, but you want enough cores to handle your Docker containers without choking.
RAM: 32GB DDR4/DDR5. Running Docker, Python environments, and loading model weights into system memory before pushing them to the GPU requires a decent chunk of RAM.
GPU (The Star of the Show): An NVIDIA GPU with at least 12GB of VRAM. The RTX 3060 12GB is the undisputed budget king of homelab AI, but if you want faster inference for your local vision model, an RTX 4070 Ti (16GB) or a used RTX 3090 (24GB) will let you run larger vision models like LLaVA-13B or even Qwen-VL with plenty of context room to spare.
Storage: A basic NVMe SSD. You’ll be generating and deleting hundreds of images during the testing phase, so you want fast I/O.

We will be using Docker to containerize our local vision model. Ollama is our tool of choice here because it makes spinning up local LLMs incredibly simple.

The Walkthrough: Setting Up the Canvas

Let’s get into the practical implementation. We’re going to write a Python script that orchestrates this entire dance between the cloud API and our local hardware.

Step 1: Set up the Local Vision Critic

First, ensure Docker is installed on your homelab server. Pull and run a local vision model using Ollama. Open your terminal and run:

1
2
3
4
5
# Pull a lightweight but capable vision model
ollama run llava:7b

# Keep it running in the background, or start the Ollama service
sudo systemctl start ollama

This local model will act as our art critic. It doesn’t need to be as smart as Opus 5.5; it just needs to accurately describe the shapes, colors, and layout of the image so Opus knows if its code worked.

Step 2: Define the Canvas API

We need to give Opus 5.5 a constrained environment. If we just tell it to “write Python code to draw a cat,” it might use obscure libraries we don’t have installed. Instead, we define a strict set of functions using Python’s Pillow (PIL) library.

Create a file named canvas_api.py:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
from PIL import Image, ImageDraw

class Canvas:
    def __init__(self, width=500, height=500):
        self.width = width
        self.height = height
        self.img = Image.new('RGB', (width, height), color='white')
        self.draw = ImageDraw.Draw(self.img)

    def set_background(self, color):
        self.draw.rectangle([0, 0, self.width, self.height], fill=color)

    def draw_circle(self, x, y, radius, color):
        self.draw.ellipse([x-radius, y-radius, x+radius, y+radius], fill=color)

    def draw_rectangle(self, x1, y1, x2, y2, color):
        self.draw.rectangle([x1, y1, x2, y2], fill=color)

    def draw_line(self, x1, y1, x2, y2, width, color):
        self.draw.line([x1, y1, x2, y2], fill=color, width=width)

    def save(self, filename):
        self.img.save(filename)

This is the “brush” Opus 5.5 will use. It’s simple, predictable, and forces the LLM to think in raw coordinates.

Step 3: The Agentic Orchestrator

Now for the magic. We need a master script that talks to Opus 5.5, extracts the code, runs it, shows the image to our local LLaVA model, and feeds the critique back to Opus.

Here is the core logic for our orchestrator (painter.py). You will need the anthropic and requests Python packages installed (pip install anthropic requests pillow).

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
import anthropic
import requests
import base64
import json
import re

# Initialize the Anthropic client
client = anthropic.Anthropic(api_key="YOUR_ANTHROPIC_API_KEY")

# System prompt defining the rules for Opus 5.5
SYSTEM_PROMPT = """
You are an expert digital artist operating a simulated canvas.
You have access to a Python class called `Canvas` with the following methods:
- set_background(color)
- draw_circle(x, y, radius, color)
- draw_rectangle(x1, y1, x2, y2, color)
- draw_line(x1, y1, x2, y2, width, color)
- save(filename)

The canvas is 500x500 pixels. (0,0) is top-left.
You must write a complete, executable Python script using this class to draw the requested image.
Only output Python code inside ```python blocks. Do not include explanations.
"""

def get_opus_code(prompt, critique=None):
    """Asks Opus 5.5 to generate or refine the painting code."""
    messages = [{"role": "user", "content": prompt}]
    
    if critique:
        messages.append({
            "role": "user", 
            "content": f"You previously generated code, but a local vision critic saw this: '{critique}'. Please rewrite the code to fix any issues and better match the original prompt."
        })

    response = client.messages.create(
        model="claude-3-opus-5-5-20241022", # Hypothetical Opus 5.5 model ID
        max_tokens=2000,
        system=SYSTEM_PROMPT,
        messages=messages
    )
    
    # Extract Python code from the response
    text = response.content[0].text
    code_match = re.search(r'```python\n(.*?)\n```', text, re.DOTALL)
    if code_match:
        return code_match.group(1)
    return text

def execute_code(code):
    """Runs the code generated by Opus and saves the image."""
    # We inject the Canvas class definition into the executed environment
    full_code = open('canvas_api.py').read() + "\n" + code + "\ncanvas.save('current_art.png')"
    exec(full_code, globals())

def get_local_critique(image_path):
    """Sends the image to our local LLaVA model running on homelab hardware."""
    with open(image_path, "rb") as image_file:
        img_base64 = base64.b64encode(image_file.read()).decode('utf-8')
    
    payload = {
        "model": "llava:7b",
        "prompt": "Describe this image in detail. What objects, shapes, and colors do you see? Is there a cat?",
        "images": [img_base64]
    }
    
    response = requests.post("http://localhost:11434/api/generate", json=payload)
    # Ollama streams responses, so we parse the JSON lines
    critique = ""
    for line in response.iter_lines():
        if line:
            critique += json.loads(line).get("response", "")
    
    return critique

# The Main Agentic Loop
target_image = "A black cat sitting on a brown windowsill looking at a yellow moon."
max_iterations = 3

for i in range(max_iterations):
    print(f"--- Iteration {i+1} ---")
    
    if i == 0:
        code = get_opus_code(target_image)
    else:
        code = get_opus_code(target_image, critique=critique)
        
    print("Opus 5.5 generated code. Executing locally...")
    execute_code(code)
    
    print("Getting critique from local hardware (LLaVA)...")
    critique = get_local_critique("current_art.png")
    print(f"Local Critique: {critique}")
    
    if "cat" in critique.lower() and "moon" in critique.lower():
        print("Success! The local critic confirms the image matches the prompt.")
        break

Step 4: Running the Loop

When you run python painter.py, here is what happens under the hood:

The script pings Anthropic’s API. Opus 5.5 receives the prompt and outputs a block of Python code. It might start with a black circle for the cat’s body, a brown rectangle for the sill, and a yellow circle for the moon.
Your local homelab CPU executes that Python code. The Pillow library renders the pixels and saves current_art.png to your NVMe drive.
The script loads that image, converts it to base64, and sends it to localhost:11434—your local Ollama instance.
Your local GPU (let’s say your RTX 3060) spins up, loads the LLaVA weights into VRAM, processes the image, and generates a text description: “I see a yellow circle, a black circle, and a brown rectangle. It does not look like a cat.”
This critique is sent back to Opus 5.5. Opus realizes its simple shapes didn’t read as a cat. In the next iteration, it might add triangles for ears, green circles for eyes, and whiskers using draw_line.
The loop repeats until the local vision model is satisfied.

Why This Matters for Homelabbers

You might be wondering, “Why go through all this trouble just to get an AI to draw a blocky cat?”

The answer is that this project is a microcosm of modern AI infrastructure. We are combining cloud-based proprietary models with local, self-hosted open-source models. This hybrid architecture is exactly where the cutting edge of homelab AI is right now.

By running the vision critic locally, you maintain privacy (you aren’t uploading generated images to a third-party vision API), you save money (local inference is free after the hardware cost), and you get to utilize that beautiful GPU sitting in your server rack.

Furthermore, this teaches you how to build robust agentic loops. The same architecture—Cloud Brain + Local Executor + Local Evaluator—can be repurposed for incredibly practical tasks. You could swap the “Canvas” for a local Docker API, asking Opus to write Ansible scripts to configure your homelab, and using a local model to verify the configurations. The possibilities are endless.

Summary and Call to Action

Giving Opus 5.5 a simulated paint canvas is a fantastic way to explore spatial reasoning in large language models. By building an agentic loop, we allow a cloud-based LLM to generate code, execute it on our local homelab hardware, and receive feedback from a self-hosted vision model.

It’s a perfect blend of cloud computing and local self-hosting, pushing your hardware to do more than just host Jellyfin and Pi-hole.

What would you ask Opus 5.5 to paint? More importantly, what other agentic loops are you building in your homelab? Drop a comment down below or join our Discord community to share your local AI setups. If you found this walkthrough helpful, hit that subscribe button on the YouTube channel so you don’t miss our next deep dive into homelab hardware and self-hosted AI.

Until next time, keep your containers running and your GPUs cool!

Show HN: Giving Opus 5.5 A Simulated Paint Canvas#

The Concept: How an LLM Paints#

Hardware Requirements: Building the Local Studio#

The Walkthrough: Setting Up the Canvas#

Step 1: Set up the Local Vision Critic#

Step 2: Define the Canvas API#

Step 3: The Agentic Orchestrator#

Step 4: Running the Loop#

Why This Matters for Homelabbers#

Summary and Call to Action#