startrail-org--pixelrag/skill/pixelrag.md

---

name: pixelrag description: Visual search over documents. Use when the user wants to capture screenshots of web pages, search visual content, or build visual retrieval indexes. Triggers on: "screenshot this URL", "search Wikipedia visually", "find documents about X", "capture this page", "build a visual index".

PixelRAG — Visual Retrieval-Augmented Generation

You have access to a visual document retrieval system. Use it when the user needs to:

• Capture a web page or document as tiled screenshot images
• Search for visually relevant content in pre-built indexes (Wikipedia, news, custom)
• Build a searchable visual index from documents

Available Tools

1. Capture a URL

Render any web page to tiled JPEG screenshots:

cd ~/pixelrag
uv run pixelshot <URL> --output ./tiles

Or from Python:

from pixelrag_render import render_url
tiles = render_url("https://en.wikipedia.org/wiki/Python", "./tiles")

Output: {output_dir}/{stem}.png.tiles/tile_NNNN.jpg + tiles.json manifest.

2. Search an Index

Query the running search API (must be started first):

curl -s -X POST http://localhost:30001/search \
    -H "Content-Type: application/json" \
    -d '{"queries": [{"text": "YOUR QUERY"}], "n_docs": 5}'

The API returns JSON with hits:

{
  "results": [{
    "hits": [
      {"score": 0.73, "url": "https://en.wikipedia.org/wiki/...", "article_id": 123, ...}
    ]
  }]
}

Available endpoints (if running):

• :30001 — Wikipedia text chunks (15.7M vectors)
• :30002 — Wikipedia pixel screenshots (28M vectors)
• :30003 — Wikipedia LoRA+ViT pixel (28M vectors)

3. Build an Index

Create a searchable visual index from any document source:

cd ~/pixelrag

# Create pixelrag.yaml
cat > pixelrag.yaml << 'EOF'
source:
  type: local        # or: kiwix, web, pdf
  path: ./my_docs

embed:
  model: Qwen/Qwen3-VL-Embedding-2B
  device: cpu         # or: cuda

output: ./my_index
EOF

uv run pixelrag index build --config pixelrag.yaml --limit 100

Then serve it:

PIXELRAG_INDEX_DIR=./my_index PIXELRAG_ARTICLES_JSON=./my_index/articles.json \
uv run pixelrag serve --port 31337

4. Start/Check Serving

# Check if search API is running
curl -s http://localhost:30001/health

# Start serving a pre-built index
PIXELRAG_INDEX_DIR=/home/yichuan/pixelrag-data/text_search_index_1024 \
PIXELRAG_ARTICLES_JSON=/home/yichuan/pixelrag-data/articles.json \
uv run pixelrag serve --port 30001 &

When to Use

• User asks to find information about a topic → search the index
• User shares a URL and wants to see/capture it → use ingest
• User has documents and wants them searchable → build an index
• User asks about Wikipedia content → search the pre-built Wikipedia index
• User wants to compare visual vs text retrieval → search both :30001 (text) and :30002 (pixel)

Tips

• The search API embeds queries on CPU (~1-2s per query). For faster queries, use GPU.
• Pre-built Wikipedia indexes are at /home/yichuan/pixelrag-data/.
• The ingest CDP backend is fastest (~1s per page). Playwright backend has more options.
• For large-scale embedding, use GPU machines with pixelrag embed (vLLM/sglang backend).