docs: make Chinese README the default
This commit is contained in:
@@ -1,9 +1,15 @@
|
||||
<!-- WEHUB_ZH_README -->
|
||||
> [!NOTE]
|
||||
> 本文档由 WeHub 基于上游 README 翻译整理,属于社区翻译,非官方中文文档。
|
||||
> [English](./README.en.md) · [原始项目](https://github.com/StarTrail-org/PixelRAG) · [上游 README](https://github.com/StarTrail-org/PixelRAG/blob/HEAD/README.md)
|
||||
> 原作者、版权与许可证归属以原始项目及本仓库 LICENSE 文件为准。
|
||||
|
||||
<p align="center">
|
||||
<img src="docs/assets/banner.png" alt="PixelRAG — Visual Retrieval-Augmented Generation" width="100%">
|
||||
<img src="docs/assets/banner.png" alt="PixelRAG — 视觉检索增强生成(Visual Retrieval-Augmented Generation)" width="100%">
|
||||
</p>
|
||||
<p align="center">
|
||||
Official codebase for <b><a href="https://arxiv.org/abs/2606.28344">PIXELRAG: Web Screenshots Beat Text for
|
||||
Retrieval-Augmented Generation</a></b>
|
||||
<b><a href="https://arxiv.org/abs/2606.28344">PIXELRAG: Web Screenshots Beat Text for
|
||||
Retrieval-Augmented Generation</a></b> 的官方代码库
|
||||
</p>
|
||||
<p align="center">
|
||||
<a href="https://yichuan-w.github.io/">Yichuan Wang</a>*,
|
||||
@@ -16,8 +22,8 @@ Retrieval-Augmented Generation</a></b>
|
||||
<a href="https://people.eecs.berkeley.edu/~jegonzal/">Joseph E. Gonzalez</a>†,
|
||||
<a href="https://www.sewonmin.com/">Sewon Min</a>†
|
||||
</p>
|
||||
<p align="center"><sub>* Equal contribution † Equal advising</sub><br><sub>Work done at <a href="https://sky.cs.berkeley.edu/">Berkeley SkyLab</a> & <a href="https://bair.berkeley.edu/">BAIR</a> & <a href="https://nlp.cs.berkeley.edu/">Berkeley NLP</a></sub></p>
|
||||
<p align="center">Search any document by how it <em>looks</em>, not just the text it contains.</p>
|
||||
<p align="center"><sub>* 同等贡献 † 同等指导</sub><br><sub>工作完成于 <a href="https://sky.cs.berkeley.edu/">Berkeley SkyLab</a> & <a href="https://bair.berkeley.edu/">BAIR</a> & <a href="https://nlp.cs.berkeley.edu/">Berkeley NLP</a></sub></p>
|
||||
<p align="center">按文档的<em>外观</em>搜索,而不仅仅看它包含的文字。</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="https://github.com/StarTrail-org/PixelRAG/actions/workflows/ci.yml"><img src="https://github.com/StarTrail-org/PixelRAG/actions/workflows/ci.yml/badge.svg" alt="CI"></a>
|
||||
@@ -28,10 +34,10 @@ Retrieval-Augmented Generation</a></b>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
<a href="#what-it-is">What it is</a> ·
|
||||
<a href="#give-claude-eyes">Give Claude eyes</a> ·
|
||||
<a href="#how-it-works">How it works</a> ·
|
||||
<a href="#pipelines">Pipelines</a>
|
||||
<a href="#what-it-is">它是什么</a> ·
|
||||
<a href="#give-claude-eyes">给 Claude 一双「眼睛」</a> ·
|
||||
<a href="#how-it-works">工作原理</a> ·
|
||||
<a href="#pipelines">流水线</a>
|
||||
</p>
|
||||
|
||||
---
|
||||
@@ -40,7 +46,7 @@ Retrieval-Augmented Generation</a></b>
|
||||
pip install pixelrag
|
||||
```
|
||||
|
||||
The two core operations — **render** a page to screenshots, **search** a visual index:
|
||||
两大核心操作——将页面**渲染(render)**为截图,并在视觉索引中**搜索(search)**:
|
||||
|
||||
```bash
|
||||
# Render any page or document to screenshot tiles
|
||||
@@ -52,30 +58,19 @@ curl -X POST https://api.pixelrag.ai/search \
|
||||
-d '{"queries": [{"text": "What is the capital of France?"}], "n_docs": 5}'
|
||||
```
|
||||
|
||||
> **Live, hosted endpoint** — [`https://api.pixelrag.ai`](https://api.pixelrag.ai/status) serves a
|
||||
> pre-built index of **8.28M Wikipedia pages**. No setup, no API key. It even takes an image as the query
|
||||
> ([visual search](https://pixelrag.ai/docs#search)) — see the **[API reference →](https://pixelrag.ai/docs)**.
|
||||
> **在线托管端点** — [`https://api.pixelrag.ai`](https://api.pixelrag.ai/status) 提供由 **8.28M** 篇 Wikipedia 页面构成的预构建索引。无需配置,无需 API 密钥。查询甚至可以是图片([视觉搜索(visual search)](https://pixelrag.ai/docs#search)) — 详见 **[API 参考 →](https://pixelrag.ai/docs)**.
|
||||
|
||||
Or try it in the browser at **[pixelrag.ai](https://pixelrag.ai)**, or run the demo notebook in
|
||||
Colab [](https://colab.research.google.com/github/StarTrail-org/PixelRAG/blob/main/demos/quickstart.ipynb) — it
|
||||
renders a page and searches the hosted index, with the images inline.
|
||||
或在浏览器中访问 **[pixelrag.ai](https://pixelrag.ai)**,,或在 Colab 中运行演示 notebook [](https://colab.research.google.com/github/StarTrail-org/PixelRAG/blob/main/demos/quickstart.ipynb) — 它会渲染页面并搜索托管索引,图像内联显示。
|
||||
|
||||
## What it is
|
||||
## 它是什么
|
||||
|
||||
PixelRAG renders documents — web pages, PDFs, images — as screenshots and retrieves over the
|
||||
images directly. Visual structure that HTML parsing throws away — tables, charts, layout,
|
||||
infographics — stays intact, so the reader model can actually answer questions about it.
|
||||
Wikipedia's 8.28M articles ship as a pre-built index; the pipeline itself is general-purpose.
|
||||
PixelRAG 将文档——网页、PDF、图像——渲染为截图,并直接在图像上进行检索。HTML 解析会丢弃的视觉结构——表格、图表、版式、信息图——得以保留,因此阅读模型能够真正回答相关问题。Wikipedia 的 8.28M 篇文章以预构建索引形式提供;流水线本身则是通用型的。
|
||||
|
||||
## Give Claude eyes
|
||||
## 给 Claude 一双「眼睛」
|
||||
|
||||
The renderer also ships as a Claude Code plugin — the **pixelbrowse** skill. Instead of fetching
|
||||
raw HTML, Claude screenshots a page with `pixelshot` and _reads the image_, so it sees
|
||||
charts, diagrams, tables, and layout the way a person does.
|
||||
渲染器还以 Claude Code 插件形式提供——**pixelbrowse** 技能。Claude 不再抓取原始 HTML,而是用 `pixelshot` 对页面截图并*阅读图像*,从而像人一样看到图表、示意图、表格和版式。
|
||||
|
||||
Install it — no clone needed. Install the `pixelshot` CLI so it's on your `PATH`
|
||||
(use `uv tool` or `pipx` to keep it isolated yet always available to Claude — a
|
||||
plain `pip install` into a project venv may leave `pixelshot` off `PATH`):
|
||||
安装即可——无需克隆仓库。安装 `pixelshot` CLI,使其位于你的 `PATH` 上(使用 `uv tool` 或 `pipx`,既可隔离环境,又随时可供 Claude 使用——仅向项目 venv 执行普通 `pip install` 可能会使 `pixelshot` 不在 `PATH` 中):
|
||||
|
||||
```bash
|
||||
uv tool install pixelrag # pixelshot on PATH (or: pipx install pixelrag)
|
||||
@@ -83,50 +78,45 @@ claude plugin marketplace add StarTrail-org/PixelRAG
|
||||
claude plugin install pixelbrowse@pixelrag-plugins
|
||||
```
|
||||
|
||||
Then just ask Claude to look at a page:
|
||||
然后直接让 Claude 查看某个页面即可:
|
||||
|
||||
```bash
|
||||
claude -p "screenshot https://news.ycombinator.com and summarize the top stories"
|
||||
claude -p "screenshot https://arxiv.org/abs/2404.12387 and explain the key findings"
|
||||
```
|
||||
|
||||
Or use the slash command in an interactive session: `/screenshot https://example.com`.
|
||||
No MCP server, no backend: the skill just calls `pixelshot` (Playwright/CDP) on your machine.
|
||||
或在交互式会话中使用斜杠命令:`/screenshot https://example.com`。
|
||||
无需 MCP 服务器,也无需后端:该技能只在你本机调用 `pixelshot`(Playwright/CDP)。
|
||||
|
||||
## How it works
|
||||
## 工作原理
|
||||
|
||||
<p align="center">
|
||||
<img src="docs/assets/pipeline.png" alt="Text-based RAG parses to text and loses the table; PixelRAG renders to screenshot tiles and keeps it" width="100%">
|
||||
<img src="docs/assets/pipeline.png" alt="基于文本的 RAG 解析为文本并丢失表格;PixelRAG 渲染为截图分块并保留表格" width="100%">
|
||||
</p>
|
||||
|
||||
Text-based RAG parses the page to text chunks and **loses the table** — the reader can't find the
|
||||
answer. PixelRAG renders the page to **screenshot tiles**, retrieves the right tile, and the reader
|
||||
reads the number straight off the image.
|
||||
基于文本的 RAG 将页面解析为文本块并**丢失表格**——阅读器找不到答案。PixelRAG 将页面渲染为**截图分块(screenshot tiles)**,检索到正确的分块后,阅读器可直接从图像中读取数字。
|
||||
|
||||
Two pieces make this work: (1) rendering documents to images instead of parsing them to text, and
|
||||
(2) a `Qwen3-VL-Embedding` model, LoRA-fine-tuned on screenshot data, that embeds page images into
|
||||
a space where visual content is retrievable.
|
||||
实现这一点需要两个组件:(1) 将文档渲染为图像,而非解析为文本;(2) 一个在截图数据上经 LoRA 微调的 `Qwen3-VL-Embedding` 模型,将页面图像嵌入到可视内容可检索的空间中。
|
||||
|
||||
## Pipelines
|
||||
## 流水线
|
||||
|
||||
Capture is the standalone `pixelshot` command; the rest of the pipeline runs through the
|
||||
`pixelrag` umbrella — `pixelrag <stage>`. Install only the stages you need:
|
||||
Capture 是独立的 `pixelshot` 命令;流水线的其余部分通过 `pixelrag` 总入口运行——`pixelrag <stage>`。只需安装你需要的阶段:
|
||||
|
||||
| Command | What it does | Install |
|
||||
| 命令 | 功能 | 安装 |
|
||||
| ------------------------------------------ | --------------------------------------------------------------- | ------------------------------- |
|
||||
| `pixelshot` | Document → image tiles (Playwright CDP, PDF) | `pip install pixelrag` |
|
||||
| `pixelrag chunk` · `embed` · `build-index` | Tiles → vectors → FAISS index | `pip install 'pixelrag[embed]'` |
|
||||
| `pixelrag index` | Orchestrates the full pipeline: source → ingest → embed → index | `pip install 'pixelrag[index]'` |
|
||||
| `pixelrag serve` | FAISS search API (FastAPI, CPU or GPU) | `pip install 'pixelrag[serve]'` |
|
||||
| `pixelshot` | 文档 → 图像分块(Playwright CDP、PDF) | `pip install pixelrag` |
|
||||
| `pixelrag chunk` · `embed` · `build-index` | 分块 → 向量 → FAISS 索引 | `pip install 'pixelrag[embed]'` |
|
||||
| `pixelrag index` | 编排完整流水线:源 → 摄取 → 嵌入 → 索引 | `pip install 'pixelrag[index]'` |
|
||||
| `pixelrag serve` | FAISS 搜索 API(FastAPI,CPU 或 GPU) | `pip install 'pixelrag[serve]'` |
|
||||
|
||||
```
|
||||
render ←── index ──→ embed serve (independent) train → serve (HTTP)
|
||||
```
|
||||
|
||||
**`train` is a separate uv project** with its own pinned env (`torch==2.9.1+cu129`,
|
||||
`transformers==4.57.1`, cuDNN 9.20) — install it from inside `train/`, not from the root.
|
||||
**`train` 是一个独立的 uv 项目**,拥有各自锁定的环境(`torch==2.9.1+cu129`、
|
||||
`transformers==4.57.1`、cuDNN 9.20)——请在 `train/` 内部安装,而非在仓库根目录。
|
||||
|
||||
### Search a pre-built index
|
||||
### 搜索预构建索引
|
||||
|
||||
```bash
|
||||
pip install 'pixelrag[serve]'
|
||||
@@ -144,9 +134,9 @@ curl -X POST http://localhost:30001/search \
|
||||
-d '{"queries": [{"text": "What is the capital of France?"}], "n_docs": 5}'
|
||||
```
|
||||
|
||||
### Build an index from your own documents
|
||||
### 用自己的文档构建索引
|
||||
|
||||
Works on **Linux (CUDA)** and **macOS (Apple Silicon / MPS)** — `device: auto` picks the best backend.
|
||||
适用于 **Linux(CUDA)** 和 **macOS(Apple Silicon / MPS)**——`device: auto` 会自动选择最佳后端。
|
||||
|
||||
```bash
|
||||
pip install 'pixelrag[index]'
|
||||
@@ -170,9 +160,9 @@ pixelrag serve --index-dir ./my_index --port 30001
|
||||
```
|
||||
|
||||
<details>
|
||||
<summary><strong>Try it: index a PDF and search it locally</strong></summary>
|
||||
<summary><strong>试试看:索引 PDF 并在本地搜索</strong></summary>
|
||||
|
||||
No GPU required — runs on macOS (Apple Silicon) or any machine with Python 3.10+.
|
||||
无需 GPU — 可在 macOS(Apple Silicon)或任何装有 Python 3.10+ 的机器上运行。
|
||||
|
||||
```bash
|
||||
pip install 'pixelrag[index]'
|
||||
@@ -207,7 +197,7 @@ curl -X POST http://localhost:30001/search \
|
||||
|
||||
</details>
|
||||
|
||||
### Render a page programmatically
|
||||
### 以编程方式渲染页面
|
||||
|
||||
```python
|
||||
from pixelrag_render import render_url
|
||||
@@ -216,7 +206,7 @@ from pixelrag_render import render_url
|
||||
tiles = render_url("https://en.wikipedia.org/wiki/Python", "./tiles")
|
||||
```
|
||||
|
||||
The same rendering is available as a CLI — `pixelshot` ships with `pip install pixelrag`:
|
||||
同样的渲染能力也可通过 CLI 使用 — `pixelshot` 随 `pip install pixelrag` 一起提供:
|
||||
|
||||
```bash
|
||||
# Web page → tiles (headless Chromium via CDP)
|
||||
@@ -230,16 +220,11 @@ pixelshot paper.pdf -o ./tiles --dpi 200
|
||||
pixelshot https://github.com/StarTrail-org/PixelRAG paper.pdf -o ./tiles
|
||||
```
|
||||
|
||||
> **Chrome on Windows/macOS** — the bundled turbo `headless_shell` auto-installs on
|
||||
> **linux-x64** only. Elsewhere, `pixelshot` uses your system Chrome/Chromium (or
|
||||
> Playwright's Chromium), auto-detected from the standard install locations. Point it at
|
||||
> a specific binary with `CHROME_PATH=/path/to/chrome` if it isn't found automatically.
|
||||
> Each render runs in an isolated, throwaway Chrome profile, so it works even while you
|
||||
> have Chrome open.
|
||||
> **Windows/macOS 上的 Chrome** — 捆绑的 turbo `headless_shell` 仅在 **linux-x64** 上自动安装。在其他平台上,`pixelshot` 会使用你系统中的 Chrome/Chromium(或 Playwright 的 Chromium),从标准安装位置自动检测。若无法自动找到,可通过 `CHROME_PATH=/path/to/chrome` 指定具体可执行文件路径。每次渲染都在隔离的临时 Chrome 配置文件中运行,因此即使你正在使用 Chrome 也能正常工作。
|
||||
|
||||
### Embed tools (standalone)
|
||||
### 嵌入工具(独立运行)
|
||||
|
||||
Each stage runs independently, without the orchestrator:
|
||||
各阶段可独立运行,无需编排器(orchestrator):
|
||||
|
||||
```bash
|
||||
pip install 'pixelrag[embed]'
|
||||
@@ -249,24 +234,17 @@ pixelrag embed --shard-dir ./tiles --output-dir ./embeddings --gpu-ids 0,1
|
||||
pixelrag build-index --embeddings-dir ./embeddings --output-dir ./index
|
||||
```
|
||||
|
||||
### Training
|
||||
### 训练
|
||||
|
||||
Fine-tuning lives in `train/` — a **separate uv project** (`wiki-screenshot-training`) with its own
|
||||
pinned env. It LoRA-fine-tunes `Qwen/Qwen3-VL-Embedding-2B` for webpage retrieval; run it from
|
||||
inside `train/` (`cd train && uv sync`). See [`train/README.md`](train/README.md) for the full recipe.
|
||||
微调代码位于 `train/` — 这是一个 **独立的 uv 项目**(`wiki-screenshot-training`),拥有自己固定版本的环境。它对 `Qwen/Qwen3-VL-Embedding-2B` 进行 LoRA 微调,用于网页检索;请在 `train/`(`cd train && uv sync`)内运行。完整流程请参阅 [`train/README.md`](train/README.md)。
|
||||
|
||||
You don't need to retrain to use the model — the trained adapters are published at
|
||||
[`Chrisyichuan/wiki-screenshot-embedding-lora`](https://huggingface.co/Chrisyichuan/wiki-screenshot-embedding-lora/tree/main/lora_vit/ckpt200).
|
||||
使用模型无需自行重新训练 — 训练好的适配器已发布在 [`Chrisyichuan/wiki-screenshot-embedding-lora`](https://huggingface.co/Chrisyichuan/wiki-screenshot-embedding-lora/tree/main/lora_vit/ckpt200).
|
||||
|
||||
We also release the full training set
|
||||
([`Chrisyichuan/screenshot-training-natural-filtered-v2`](https://huggingface.co/datasets/Chrisyichuan/screenshot-training-natural-filtered-v2)),
|
||||
so you can adapt other backbones yourself — a larger Qwen, or any other embedding model.
|
||||
The data curation pipeline (LLM-augmented query generation, filtering, hard-negative mining)
|
||||
is documented in [`train/docs/synthetic_data_pipeline.md`](train/docs/synthetic_data_pipeline.md).
|
||||
我们还发布了完整训练集([`Chrisyichuan/screenshot-training-natural-filtered-v2`](https://huggingface.co/datasets/Chrisyichuan/screenshot-training-natural-filtered-v2)),),方便你自行适配其他骨干模型 — 例如更大的 Qwen,或任意其他嵌入模型。数据整理流水线(LLM 增强的查询生成、过滤、困难负样本挖掘)详见 [`train/docs/synthetic_data_pipeline.md`](train/docs/synthetic_data_pipeline.md)。
|
||||
|
||||
## Citation
|
||||
## 引用
|
||||
|
||||
If you find PixelRAG useful, please cite our paper:
|
||||
如果你觉得 PixelRAG 有用,请引用我们的论文:
|
||||
|
||||
```bibtex
|
||||
@misc{wang2026pixelragwebscreenshotsbeat,
|
||||
@@ -280,17 +258,17 @@ If you find PixelRAG useful, please cite our paper:
|
||||
}
|
||||
```
|
||||
|
||||
## Acknowledgments
|
||||
## 致谢
|
||||
|
||||
Thanks to [Rulin Shao](https://rulinshao.github.io/) for support.
|
||||
感谢 [Rulin Shao](https://rulinshao.github.io/) 的支持。
|
||||
|
||||
Thanks also to [Claude Code](https://github.com/anthropics/claude-code) and
|
||||
[OpenAI Codex](https://github.com/openai/codex) for supporting open-source contributors with credits and plans,
|
||||
which we earned by working on [LEANN](https://github.com/StarTrail-org/LEANN).
|
||||
同时感谢 [Claude Code](https://github.com/anthropics/claude-code) 和
|
||||
[OpenAI Codex](https://github.com/openai/codex) 为开源贡献者提供积分与方案支持,
|
||||
这些支持是我们在参与 [LEANN](https://github.com/StarTrail-org/LEANN). 工作期间获得的。
|
||||
|
||||
This work is done by the [Berkeley Sky Computing Lab](https://sky.cs.berkeley.edu/),
|
||||
[BAIR](https://bair.berkeley.edu/), and the [Berkeley NLP Group](https://nlp.cs.berkeley.edu/).
|
||||
本工作由 [Berkeley Sky Computing Lab](https://sky.cs.berkeley.edu/),
|
||||
[BAIR](https://bair.berkeley.edu/), 与 [Berkeley NLP Group](https://nlp.cs.berkeley.edu/). 完成。
|
||||
|
||||
## License
|
||||
## 许可证
|
||||
|
||||
Apache-2.0
|
||||
|
||||
Reference in New Issue
Block a user