docs: make Chinese README the default
/ test (push) Failing after 1s
/ build-and-push-to-ghcr (push) Has been skipped

This commit is contained in:
wehub-resource-sync
2026-07-13 10:13:56 +00:00
parent f0db2f4b22
commit ec513baf7b
+138 -132
View File
@@ -1,80 +1,86 @@
<!-- WEHUB_ZH_README -->
> [!NOTE]
> 本文档由 WeHub 基于上游 README 翻译整理,属于社区翻译,非官方中文文档。
> [English](./README.en.md) · [原始项目](https://github.com/jina-ai/reader) · [上游 README](https://github.com/jina-ai/reader/blob/HEAD/README.md)
> 原作者、版权与许可证归属以原始项目及本仓库 LICENSE 文件为准。
# Reader
[![codecov](https://codecov.io/gh/jina-ai/reader/branch/main/graph/badge.svg)](https://codecov.io/gh/jina-ai/reader)
[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/jina-ai/reader)
Your LLMs deserve better input.
你的 LLM 值得获得更好的输入。
Reader does two things:
- **Read**: It converts any URL to an **LLM-friendly** input with `https://r.jina.ai/https://your.url`. Get improved output for your agent and RAG systems at no cost.
- **Search**: It searches the web for a given query with `https://s.jina.ai/your+query`. This allows your LLMs to access the latest world knowledge from the web.
Reader 提供两项能力:
- **Read(读取)**:它将任意 URL 转换为适合 **LLM** 使用的输入,采用 `https://r.jina.ai/https://your.url`。无需额外成本,即可为你的智能体(agent)和 RAG 系统获得更优的输出。
- **Search(搜索)**:它使用 `https://s.jina.ai/your+query` 针对给定查询搜索网络。这使你的 LLM 能够访问来自网络的最新世界知识。
Check out [the live demo](https://jina.ai/reader#demo)
请查看[在线演示](https://jina.ai/reader#demo)
Or just visit these URLs (**Read**) https://r.jina.ai/https://github.com/jina-ai/reader, (**Search**) https://s.jina.ai/Who%20will%20win%202024%20US%20presidential%20election%3F and see yourself.
或直接访问这些 URL**Read**https://r.jina.ai/https://github.com/jina-ai/reader,**Search**https://s.jina.ai/Who%20will%20win%202024%20US%20presidential%20election%3F 亲自体验。
> Feel free to use Reader API in production. It is free, stable and scalable. We are maintaining it actively as one of the core products of Jina AI. [Check out rate limit](https://jina.ai/reader#pricing)
> 欢迎在生产环境中使用 Reader API。它免费、稳定且可扩展。我们将其作为 Jina AI 的核心产品之一积极维护。[查看速率限制](https://jina.ai/reader#pricing)
<img width="973" alt="image" src="https://github.com/jina-ai/reader/assets/2041322/2067c7a2-c12e-4465-b107-9a16ca178d41">
<img width="973" alt="image" src="https://github.com/jina-ai/reader/assets/2041322/675ac203-f246-41c2-b094-76318240159f">
> This repository is the open source branch of the codebase behind `https://r.jina.ai` and `https://s.jina.ai`. It runs in stateless or bucket-cached mode; the MongoDB-backed SaaS storage layer is not included here.
> 本仓库是 `https://r.jina.ai` `https://s.jina.ai` 背后代码库的开源分支。它以无状态或 bucket 缓存模式运行;此处不包含基于 MongoDB 的 SaaS 存储层。
## Updates
## 更新
- **2026-04** — Re-synchronized the open source branch with the SaaS code. The MongoDB-backed storage layer is stripped; the oss branch runs in stateless mode out of the box, with optional MinIO/S3-compatible bucket caching via `docker compose`. See [Local development](#local-development).
- **2025-12** — Storage layer decoupled and binary file uploads landed. PDFs and MS Office documents (Word, Excel, PowerPoint) can now be POSTed directly via the `file` body field — no need to host them first. See [cookbooks.md](./cookbooks.md#pdf-ms-office-and-raw-html-uploads).
- **2025-03** — Major refactor: Reader is no longer a Firebase application. The SaaS migrated off Firestore + Cloud Functions to a Cloud Run image with MongoDB Atlas, removing the platform-coupled bits and unblocking the local-Docker path above.
- **2024-05** — `s.jina.ai` launched, extending Reader from URL→markdown to search→markdown. PDFs added the same month — any URL ending in `.pdf` is parsed with PDF.js and returned as markdown.
- **2024-04** — Reader released and `r.jina.ai` went live as Jina AI's first SaaS API for converting URLs to LLM-friendly input.
- **2026-04** — 已将开源分支与 SaaS 代码重新同步。基于 MongoDB 的存储层已剥离;oss 分支开箱即可以无状态模式运行,并可通过 `docker compose` 可选启用 MinIO/S3 兼容的 bucket 缓存。参见[本地开发](#local-development)
- **2025-12** — 存储层解耦,并支持二进制文件上传。PDF 及 MS Office 文档(WordExcelPowerPoint)现可直接通过 `file` 请求体字段 POST 上传——无需事先托管。参见 [cookbooks.md](./cookbooks.md#pdf-ms-office-and-raw-html-uploads)
- **2025-03** — 重大重构:Reader 不再是 Firebase 应用。SaaS 已从 Firestore + Cloud Functions 迁移至搭载 MongoDB Atlas 的 Cloud Run 镜像,移除了与平台耦合的部分,并打通了上文所述的本地 Docker 路径。
- **2024-05** — `s.jina.ai` 上线,将 Reader URL→markdown 扩展至 search→markdown。同月新增 PDF 支持——任何以 `.pdf` 结尾的 URL 均使用 PDF.js 解析并返回为 markdown
- **2024-04** — Reader 发布,`r.jina.ai` 作为 Jina AI 首个将 URL 转换为 LLM 友好输入的 SaaS API 正式上线。
## What Reader can read
## Reader 可读取的内容
- **Web pages** — rendered with headless Chrome, or fetched lightweight via `curl-impersonate`. Reader picks intelligently between the two.
- **PDFs** — any URL, parsed with PDF.js. [See this NASA PDF result](https://r.jina.ai/https://www.nasa.gov/wp-content/uploads/2023/01/55583main_vision_space_exploration2.pdf) vs [the original](https://www.nasa.gov/wp-content/uploads/2023/01/55583main_vision_space_exploration2.pdf).
- **MS Office documents** — Word, Excel, PowerPoint, converted via LibreOffice and then processed as HTML/PDF.
- **Images** — captioned by a vision-language model, so your downstream text-only LLM gets *just enough* hints to reason about them.
- **网页** — 通过无头 Chrome 渲染,或通过 `curl-impersonate` 轻量抓取。Reader 会智能地在两者之间选择。
- **PDF** — 任意 URL,使用 PDF.js 解析。[查看此 NASA PDF 结果](https://r.jina.ai/https://www.nasa.gov/wp-content/uploads/2023/01/55583main_vision_space_exploration2.pdf) 与[原始文件](https://www.nasa.gov/wp-content/uploads/2023/01/55583main_vision_space_exploration2.pdf).
- **MS Office 文档** — WordExcelPowerPoint,经 LibreOffice 转换后按 HTML/PDF 处理。
- **图片** — 由视觉-语言模型(vision-language model)生成说明文字,使你的下游纯文本 LLM 获得*恰到好处*的提示以进行推理。
## Usage
## 用法
### Using `r.jina.ai` for single URL fetching
Simply prepend `https://r.jina.ai/` to any URL. For example, to convert the URL `https://en.wikipedia.org/wiki/Artificial_intelligence` to an LLM-friendly input, use the following URL:
### 使用 `r.jina.ai` 抓取单个 URL
只需在任意 URL 前添加 `https://r.jina.ai/` 前缀。例如,要将 URL `https://en.wikipedia.org/wiki/Artificial_intelligence` 转换为 LLM 友好输入,可使用以下 URL
[https://r.jina.ai/https://en.wikipedia.org/wiki/Artificial_intelligence](https://r.jina.ai/https://en.wikipedia.org/wiki/Artificial_intelligence)
### [Using `r.jina.ai` for a full website fetching (Google Colab)](https://colab.research.google.com/drive/1uoBy6_7BhxqpFQ45vuhgDDDGwstaCt4P#scrollTo=5LQjzJiT9ewT)
### [使用 `r.jina.ai` 抓取完整网站(Google Colab](https://colab.research.google.com/drive/1uoBy6_7BhxqpFQ45vuhgDDDGwstaCt4P#scrollTo=5LQjzJiT9ewT)
### Using `s.jina.ai` for web search
Simply prepend `https://s.jina.ai/` to your search query. Note that if you are using this in the code, make sure to encode your search query first, e.g. if your query is `Who will win 2024 US presidential election?` then your url should look like:
### 使用 `s.jina.ai` 进行网络搜索
只需在搜索查询前添加 `https://s.jina.ai/` 前缀。注意,若在代码中使用,请先对搜索查询进行编码,例如若查询为 `Who will win 2024 US presidential election?`,则 URL 应如下所示:
[https://s.jina.ai/Who%20will%20win%202024%20US%20presidential%20election%3F](https://s.jina.ai/Who%20will%20win%202024%20US%20presidential%20election%3F)
Behind the scenes, Reader searches the web, fetches the top 5 results, visits each URL, and applies `r.jina.ai` to it. This is different from many `web search function-calling` in agent/RAG frameworks, which often return only the title, URL, and description provided by the search engine API. If you want to read one result more deeply, you have to fetch the content yourself from that URL. With Reader, `http://s.jina.ai` automatically fetches the content from the top 5 search result URLs for you (reusing the tech stack behind `http://r.jina.ai`). This means you don't have to handle browser rendering, blocking, or any issues related to JavaScript and CSS yourself.
在幕后,Reader 会搜索网络、获取前 5 条结果、访问每个 URL,并对其应用 `r.jina.ai`。这与许多智能体/RAG 框架中的 `web search function-calling` 不同——后者通常只返回搜索引擎 API 提供的标题、URL 和描述。若要对某条结果做更深入阅读,你必须自行从该 URL 抓取内容。使用 Reader 时,`http://s.jina.ai` 会自动为你从前 5 条搜索结果 URL 抓取内容(复用 `http://r.jina.ai` 背后的技术栈)。这意味着你无需自行处理浏览器渲染、拦截,或与 JavaScript CSS 相关的任何问题。
### Using `s.jina.ai` for in-site search
Simply specify `site` in the query parameters such as:
### 使用 `s.jina.ai` 进行站内搜索
只需在查询参数中指定 `site`,例如:
```bash
curl 'https://s.jina.ai/When%20was%20Jina%20AI%20founded%3F?site=jina.ai&site=github.com'
```
### [Interactive Code Snippet Builder](https://jina.ai/reader#apiform)
### [交互式代码片段构建器](https://jina.ai/reader#apiform)
We highly recommend using the code builder to explore different parameter combinations of the Reader API.
我们强烈建议使用代码构建器探索 Reader API 的不同参数组合。
<a href="https://jina.ai/reader#apiform"><img width="973" alt="image" src="https://github.com/jina-ai/reader/assets/2041322/a490fd3a-1c4c-4a3f-a95a-c481c2a8cc8f"></a>
### Using request headers
### 使用请求头
You can control the behavior of the Reader API using request headers. The list below covers the most useful ones — for the full surface with up-to-date defaults and validation rules, see the live API docs at [https://r.jina.ai/docs](https://r.jina.ai/docs), or the source of truth in [`src/dto/crawler-options.ts`](./src/dto/crawler-options.ts).
你可以通过请求头控制 Reader API 的行为。下列列表涵盖最实用的选项——完整接口及最新默认值与校验规则,请参见 [https://r.jina.ai/docs](https://r.jina.ai/docs), 上的在线 API 文档,或 [`src/dto/crawler-options.ts`](./src/dto/crawler-options.ts) 中的权威源码。
- `x-respond-with`select the output format.
- `markdown` returns markdown *without* going through `readability`
- `html` returns `documentElement.outerHTML`
- `text` returns `document.body.innerText`
- `screenshot` returns the URL of the webpage's screenshot
- `pageshot` similar to `screenshot` but tries to capture the whole page instead of just the viewport
- `frontmatter` returns **Markdown with a YAML frontmatter block**. The default plain-text response uses a custom `Title: …` / `URL Source: …` header format; `frontmatter` replaces that with a front matter block. Example:
- `x-respond-with`选择输出格式。
- `markdown` 返回 markdown*不*经过 `readability`
- `html` 返回 `documentElement.outerHTML`
- `text` 返回 `document.body.innerText`
- `screenshot` 返回网页截图的 URL
- `pageshot` 类似于 `screenshot`,但会尝试截取整页而非仅视口
- `frontmatter` 返回**带 YAML frontmatter 块的 Markdown**。默认纯文本响应使用自定义的 `Title: …` / `URL Source: …` 头格式;`frontmatter` 会将其替换为 front matter 块。示例:
```bash
curl -H 'X-Respond-With: frontmatter' 'https://r.jina.ai/https://example.com'
@@ -92,67 +98,67 @@ You can control the behavior of the Reader API using request headers. The list b
This domain is for use in illustrative examples in documents. ...
```
- `markdown+frontmatter` — like `frontmatter` but covers the full page without readability filtering.
- `x-engine` — enforces a fetching engine: `browser` (headless Chrome), `curl` (lightweight, no JS), or `auto` (the default — Combined use of both browser and curl).
- `x-proxy-url` — route the traffic through your designated proxy.
- `x-cache-tolerance` — integer seconds; how stale a cached page is acceptable.
- `x-no-cache: true` — bypass the cached page (lifetime 3600s). Equivalent to `x-cache-tolerance: 0`.
- `x-target-selector` — a CSS selector. Reader returns content within the matched element instead of the full page. Useful when automatic content extraction misses what you want.
- `x-wait-for-selector` — a CSS selector. Reader waits until the matched element is rendered before returning. If `x-target-selector` is set, this can be omitted to wait for the same element.
- `x-timeout` — integer seconds (max 180). When set, Reader will not return early; it waits for network idle or until the timeout is reached.
- `x-max-tokens` — integer (≥500). Trim the response so it never exceeds this many tokens. Useful as a per-request guardrail when feeding a fixed-size context window — Reader truncates rather than rejects.
- `x-token-budget` — integer. Reject the request if the resulting content would exceed this many tokens. Use this when *over*-budget output is worse than no output (e.g. cost control). Ignored on the search endpoint.
- `x-respond-timing` — explicit control over *when* Reader is willing to return. Trade off latency against completeness:
- `html` — return as soon as the raw HTML lands. No JS execution, no waiting.
- `visible-content` — return the moment readable content is parseable. Lowest latency that still produces text.
- `mutation-idle` — wait for DOM mutations to settle for ≥0.2s. Good default for SPAs that lazy-render above the fold.
- `resource-idle` — wait for content-affecting resources to finish loading (≥0.5s quiet). The default heuristic for content-shaped requests.
- `media-idle` — wait for media (images, video, fonts) to also finish. Use with `screenshot` / `pageshot` / `vlm`.
- `network-idle` — full `networkidle0`. Slowest, most complete. Implied when `x-timeout` ≥ 20.
- `markdown+frontmatter` — 类似 `frontmatter`,但覆盖整页且不做可读性过滤。
- `x-engine` — 强制指定抓取引擎:`browser`(无头 Chrome)、`curl`(轻量,无 JS),或 `auto`(默认——浏览器与 curl 组合使用)。
- `x-proxy-url` — 通过你指定的代理路由流量。
- `x-cache-tolerance` — 整数秒;可接受的缓存页面陈旧程度。
- `x-no-cache: true` — 绕过缓存页面(生命周期 3600s)。等效于 `x-cache-tolerance: 0`
- `x-target-selector` — CSS 选择器。Reader 返回匹配元素内的内容,而非整页。当自动内容提取遗漏了你所需内容时很有用。
- `x-wait-for-selector` — CSS 选择器。Reader 会等待匹配元素渲染完成后再返回。若已设置 `x-target-selector`,可省略此项以等待同一元素。
- `x-timeout` — 整数秒(最大 180)。设置后 Reader 不会提前返回;会等待网络空闲或直至超时。
- `x-max-tokens` — 整数(≥500)。裁剪响应,使其不超过该 token 数。在向固定大小的上下文窗口喂入数据时,可作为按请求的防护栏——Reader 会截断而非拒绝。
- `x-token-budget` — 整数。若结果内容将超过该 token 数则拒绝请求。当*超出*预算的输出比无输出更糟时使用(例如成本控制)。在搜索端点上会被忽略。
- `x-respond-timing` — 显式控制 Reader *何时*愿意返回。在延迟与完整性之间权衡:
- `html` — 原始 HTML 一到达即返回。不执行 JS,不等待。
- `visible-content` — 可读内容可解析的瞬间即返回。仍能产出文本的最低延迟。
- `mutation-idle` — 等待 DOM 变更稳定 ≥0.2s。适合首屏懒渲染的 SPA 的良好默认值。
- `resource-idle` — 等待影响内容的资源加载完成(安静 ≥0.5s)。面向内容形态请求的默认启发式策略。
- `media-idle` — 同时等待媒体(图片、视频、字体)加载完成。可与 `screenshot` / `pageshot` / `vlm` 配合使用。
- `network-idle` — 完整 `networkidle0`。最慢,最完整。当 `x-timeout` ≥ 20 时隐含启用。
When omitted, Reader picks one based on `x-respond-with`, `x-timeout`, and `x-with-iframe`. See `presumedRespondTiming` in [src/dto/crawler-options.ts](./src/dto/crawler-options.ts) for the exact rules.
- `x-with-generated-alt: true` — caption images on the page with a VLM.
- `x-retain-images` — control how images survive into the output:
- `all` (default) — keep `![alt](url)` markdown for every image.
- `none` — drop images entirely.
- `alt` — keep alt text only, no URLs. Cheap on tokens; useful when the downstream LLM has no use for the image link.
- `x-retain-links` — control how links survive into the output:
- `all` (default) — keep `[text](url)` markdown.
- `none` — drop links entirely.
- `text` — keep link anchor text only, drop URLs. Best for embedding / semantic-index pipelines where URLs are noise.
- `gpt-oss` — emit citations in gpt-oss's `【{id}†...】` format and append a numbered URL footer (also auto-enables `x-with-links-summary`).
- `x-retain-media` — control how `<video>`, `<audio>`, and embedded video iframes (`<iframe>` from YouTube, Vimeo, Bilibili, etc.) appear in the output:
- `link` (default) — markdown link, e.g. `[Video 1](url)`. Embedded iframes are rewritten to their canonical watch URL. Respects `x-md-link-style`.
- `none` — drop media entirely; non-video iframes fall back to their inner text content.
- `text` — bare label only, e.g. `Video 1` or `Audio 1`. No URL.
- `image` — markdown image syntax, e.g. `![Video 1](url)`.
- `html` — the original HTML element with cosmetic attributes (`class`, `id`, `style`, `data-*`, `aria-*`) stripped. Embedded video iframes keep their original embed `src` rather than the canonical watch URL.
- `x-with-links-summary` / `x-with-images-summary` — append a deduplicated footer of all links / images to the output. Combine with `x-retain-links: text` or `x-retain-images: alt` to get inline anchor/alt text plus *one* canonical URL list at the end — convenient when you want the model to see URLs without paying for them inline. `x-with-links-summary: all` keeps every link instead of only the unique ones.
- `x-markdown-chunking` — opt-in semantic chunking of the markdown response. Returns a JSON array (or ``-delimited text) of chunks instead of one blob:
- `true` / `h1` … `h5` — heading-based split at the given heading level (e.g. `h3` chunks at `#`, `##`, and `###`).
- `structured` / `s1` … `s5` — block-level structured split. `s1` is coarsest, `s5` finest.
- `x-preset` — apply a pre-packaged option bundle for common scenarios. Preset values only take effect for options the caller does *not* set explicitly (via body or another header). See [cookbooks.md](./cookbooks.md#using-presets) for examples.
- `reader` — for displaying content to human users.
- `index` — for semantic indexing / embedding pipelines.
- `research` — for AI research agents needing structured, citable output.
- `agent` — for AI agents doing everyday browsing tasks.
- `spider` — for recursive site crawling with a full link inventory.
- `x-detach-invisibles` — detach elements with eventual `display:none` before snapshotting. Implies browser engine; disables caching.
- `x-set-cookie` — forward cookie settings. Requests with cookies are not cached.
- `x-md-*` — fine-tune markdown output (heading style, bullet markers, link style, etc.). See [src/dto/turndown-tweakable-options.ts](./src/dto/turndown-tweakable-options.ts).
省略时,Reader 会根据 `x-respond-with``x-timeout` `x-with-iframe` 选择其一。具体规则请参阅 [src/dto/crawler-options.ts](./src/dto/crawler-options.ts) 中的 `presumedRespondTiming`。
- `x-with-generated-alt: true` — 使用 VLM(视觉语言模型)为页面上的图片生成说明文字。
- `x-retain-images` — 控制图片如何保留在输出中:
- `all`(默认)— 为每张图片保留 `![alt](url)` markdown
- `none` — 完全丢弃图片。
- `alt` — 仅保留 alt 文本,不含 URL。节省 token;当下游 LLM 不需要图片链接时很有用。
- `x-retain-links` — 控制链接如何保留在输出中:
- `all`(默认)— 保留 `[text](url)` markdown
- `none` — 完全丢弃链接。
- `text` — 仅保留链接锚文本,丢弃 URL。最适合 embedding / 语义索引(semantic indexing)流水线,其中 URL 是噪声。
- `gpt-oss` — gpt-oss `【{id}†...】` 格式输出引用,并附加带编号的 URL 页脚(同时自动启用 `x-with-links-summary`)。
- `x-retain-media` — 控制 `<video>``<audio>` 以及嵌入式视频 iframe(来自 YouTubeVimeoBilibili 等的 `<iframe>`)在输出中的呈现方式:
- `link`(默认)— markdown 链接,例如 `[Video 1](url)`。嵌入式 iframe 会重写为其规范观看 URL。遵守 `x-md-link-style`
- `none` — 完全丢弃媒体;非视频 iframe 回退为其内部文本内容。
- `text` — 仅纯标签,例如 `Video 1` `Audio 1`。不含 URL
- `image` — markdown 图片语法,例如 `![Video 1](url)`
- `html` — 保留原始 HTML 元素,但剥离装饰性属性(`class``id``style``data-*``aria-*`)。嵌入式视频 iframe 保留其原始嵌入 `src`,而非规范观看 URL
- `x-with-links-summary` / `x-with-images-summary` — 在输出末尾附加去重后的所有链接 / 图片页脚。与 `x-retain-links: text` `x-retain-images: alt` 组合使用,可获得行内锚文本/alt 文本,并在末尾附带*一份*规范 URL 列表——当你希望模型能看到 URL 又不想在行内为此付出 token 成本时非常方便。`x-with-links-summary: all` 会保留所有链接,而非仅保留唯一链接。
- `x-markdown-chunking` — 可选启用对 markdown 响应的语义分块。返回 JSON 数组(或以 `` 分隔的文本)形式的块,而非一整块内容:
- `true` / `h1` … `h5` — 按给定标题级别进行基于标题的分割(例如 `h3` 在 `#``##` `###` 处分块)。
- `structured` / `s1` … `s5` — 块级结构化分割。`s1` 最粗,`s5` 最细。
- `x-preset` — 为常见场景应用预置选项组合。预设值仅对调用方*未*显式设置(通过 body 或其他 header)的选项生效。示例请参阅 [cookbooks.md](./cookbooks.md#using-presets)
- `reader` — 面向向人类用户展示内容。
- `index` — 面向语义索引 / embedding 流水线。
- `research` — 面向需要结构化、可引用输出的 AI 研究代理。
- `agent` — 面向执行日常浏览任务的 AI 代理。
- `spider` — 面向递归站点爬取,并生成完整链接清单。
- `x-detach-invisibles` — 在快照前分离带有潜在 `display:none` 的元素。意味着使用浏览器引擎;禁用缓存。
- `x-set-cookie` — 转发 cookie 设置。带 cookie 的请求不会被缓存。
- `x-md-*` — 微调 markdown 输出(标题样式、列表标记、链接样式等)。请参阅 [src/dto/turndown-tweakable-options.ts](./src/dto/turndown-tweakable-options.ts)
### Using `r.jina.ai` for single page application (SPA) fetching
Many websites nowadays rely on JavaScript frameworks and client-side rendering, usually known as Single Page Applications (SPA). Thanks to [Puppeteer](https://github.com/puppeteer/puppeteer) and headless Chrome, Reader natively supports fetching these websites. However, due to specific approaches some SPAs are developed with, there may be some extra precautions to take.
### 使用 `r.jina.ai` 抓取单页应用(Single Page ApplicationSPA
如今许多网站依赖 JavaScript 框架和客户端渲染,通常称为单页应用(Single Page ApplicationSPA)。得益于 [Puppeteer](https://github.com/puppeteer/puppeteer) 和无头 Chromeheadless Chrome),Reader 原生支持抓取这些网站。不过,由于部分 SPA 的特定开发方式,可能需要采取一些额外预防措施。
#### SPAs with hash-based routing
By definition of the web standards, content after `#` in a URL is not sent to the server. To mitigate this, use `POST` with the `url` parameter in the body:
#### 基于 hash 路由的 SPA
根据 Web 标准定义,URL 中 `#` 之后的内容不会发送到服务器。为缓解此问题,请在 body 中使用 `POST` 并配合 `url` 参数:
```bash
curl -X POST 'https://r.jina.ai/' -d 'url=https://example.com/#/route'
```
#### SPAs with preloading contents
Some SPAs (and even some non-SPAs) show preload content before later loading the main content dynamically. In this case, Reader may capture the preload content instead. Two ways to mitigate:
#### 带预加载内容的 SPA
某些 SPA(甚至有些非 SPA)会先显示预加载内容,之后再动态加载主要内容。此时 Reader 可能会抓取到预加载内容。有两种缓解方式:
```bash
# wait for network idle or until timeout
@@ -165,17 +171,17 @@ curl 'https://r.jina.ai/https://example.com/' -H 'x-wait-for-selector: #content'
curl 'https://r.jina.ai/https://example.com/' -H 'x-timeout: 30' -H 'x-wait-for-selector: non-existent-element'
```
### JSON mode
### JSON 模式
Use the accept-header to control the output format:
使用 accept-header 控制输出格式:
```bash
curl -H "Accept: application/json" https://r.jina.ai/https://en.m.wikipedia.org/wiki/Main_Page
```
### Generated alt
### 自动生成 alt
All images on a page that lack an `alt` tag can be auto-captioned by a VLM (vision-language model) and formatted as `![Image [idx]: [VLM_caption]](img_URL)`. This should give your downstream text-only LLM *just enough* hints to include those images in reasoning, selection, and summarization:
页面上所有缺少 `alt` 标签的图片,都可由 VLM(视觉语言模型,vision-language model)自动生成说明文字,并格式化为 `![Image [idx]: [VLM_caption]](img_URL)`。这应能为下游纯文本 LLM 提供*足够*的线索,使其在推理、筛选和摘要中纳入这些图片:
```bash
curl -H "X-With-Generated-Alt: true" https://r.jina.ai/https://en.m.wikipedia.org/wiki/Main_Page
@@ -183,41 +189,41 @@ curl -H "X-With-Generated-Alt: true" https://r.jina.ai/https://en.m.wikipedia.or
## Cookbooks
For pipeline-specific recipes — RAG, semantic indexing, deep research, agentic browsing, visual snapshots, PDF/Office/HTML uploads, and more — see [cookbooks.md](./cookbooks.md). Each entry is a short curl example with the header combination that fits the use case and a paragraph explaining the trade-offs.
针对各类流水线场景的配方 — RAG、语义索引、深度研究、代理式浏览、可视化快照、PDF/Office/HTML 上传等 — 请参阅 [cookbooks.md](./cookbooks.md)。每条目都是一个简短的 curl 示例,包含适用于该场景的 header 组合,以及一段说明权衡取舍的文字。
## Self-host with Docker
## 使用 Docker 自托管
A prebuilt image of the open-source branch is published to GitHub Container Registry. It bundles headless Chrome, LibreOffice, and CJK fonts, so you can run Reader without building it yourself.
开源分支的预构建镜像已发布到 GitHub Container Registry。它捆绑了 headless ChromeLibreOffice 和 CJK 字体,因此你可以无需自行构建即可运行 Reader。
```bash
docker pull ghcr.io/jina-ai/reader:oss
```
### Run
### 运行
The image exposes two ports:
该镜像暴露两个端口:
- `8080` — **h2c** (HTTP/2 cleartext). Production-grade, multiplexed; this is what Cloud Run talks to. Plain `curl` won't speak it without `--http2-prior-knowledge`.
- `8081` — **HTTP/1.1** fallback. Same handler, same routes; use this from anything that doesn't speak h2c.
- `8080` — **h2c**HTTP/2 cleartext,HTTP/2 明文)。生产级、多路复用;Cloud Run 即通过此端口通信。普通 `curl` 若无 `--http2-prior-knowledge` 将无法使用它。
- `8081` — **HTTP/1.1** 回退。同一处理器、同一路由;从不支持 h2c 的客户端使用此端口。
For a quick try-out from `curl` or a browser, map the HTTP/1.1 port:
若要从 `curl` 或浏览器快速试用,请映射 HTTP/1.1 端口:
```bash
docker run --rm -p 3000:8081 ghcr.io/jina-ai/reader:oss
# then: curl http://localhost:3000/https://example.com
```
For load-testing or production-shape traffic, map the h2c port instead (or both):
若要进行负载测试或模拟生产流量,请改映射 h2c 端口(或同时映射两者):
```bash
docker run --rm -p 3000:8080 -p 3001:8081 ghcr.io/jina-ai/reader:oss
```
With no extra config the container is fully stateless — every request hits the live URL, no cache, no rate limiting. That's the right default for a quick try-out, CI, or throwaway environments.
无需额外配置时,容器完全无状态 — 每个请求都访问实时 URL,无缓存、无限流。这对快速试用、CI 或一次性环境是合适的默认设置。
### Run with caching
### 带缓存运行
Point Reader at an S3-compatible bucket to cache fetched pages and reuse them across requests:
Reader 指向 S3 兼容存储桶,以缓存已抓取的页面并在请求间复用:
```bash
docker run --rm -p 3000:8081 \
@@ -228,13 +234,13 @@ docker run --rm -p 3000:8081 \
ghcr.io/jina-ai/reader:oss
```
See [CONTRIBUTING.md](./CONTRIBUTING.md#environment-variables) for the full env-var table.
完整环境变量表请参阅 [CONTRIBUTING.md](./CONTRIBUTING.md#environment-variables)
## Local development
## 本地开发
Requirements:
要求:
- nvm use
- Docker *(optional — only if you want a local MinIO bucket cache)*
- Docker *(可选 — 仅当你需要本地 MinIO 存储桶缓存时)*
```bash
git clone git@github.com:jina-ai/reader.git
@@ -244,45 +250,45 @@ npm install
docker compose up -d
```
Then either press `F5` in VSCode to launch the debugger, or after setting up the appropriate environment variables:
然后在 VSCode 中按 `F5` 启动调试器,或在设置好相应环境变量后:
```bash
npm run dev
```
For a deeper tour of the codebase — engines, formatting profiles, abuse alleviation, deployment topology — see [architecture.md](./architecture.md). For dev workflow, env vars, and tests, see [CONTRIBUTING.md](./CONTRIBUTING.md).
若要更深入地了解代码库 — 引擎、格式化配置、滥用缓解、部署拓扑等 — 请参阅 [architecture.md](./architecture.md)。开发工作流、环境变量和测试请参阅 [CONTRIBUTING.md](./CONTRIBUTING.md)
### Licensed assets
### 许可资产
A few non-redistributable artifacts live in `licensed/` and are needed at build/runtime:
少量不可再分发的构件位于 `licensed/`,在构建/运行时需要:
- `GeoLite2-City.mmdb` and `geolite2-asn.mmdb` — MaxMind GeoLite databases (geolocation + ASN lookups).
- `SourceHanSansSC-Regular.otf` — Source Han Sans (CJK rendering for PDFs/screenshots).
- `gsa_useragents.txt` — user-agent list used by the curl engine.
- `GeoLite2-City.mmdb` `geolite2-asn.mmdb` — MaxMind GeoLite 数据库(地理定位 + ASN 查询)。
- `SourceHanSansSC-Regular.otf` — Source Han Sans(用于 PDF/截图的 CJK 渲染)。
- `gsa_useragents.txt` — curl 引擎使用的 user-agent 列表。
Fetch them in one shot:
一次性获取:
```bash
npm run assets:download
```
The script (`download-external-assets.sh`) is idempotent — it skips files already present and exits 0 even on partial network failure. Set `FORCE_DOWNLOAD_EXTERNAL=1` to overwrite, or `SKIP_DOWNLOAD_EXTERNAL=1` to bypass entirely if you supply your own copies. The repo's CI fetches the same URLs inline; this script exists for local convenience.
脚本(`download-external-assets.sh`)具有幂等性 — 会跳过已存在的文件,即使出现部分网络失败也会以退出码 0 结束。设置 `FORCE_DOWNLOAD_EXTERNAL=1` 可覆盖;若你自备副本,可设置 `SKIP_DOWNLOAD_EXTERNAL=1` 完全跳过。仓库的 CI 会内联获取相同 URL;此脚本仅为本地便利而设。
## How it works
## 工作原理
[![Ask DeepWiki](https://deepwiki.com/badge.svg)](https://deepwiki.com/jina-ai/reader)
## Having trouble on some websites?
## 某些网站遇到问题?
Some sites push back against scrapers — bot challenges, geo blocks, stale CDN edges. A few knobs to try, in roughly increasing order of "this is bothering me":
部分网站会抵制爬虫 — bot 挑战、地域封锁、过时的 CDN 边缘节点。可按大致递增的「这开始让我头疼」顺序尝试以下选项:
- **Use an API key.** Anonymous traffic is the most aggressively rate-limited and lands in the lowest-trust pool. Authenticated requests get a higher quota and access to features like the internal proxy. Get one at [jina.ai/reader](https://jina.ai/reader#pricing).
- **Bypass the cache** with `-H 'x-no-cache: true'`. If a stale or already-blocked response got cached, this forces a fresh fetch.
- **Force the browser engine** with `-H 'x-engine: browser'`. The default `auto` engine prefers the lightweight curl path when it can; some sites only serve real content to a JS-capable browser.
- **Route through the SaaS proxy** with `-H 'x-proxy: auto'` (key required). Reader's hosted proxy pool rotates residential / datacenter IPs and handles common anti-bot challenges automatically. You can also pin a country, e.g. `x-proxy: us` (see [Geo- and locale-sensitive scraping](./cookbooks.md#geo--and-locale-sensitive-scraping)).
- **Bring your own proxy** with `-H 'x-proxy-url: <url>'`. As a last resort — when even the hosted proxy can't get through — buy a residential or ISP-grade proxy from a third-party provider (BrightData, Thordata, Oxylabs, etc.) and pass the URL directly. Supports `http`, `https`, `socks4`, `socks5`; for auth use `https://user:pass@host:port`.
- **使用 API key** 匿名流量受到最严格的速率限制,且会落入最低信任池。经身份验证的请求可获得更高配额,并可使用内部代理等功能。前往 [jina.ai/reader](https://jina.ai/reader#pricing). 获取。
- **绕过缓存**:使用 `-H 'x-no-cache: true'`。若陈旧或已被封禁的响应已被缓存,此操作可强制重新获取。
- **强制使用浏览器引擎**:使用 `-H 'x-engine: browser'`。默认的 `auto` 引擎在可行时优先走轻量 curl 路径;部分网站仅向支持 JS 的浏览器提供真实内容。
- **通过 SaaS 代理路由**:使用 `-H 'x-proxy: auto'`(需要 key)。Reader 托管的代理池会轮换住宅/数据中心 IP,并自动处理常见的反 bot 挑战。也可固定国家/地区,例如 `x-proxy: us`(参见[地理与区域敏感爬取](./cookbooks.md#geo--and-locale-sensitive-scraping))。
- **自带代理**:使用 `-H 'x-proxy-url: <url>'`。作为最后手段 — 当托管代理也无法穿透时 — 从第三方供应商(BrightDataThordataOxylabs 等)购买住宅或 ISP 级代理,并直接传入 URL。支持 `http``https``socks4``socks5`;身份验证请使用 `https://user:pass@host:port`
If none of those help, please open an issue with the URL and the headers you tried — we'll take a look.
若以上均无效,请提交 issue,附上 URL 与你尝试过的 headers — 我们会排查。
## License
## 许可证
Reader is backed by [Jina AI](https://jina.ai) and licensed under [Apache-2.0](./LICENSE).
Reader [Jina AI](https://jina.ai) 提供支持,并根据 [Apache-2.0](./LICENSE) 授权。