docs: make Chinese README the default

This commit is contained in:
wehub-resource-sync
2026-07-13 09:50:49 +00:00
parent 83bc768741
commit 39fdd8baa4
+154 -141
View File
@@ -1,3 +1,9 @@
<!-- WEHUB_ZH_README -->
> [!NOTE]
> 本文档由 WeHub 基于上游 README 翻译整理,属于社区翻译,非官方中文文档。
> [English](./README.en.md) · [原始项目](https://github.com/D4Vinci/Scrapling) · [上游 README](https://github.com/D4Vinci/Scrapling/blob/HEAD/README.md)
> 原作者、版权与许可证归属以原始项目及本仓库 LICENSE 文件为准。
<!-- mcp-name: io.github.D4Vinci/Scrapling -->
<h1 align="center">
@@ -8,7 +14,7 @@
</picture>
</a>
<br>
<small>Effortless Web Scraping for the Modern Web</small>
<small>面向现代网页的轻松爬取</small>
</h1>
<p align="center">
@@ -38,24 +44,24 @@
</p>
<p align="center">
<a href="https://scrapling.readthedocs.io/en/latest/parsing/selection.html"><strong>Selection methods</strong></a>
<a href="https://scrapling.readthedocs.io/en/latest/parsing/selection.html"><strong>选择方法</strong></a>
&middot;
<a href="https://scrapling.readthedocs.io/en/latest/fetching/choosing.html"><strong>Fetchers</strong></a>
&middot;
<a href="https://scrapling.readthedocs.io/en/latest/spiders/architecture.html"><strong>Spiders</strong></a>
&middot;
<a href="https://scrapling.readthedocs.io/en/latest/spiders/proxy-blocking.html"><strong>Proxy Rotation</strong></a>
<a href="https://scrapling.readthedocs.io/en/latest/spiders/proxy-blocking.html"><strong>代理轮换</strong></a>
&middot;
<a href="https://scrapling.readthedocs.io/en/latest/cli/overview.html"><strong>CLI</strong></a>
&middot;
<a href="https://scrapling.readthedocs.io/en/latest/ai/mcp-server.html"><strong>MCP</strong></a>
</p>
Scrapling is an adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl.
Scrapling 是一款自适应 Web Scraping(网页爬取)框架,可处理从单次请求到大规模爬取的一切场景。
Its parser learns from website changes and automatically relocates your elements when pages update. Its fetchers bypass anti-bot systems like Cloudflare Turnstile out of the box. And its spider framework lets you scale up to concurrent, multi-session crawls with pause/resume and automatic proxy rotation - all in a few lines of Python. One library, zero compromises.
其解析器会学习网站变化,并在页面更新时自动重新定位你的元素。其 Fetcher 可开箱即用地绕过 Cloudflare Turnstile 等反机器人系统。其 Spider 框架让你只需几行 Python 代码,即可扩展到并发、多会话爬取,并支持暂停/恢复与自动代理轮换——一个库,零妥协。
Blazing fast crawls with real-time stats and streaming. Built by Web Scrapers for Web Scrapers and regular users, there's something for everyone.
极速爬取,实时统计与流式输出。由网页爬取从业者打造,兼顾专业用户与普通用户,人人适用。
```python
from scrapling.fetchers import Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher
@@ -64,7 +70,7 @@ p = StealthyFetcher.fetch('https://example.com', headless=True, network_idle=Tru
products = p.css('.product', auto_save=True) # Scrape data that survives website design changes!
products = p.css('.product', adaptive=True) # Later, if the website structure changes, pass `adaptive=True` to find them!
```
Or scale up to full crawls
或扩展到完整爬取
```python
from scrapling.spiders import Spider, Response
@@ -93,7 +99,7 @@ MySpider().start()
<img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/proxidize.png">
</a>
</td>
<td> <a href="https://proxidize.com/?utm_source=github&utm_medium=sponsorship&utm_campaign=scrapling&utm_content=d4vinci" target="_blank"><b>Proxidize</b></a> provides mobile and residential proxies for scraping, browser automation, SEO monitoring, AI agents, and data collection. <i>Use code <b>scrapling20</b> for 20% off</i>.
<td> <a href="https://proxidize.com/?utm_source=github&utm_medium=sponsorship&utm_campaign=scrapling&utm_content=d4vinci" target="_blank"><b>Proxidize</b></a> 提供移动与住宅代理,适用于爬取、浏览器自动化、SEO 监控、AI Agent 与数据采集。<i>使用优惠码 <b>scrapling20</b> 可享 8 折</i>
</td>
</tr>
<tr>
@@ -102,7 +108,7 @@ MySpider().start()
<img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/coldproxy.png">
</a>
</td>
<td> <a href="https://coldproxy.com/" target="_blank"><b>ColdProxy</b></a> provides residential and datacenter proxies for stable web scraping, public data collection, and geo-targeted testing across 195+ countries.
<td> <a href="https://coldproxy.com/" target="_blank"><b>ColdProxy</b></a> 提供住宅与数据中心代理,用于稳定的网页爬取、公开数据采集,以及覆盖 195+ 个国家的地理定向测试。
</td>
</tr>
<tr>
@@ -111,9 +117,9 @@ MySpider().start()
<img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/HyperSolutions.png">
</a>
</td>
<td> Scrapling handles Cloudflare Turnstile. For enterprise-grade protection, <a href="https://hypersolutions.co?utm_source=github&utm_medium=readme&utm_campaign=scrapling">
<td> Scrapling 可处理 Cloudflare Turnstile。针对企业级防护,<a href="https://hypersolutions.co?utm_source=github&utm_medium=readme&utm_campaign=scrapling">
<b>Hyper Solutions</b>
</a> provides API endpoints that generate valid antibot tokens for <b>Akamai</b>, <b>DataDome</b>, <b>Kasada</b>, and <b>Incapsula</b>. Simple API calls, no browser automation required. </td>
</a> 提供 API 端点,可为 <b>Akamai</b><b>DataDome</b><b>Kasada</b> <b>Incapsula</b> 生成有效的反机器人令牌。简单 API 调用,无需浏览器自动化。 </td>
</tr>
<tr>
<td width="200">
@@ -124,8 +130,8 @@ MySpider().start()
<td>
<a href="https://evomi.com?utm_source=github&utm_medium=banner&utm_campaign=d4vinci-scrapling">
<b>Evomi</b>
</a>: residential proxies from $0.49/GB. Scraping browser with fully spoofed Chromium, residential IPs, auto CAPTCHA solving, and anti-bot bypass. </br>
<b>Scraper API for hassle-free results. MCP and N8N integrations are available.</b>
</a>:住宅代理低至 $0.49/GB。爬取浏览器配备完整伪装的 Chromium、住宅 IP、自动 CAPTCHA 求解与反机器人绕过。 </br>
<b>提供 Scraper API,轻松获取结果。支持 MCP N8N 集成。</b>
</td>
</tr>
<tr>
@@ -135,7 +141,7 @@ MySpider().start()
</a>
</td>
<td>
<a href="https://tikhub.io/?utm_source=github.com/D4Vinci/Scrapling&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad" target="_blank">TikHub.io</a> provides 900+ stable APIs across 16+ platforms including TikTok, X, YouTube & Instagram, with 40M+ datasets. <br /> Also offers <a href="https://ai.tikhub.io/?ref=KarimShoair" target="_blank">DISCOUNTED AI models</a> - Claude, GPT, GEMINI & more up to 71% off.
<a href="https://tikhub.io/?utm_source=github.com/D4Vinci/Scrapling&utm_medium=marketing_social&utm_campaign=retargeting&utm_content=carousel_ad" target="_blank">TikHub.io</a> 提供 900+ 稳定 API,覆盖 TikTok、X、YouTube Instagram 等 16+ 平台,拥有 40M+ 数据集。<br /> 另提供<a href="https://ai.tikhub.io/?ref=KarimShoair" target="_blank">折扣 AI 模型</a>——ClaudeGPTGEMINI 等,最高可省 71%。
</td>
</tr>
<tr>
@@ -145,8 +151,8 @@ MySpider().start()
</a>
</td>
<td>
Close your laptop. Your scrapers keep running. <br />
<a href="https://petrosky.io/d4vinci" target="_blank">PetroSky VPS</a> - cloud servers built for nonstop automation. Windows and Linux machines with full control. From €6.99/mo.
合上笔记本,爬虫仍在运行。<br />
<a href="https://petrosky.io/d4vinci" target="_blank">PetroSky VPS</a>——专为不间断自动化打造的云服务器。提供 Windows Linux 机器,完全可控。起价 €6.99/月。
</td>
</tr>
<tr>
@@ -156,7 +162,7 @@ MySpider().start()
</a>
</td>
<td>
Read a full review of <a href="https://substack.thewebscraping.club/p/scrapling-hands-on-guide?utm_source=github&utm_medium=repo&utm_campaign=scrapling" target="_blank">Scrapling on The Web Scraping Club</a> (Nov 2025), the #1 newsletter dedicated to Web Scraping.
阅读 <a href="https://substack.thewebscraping.club/p/scrapling-hands-on-guide?utm_source=github&utm_medium=repo&utm_campaign=scrapling" target="_blank">The Web Scraping Club 对 Scrapling 的完整评测</a>2025 年 11 月),这是专注于 Web Scraping 的头号 Newsletter。
</td>
</tr>
<tr>
@@ -166,7 +172,7 @@ MySpider().start()
</a>
</td>
<td>
<a href="https://www.swiftproxy.net/?ref=D4Vinci" target="_blank">Swiftproxy</a> provides scalable residential proxies with 80M+ IPs across 195+ countries, delivering fast, reliable connections, automatic rotation, and strong anti-block performance. Free trial available.
<a href="https://www.swiftproxy.net/?ref=D4Vinci" target="_blank">Swiftproxy</a> 提供可扩展的住宅代理,拥有 80M+ IP,覆盖 195+ 个国家,连接快速可靠,支持自动轮换,具备出色的防封性能。提供免费试用。
</td>
</tr>
<tr>
@@ -176,81 +182,81 @@ MySpider().start()
</a>
</td>
<td>
<a href="https://go.nodemaven.com/scraplingjune" target="_blank">NodeMaven</a> - reliable proxy provider with the highest quality IP on the market. Use promo code SCRAPLING35 for 35% discount on proxies.
<a href="https://go.nodemaven.com/scraplingjune" target="_blank">NodeMaven</a>——可靠的代理提供商,拥有市场上质量最高的 IP。使用优惠码 SCRAPLING35 可享代理 35% 折扣。
</td>
</tr>
</table>
<i><sub>Do you want to show your ad here? Click [here](https://github.com/sponsors/D4Vinci/sponsorships?tier_id=586646)</sub></i>
# Sponsors
<i><sub>您想在此展示广告?点击[此处](https://github.com/sponsors/D4Vinci/sponsorships?tier_id=586646))</sub></i>
# 赞助商
<!-- sponsors -->
<a href="https://serpapi.com/?utm_source=scrapling" target="_blank" title="Scrape Google and other search engines with SerpApi"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/SerpApi.png"></a>
<a href="https://visit.decodo.com/Dy6W0b" target="_blank" title="Try the Most Efficient Residential Proxies for Free"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/decodo.png"></a>
<a href="https://hasdata.com/?utm_source=github&utm_medium=banner&utm_campaign=D4Vinci" target="_blank" title="The web scraping service that actually beats anti-bot systems!"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/hasdata.png"></a>
<a href="https://proxyempire.io/?ref=scrapling&utm_source=scrapling" target="_blank" title="Collect The Data Your Project Needs with the Best Residential Proxies"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/ProxyEmpire.png"></a>
<a href="https://www.webshare.io/?referral_code=48r2m2cd5uz1" target="_blank" title="The Most Reliable Proxy with Unparalleled Performance"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/webshare.png"></a>
<a href="https://proxiware.com/?ref=scrapling" target="_blank" title="Collect Any Data. At Any Scale."><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/proxiware.png"></a>
<a href="https://serpapi.com/?utm_source=scrapling" target="_blank" title="使用 SerpApi 抓取 Google 及其他搜索引擎"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/SerpApi.png"></a>
<a href="https://visit.decodo.com/Dy6W0b" target="_blank" title="免费试用最高效的住宅代理"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/decodo.png"></a>
<a href="https://hasdata.com/?utm_source=github&utm_medium=banner&utm_campaign=D4Vinci" target="_blank" title="真正能击败反爬系统的网页抓取服务!"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/hasdata.png"></a>
<a href="https://proxyempire.io/?ref=scrapling&utm_source=scrapling" target="_blank" title="用最好的住宅代理收集项目所需数据"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/ProxyEmpire.png"></a>
<a href="https://www.webshare.io/?referral_code=48r2m2cd5uz1" target="_blank" title="最可靠的代理,性能无与伦比"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/webshare.png"></a>
<a href="https://proxiware.com/?ref=scrapling" target="_blank" title="任意规模,收集任意数据。"><img src="https://raw.githubusercontent.com/D4Vinci/Scrapling/main/images/proxiware.png"></a>
<!-- /sponsors -->
<i><sub>Do you want to show your ad here? Click [here](https://github.com/sponsors/D4Vinci) and choose the tier that suits you!</sub></i>
<i><sub>您想在此展示广告?点击[此处](https://github.com/sponsors/D4Vinci))选择适合您的赞助等级!</sub></i>
---
## Key Features
## 核心特性
### Spiders - A Full Crawling Framework
- 🕷️ **Scrapy-like Spider API**: Define spiders with `start_urls`, async `parse` callbacks, and `Request`/`Response` objects.
-**Concurrent Crawling**: Configurable concurrency limits, per-domain throttling, and download delays.
- 🔄 **Multi-Session Support**: Unified interface for HTTP requests, and stealthy headless browsers in a single spider - route requests to different sessions by ID.
- 💾 **Pause & Resume**: Checkpoint-based crawl persistence. Press Ctrl+C for a graceful shutdown; restart to resume from where you left off.
- 📡 **Streaming Mode**: Stream scraped items as they arrive via `async for item in spider.stream()` with real-time stats - ideal for UI, pipelines, and long-running crawls.
- 🛡️ **Blocked Request Detection**: Automatic detection and retry of blocked requests with customizable logic.
- 🤖 **Robots.txt Compliance**: Optional `robots_txt_obey` flag that respects `Disallow`, `Crawl-delay`, and `Request-rate` directives with per-domain caching.
- 🧪 **Development Mode**: Cache responses to disk on the first run and replay them on subsequent runs - iterate on your `parse()` logic without re-hitting the target servers.
- 📦 **Built-in Export**: Export results through hooks and your own pipeline or the built-in JSON/JSONL with `result.items.to_json()` / `result.items.to_jsonl()` respectively.
### Spiders——完整的爬虫框架
- 🕷️ **Scrapy Spider API**:使用 `start_urls` 定义爬虫,配合异步 `parse` 回调以及 `Request` / `Response` 对象。
-**并发抓取**:可配置的并发限制、按域名限速以及下载延迟。
- 🔄 **多会话支持**:在单个爬虫中统一 HTTP 请求与隐身无头浏览器接口——按 ID 将请求路由到不同会话。
- 💾 **暂停与恢复**:基于检查点的爬虫持久化。按 Ctrl+C 优雅关闭;重新启动后从上次中断处继续。
- 📡 **流式模式**:通过 `async for item in spider.stream()` 实时流式传输抓取到的数据项,并提供实时统计信息——非常适合 UI、数据管道以及长时间运行的爬取任务。
- 🛡️ **请求被拦截检测**:自动检测并重试被拦截的请求,支持自定义逻辑。
- 🤖 **Robots.txt 合规**:可选的 `robots_txt_obey` 标志,遵守 `Disallow``Crawl-delay` `Request-rate` 指令,并按域名缓存。
- 🧪 **开发模式**:首次运行时将响应缓存到磁盘,后续运行直接回放——在不重复访问目标服务器的情况下迭代你的 `parse()` 逻辑。
- 📦 **内置导出**:通过钩子函数和你自己的数据管道,或内置的 JSON/JSONL(分别对应 `result.items.to_json()` / `result.items.to_jsonl()`)导出结果。
### Advanced Websites Fetching with Session Support
- **HTTP Requests**: Fast and stealthy HTTP requests with the `Fetcher` class. Can impersonate browsers' TLS fingerprint, headers, and use HTTP/3.
- **Dynamic Loading**: Fetch dynamic websites with full browser automation through the `DynamicFetcher` class supporting Playwright's Chromium and Google's Chrome.
- **Anti-bot Bypass**: Advanced stealth capabilities with `StealthyFetcher` and fingerprint spoofing. Can easily bypass all types of Cloudflare's Turnstile/Interstitial with automation.
- **Session Management**: Persistent session support with `FetcherSession`, `StealthySession`, and `DynamicSession` classes for cookie and state management across requests.
- **Proxy Rotation**: Built-in `ProxyRotator` with cyclic or custom rotation strategies across all session types, plus per-request proxy overrides.
- **Domain & Ad Blocking**: Block requests to specific domains (and their subdomains) or enable built-in ad blocking (~3,500 known ad/tracker domains) in browser-based fetchers.
- **DNS Leak Prevention**: Optional DNS-over-HTTPS support to route DNS queries through Cloudflare's DoH, preventing DNS leaks when using proxies.
- **Async Support**: Complete async support across all fetchers and dedicated async session classes.
### 高级网站抓取与会话支持
- **HTTP 请求**:使用 `Fetcher` 类实现快速且隐蔽的 HTTP 请求。可模拟浏览器 TLS 指纹、请求头,并使用 HTTP/3
- **动态加载**:通过 `DynamicFetcher` 类以完整的浏览器自动化抓取动态网站,支持 Playwright Chromium Google Chrome
- **反爬绕过**:通过 `StealthyFetcher` 和指纹伪造实现高级隐身能力。可通过自动化轻松绕过各类 Cloudflare Turnstile/插页式验证。
- **会话管理**:通过 `FetcherSession``StealthySession` `DynamicSession` 类实现持久化会话支持,用于跨请求的 Cookie 和状态管理。
- **代理轮换**:内置 `ProxyRotator`,支持循环或自定义轮换策略,适用于所有会话类型,并支持按请求覆盖代理。
- **域名与广告拦截**:拦截对特定域名(及其子域名)的请求,或在基于浏览器的抓取器中启用内置广告拦截(约 3,500 个已知广告/跟踪域名)。
- **DNS 泄露防护**:可选的 DNS-over-HTTPS 支持,通过 Cloudflare DoH 路由 DNS 查询,防止使用代理时发生 DNS 泄露。
- **异步支持**:所有抓取器均支持完整的异步操作,并提供专用的异步会话类。
### Adaptive Scraping & AI Integration
- 🔄 **Smart Element Tracking**: Relocate elements after website changes using intelligent similarity algorithms.
- 🎯 **Smart Flexible Selection**: CSS selectors, XPath selectors, filter-based search, text search, regex search, and more.
- 🔍 **Find Similar Elements**: Automatically locate elements similar to found elements.
- 🤖 **MCP Server to be used with AI**: Built-in MCP server for AI-assisted Web Scraping and data extraction. The MCP server features powerful, custom capabilities that leverage Scrapling to extract targeted content before passing it to the AI (Claude/Cursor/etc), thereby speeding up operations and reducing costs by minimizing token usage. ([demo video](https://www.youtube.com/watch?v=qyFk3ZNwOxE))
### 自适应抓取与 AI 集成
- 🔄 **智能元素追踪**:网站变更后,使用智能相似度算法重新定位元素。
- 🎯 **智能灵活选择**CSS 选择器、XPath 选择器、基于过滤的搜索、文本搜索、正则表达式搜索等。
- 🔍 **查找相似元素**:自动定位与已找到元素相似的元素。
- 🤖 **用于 AI 的 MCP 服务器**:内置 MCP 服务器,用于 AI 辅助的网页抓取和数据提取。该 MCP 服务器具备强大的自定义能力,利用 Scrapling 在将目标内容传递给 AIClaude/Cursor 等)之前进行提取,从而加速操作并通过减少 token 用量降低成本。([演示视频](https://www.youtube.com/watch?v=qyFk3ZNwOxE)))
### High-Performance & battle-tested Architecture
- 🚀 **Lightning Fast**: Optimized performance outperforming most Python scraping libraries.
- 🔋 **Memory Efficient**: Optimized data structures and lazy loading for a minimal memory footprint.
-**Fast JSON Serialization**: 10x faster than the standard library.
- 🏗️ **Battle tested**: Not only does Scrapling have 92% test coverage and full type hints coverage, but it has been used daily by hundreds of Web Scrapers over the past year.
### 高性能与久经考验的架构
- 🚀 **极速**:优化后的性能超越大多数 Python 抓取库。
- 🔋 **内存高效**:优化的数据结构和延迟加载,实现最小的内存占用。
-**快速 JSON 序列化**:比标准库快 10 倍。
- 🏗️ **久经考验**Scrapling 不仅拥有 92% 的测试覆盖率和完整的类型提示覆盖,在过去一年中更被数百位网页抓取工程师日常使用。
### Developer/Web Scraper Friendly Experience
- 🎯 **Interactive Web Scraping Shell**: Optional built-in IPython shell with Scrapling integration, shortcuts, and new tools to speed up Web Scraping scripts development, like converting curl requests to Scrapling requests and viewing requests results in your browser.
- 🚀 **Use it directly from the Terminal**: Optionally, you can use Scrapling to scrape a URL without writing a single line of code!
- 🛠️ **Rich Navigation API**: Advanced DOM traversal with parent, sibling, and child navigation methods.
- 🧬 **Enhanced Text Processing**: Built-in regex, cleaning methods, and optimized string operations.
- 📝 **Auto Selector Generation**: Generate robust CSS/XPath selectors for any element.
- 🔌 **Familiar API**: Similar to Scrapy/BeautifulSoup with the same pseudo-elements used in Scrapy/Parsel.
- 📘 **Complete Type Coverage**: Full type hints for excellent IDE support and code completion. The entire codebase is automatically scanned with **PyRight** and **MyPy** with each change.
- 🔋 **Ready Docker image**: With each release, a Docker image containing all browsers is automatically built and pushed.
### 开发者/网页抓取工程师友好体验
- 🎯 **交互式网页抓取 Shell**:可选的内置 IPython Shell,集成 Scrapling、快捷操作和新工具,加速网页抓取脚本开发,例如将 curl 请求转换为 Scrapling 请求,以及在浏览器中查看请求结果。
- 🚀 **直接在终端中使用**:可选地,你可以使用 Scrapling 抓取 URL,而无需编写一行代码!
- 🛠️ **丰富的导航 API**:高级 DOM 遍历,支持父级、兄弟节点和子节点导航方法。
- 🧬 **增强文本处理**:内置正则表达式、清洗方法和优化的字符串操作。
- 📝 **自动选择器生成**:为任意元素生成鲁棒的 CSS/XPath 选择器。
- 🔌 **熟悉的 API**:与 Scrapy/BeautifulSoup 类似,使用与 Scrapy/Parsel 相同的伪元素。
- 📘 **完整的类型覆盖**:全面的类型提示,提供出色的 IDE 支持和代码补全。每次变更都会通过 **PyRight** **MyPy** 自动扫描整个代码库。
- 🔋 **即用型 Docker 镜像**:每次发布都会自动构建并推送包含所有浏览器的 Docker 镜像。
## Getting Started
## 快速入门
Let's give you a quick glimpse of what Scrapling can do without deep diving.
让我们快速了解一下 Scrapling 的功能,无需深入细节。
### Basic Usage
HTTP requests with session support
### 基础用法
带会话支持的 HTTP 请求
```python
from scrapling.fetchers import Fetcher, FetcherSession
@@ -262,7 +268,7 @@ with FetcherSession(impersonate='chrome') as session: # Use latest version of C
page = Fetcher.get('https://quotes.toscrape.com/')
quotes = page.css('.quote .text::text').getall()
```
Advanced stealth mode
高级隐身模式
```python
from scrapling.fetchers import StealthyFetcher, StealthySession
@@ -274,7 +280,7 @@ with StealthySession(headless=True, solve_cloudflare=True) as session: # Keep t
page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare')
data = page.css('#padded_content a').getall()
```
Full browser automation
完整浏览器自动化
```python
from scrapling.fetchers import DynamicFetcher, DynamicSession
@@ -287,8 +293,8 @@ page = DynamicFetcher.fetch('https://quotes.toscrape.com/')
data = page.css('.quote .text::text').getall()
```
### Spiders
Build full crawlers with concurrent requests, multiple session types, and pause/resume:
### Spiders(爬虫)
使用并发请求、多种会话类型和暂停/恢复功能构建完整的爬虫:
```python
from scrapling.spiders import Spider, Request, Response
@@ -312,7 +318,7 @@ result = QuotesSpider().start()
print(f"Scraped {len(result.items)} quotes")
result.items.to_json("quotes.json")
```
Use multiple session types in a single spider:
在单个爬虫中使用多种会话类型:
```python
from scrapling.spiders import Spider, Request, Response
from scrapling.fetchers import FetcherSession, AsyncStealthySession
@@ -333,13 +339,13 @@ class MultiSessionSpider(Spider):
else:
yield Request(link, sid="fast", callback=self.parse) # explicit callback
```
Pause and resume long crawls with checkpoints by running the spider like this:
通过检查点暂停和恢复长时间运行的爬取任务,像这样运行爬虫:
```python
QuotesSpider(crawldir="./crawl_data").start()
```
Press Ctrl+C to pause gracefully - progress is saved automatically. Later, when you start the spider again, pass the same `crawldir`, and it will resume from where it stopped.
Ctrl+C 优雅暂停——进度会自动保存。之后,当你再次启动爬虫时,传入相同的 `crawldir`,它就会从停止的地方恢复。
### Advanced Parsing & Navigation
### 高级解析与导航
```python
from scrapling.fetchers import Fetcher
@@ -368,15 +374,15 @@ parent_container = first_quote.parent
similar_elements = first_quote.find_similar()
below_elements = first_quote.below_elements()
```
You can use the parser right away if you don't want to fetch websites like below:
如果你不想抓取网站,可以直接使用解析器,如下所示:
```python
from scrapling.parser import Selector
page = Selector("<html>...</html>")
```
And it works precisely the same way!
效果完全一样!
### Async Session Management Examples
### 异步会话管理示例
```python
import asyncio
from scrapling.fetchers import FetcherSession, AsyncStealthySession, AsyncDynamicSession
@@ -399,17 +405,17 @@ async with AsyncStealthySession(max_pages=2) as session:
print(session.get_pool_stats())
```
## CLI & Interactive Shell
## CLI 与交互式 Shell
Scrapling includes a powerful command-line interface:
Scrapling 包含一个强大的命令行界面:
[![asciicast](https://asciinema.org/a/736339.svg)](https://asciinema.org/a/736339)
Launch the interactive Web Scraping shell
启动交互式 Web 抓取 Shell
```bash
scrapling shell
```
Extract pages to a file directly without programming (Extracts the content inside the `body` tag by default). If the output file ends with `.txt`, then the text content of the target will be extracted. If it ends in `.md`, it will be a Markdown representation of the HTML content; if it ends in `.html`, it will be the HTML content itself.
无需编程即可直接将页面提取到文件(默认提取 `body` 标签内的内容)。如果输出文件以 `.txt` 结尾,则提取目标的文本内容;如果以 `.md` 结尾,则输出 HTML 内容的 Markdown 表示形式;如果以 `.html` 结尾,则直接输出 HTML 内容本身。
```bash
scrapling extract get 'https://example.com' content.md
scrapling extract get 'https://example.com' content.txt --css-selector '#fromSkipToProducts' --impersonate 'chrome' # All elements matching the CSS selector '#fromSkipToProducts'
@@ -418,106 +424,111 @@ scrapling extract stealthy-fetch 'https://nopecha.com/demo/cloudflare' captchas.
```
> [!NOTE]
> There are many additional features, but we want to keep this page concise, including the MCP server and the interactive Web Scraping Shell. Check out the full documentation [here](https://scrapling.readthedocs.io/en/latest/)
> 还有很多其他功能,但我们希望此页面保持简洁,包括 MCP 服务器和交互式 Web 抓取 Shell。请在此处查看完整文档:[here](https://scrapling.readthedocs.io/en/latest/)
## Performance Benchmarks
## 性能基准测试
Scrapling isn't just powerful-it's also blazing fast. The following benchmarks compare Scrapling's parser with the latest versions of other popular libraries.
Scrapling 不仅功能强大,而且速度极快。以下基准测试将 Scrapling 的解析器与其他流行库的最新版本进行了对比。
### Text Extraction Speed Test (5000 nested elements)
### 文本提取速度测试(5000 个嵌套元素)
| # | Library | Time (ms) | vs Scrapling |
|---|:-----------------:|:---------:|:------------:|
| 1 | Scrapling | 1.98 | 1.0x |
| 2 | Parsel/Scrapy | 1.99 | 1.005 |
| 3 | Raw Lxml | 2.48 | 1.253 |
| 4 | PyQuery | 23.15 | ~12x |
| 5 | Selectolax | 196.09 | ~99x |
| 6 | MechanicalSoup | 1531.24 | ~773.4x |
| 7 | BS4 with Lxml | 1535.19 | ~775.3x |
| 8 | BS4 with html5lib | 3388.16 | ~1711.2x |
| # | 库名称 | 时间 (ms) | Scrapling 对比 |
|---|:----------------:|:---------:|:----------------:|
| 1 | Scrapling | 1.98 | 1.0x |
| 2 | Parsel/Scrapy | 1.99 | 1.005x |
| 3 | Raw Lxml | 2.48 | 1.253x |
| 4 | PyQuery | 23.15 | ~12x |
| 5 | Selectolax | 196.09 | ~99x |
| 6 | MechanicalSoup | 1531.24 | ~773.4x |
| 7 | BS4 with Lxml | 1535.19 | ~775.3x |
| 8 | BS4 with html5lib | 3388.16 | ~1711.2x |
### 元素相似度与文本搜索性能
### Element Similarity & Text Search Performance
Scrapling 的自适应元素查找能力显著优于其他替代方案:
Scrapling's adaptive element finding capabilities significantly outperform alternatives:
| 库 | 时间 (ms) | 与 Scrapling 对比 |
|-------------|:---------:|:----------------:|
| Scrapling | 2.29 | 1.0x |
| AutoScraper | 12.46 | 5.441x |
| Library | Time (ms) | vs Scrapling |
|-------------|:---------:|:------------:|
| Scrapling | 2.29 | 1.0x |
| AutoScraper | 12.46 | 5.441x |
> 所有基准测试均为 100 次以上运行的平均值。测试方法详见 [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py)。
## 安装
> All benchmarks represent averages of 100+ runs. See [benchmarks.py](https://github.com/D4Vinci/Scrapling/blob/main/benchmarks.py) for methodology.
## Installation
Scrapling requires Python 3.10 or higher:
Scrapling 需要 Python 3.10 或更高版本:
```bash
pip install scrapling
```
> [!IMPORTANT]
> This installation only includes the parser engine and its dependencies, without any fetchers or commandline dependencies. So importing anything from `scrapling.fetchers` or `scrapling.spiders`, like in the examples above, will raise `ModuleNotFoundError` with this installation alone. If you are going to use any of the fetchers or spiders, install the fetchers' dependencies first as shown below.
> 此安装仅包含解析器引擎及其依赖项,不包含任何抓取器(fetchers)或命令行依赖项。因此,仅凭此安装,从 `scrapling.fetchers` `scrapling.spiders` 导入任何内容(如上述示例所示)都会引发 `ModuleNotFoundError`。如果你要使用任何抓取器或爬虫,请先按照下方说明安装抓取器的依赖项。
### Optional Dependencies
### 可选依赖项
1. If you are going to use any of the extra features below, the fetchers, or their classes, you will need to install fetchers' dependencies and their browser dependencies as follows:
1. 如果你要使用以下任何额外功能、抓取器或其类,则需要安装抓取器的依赖项及其浏览器依赖项,具体如下:
```bash
pip install "scrapling[fetchers]"
scrapling install # normal install
scrapling install --force # force reinstall
scrapling install # 正常安装
scrapling install --force # 强制重新安装
```
This downloads all browsers, along with their system dependencies and fingerprint manipulation dependencies.
这将下载所有浏览器及其系统依赖项和指纹操作依赖项。
Or you can install them from the code instead of running a command like this:
或者你也可以通过代码安装,而不是运行命令:
```python
from scrapling.cli import install
install([], standalone_mode=False) # normal install
install(["--force"], standalone_mode=False) # force reinstall
install([], standalone_mode=False) # 正常安装
install(["--force"], standalone_mode=False) # 强制重新安装
```
2. Extra features:
- Install the MCP server feature:
2. 额外功能:
- 安装 MCP 服务器功能:
```bash
pip install "scrapling[ai]"
```
- Install shell features (Web Scraping shell and the `extract` command):
- 安装 Shell 功能(Web 抓取 Shell 和 `extract` 命令):
```bash
pip install "scrapling[shell]"
```
- Install everything:
- 安装全部功能:
```bash
pip install "scrapling[all]"
```
Remember that you need to install the browser dependencies with `scrapling install` after any of these extras (if you didn't already)
请注意,在安装上述任意附加功能后,你还需要使用 `scrapling install` 安装浏览器依赖项(如果尚未安装)。
### Docker
You can also install a Docker image with all extras and browsers with the following command from DockerHub:
你也可以通过 DockerHub 使用以下命令安装包含所有扩展和浏览器的 Docker 镜像:
```bash
docker pull pyd4vinci/scrapling
```
Or download it from the GitHub registry:
或者从 GitHub 容器镜像仓库下载:
```bash
docker pull ghcr.io/d4vinci/scrapling:latest
```
This image is automatically built and pushed using GitHub Actions and the repository's main branch.
## Contributing
该镜像通过 GitHub Actions 自动构建并推送,基于仓库的主分支。
We welcome contributions! Please read our [contributing guidelines](https://github.com/D4Vinci/Scrapling/blob/main/CONTRIBUTING.md) before getting started.
## 贡献
## Disclaimer
欢迎贡献!在开始之前,请阅读我们的[贡献指南](https://github.com/D4Vinci/Scrapling/blob/main/CONTRIBUTING.md)。
## 免责声明
> [!CAUTION]
> This library is provided for educational and research purposes only. By using this library, you agree to comply with local and international data scraping and privacy laws. The authors and contributors are not responsible for any misuse of this software. Always respect the terms of service of websites and robots.txt files.
> 本库仅用于教育和研究目的。使用本库即表示您同意遵守当地及国际的数据抓取与隐私法律。作者和贡献者不对任何滥用此软件的行为负责。请始终遵守网站的服務条款及 robots.txt 文件。
## 🎓 引用
如果您在研究中使用了本库,请按以下参考文献引用我们:
## 🎓 Citations
If you have used our library for research purposes please quote us with the following reference:
```text
@misc{scrapling,
author = {Karim Shoair},
@@ -528,14 +539,16 @@ If you have used our library for research purposes please quote us with the foll
}
```
## License
## 许可证
This work is licensed under the BSD-3-Clause License.
本项目基于 BSD-3-Clause 许可证授权。
## Acknowledgments
## 致谢
This project includes code adapted from:
- Parsel (BSD License)-Used for [translator](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) submodule
本项目包含来自以下项目的改编代码:
- ParselBSD 许可证)——用于[翻译器](https://github.com/D4Vinci/Scrapling/blob/main/scrapling/core/translator.py) 子模块
---
<div align="center"><small>Designed & crafted with ❤️ by Karim Shoair.</small></div><br>
<div align="center"><small>由 Karim Shoair 设计与开发,倾注 ❤️。</small></div><br>
USD 预算:$0/$3;剩余 $3。