chore: import upstream snapshot with attribution
Tests / tests (map[TOXENV:py310], macos-latest, 3.10) (push) Has been cancelled
Tests / tests (map[TOXENV:py311], macos-latest, 3.11) (push) Has been cancelled
Tests / tests (map[TOXENV:py312], macos-latest, 3.12) (push) Has been cancelled
Tests / tests (map[TOXENV:py313], macos-latest, 3.13) (push) Has been cancelled
Tests / tests (map[TOXENV:py310], macos-latest, 3.10) (push) Has been cancelled
Tests / tests (map[TOXENV:py311], macos-latest, 3.11) (push) Has been cancelled
Tests / tests (map[TOXENV:py312], macos-latest, 3.12) (push) Has been cancelled
Tests / tests (map[TOXENV:py313], macos-latest, 3.13) (push) Has been cancelled
This commit is contained in:
@@ -0,0 +1,26 @@
|
||||
---
|
||||
search:
|
||||
exclude: true
|
||||
---
|
||||
|
||||
# Custom Types API Reference
|
||||
|
||||
Here's the reference information for all custom types of classes Scrapling implemented, with all their parameters, attributes, and methods.
|
||||
|
||||
You can import all of them directly like below:
|
||||
|
||||
```python
|
||||
from scrapling.core.custom_types import TextHandler, TextHandlers, AttributesHandler
|
||||
```
|
||||
|
||||
## ::: scrapling.core.custom_types.TextHandler
|
||||
handler: python
|
||||
:docstring:
|
||||
|
||||
## ::: scrapling.core.custom_types.TextHandlers
|
||||
handler: python
|
||||
:docstring:
|
||||
|
||||
## ::: scrapling.core.custom_types.AttributesHandler
|
||||
handler: python
|
||||
:docstring:
|
||||
@@ -0,0 +1,63 @@
|
||||
---
|
||||
search:
|
||||
exclude: true
|
||||
---
|
||||
|
||||
# Fetchers Classes
|
||||
|
||||
Here's the reference information for all fetcher-type classes' parameters, attributes, and methods.
|
||||
|
||||
You can import all of them directly like below:
|
||||
|
||||
```python
|
||||
from scrapling.fetchers import (
|
||||
Fetcher, AsyncFetcher, StealthyFetcher, DynamicFetcher,
|
||||
FetcherSession, AsyncStealthySession, StealthySession, DynamicSession, AsyncDynamicSession
|
||||
)
|
||||
```
|
||||
|
||||
## ::: scrapling.fetchers.Fetcher
|
||||
handler: python
|
||||
:docstring:
|
||||
|
||||
## ::: scrapling.fetchers.AsyncFetcher
|
||||
handler: python
|
||||
:docstring:
|
||||
|
||||
## ::: scrapling.fetchers.DynamicFetcher
|
||||
handler: python
|
||||
:docstring:
|
||||
|
||||
## ::: scrapling.fetchers.StealthyFetcher
|
||||
handler: python
|
||||
:docstring:
|
||||
|
||||
|
||||
## Session Classes
|
||||
|
||||
### HTTP Sessions
|
||||
|
||||
## ::: scrapling.fetchers.FetcherSession
|
||||
handler: python
|
||||
:docstring:
|
||||
|
||||
### Stealth Sessions
|
||||
|
||||
## ::: scrapling.fetchers.StealthySession
|
||||
handler: python
|
||||
:docstring:
|
||||
|
||||
## ::: scrapling.fetchers.AsyncStealthySession
|
||||
handler: python
|
||||
:docstring:
|
||||
|
||||
### Dynamic Sessions
|
||||
|
||||
## ::: scrapling.fetchers.DynamicSession
|
||||
handler: python
|
||||
:docstring:
|
||||
|
||||
## ::: scrapling.fetchers.AsyncDynamicSession
|
||||
handler: python
|
||||
:docstring:
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
---
|
||||
search:
|
||||
exclude: true
|
||||
---
|
||||
|
||||
# MCP Server API Reference
|
||||
|
||||
The **Scrapling MCP Server** provides nine powerful tools for web scraping through the Model Context Protocol (MCP). This server integrates Scrapling's capabilities directly into AI chatbots and agents, allowing conversational web scraping with advanced anti-bot bypass features.
|
||||
|
||||
You can start the MCP server by running:
|
||||
|
||||
```bash
|
||||
scrapling mcp
|
||||
```
|
||||
|
||||
Or import the server class directly:
|
||||
|
||||
```python
|
||||
from scrapling.core.ai import ScraplingMCPServer
|
||||
|
||||
server = ScraplingMCPServer()
|
||||
server.serve(http=False, host="0.0.0.0", port=8000)
|
||||
```
|
||||
|
||||
To set a custom Chromium-compatible browser executable for browser-based MCP tools, pass `executable_path`:
|
||||
|
||||
```python
|
||||
server = ScraplingMCPServer(executable_path="/path/to/chromium")
|
||||
```
|
||||
|
||||
## Response Model
|
||||
|
||||
The standardized response structure that's returned by all MCP server tools:
|
||||
|
||||
## ::: scrapling.core.ai.ResponseModel
|
||||
handler: python
|
||||
:docstring:
|
||||
|
||||
## Session Models
|
||||
|
||||
Model classes for session management:
|
||||
|
||||
## ::: scrapling.core.ai.SessionInfo
|
||||
handler: python
|
||||
:docstring:
|
||||
|
||||
## ::: scrapling.core.ai.SessionCreatedModel
|
||||
handler: python
|
||||
:docstring:
|
||||
|
||||
## ::: scrapling.core.ai.SessionClosedModel
|
||||
handler: python
|
||||
:docstring:
|
||||
|
||||
## MCP Server Class
|
||||
|
||||
The main MCP server class that provides all web scraping tools:
|
||||
|
||||
## ::: scrapling.core.ai.ScraplingMCPServer
|
||||
handler: python
|
||||
:docstring:
|
||||
@@ -0,0 +1,18 @@
|
||||
---
|
||||
search:
|
||||
exclude: true
|
||||
---
|
||||
|
||||
# Proxy Rotation
|
||||
|
||||
The `ProxyRotator` class provides thread-safe proxy rotation for any fetcher or session.
|
||||
|
||||
You can import it directly like below:
|
||||
|
||||
```python
|
||||
from scrapling.fetchers import ProxyRotator
|
||||
```
|
||||
|
||||
## ::: scrapling.engines.toolbelt.proxy_rotation.ProxyRotator
|
||||
handler: python
|
||||
:docstring:
|
||||
@@ -0,0 +1,18 @@
|
||||
---
|
||||
search:
|
||||
exclude: true
|
||||
---
|
||||
|
||||
# Response Class
|
||||
|
||||
The `Response` class wraps HTTP responses returned by all fetchers, providing access to status, headers, body, cookies, and a `Selector` for parsing.
|
||||
|
||||
You can import the `Response` class like below:
|
||||
|
||||
```python
|
||||
from scrapling.engines.toolbelt.custom import Response
|
||||
```
|
||||
|
||||
## ::: scrapling.engines.toolbelt.custom.Response
|
||||
handler: python
|
||||
:docstring:
|
||||
@@ -0,0 +1,25 @@
|
||||
---
|
||||
search:
|
||||
exclude: true
|
||||
---
|
||||
|
||||
# Selector Class
|
||||
|
||||
The `Selector` class is the core parsing engine in Scrapling that provides HTML parsing and element selection capabilities.
|
||||
|
||||
Here's the reference information for the `Selector` class, with all its parameters, attributes, and methods.
|
||||
|
||||
You can import the `Selector` class directly from `scrapling`:
|
||||
|
||||
```python
|
||||
from scrapling.parser import Selector
|
||||
```
|
||||
|
||||
## ::: scrapling.parser.Selector
|
||||
handler: python
|
||||
:docstring:
|
||||
|
||||
## ::: scrapling.parser.Selectors
|
||||
handler: python
|
||||
:docstring:
|
||||
|
||||
@@ -0,0 +1,42 @@
|
||||
---
|
||||
search:
|
||||
exclude: true
|
||||
---
|
||||
|
||||
# Spider Classes
|
||||
|
||||
Here's the reference information for the spider framework classes' parameters, attributes, and methods.
|
||||
|
||||
You can import them directly like below:
|
||||
|
||||
```python
|
||||
from scrapling.spiders import Spider, Request, CrawlResult, SessionManager, Response
|
||||
```
|
||||
|
||||
## ::: scrapling.spiders.Spider
|
||||
handler: python
|
||||
:docstring:
|
||||
|
||||
## ::: scrapling.spiders.Request
|
||||
handler: python
|
||||
:docstring:
|
||||
|
||||
## Result Classes
|
||||
|
||||
## ::: scrapling.spiders.result.CrawlResult
|
||||
handler: python
|
||||
:docstring:
|
||||
|
||||
## ::: scrapling.spiders.result.CrawlStats
|
||||
handler: python
|
||||
:docstring:
|
||||
|
||||
## ::: scrapling.spiders.result.ItemList
|
||||
handler: python
|
||||
:docstring:
|
||||
|
||||
## Session Management
|
||||
|
||||
## ::: scrapling.spiders.session.SessionManager
|
||||
handler: python
|
||||
:docstring:
|
||||
Reference in New Issue
Block a user