wehub/d4vinci--scrapling
1
0
Fork 0
Code Issues Pull Requests Actions Deployments Models Agent Notes Packages Projects Releases Wiki Activity

chore: import upstream snapshot with attribution
Tests / tests (map[TOXENV:py310], macos-latest, 3.10) (push) Has been cancelled
Details
Tests / tests (map[TOXENV:py311], macos-latest, 3.11) (push) Has been cancelled
Details
Tests / tests (map[TOXENV:py312], macos-latest, 3.12) (push) Has been cancelled
Details
Tests / tests (map[TOXENV:py313], macos-latest, 3.13) (push) Has been cancelled
Details

Browse Source
This commit is contained in:
wehub-resource-sync
2026-07-13 12:01:50 +08:00
commit 247153575d
244 changed files with 44853 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/cli/extract-commands.md
+371
View File
@@ -0,0 +1,371 @@
# Scrapling Extract Command Guide
**Web Scraping through the terminal without requiring any programming!**
The `scrapling extract` command lets you download and extract content from websites directly from your terminal without writing any code. Ideal for beginners, researchers, and anyone requiring rapid web data extraction.
!!! success "Prerequisites"
1. You've completed or read the [Fetchers basics](../fetching/choosing.md) page to understand what the [Response object](../fetching/choosing.md#response-object) is and which fetcher to use.
2. You've completed or read the [Querying elements](../parsing/selection.md) page to understand how to find/extract elements from the [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object) object.
3. You've completed or read the [Main classes](../parsing/main_classes.md) page to know what properties/methods the [Response](../fetching/choosing.md#response-object) class is inheriting from the [Selector](../parsing/main_classes.md#selector) class.
4. You've completed or read at least one page from the fetchers section to use here for requests: [HTTP requests](../fetching/static.md), [Dynamic websites](../fetching/dynamic.md), or [Dynamic websites with hard protections](../fetching/stealthy.md).
## What is the Extract Command group?
The extract command is a set of simple terminal tools that:
- **Downloads web pages** and saves their content to files.
- **Converts HTML to readable formats** like Markdown, keeps it as HTML, or just extracts the text content of the page.
- **Supports custom CSS selectors** to extract specific parts of the page.
- **Handles HTTP requests and fetching through browsers**
- **Highly customizable** with custom headers, cookies, proxies, and the rest of the options. Almost all the options available through the code are also accessible through the command line.
!!! tip "AI-Targeted Mode"
All extract commands support an `--ai-targeted` flag. When enabled, it extracts only the main body content, strips noise tags (script, style, noscript, svg), removes hidden elements that could be used for prompt injection (CSS-hidden, aria-hidden, template tags), strips zero-width unicode characters, and removes HTML comments. For browser commands (`fetch`/`stealthy-fetch`), it also automatically enables ad blocking. This is ideal when the output is destined for an AI model.
## Quick Start
- **Basic Website Download**
Download a website's text content as clean, readable text:
```bash
scrapling extract get "https://example.com" page_content.txt
```
This makes an HTTP GET request and saves the webpage's text content to `page_content.txt`.
- **Save as Different Formats**
Choose your output format by changing the file extension:
```bash
# Convert the HTML content to Markdown, then save it to the file (great for documentation)
scrapling extract get "https://blog.example.com" article.md
# Save the HTML content as it is to the file
scrapling extract get "https://example.com" page.html
# Save a clean version of the text content of the webpage to the file
scrapling extract get "https://example.com" content.txt
# Or use the Docker image with something like this:
docker run -v $(pwd)/output:/output pyd4vinci/scrapling extract get "https://blog.example.com" /output/article.md
```
- **Extract Specific Content**
All commands can use CSS selectors to extract specific parts of the page through `--css-selector` or `-s` as you will see in the examples below.
## Available Commands
You can display the available commands through `scrapling extract --help` to get the following list:
```bash
Usage: scrapling extract [OPTIONS] COMMAND [ARGS]...
Fetch web pages using various fetchers and extract full/selected HTML content as HTML, Markdown, or extract text content.
Options:
--help Show this message and exit.
Commands:
get Perform a GET request and save the content to a file.
post Perform a POST request and save the content to a file.
put Perform a PUT request and save the content to a file.
delete Perform a DELETE request and save the content to a file.
fetch Use DynamicFetcher to fetch content with browser...
stealthy-fetch Use StealthyFetcher to fetch content with advanced...
```
We will go through each command in detail below.
### HTTP Requests
1. **GET Request**
The most common command for downloading website content:
```bash
scrapling extract get [URL] [OUTPUT_FILE] [OPTIONS]
```
**Examples:**
```bash
# Basic download
scrapling extract get "https://news.site.com" news.md
# Download with custom timeout
scrapling extract get "https://example.com" content.txt --timeout 60
# Extract only specific content using CSS selectors
scrapling extract get "https://blog.example.com" articles.md --css-selector "article"
# Send a request with cookies
scrapling extract get "https://scrapling.requestcatcher.com" content.md --cookies "session=abc123; user=john"
# Add user agent
scrapling extract get "https://api.site.com" data.json -H "User-Agent: MyBot 1.0"
# Add multiple headers
scrapling extract get "https://site.com" page.html -H "Accept: text/html" -H "Accept-Language: en-US"
```
Get the available options for the command with `scrapling extract get --help` as follows:
```bash
Usage: scrapling extract get [OPTIONS] URL OUTPUT_FILE
Perform a GET request and save the content to a file.
The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.
Options:
-H, --headers TEXT HTTP headers in format "Key: Value" (can be used multiple times)
--cookies TEXT Cookies string in format "name1=value1;name2=value2"
--timeout INTEGER Request timeout in seconds (default: 30)
--proxy TEXT Proxy URL in format "http://username:password@host:port"
-s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.
-p, --params TEXT Query parameters in format "key=value" (can be used multiple times)
--follow-redirects / --no-follow-redirects Whether to follow redirects (default: True)
--verify / --no-verify Whether to verify SSL certificates (default: True)
--impersonate TEXT Browser to impersonate (e.g., chrome, firefox).
--stealthy-headers / --no-stealthy-headers Use stealthy browser headers (default: True)
--ai-targeted Extract only main content and sanitize hidden elements for AI consumption (default: False)
--help Show this message and exit.
```
Note that the options will work in the same way for all other request commands, so no need to repeat them.
2. **Post Request**
```bash
scrapling extract post [URL] [OUTPUT_FILE] [OPTIONS]
```
**Examples:**
```bash
# Submit form data
scrapling extract post "https://api.site.com/search" results.html --data "query=python&type=tutorial"
# Send JSON data
scrapling extract post "https://api.site.com" response.json --json '{"username": "test", "action": "search"}'
```
Get the available options for the command with `scrapling extract post --help` as follows:
```bash
Usage: scrapling extract post [OPTIONS] URL OUTPUT_FILE
Perform a POST request and save the content to a file.
The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.
Options:
-d, --data TEXT Form data to include in the request body (as string, ex: "param1=value1&param2=value2")
-j, --json TEXT JSON data to include in the request body (as string)
-H, --headers TEXT HTTP headers in format "Key: Value" (can be used multiple times)
--cookies TEXT Cookies string in format "name1=value1;name2=value2"
--timeout INTEGER Request timeout in seconds (default: 30)
--proxy TEXT Proxy URL in format "http://username:password@host:port"
-s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.
-p, --params TEXT Query parameters in format "key=value" (can be used multiple times)
--follow-redirects / --no-follow-redirects Whether to follow redirects (default: True)
--verify / --no-verify Whether to verify SSL certificates (default: True)
--impersonate TEXT Browser to impersonate (e.g., chrome, firefox).
--stealthy-headers / --no-stealthy-headers Use stealthy browser headers (default: True)
--ai-targeted Extract only main content and sanitize hidden elements for AI consumption (default: False)
--help Show this message and exit.
```
3. **Put Request**
```bash
scrapling extract put [URL] [OUTPUT_FILE] [OPTIONS]
```
**Examples:**
```bash
# Send data
scrapling extract put "https://scrapling.requestcatcher.com/put" results.html --data "update=info" --impersonate "firefox"
# Send JSON data
scrapling extract put "https://scrapling.requestcatcher.com/put" response.json --json '{"username": "test", "action": "search"}'
```
Get the available options for the command with `scrapling extract put --help` as follows:
```bash
Usage: scrapling extract put [OPTIONS] URL OUTPUT_FILE
Perform a PUT request and save the content to a file.
The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.
Options:
-d, --data TEXT Form data to include in the request body
-j, --json TEXT JSON data to include in the request body (as string)
-H, --headers TEXT HTTP headers in format "Key: Value" (can be used multiple times)
--cookies TEXT Cookies string in format "name1=value1;name2=value2"
--timeout INTEGER Request timeout in seconds (default: 30)
--proxy TEXT Proxy URL in format "http://username:password@host:port"
-s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.
-p, --params TEXT Query parameters in format "key=value" (can be used multiple times)
--follow-redirects / --no-follow-redirects Whether to follow redirects (default: True)
--verify / --no-verify Whether to verify SSL certificates (default: True)
--impersonate TEXT Browser to impersonate (e.g., chrome, firefox).
--stealthy-headers / --no-stealthy-headers Use stealthy browser headers (default: True)
--ai-targeted Extract only main content and sanitize hidden elements for AI consumption (default: False)
--help Show this message and exit.
```
4. **Delete Request**
```bash
scrapling extract delete [URL] [OUTPUT_FILE] [OPTIONS]
```
**Examples:**
```bash
# Send data
scrapling extract delete "https://scrapling.requestcatcher.com/delete" results.html
# Send JSON data
scrapling extract delete "https://scrapling.requestcatcher.com/" response.txt --impersonate "chrome"
```
Get the available options for the command with `scrapling extract delete --help` as follows:
```bash
Usage: scrapling extract delete [OPTIONS] URL OUTPUT_FILE
Perform a DELETE request and save the content to a file.
The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.
Options:
-H, --headers TEXT HTTP headers in format "Key: Value" (can be used multiple times)
--cookies TEXT Cookies string in format "name1=value1;name2=value2"
--timeout INTEGER Request timeout in seconds (default: 30)
--proxy TEXT Proxy URL in format "http://username:password@host:port"
-s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.
-p, --params TEXT Query parameters in format "key=value" (can be used multiple times)
--follow-redirects / --no-follow-redirects Whether to follow redirects (default: True)
--verify / --no-verify Whether to verify SSL certificates (default: True)
--impersonate TEXT Browser to impersonate (e.g., chrome, firefox).
--stealthy-headers / --no-stealthy-headers Use stealthy browser headers (default: True)
--ai-targeted Extract only main content and sanitize hidden elements for AI consumption (default: False)
--help Show this message and exit.
```
### Browsers fetching
1. **fetch - Handle Dynamic Content**
For websites that load content with dynamic content or have slight protection
```bash
scrapling extract fetch [URL] [OUTPUT_FILE] [OPTIONS]
```
**Examples:**
```bash
# Wait for JavaScript to load content and finish network activity
scrapling extract fetch "https://scrapling.requestcatcher.com/" content.md --network-idle
# Wait for specific content to appear
scrapling extract fetch "https://scrapling.requestcatcher.com/" data.txt --wait-selector ".content-loaded"
# Run in visible browser mode (helpful for debugging)
scrapling extract fetch "https://scrapling.requestcatcher.com/" page.html --no-headless --disable-resources
```
Get the available options for the command with `scrapling extract fetch --help` as follows:
```bash
Usage: scrapling extract fetch [OPTIONS] URL OUTPUT_FILE
Use DynamicFetcher to fetch content with browser automation.
The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.
Options:
--headless / --no-headless Run browser in headless mode (default: True)
--disable-resources / --enable-resources Drop unnecessary resources for speed boost (default: False)
--network-idle / --no-network-idle Wait for network idle (default: False)
--timeout INTEGER Timeout in milliseconds (default: 30000)
--wait INTEGER Additional wait time in milliseconds after page load (default: 0)
-s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.
--wait-selector TEXT CSS selector to wait for before proceeding
--locale TEXT Specify user locale. Defaults to the system default locale.
--real-chrome/--no-real-chrome If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)
--proxy TEXT Proxy URL in format "http://username:password@host:port"
-H, --extra-headers TEXT Extra headers in format "Key: Value" (can be used multiple times)
--dns-over-https / --no-dns-over-https Route DNS through Cloudflare's DoH to prevent DNS leaks when using proxies (default: False)
--block-ads / --no-block-ads Block requests to known ad and tracker domains (default: False)
--executable-path TEXT Path to a custom Chromium-compatible browser executable. Falls back to the SCRAPLING_EXECUTABLE_PATH environment variable when not set.
--ai-targeted Extract only main content and sanitize hidden elements for AI consumption (default: False)
--help Show this message and exit.
```
2. **stealthy-fetch - Bypass Protection**
For websites with anti-bot protection or Cloudflare protection
```bash
scrapling extract stealthy-fetch [URL] [OUTPUT_FILE] [OPTIONS]
```
**Examples:**
```bash
# Bypass basic protection
scrapling extract stealthy-fetch "https://scrapling.requestcatcher.com" content.md
# Solve Cloudflare challenges
scrapling extract stealthy-fetch "https://nopecha.com/demo/cloudflare" data.txt --solve-cloudflare --css-selector "#padded_content a"
# Use a proxy for anonymity.
scrapling extract stealthy-fetch "https://site.com" content.md --proxy "http://proxy-server:8080"
```
Get the available options for the command with `scrapling extract stealthy-fetch --help` as follows:
```bash
Usage: scrapling extract stealthy-fetch [OPTIONS] URL OUTPUT_FILE
Use StealthyFetcher to fetch content with advanced stealth features.
The output file path can be an HTML file, a Markdown file of the HTML content, or the text content itself. Use file extensions (`.html`/`.md`/`.txt`) respectively.
Options:
--headless / --no-headless Run browser in headless mode (default: True)
--disable-resources / --enable-resources Drop unnecessary resources for speed boost (default: False)
--block-webrtc / --allow-webrtc Block WebRTC entirely (default: False)
--solve-cloudflare / --no-solve-cloudflare Solve Cloudflare challenges (default: False)
--allow-webgl / --block-webgl Allow WebGL (default: True)
--network-idle / --no-network-idle Wait for network idle (default: False)
--real-chrome/--no-real-chrome If you have a Chrome browser installed on your device, enable this, and the Fetcher will launch an instance of your browser and use it. (default: False)
--timeout INTEGER Timeout in milliseconds (default: 30000)
--wait INTEGER Additional wait time in milliseconds after page load (default: 0)
-s, --css-selector TEXT CSS selector to extract specific content from the page. It returns all matches.
--wait-selector TEXT CSS selector to wait for before proceeding
--hide-canvas / --show-canvas Add noise to canvas operations (default: False)
--proxy TEXT Proxy URL in format "http://username:password@host:port"
-H, --extra-headers TEXT Extra headers in format "Key: Value" (can be used multiple times)
--dns-over-https / --no-dns-over-https Route DNS through Cloudflare's DoH to prevent DNS leaks when using proxies (default: False)
--block-ads / --no-block-ads Block requests to known ad and tracker domains (default: False)
--executable-path TEXT Path to a custom Chromium-compatible browser executable. Falls back to the SCRAPLING_EXECUTABLE_PATH environment variable when not set.
--ai-targeted Extract only main content and sanitize hidden elements for AI consumption (default: False)
--help Show this message and exit.
```
## When to use each command
If you are not a Web Scraping expert and can't decide what to choose, you can use the following formula to help you decide:
- Use **`get`** with simple websites, blogs, or news articles
- Use **`fetch`** with modern web apps, or sites with dynamic content
- Use **`stealthy-fetch`** with protected sites, Cloudflare, or anti-bot systems
## Legal and Ethical Considerations
⚠️ **Important Guidelines:**
- **Check robots.txt**: Visit `https://website.com/robots.txt` to see scraping rules
- **Respect rate limits**: Don't overwhelm servers with requests
- **Terms of Service**: Read and comply with website terms
- **Copyright**: Respect intellectual property rights
- **Privacy**: Be mindful of personal data protection laws
- **Commercial use**: Ensure you have permission for business purposes
---
*Happy scraping! Remember to always respect website policies and comply with all applicable laws and regulations.*
docs/cli/interactive-shell.md
+234
View File
@@ -0,0 +1,234 @@
# Scrapling Interactive Shell Guide
<script src="https://asciinema.org/a/736339.js" id="asciicast-736339" async data-autoplay="1" data-loop="1" data-cols="225" data-rows="40" data-start-at="00:06" data-speed="1.5" data-theme="tango"></script>
**Powerful Web Scraping REPL for Developers and Data Scientists**
The Scrapling Interactive Shell is an enhanced IPython-based environment designed specifically for Web Scraping tasks. It provides instant access to all Scrapling features, clever shortcuts, automatic page management, and advanced tools, such as conversion of the curl command.
!!! success "Prerequisites"
1. You've completed or read the [Fetchers basics](../fetching/choosing.md) page to understand what the [Response object](../fetching/choosing.md#response-object) is and which fetcher to use.
2. You've completed or read the [Querying elements](../parsing/selection.md) page to understand how to find/extract elements from the [Selector](../parsing/main_classes.md#selector)/[Response](../fetching/choosing.md#response-object) object.
3. You've completed or read the [Main classes](../parsing/main_classes.md) page to know what properties/methods the [Response](../fetching/choosing.md#response-object) class is inheriting from the [Selector](../parsing/main_classes.md#selector) class.
4. You've completed or read at least one page from the fetchers section to use here for requests: [HTTP requests](../fetching/static.md), [Dynamic websites](../fetching/dynamic.md), or [Dynamic websites with hard protections](../fetching/stealthy.md).
## Why use the Interactive Shell?
The interactive shell transforms web scraping from a slow script-and-run cycle into a fast, exploratory experience. It's perfect for:
- **Rapid prototyping**: Test scraping strategies instantly
- **Data exploration**: Interactively navigate and extract from websites
- **Learning Scrapling**: Experiment with features in real-time
- **Debugging scrapers**: Step through requests and inspect results
- **Converting workflows**: Transform curl commands from browser DevTools to a Fetcher request in a one-liner
## Getting Started
### Launch the Shell
```bash
# Start the interactive shell
scrapling shell
# Execute code and exit (useful for scripting)
scrapling shell -c "get('https://quotes.toscrape.com'); print(len(page.css('.quote')))"
# Set logging level
scrapling shell --loglevel info
```
Once launched, you'll see the Scrapling banner and can immediately start scraping as the video above shows:
```python
# No imports needed - everything is ready!
get('https://news.ycombinator.com')
# Explore the page structure
page.css('a')[:5] # Look at first 5 links
# Refine your selectors
stories = page.css('.titleline>a')
len(stories) # 30
# Extract specific data
for story in stories[:3]:
... title = story.text
... url = story['href']
... print(f"{title}: {url}")
# Try different approaches
titles = page.css('.titleline>a::text') # Direct text extraction
urls = page.css('.titleline>a::attr(href)') # Direct attribute extraction
```
## Built-in Shortcuts
The shell provides convenient shortcuts that eliminate boilerplate code:
- **`get(url, **kwargs)`** - HTTP GET request (instead of `Fetcher.get`)
- **`post(url, **kwargs)`** - HTTP POST request (instead of `Fetcher.post`)
- **`put(url, **kwargs)`** - HTTP PUT request (instead of `Fetcher.put`)
- **`delete(url, **kwargs)`** - HTTP DELETE request (instead of `Fetcher.delete`)
- **`fetch(url, **kwargs)`** - Browser-based fetch (instead of `DynamicFetcher.fetch`)
- **`stealthy_fetch(url, **kwargs)`** - Stealthy browser fetch (instead of `StealthyFetcher.fetch`)
The most commonly used classes are automatically available without any import, including `Fetcher`, `AsyncFetcher`, `DynamicFetcher`, `StealthyFetcher`, and `Selector`.
### Smart Page Management
The shell automatically tracks your requests and pages:
- **Current Page Access**
The `page` and `response` commands are automatically updated with the last fetched page:
```python
get('https://quotes.toscrape.com')
# 'page' and 'response' both refer to the last fetched page
page.url # 'https://quotes.toscrape.com'
response.status # Prints 200; Same as page.status
```
- **Page History**
The `pages` command keeps track of the last five pages (it's a `Selectors` object):
```python
get('https://site1.com')
get('https://site2.com')
get('https://site3.com')
# Access last 5 pages
len(pages) # `Selectors` object with `page` history -> 3
pages[0].url # First page in history -> 'https://site1.com'
pages[-1].url # Most recent page -> 'https://site3.com'
# Work with historical pages
for i, old_page in enumerate(pages):
... print(f"Page {i}: {old_page.url} - {old_page.status}")
```
## Additional helpful commands
### Page Visualization
View scraped pages in your browser:
```python
get('https://quotes.toscrape.com')
view(page) # Opens the page HTML in your default browser
```
### Curl Command Integration
The shell provides a few functions to help you convert curl commands from the browser DevTools to `Fetcher` requests: `uncurl` and `curl2fetcher`.
First, you need to copy a request as a curl command like the following:
<img src="../assets/scrapling_shell_curl.png" title="Copying a request as a curl command from Chrome" alt="Copying a request as a curl command from Chrome" style="width: 70%;"/>
- **Convert Curl command to Request Object**
```python
curl_cmd = '''curl 'https://scrapling.requestcatcher.com/post' \
... -X POST \
... -H 'Content-Type: application/json' \
... -d '{"name": "test", "value": 123}' '''
request = uncurl(curl_cmd)
request.method # -> 'post'
request.url # -> 'https://scrapling.requestcatcher.com/post'
request.headers # -> {'Content-Type': 'application/json'}
```
- **Execute Curl Command Directly**
```python
# Convert and execute in one step
curl2fetcher(curl_cmd)
page.status # -> 200
page.json()['json'] # -> {'name': 'test', 'value': 123}
```
### IPython Features
The shell inherits all IPython capabilities:
```python
# Magic commands
%time page = get('https://example.com') # Time execution
%history # Show command history
%save filename.py 1-10 # Save commands 1-10 to file
# Tab completion works everywhere
page.c<TAB> # Shows: css, cookies, headers, etc.
Fetcher.<TAB> # Shows all Fetcher methods
# Object inspection
get? # Show get documentation
```
## Examples
Here are a few examples generated via AI:
#### E-commerce Data Collection
```python
# Start with product listing page
catalog = get('https://shop.example.com/products')
# Find product links
product_links = catalog.css('.product-link::attr(href)')
print(f"Found {len(product_links)} products")
# Sample a few products first
for link in product_links[:3]:
... product = get(f"https://shop.example.com{link}")
... name = product.css('.product-name::text').get('')
... price = product.css('.price::text').get('')
... print(f"{name}: {price}")
# Scale up with sessions for efficiency
from scrapling.fetchers import FetcherSession
with FetcherSession() as session:
... products = []
... for link in product_links:
... product = session.get(f"https://shop.example.com{link}")
... products.append({
... 'name': product.css('.product-name::text').get(''),
... 'price': product.css('.price::text').get(''),
... 'url': link
... })
```
#### API Integration and Testing
```python
>>> # Test API endpoints interactively
>>> response = get('https://jsonplaceholder.typicode.com/posts/1')
>>> response.json()
{'userId': 1, 'id': 1, 'title': 'sunt aut...', 'body': 'quia et...'}
>>> # Test POST requests
>>> new_post = post('https://jsonplaceholder.typicode.com/posts',
... json={'title': 'Test Post', 'body': 'Test content', 'userId': 1})
>>> new_post.json()['id']
101
>>> # Test with different data
>>> updated = put(f'https://jsonplaceholder.typicode.com/posts/{new_post.json()["id"]}',
... json={'title': 'Updated Title'})
```
## Getting Help
If you need help other than what is available in-terminal, you can:
- [Scrapling Documentation](https://scrapling.readthedocs.io/)
- [Discord Community](https://discord.gg/EMgGbDceNQ)
- [GitHub Issues](https://github.com/D4Vinci/Scrapling/issues)
And that's it! Happy scraping! The shell makes web scraping as easy as a conversation.
docs/cli/overview.md
+30
View File
@@ -0,0 +1,30 @@
# Command Line Interface
Since v0.3, Scrapling includes a powerful command-line interface that provides three main capabilities:
1. **Interactive Shell**: An interactive Web Scraping shell based on IPython that provides many shortcuts and useful tools
2. **Extract Commands**: Scrape websites from the terminal without any programming
3. **Utility Commands**: Installation and management tools
```bash
# Launch interactive shell
scrapling shell
# Convert the content of a page to markdown and save it to a file
scrapling extract get "https://example.com" content.md
# Get help for any command
scrapling --help
scrapling extract --help
```
## Requirements
This section requires you to install the extra `shell` dependency group, like the following:
```bash
pip install "scrapling[shell]"
```
and the installation of the fetchers' dependencies with the following command
```bash
scrapling install
```
This downloads all browsers, along with their system dependencies and fingerprint manipulation dependencies.