wehub-resource-sync
247153575d
Tests / tests (map[TOXENV:py310], macos-latest, 3.10) (push) Has been cancelled
Tests / tests (map[TOXENV:py311], macos-latest, 3.11) (push) Has been cancelled
Tests / tests (map[TOXENV:py312], macos-latest, 3.12) (push) Has been cancelled
Tests / tests (map[TOXENV:py313], macos-latest, 3.13) (push) Has been cancelled
342 lines
17 KiB
Markdown
342 lines
17 KiB
Markdown
## Pick Your Path
|
|
|
|
Not sure where to start? Pick the path that matches what you're trying to do:
|
|
|
|
| I want to... | Start here |
|
|
|:---|:---|
|
|
| **Parse HTML** I already have | [Querying elements](parsing/selection.md): CSS, XPath, and text-based selection |
|
|
| **Quickly scrape a page** and prototype | Pick a [fetcher](fetching/choosing.md) and test right away, or launch the [interactive shell](cli/interactive-shell.md) |
|
|
| **Build a crawler** that scales | [Spiders](spiders/getting-started.md): concurrent, multi-session crawls with pause/resume |
|
|
| **Scrape without writing code** | [CLI extract commands](cli/extract-commands.md) or hook up the [MCP server](ai/mcp-server.md) to your favourite AI tool |
|
|
| **Migrate** from another library | [From BeautifulSoup](tutorials/migrating_from_beautifulsoup.md) or [Scrapy comparison](spiders/architecture.md#comparison-with-scrapy) |
|
|
|
|
---
|
|
|
|
We will start by quickly reviewing the parsing capabilities. Then we will fetch websites using custom browsers, make requests, and parse the responses.
|
|
|
|
Here's an HTML document generated by ChatGPT that we will be using as an example throughout this page:
|
|
```html
|
|
<html>
|
|
<head>
|
|
<title>Complex Web Page</title>
|
|
<style>
|
|
.hidden { display: none; }
|
|
</style>
|
|
</head>
|
|
<body>
|
|
<header>
|
|
<nav>
|
|
<ul>
|
|
<li> <a href="#home">Home</a> </li>
|
|
<li> <a href="#about">About</a> </li>
|
|
<li> <a href="#contact">Contact</a> </li>
|
|
</ul>
|
|
</nav>
|
|
</header>
|
|
<main>
|
|
<section id="products" schema='{"jsonable": "data"}'>
|
|
<h2>Products</h2>
|
|
<div class="product-list">
|
|
<article class="product" data-id="1">
|
|
<h3>Product 1</h3>
|
|
<p class="description">This is product 1</p>
|
|
<span class="price">$10.99</span>
|
|
<div class="hidden stock">In stock: 5</div>
|
|
</article>
|
|
|
|
<article class="product" data-id="2">
|
|
<h3>Product 2</h3>
|
|
<p class="description">This is product 2</p>
|
|
<span class="price">$20.99</span>
|
|
<div class="hidden stock">In stock: 3</div>
|
|
</article>
|
|
|
|
<article class="product" data-id="3">
|
|
<h3>Product 3</h3>
|
|
<p class="description">This is product 3</p>
|
|
<span class="price">$15.99</span>
|
|
<div class="hidden stock">Out of stock</div>
|
|
</article>
|
|
</div>
|
|
</section>
|
|
|
|
<section id="reviews">
|
|
<h2>Customer Reviews</h2>
|
|
<div class="review-list">
|
|
<div class="review" data-rating="5">
|
|
<p class="review-text">Great product!</p>
|
|
<span class="reviewer">John Doe</span>
|
|
</div>
|
|
<div class="review" data-rating="4">
|
|
<p class="review-text">Good value for money.</p>
|
|
<span class="reviewer">Jane Smith</span>
|
|
</div>
|
|
</div>
|
|
</section>
|
|
</main>
|
|
<script id="page-data" type="application/json">
|
|
{
|
|
"lastUpdated": "2024-09-22T10:30:00Z",
|
|
"totalProducts": 3
|
|
}
|
|
</script>
|
|
</body>
|
|
</html>
|
|
```
|
|
Starting with loading raw HTML above like this
|
|
```python
|
|
from scrapling.parser import Selector
|
|
page = Selector(html_doc)
|
|
page # <data='<html><head><title>Complex Web Page</tit...'>
|
|
```
|
|
Get all text content on the page recursively
|
|
```python
|
|
page.get_all_text(ignore_tags=('script', 'style'))
|
|
# 'Complex Web Page\nHome\nAbout\nContact\nProducts\nProduct 1\nThis is product 1\n$10.99\nIn stock: 5\nProduct 2\nThis is product 2\n$20.99\nIn stock: 3\nProduct 3\nThis is product 3\n$15.99\nOut of stock\nCustomer Reviews\nGreat product!\nJohn Doe\nGood value for money.\nJane Smith'
|
|
```
|
|
|
|
## Finding elements
|
|
If there's an element you want to find on the page, you will find it! Your creativity level is the only limitation!
|
|
|
|
Finding the first HTML `section` element
|
|
```python
|
|
section_element = page.find('section')
|
|
# <data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>
|
|
```
|
|
Find all `section` elements
|
|
```python
|
|
section_elements = page.find_all('section')
|
|
# [<data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>, <data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>]
|
|
```
|
|
Find all `section` elements whose `id` attribute value is `products`.
|
|
```python
|
|
section_elements = page.find_all('section', {'id':"products"})
|
|
# Same as
|
|
section_elements = page.find_all('section', id="products")
|
|
# [<data='<section id="products" schema='{"jsonabl...' parent='<main><section id="products" schema='{"j...'>]
|
|
```
|
|
Find all `section` elements whose `id` attribute value contains `product`.
|
|
```python
|
|
section_elements = page.find_all('section', {'id*':"product"})
|
|
```
|
|
Find all `h3` elements whose text content matches this regex `Product \d`
|
|
```python
|
|
page.find_all('h3', re.compile(r'Product \d'))
|
|
# [<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>, <data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>]
|
|
```
|
|
Find all `h3` and `h2` elements whose text content matches the regex `Product` only
|
|
```python
|
|
page.find_all(['h3', 'h2'], re.compile(r'Product'))
|
|
# [<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>, <data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>, <data='<h2>Products</h2>' parent='<section id="products" schema='{"jsonabl...'>]
|
|
```
|
|
Find all elements whose text content matches exactly `Products` (Whitespaces are not taken into consideration)
|
|
```python
|
|
page.find_by_text('Products', first_match=False)
|
|
# [<data='<h2>Products</h2>' parent='<section id="products" schema='{"jsonabl...'>]
|
|
```
|
|
Or find all elements whose text content matches regex `Product \d`
|
|
```python
|
|
page.find_by_regex(r'Product \d', first_match=False)
|
|
# [<data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>, <data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>]
|
|
```
|
|
Find all elements that are similar to the element you want
|
|
```python
|
|
target_element = page.find_by_regex(r'Product \d', first_match=True)
|
|
# <data='<h3>Product 1</h3>' parent='<article class="product" data-id="1"><h3...'>
|
|
target_element.find_similar()
|
|
# [<data='<h3>Product 2</h3>' parent='<article class="product" data-id="2"><h3...'>, <data='<h3>Product 3</h3>' parent='<article class="product" data-id="3"><h3...'>]
|
|
```
|
|
Find the first element that matches a CSS selector
|
|
```python
|
|
page.css('.product-list [data-id="1"]')[0]
|
|
# <data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>
|
|
```
|
|
Find all elements that match a CSS selector
|
|
```python
|
|
page.css('.product-list article')
|
|
# [<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]
|
|
```
|
|
Find the first element that matches an XPath selector
|
|
```python
|
|
page.xpath("//*[@id='products']/div/article")[0]
|
|
# <data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>
|
|
```
|
|
Find all elements that match an XPath selector
|
|
```python
|
|
page.xpath("//*[@id='products']/div/article")
|
|
# [<data='<article class="product" data-id="1"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="2"><h3...' parent='<div class="product-list"> <article clas...'>, <data='<article class="product" data-id="3"><h3...' parent='<div class="product-list"> <article clas...'>]
|
|
```
|
|
|
|
With this, we just scratched the surface of these functions; more advanced options with these selection methods are shown later.
|
|
## Accessing elements' data
|
|
It's as simple as
|
|
```python
|
|
>>> section_element.tag
|
|
'section'
|
|
>>> print(section_element.attrib)
|
|
{'id': 'products', 'schema': '{"jsonable": "data"}'}
|
|
>>> section_element.attrib['schema'].json() # If an attribute value can be converted to json, then use `.json()` to convert it
|
|
{'jsonable': 'data'}
|
|
>>> section_element.text # Direct text content
|
|
''
|
|
>>> section_element.get_all_text() # All text content recursively
|
|
'Products\nProduct 1\nThis is product 1\n$10.99\nIn stock: 5\nProduct 2\nThis is product 2\n$20.99\nIn stock: 3\nProduct 3\nThis is product 3\n$15.99\nOut of stock'
|
|
>>> section_element.html_content # The HTML content of the element
|
|
'<section id="products" schema=\'{"jsonable": "data"}\'><h2>Products</h2>\n <div class="product-list">\n <article class="product" data-id="1"><h3>Product 1</h3>\n <p class="description">This is product 1</p>\n <span class="price">$10.99</span>\n <div class="hidden stock">In stock: 5</div>\n </article><article class="product" data-id="2"><h3>Product 2</h3>\n <p class="description">This is product 2</p>\n <span class="price">$20.99</span>\n <div class="hidden stock">In stock: 3</div>\n </article><article class="product" data-id="3"><h3>Product 3</h3>\n <p class="description">This is product 3</p>\n <span class="price">$15.99</span>\n <div class="hidden stock">Out of stock</div>\n </article></div>\n </section>'
|
|
>>> print(section_element.prettify()) # The prettified version
|
|
'''
|
|
<section id="products" schema='{"jsonable": "data"}'><h2>Products</h2>
|
|
<div class="product-list">
|
|
<article class="product" data-id="1"><h3>Product 1</h3>
|
|
<p class="description">This is product 1</p>
|
|
<span class="price">$10.99</span>
|
|
<div class="hidden stock">In stock: 5</div>
|
|
</article><article class="product" data-id="2"><h3>Product 2</h3>
|
|
<p class="description">This is product 2</p>
|
|
<span class="price">$20.99</span>
|
|
<div class="hidden stock">In stock: 3</div>
|
|
</article><article class="product" data-id="3"><h3>Product 3</h3>
|
|
<p class="description">This is product 3</p>
|
|
<span class="price">$15.99</span>
|
|
<div class="hidden stock">Out of stock</div>
|
|
</article>
|
|
</div>
|
|
</section>
|
|
'''
|
|
>>> section_element.path # All the ancestors in the DOM tree of this element
|
|
[<data='<main><section id="products" schema='{"j...' parent='<body> <header><nav><ul><li> <a href="#h...'>,
|
|
<data='<body> <header><nav><ul><li> <a href="#h...' parent='<html><head><title>Complex Web Page</tit...'>,
|
|
<data='<html><head><title>Complex Web Page</tit...'>]
|
|
>>> section_element.generate_css_selector
|
|
'#products'
|
|
>>> section_element.generate_full_css_selector
|
|
'body > main > #products > #products'
|
|
>>> section_element.generate_xpath_selector
|
|
"//*[@id='products']"
|
|
>>> section_element.generate_full_xpath_selector
|
|
"//body/main/*[@id='products']"
|
|
```
|
|
|
|
## Navigation
|
|
Using the elements we found above
|
|
|
|
```python
|
|
>>> section_element.parent
|
|
<data='<main><section id="products" schema='{"j...' parent='<body> <header><nav><ul><li> <a href="#h...'>
|
|
>>> section_element.parent.tag
|
|
'main'
|
|
>>> section_element.parent.parent.tag
|
|
'body'
|
|
>>> section_element.children
|
|
[<data='<h2>Products</h2>' parent='<section id="products" schema='{"jsonabl...'>,
|
|
<data='<div class="product-list"> <article clas...' parent='<section id="products" schema='{"jsonabl...'>]
|
|
>>> section_element.siblings
|
|
[<data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>]
|
|
>>> section_element.next # gets the next element, the same logic applies to `quote.previous`.
|
|
<data='<section id="reviews"><h2>Customer Revie...' parent='<main><section id="products" schema='{"j...'>
|
|
>>> section_element.children.css('h2::text').getall()
|
|
['Products']
|
|
>>> page.css('[data-id="1"]')[0].has_class('product')
|
|
True
|
|
```
|
|
If your case needs more than the element's parent, you can iterate over the whole ancestors' tree of any element, like the one below
|
|
```python
|
|
for ancestor in section_element.iterancestors():
|
|
# do something with it...
|
|
```
|
|
You can search for a specific ancestor of an element that satisfies a function; all you need to do is pass a function that takes a `Selector` object as an argument and returns `True` if the condition is satisfied or `False` otherwise, like below:
|
|
```python
|
|
>>> section_element.find_ancestor(lambda ancestor: ancestor.css('nav'))
|
|
<data='<body> <header><nav><ul><li> <a href="#h...' parent='<html><head><title>Complex Web Page</tit...'>
|
|
```
|
|
|
|
## Fetching websites
|
|
Instead of passing the raw HTML to Scrapling, you can retrieve a website's response directly via HTTP requests or by fetching it in a browser.
|
|
|
|
A fetcher is made for every use case.
|
|
|
|
### HTTP Requests
|
|
For simple HTTP requests, there's a `Fetcher` class that can be imported and used as below:
|
|
```python
|
|
from scrapling.fetchers import Fetcher
|
|
page = Fetcher.get('https://scrapling.requestcatcher.com/get', impersonate="chrome")
|
|
```
|
|
With that out of the way, here's how to do all HTTP methods:
|
|
```python
|
|
from scrapling.fetchers import Fetcher
|
|
page = Fetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)
|
|
page = Fetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')
|
|
page = Fetcher.put('https://scrapling.requestcatcher.com/put', data={'key': 'value'})
|
|
page = Fetcher.delete('https://scrapling.requestcatcher.com/delete')
|
|
```
|
|
For Async requests, you will replace the import like below:
|
|
```python
|
|
from scrapling.fetchers import AsyncFetcher
|
|
page = await AsyncFetcher.get('https://scrapling.requestcatcher.com/get', stealthy_headers=True)
|
|
page = await AsyncFetcher.post('https://scrapling.requestcatcher.com/post', data={'key': 'value'}, proxy='http://username:password@localhost:8030')
|
|
page = await AsyncFetcher.put('https://scrapling.requestcatcher.com/put', data={'key': 'value'})
|
|
page = await AsyncFetcher.delete('https://scrapling.requestcatcher.com/delete')
|
|
```
|
|
|
|
!!! note "Notes:"
|
|
|
|
1. You have the `stealthy_headers` argument, which, when enabled, makes requests to generate real browser headers and use them, including a Google referer header. It's enabled by default.
|
|
2. The `impersonate` argument lets you fake the TLS fingerprint for a specific browser version.
|
|
3. There's also the `http3` argument, which, when enabled, makes the fetcher use HTTP/3 for requests, which makes your requests more authentic
|
|
|
|
This is just the tip of the iceberg with this fetcher; check out the rest from [here](fetching/static.md)
|
|
|
|
### Dynamic loading
|
|
We have you covered if you deal with dynamic websites like most today!
|
|
|
|
The `DynamicFetcher` class (formerly `PlayWrightFetcher`) offers many options for fetching and loading web pages using Chromium-based browsers.
|
|
```python
|
|
from scrapling.fetchers import DynamicFetcher
|
|
page = DynamicFetcher.fetch('https://quotes.toscrape.com/js/', disable_resources=True, block_ads=True)
|
|
print(len(page.css(".quote"))) # -> 10
|
|
|
|
# The async version of fetch
|
|
page = await DynamicFetcher.async_fetch('https://quotes.toscrape.com/js/', disable_resources=True, block_ads=True)
|
|
print(len(page.css(".quote"))) # -> 10
|
|
```
|
|
It's built on top of [Playwright](https://playwright.dev/python/), and it's currently providing two main run options that can be mixed as you want:
|
|
|
|
- Vanilla Playwright without any modifications other than the ones you chose. It uses the Chromium browser.
|
|
- Real browsers like your Chrome browser by passing the `real_chrome` argument or the CDP URL of your browser to be controlled by the Fetcher, and most of the options can be enabled on it.
|
|
|
|
|
|
Again, this is just the tip of the iceberg with this fetcher. Check out the rest from [here](fetching/dynamic.md) for all details and the complete list of arguments.
|
|
|
|
### Dynamic anti-protection loading
|
|
We also have you covered if you deal with dynamic websites with annoying anti-protections!
|
|
|
|
The `StealthyFetcher` class uses a stealthy version of the `DynamicFetcher` explained above.
|
|
|
|
Some of the things it does:
|
|
|
|
1. It easily bypasses all types of Cloudflare's Turnstile/Interstitial automatically.
|
|
2. It bypasses CDP runtime leaks and WebRTC leaks.
|
|
3. It isolates JS execution, removes many Playwright fingerprints, and stops detection through some of the known behaviors that bots do.
|
|
4. It generates canvas noise to prevent fingerprinting through canvas.
|
|
5. It automatically patches known methods to detect running in headless mode and provides an option to defeat timezone mismatch attacks.
|
|
6. and other anti-protection options...
|
|
|
|
```python
|
|
from scrapling.fetchers import StealthyFetcher
|
|
page = StealthyFetcher.fetch('https://www.browserscan.net/bot-detection') # Running headless by default
|
|
page.status == 200 # -> True
|
|
|
|
page = StealthyFetcher.fetch('https://nopecha.com/demo/cloudflare', solve_cloudflare=True) # Solve Cloudflare captcha automatically if presented
|
|
page.status == 200 # -> True
|
|
|
|
page = StealthyFetcher.fetch('https://www.browserscan.net/bot-detection', block_webrtc=True, hide_canvas=True, dns_over_https=True) # and the rest of arguments...
|
|
# The async version of fetch
|
|
page = await StealthyFetcher.async_fetch('https://www.browserscan.net/bot-detection')
|
|
page.status == 200 # -> True
|
|
```
|
|
|
|
Again, this is just the tip of the iceberg with this fetcher. Check out the rest from [here](fetching/stealthy.md) for all details and the complete list of arguments.
|
|
|
|
---
|
|
|
|
That's Scrapling at a glance. If you want to learn more, continue to the next section. |