chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,400 @@
|
||||
# Advanced Adaptive Strategies
|
||||
|
||||
## Overview
|
||||
|
||||
While the default adaptive crawling configuration works well for most use cases, understanding the underlying strategies and scoring mechanisms allows you to fine-tune the crawler for specific domains and requirements.
|
||||
|
||||
## The Three-Layer Scoring System
|
||||
|
||||
### 1. Coverage Score
|
||||
|
||||
Coverage measures how comprehensively your knowledge base covers the query terms and related concepts.
|
||||
|
||||
#### Mathematical Foundation
|
||||
|
||||
```python
|
||||
Coverage(K, Q) = Σ(t ∈ Q) score(t, K) / |Q|
|
||||
|
||||
where score(t, K) = doc_coverage(t) × (1 + freq_boost(t))
|
||||
```
|
||||
|
||||
#### Components
|
||||
|
||||
- **Document Coverage**: Percentage of documents containing the term
|
||||
- **Frequency Boost**: Logarithmic bonus for term frequency
|
||||
- **Query Decomposition**: Handles multi-word queries intelligently
|
||||
|
||||
#### Tuning Coverage
|
||||
|
||||
```python
|
||||
# For technical documentation with specific terminology
|
||||
config = AdaptiveConfig(
|
||||
confidence_threshold=0.85, # Require high coverage
|
||||
top_k_links=5 # Cast wider net
|
||||
)
|
||||
|
||||
# For general topics with synonyms
|
||||
config = AdaptiveConfig(
|
||||
confidence_threshold=0.6, # Lower threshold
|
||||
top_k_links=2 # More focused
|
||||
)
|
||||
```
|
||||
|
||||
### 2. Consistency Score
|
||||
|
||||
Consistency evaluates whether the information across pages is coherent and non-contradictory.
|
||||
|
||||
#### How It Works
|
||||
|
||||
1. Extracts key statements from each document
|
||||
2. Compares statements across documents
|
||||
3. Measures agreement vs. contradiction
|
||||
4. Returns normalized score (0-1)
|
||||
|
||||
#### Practical Impact
|
||||
|
||||
- **High consistency (>0.8)**: Information is reliable and coherent
|
||||
- **Medium consistency (0.5-0.8)**: Some variation, but generally aligned
|
||||
- **Low consistency (<0.5)**: Conflicting information, need more sources
|
||||
|
||||
### 3. Saturation Score
|
||||
|
||||
Saturation detects when new pages stop providing novel information.
|
||||
|
||||
#### Detection Algorithm
|
||||
|
||||
```python
|
||||
# Tracks new unique terms per page
|
||||
new_terms_page_1 = 50
|
||||
new_terms_page_2 = 30 # 60% of first
|
||||
new_terms_page_3 = 15 # 50% of second
|
||||
new_terms_page_4 = 5 # 33% of third
|
||||
# Saturation detected: rapidly diminishing returns
|
||||
```
|
||||
|
||||
#### Configuration
|
||||
|
||||
```python
|
||||
config = AdaptiveConfig(
|
||||
min_gain_threshold=0.1 # Stop if <10% new information
|
||||
)
|
||||
```
|
||||
|
||||
## Link Ranking Algorithm
|
||||
|
||||
### Expected Information Gain
|
||||
|
||||
Each uncrawled link is scored based on:
|
||||
|
||||
```python
|
||||
ExpectedGain(link) = Relevance × Novelty × Authority
|
||||
```
|
||||
|
||||
#### 1. Relevance Scoring
|
||||
|
||||
Uses BM25 algorithm on link preview text:
|
||||
|
||||
```python
|
||||
relevance = BM25(link.preview_text, query)
|
||||
```
|
||||
|
||||
Factors:
|
||||
- Term frequency in preview
|
||||
- Inverse document frequency
|
||||
- Preview length normalization
|
||||
|
||||
#### 2. Novelty Estimation
|
||||
|
||||
Measures how different the link appears from already-crawled content:
|
||||
|
||||
```python
|
||||
novelty = 1 - max_similarity(preview, knowledge_base)
|
||||
```
|
||||
|
||||
Prevents crawling duplicate or highly similar pages.
|
||||
|
||||
#### 3. Authority Calculation
|
||||
|
||||
URL structure and domain analysis:
|
||||
|
||||
```python
|
||||
authority = f(domain_rank, url_depth, url_structure)
|
||||
```
|
||||
|
||||
Factors:
|
||||
- Domain reputation
|
||||
- URL depth (fewer slashes = higher authority)
|
||||
- Clean URL structure
|
||||
|
||||
## Domain-Specific Configurations
|
||||
|
||||
### Technical Documentation
|
||||
|
||||
```python
|
||||
tech_doc_config = AdaptiveConfig(
|
||||
confidence_threshold=0.85,
|
||||
max_pages=30,
|
||||
top_k_links=3,
|
||||
min_gain_threshold=0.05 # Keep crawling for small gains
|
||||
)
|
||||
```
|
||||
|
||||
Rationale:
|
||||
- High threshold ensures comprehensive coverage
|
||||
- Lower gain threshold captures edge cases
|
||||
- Moderate link following for depth
|
||||
|
||||
### News & Articles
|
||||
|
||||
```python
|
||||
news_config = AdaptiveConfig(
|
||||
confidence_threshold=0.6,
|
||||
max_pages=10,
|
||||
top_k_links=5,
|
||||
min_gain_threshold=0.15 # Stop quickly on repetition
|
||||
)
|
||||
```
|
||||
|
||||
Rationale:
|
||||
- Lower threshold (articles often repeat information)
|
||||
- Higher gain threshold (avoid duplicate stories)
|
||||
- More links per page (explore different perspectives)
|
||||
|
||||
### E-commerce
|
||||
|
||||
```python
|
||||
ecommerce_config = AdaptiveConfig(
|
||||
confidence_threshold=0.7,
|
||||
max_pages=20,
|
||||
top_k_links=2,
|
||||
min_gain_threshold=0.1
|
||||
)
|
||||
```
|
||||
|
||||
Rationale:
|
||||
- Balanced threshold for product variations
|
||||
- Focused link following (avoid infinite products)
|
||||
- Standard gain threshold
|
||||
|
||||
### Research & Academic
|
||||
|
||||
```python
|
||||
research_config = AdaptiveConfig(
|
||||
confidence_threshold=0.9,
|
||||
max_pages=50,
|
||||
top_k_links=4,
|
||||
min_gain_threshold=0.02 # Very low - capture citations
|
||||
)
|
||||
```
|
||||
|
||||
Rationale:
|
||||
- Very high threshold for completeness
|
||||
- Many pages allowed for thorough research
|
||||
- Very low gain threshold to capture references
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
### Memory Management
|
||||
|
||||
```python
|
||||
# For large crawls, use streaming
|
||||
config = AdaptiveConfig(
|
||||
max_pages=100,
|
||||
save_state=True,
|
||||
state_path="large_crawl.json"
|
||||
)
|
||||
|
||||
# Periodically clean state
|
||||
if len(state.knowledge_base) > 1000:
|
||||
# Keep only the top 500 most relevant docs
|
||||
top_content = adaptive.get_relevant_content(top_k=500)
|
||||
keep_indices = {d["index"] for d in top_content}
|
||||
state.knowledge_base = [
|
||||
doc for i, doc in enumerate(state.knowledge_base) if i in keep_indices
|
||||
]
|
||||
```
|
||||
|
||||
### Parallel Processing
|
||||
|
||||
```python
|
||||
# Use multiple start points
|
||||
start_urls = [
|
||||
"https://docs.example.com/intro",
|
||||
"https://docs.example.com/api",
|
||||
"https://docs.example.com/guides"
|
||||
]
|
||||
|
||||
# Crawl in parallel
|
||||
tasks = [
|
||||
adaptive.digest(url, query)
|
||||
for url in start_urls
|
||||
]
|
||||
results = await asyncio.gather(*tasks)
|
||||
```
|
||||
|
||||
## Debugging & Analysis
|
||||
|
||||
### Enable Verbose Logging
|
||||
|
||||
```python
|
||||
import logging
|
||||
|
||||
logging.basicConfig(level=logging.DEBUG)
|
||||
adaptive = AdaptiveCrawler(crawler, config, verbose=True)
|
||||
```
|
||||
|
||||
### Analyze Crawl Patterns
|
||||
|
||||
```python
|
||||
# After crawling
|
||||
state = await adaptive.digest(start_url, query)
|
||||
|
||||
# Analyze link selection
|
||||
print("Link selection order:")
|
||||
for i, url in enumerate(state.crawl_order):
|
||||
print(f"{i+1}. {url}")
|
||||
|
||||
# Analyze term discovery
|
||||
print("\nTerm discovery rate:")
|
||||
for i, new_terms in enumerate(state.new_terms_history):
|
||||
print(f"Page {i+1}: {new_terms} new terms")
|
||||
|
||||
# Analyze score progression
|
||||
print("\nScore progression:")
|
||||
print(f"Coverage: {state.metrics['coverage_history']}")
|
||||
print(f"Saturation: {state.metrics['saturation_history']}")
|
||||
```
|
||||
|
||||
### Export for Analysis
|
||||
|
||||
```python
|
||||
# Export detailed metrics
|
||||
import json
|
||||
|
||||
metrics = {
|
||||
"query": query,
|
||||
"total_pages": len(state.crawled_urls),
|
||||
"confidence": adaptive.confidence,
|
||||
"coverage_stats": adaptive.coverage_stats,
|
||||
"crawl_order": state.crawl_order,
|
||||
"term_frequencies": dict(state.term_frequencies),
|
||||
"new_terms_history": state.new_terms_history
|
||||
}
|
||||
|
||||
with open("crawl_analysis.json", "w") as f:
|
||||
json.dump(metrics, f, indent=2)
|
||||
```
|
||||
|
||||
## Custom Strategies
|
||||
|
||||
### Implementing a Custom Strategy
|
||||
|
||||
```python
|
||||
from crawl4ai.adaptive_crawler import CrawlStrategy
|
||||
|
||||
class DomainSpecificStrategy(CrawlStrategy):
|
||||
def calculate_coverage(self, state: CrawlState) -> float:
|
||||
# Custom coverage calculation
|
||||
# e.g., weight certain terms more heavily
|
||||
pass
|
||||
|
||||
def calculate_consistency(self, state: CrawlState) -> float:
|
||||
# Custom consistency logic
|
||||
# e.g., domain-specific validation
|
||||
pass
|
||||
|
||||
def rank_links(self, links: List[Link], state: CrawlState) -> List[Link]:
|
||||
# Custom link ranking
|
||||
# e.g., prioritize specific URL patterns
|
||||
pass
|
||||
|
||||
# Use custom strategy
|
||||
adaptive = AdaptiveCrawler(
|
||||
crawler,
|
||||
config=config,
|
||||
strategy=DomainSpecificStrategy()
|
||||
)
|
||||
```
|
||||
|
||||
### Combining Strategies
|
||||
|
||||
```python
|
||||
class HybridStrategy(CrawlStrategy):
|
||||
def __init__(self):
|
||||
self.strategies = [
|
||||
TechnicalDocStrategy(),
|
||||
SemanticSimilarityStrategy(),
|
||||
URLPatternStrategy()
|
||||
]
|
||||
|
||||
def calculate_confidence(self, state: CrawlState) -> float:
|
||||
# Weighted combination of strategies
|
||||
scores = [s.calculate_confidence(state) for s in self.strategies]
|
||||
weights = [0.5, 0.3, 0.2]
|
||||
return sum(s * w for s, w in zip(scores, weights))
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### 1. Start Conservative
|
||||
|
||||
Begin with default settings and adjust based on results:
|
||||
|
||||
```python
|
||||
# Start with defaults
|
||||
result = await adaptive.digest(url, query)
|
||||
|
||||
# Analyze and adjust
|
||||
if adaptive.confidence < 0.7:
|
||||
config.max_pages += 10
|
||||
config.confidence_threshold -= 0.1
|
||||
```
|
||||
|
||||
### 2. Monitor Resource Usage
|
||||
|
||||
```python
|
||||
import psutil
|
||||
|
||||
# Check memory before large crawls
|
||||
memory_percent = psutil.virtual_memory().percent
|
||||
if memory_percent > 80:
|
||||
config.max_pages = min(config.max_pages, 20)
|
||||
```
|
||||
|
||||
### 3. Use Domain Knowledge
|
||||
|
||||
```python
|
||||
# For API documentation
|
||||
if "api" in start_url:
|
||||
config.top_k_links = 2 # APIs have clear structure
|
||||
|
||||
# For blogs
|
||||
if "blog" in start_url:
|
||||
config.min_gain_threshold = 0.2 # Avoid similar posts
|
||||
```
|
||||
|
||||
### 4. Validate Results
|
||||
|
||||
```python
|
||||
# Always validate the knowledge base
|
||||
relevant_content = adaptive.get_relevant_content(top_k=10)
|
||||
|
||||
# Check coverage
|
||||
query_terms = set(query.lower().split())
|
||||
covered_terms = set()
|
||||
|
||||
for doc in relevant_content:
|
||||
content_lower = doc['content'].lower()
|
||||
for term in query_terms:
|
||||
if term in content_lower:
|
||||
covered_terms.add(term)
|
||||
|
||||
coverage_ratio = len(covered_terms) / len(query_terms)
|
||||
print(f"Query term coverage: {coverage_ratio:.0%}")
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
- Explore [Custom Strategy Implementation](../tutorials/custom-adaptive-strategies.md)
|
||||
- Learn about [Knowledge Base Management](../tutorials/knowledge-base-management.md)
|
||||
- See [Performance Benchmarks](../benchmarks/adaptive-performance.md)
|
||||
@@ -0,0 +1,446 @@
|
||||
# Overview of Some Important Advanced Features
|
||||
(Proxy, PDF, Screenshot, SSL, Headers, & Storage State)
|
||||
|
||||
Crawl4AI offers multiple power-user features that go beyond simple crawling. This tutorial covers:
|
||||
|
||||
1. **Proxy Usage**
|
||||
2. **Capturing PDFs & Screenshots**
|
||||
3. **Handling SSL Certificates**
|
||||
4. **Custom Headers**
|
||||
5. **Session Persistence & Local Storage**
|
||||
6. **Robots.txt Compliance**
|
||||
|
||||
> **Prerequisites**
|
||||
> - You have a basic grasp of [AsyncWebCrawler Basics](../core/simple-crawling.md)
|
||||
> - You know how to run or configure your Python environment with Playwright installed
|
||||
|
||||
---
|
||||
|
||||
## 1. Proxy Usage
|
||||
|
||||
If you need to route your crawl traffic through a proxy—whether for IP rotation, geo-testing, or privacy—Crawl4AI supports it via `BrowserConfig.proxy_config`.
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig
|
||||
|
||||
async def main():
|
||||
browser_cfg = BrowserConfig(
|
||||
proxy_config={
|
||||
"server": "http://proxy.example.com:8080",
|
||||
"username": "myuser",
|
||||
"password": "mypass",
|
||||
},
|
||||
headless=True
|
||||
)
|
||||
crawler_cfg = CrawlerRunConfig(
|
||||
verbose=True
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(config=browser_cfg) as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://www.whatismyip.com/",
|
||||
config=crawler_cfg
|
||||
)
|
||||
if result.success:
|
||||
print("[OK] Page fetched via proxy.")
|
||||
print("Page HTML snippet:", result.html[:200])
|
||||
else:
|
||||
print("[ERROR]", result.error_message)
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
**Key Points**
|
||||
- **`proxy_config`** expects a dict with `server` and optional auth credentials.
|
||||
- Many commercial proxies provide an HTTP/HTTPS “gateway” server that you specify in `server`.
|
||||
- If your proxy doesn’t need auth, omit `username`/`password`.
|
||||
|
||||
---
|
||||
|
||||
## 2. Capturing PDFs & Screenshots
|
||||
|
||||
Sometimes you need a visual record of a page or a PDF “printout.” Crawl4AI can do both in one pass:
|
||||
|
||||
```python
|
||||
import os, asyncio
|
||||
from base64 import b64decode
|
||||
from crawl4ai import AsyncWebCrawler, CacheMode, CrawlerRunConfig
|
||||
|
||||
async def main():
|
||||
run_config = CrawlerRunConfig(
|
||||
cache_mode=CacheMode.BYPASS,
|
||||
screenshot=True,
|
||||
pdf=True
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://en.wikipedia.org/wiki/List_of_common_misconceptions",
|
||||
config=run_config
|
||||
)
|
||||
if result.success:
|
||||
print(f"Screenshot data present: {result.screenshot is not None}")
|
||||
print(f"PDF data present: {result.pdf is not None}")
|
||||
|
||||
if result.screenshot:
|
||||
print(f"[OK] Screenshot captured, size: {len(result.screenshot)} bytes")
|
||||
with open("wikipedia_screenshot.png", "wb") as f:
|
||||
f.write(b64decode(result.screenshot))
|
||||
else:
|
||||
print("[WARN] Screenshot data is None.")
|
||||
|
||||
if result.pdf:
|
||||
print(f"[OK] PDF captured, size: {len(result.pdf)} bytes")
|
||||
with open("wikipedia_page.pdf", "wb") as f:
|
||||
f.write(result.pdf)
|
||||
else:
|
||||
print("[WARN] PDF data is None.")
|
||||
|
||||
else:
|
||||
print("[ERROR]", result.error_message)
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
**Why PDF + Screenshot?**
|
||||
- Large or complex pages can be slow or error-prone with “traditional” full-page screenshots.
|
||||
- Exporting a PDF is more reliable for very long pages. Crawl4AI automatically converts the first PDF page into an image if you request both.
|
||||
|
||||
**Relevant Parameters**
|
||||
- **`pdf=True`**: Exports the current page as a PDF (base64-encoded in `result.pdf`).
|
||||
- **`screenshot=True`**: Creates a screenshot (base64-encoded in `result.screenshot`).
|
||||
- **`scroll_delay`**: Controls the delay (seconds) between scroll steps when taking a full-page screenshot of a tall page. Defaults to `0.2`. Increase for pages with slow-loading assets.
|
||||
- **`scan_full_page`** or advanced hooking can further refine how the crawler captures content.
|
||||
|
||||
---
|
||||
|
||||
## 3. Handling SSL Certificates
|
||||
|
||||
If you need to verify or export a site’s SSL certificate—for compliance, debugging, or data analysis—Crawl4AI can fetch it during the crawl:
|
||||
|
||||
```python
|
||||
import asyncio, os
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
|
||||
|
||||
async def main():
|
||||
tmp_dir = os.path.join(os.getcwd(), "tmp")
|
||||
os.makedirs(tmp_dir, exist_ok=True)
|
||||
|
||||
config = CrawlerRunConfig(
|
||||
fetch_ssl_certificate=True,
|
||||
cache_mode=CacheMode.BYPASS
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(url="https://example.com", config=config)
|
||||
|
||||
if result.success and result.ssl_certificate:
|
||||
cert = result.ssl_certificate
|
||||
print("\nCertificate Information:")
|
||||
print(f"Issuer (CN): {cert.issuer.get('CN', '')}")
|
||||
print(f"Valid until: {cert.valid_until}")
|
||||
print(f"Fingerprint: {cert.fingerprint}")
|
||||
|
||||
# Export in multiple formats:
|
||||
cert.to_json(os.path.join(tmp_dir, "certificate.json"))
|
||||
cert.to_pem(os.path.join(tmp_dir, "certificate.pem"))
|
||||
cert.to_der(os.path.join(tmp_dir, "certificate.der"))
|
||||
|
||||
print("\nCertificate exported to JSON/PEM/DER in 'tmp' folder.")
|
||||
else:
|
||||
print("[ERROR] No certificate or crawl failed.")
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
**Key Points**
|
||||
- **`fetch_ssl_certificate=True`** triggers certificate retrieval.
|
||||
- `result.ssl_certificate` includes methods (`to_json`, `to_pem`, `to_der`) for saving in various formats (handy for server config, Java keystores, etc.).
|
||||
|
||||
---
|
||||
|
||||
## 4. Custom Headers
|
||||
|
||||
Sometimes you need to set custom headers (e.g., language preferences, authentication tokens, or specialized user-agent strings). You can do this in multiple ways:
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
|
||||
async def main():
|
||||
# Option 1: Set headers at the crawler strategy level
|
||||
crawler1 = AsyncWebCrawler(
|
||||
# The underlying strategy can accept headers in its constructor
|
||||
crawler_strategy=None # We'll override below for clarity
|
||||
)
|
||||
crawler1.crawler_strategy.update_user_agent("MyCustomUA/1.0")
|
||||
crawler1.crawler_strategy.set_custom_headers({
|
||||
"Accept-Language": "fr-FR,fr;q=0.9"
|
||||
})
|
||||
result1 = await crawler1.arun("https://www.example.com")
|
||||
print("Example 1 result success:", result1.success)
|
||||
|
||||
# Option 2: Pass headers directly to `arun()`
|
||||
crawler2 = AsyncWebCrawler()
|
||||
result2 = await crawler2.arun(
|
||||
url="https://www.example.com",
|
||||
headers={"Accept-Language": "es-ES,es;q=0.9"}
|
||||
)
|
||||
print("Example 2 result success:", result2.success)
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
**Notes**
|
||||
- Some sites may react differently to certain headers (e.g., `Accept-Language`).
|
||||
- If you need advanced user-agent randomization or client hints, see [Identity-Based Crawling (Anti-Bot)](./identity-based-crawling.md) or use `UserAgentGenerator`.
|
||||
|
||||
---
|
||||
|
||||
## 5. Session Persistence & Local Storage
|
||||
|
||||
Crawl4AI can preserve cookies and localStorage so you can continue where you left off—ideal for logging into sites or skipping repeated auth flows.
|
||||
|
||||
### 5.1 `storage_state`
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
|
||||
async def main():
|
||||
storage_dict = {
|
||||
"cookies": [
|
||||
{
|
||||
"name": "session",
|
||||
"value": "abcd1234",
|
||||
"domain": "example.com",
|
||||
"path": "/",
|
||||
"expires": 1699999999.0,
|
||||
"httpOnly": False,
|
||||
"secure": False,
|
||||
"sameSite": "None"
|
||||
}
|
||||
],
|
||||
"origins": [
|
||||
{
|
||||
"origin": "https://example.com",
|
||||
"localStorage": [
|
||||
{"name": "token", "value": "my_auth_token"}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
# Provide the storage state as a dictionary to start "already logged in"
|
||||
async with AsyncWebCrawler(
|
||||
headless=True,
|
||||
storage_state=storage_dict
|
||||
) as crawler:
|
||||
result = await crawler.arun("https://example.com/protected")
|
||||
if result.success:
|
||||
print("Protected page content length:", len(result.html))
|
||||
else:
|
||||
print("Failed to crawl protected page")
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### 5.2 Exporting & Reusing State
|
||||
|
||||
You can sign in once, export the browser context, and reuse it later—without re-entering credentials.
|
||||
|
||||
- **`await context.storage_state(path="my_storage.json")`**: Exports cookies, localStorage, etc. to a file.
|
||||
- Provide `storage_state="my_storage.json"` on subsequent runs to skip the login step.
|
||||
|
||||
**See**: [Detailed session management tutorial](./session-management.md) or [Explanations → Browser Context & Managed Browser](./identity-based-crawling.md) for more advanced scenarios (like multi-step logins, or capturing after interactive pages).
|
||||
|
||||
---
|
||||
|
||||
## 6. Robots.txt Compliance
|
||||
|
||||
Crawl4AI supports respecting robots.txt rules with efficient caching:
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
|
||||
|
||||
async def main():
|
||||
# Enable robots.txt checking in config
|
||||
config = CrawlerRunConfig(
|
||||
check_robots_txt=True # Will check and respect robots.txt rules
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
"https://example.com",
|
||||
config=config
|
||||
)
|
||||
|
||||
if not result.success and result.status_code == 403:
|
||||
print("Access denied by robots.txt")
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
**Key Points**
|
||||
- Robots.txt files are cached locally for efficiency
|
||||
- Cache is stored in `~/.crawl4ai/robots/robots_cache.db`
|
||||
- Cache has a default TTL of 7 days
|
||||
- If robots.txt can't be fetched, crawling is allowed
|
||||
- Returns 403 status code if URL is disallowed
|
||||
|
||||
---
|
||||
|
||||
## Putting It All Together
|
||||
|
||||
Here’s a snippet that combines multiple “advanced” features (proxy, PDF, screenshot, SSL, custom headers, and session reuse) into one run. Normally, you’d tailor each setting to your project’s needs.
|
||||
|
||||
```python
|
||||
import os, asyncio
|
||||
from base64 import b64decode
|
||||
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
|
||||
|
||||
async def main():
|
||||
# 1. Browser config with proxy + headless
|
||||
browser_cfg = BrowserConfig(
|
||||
proxy_config={
|
||||
"server": "http://proxy.example.com:8080",
|
||||
"username": "myuser",
|
||||
"password": "mypass",
|
||||
},
|
||||
headless=True,
|
||||
)
|
||||
|
||||
# 2. Crawler config with PDF, screenshot, SSL, custom headers, and ignoring caches
|
||||
crawler_cfg = CrawlerRunConfig(
|
||||
pdf=True,
|
||||
screenshot=True,
|
||||
fetch_ssl_certificate=True,
|
||||
cache_mode=CacheMode.BYPASS,
|
||||
headers={"Accept-Language": "en-US,en;q=0.8"},
|
||||
storage_state="my_storage.json", # Reuse session from a previous sign-in
|
||||
verbose=True,
|
||||
)
|
||||
|
||||
# 3. Crawl
|
||||
async with AsyncWebCrawler(config=browser_cfg) as crawler:
|
||||
result = await crawler.arun(
|
||||
url = "https://secure.example.com/protected",
|
||||
config=crawler_cfg
|
||||
)
|
||||
|
||||
if result.success:
|
||||
print("[OK] Crawled the secure page. Links found:", len(result.links.get("internal", [])))
|
||||
|
||||
# Save PDF & screenshot
|
||||
if result.pdf:
|
||||
with open("result.pdf", "wb") as f:
|
||||
f.write(b64decode(result.pdf))
|
||||
if result.screenshot:
|
||||
with open("result.png", "wb") as f:
|
||||
f.write(b64decode(result.screenshot))
|
||||
|
||||
# Check SSL cert
|
||||
if result.ssl_certificate:
|
||||
print("SSL Issuer CN:", result.ssl_certificate.issuer.get("CN", ""))
|
||||
else:
|
||||
print("[ERROR]", result.error_message)
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
## 7. Anti-Bot Features (Stealth Mode & Undetected Browser)
|
||||
|
||||
Crawl4AI provides two powerful features to bypass bot detection:
|
||||
|
||||
### 7.1 Stealth Mode
|
||||
|
||||
Stealth mode uses playwright-stealth to modify browser fingerprints and behaviors. Enable it with a simple flag:
|
||||
|
||||
```python
|
||||
browser_config = BrowserConfig(
|
||||
enable_stealth=True, # Activates stealth mode
|
||||
headless=False
|
||||
)
|
||||
```
|
||||
|
||||
**When to use**: Sites with basic bot detection (checking navigator.webdriver, plugins, etc.)
|
||||
|
||||
### 7.2 Undetected Browser
|
||||
|
||||
For advanced bot detection, use the undetected browser adapter:
|
||||
|
||||
```python
|
||||
from crawl4ai import UndetectedAdapter
|
||||
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy
|
||||
|
||||
# Create undetected adapter
|
||||
adapter = UndetectedAdapter()
|
||||
strategy = AsyncPlaywrightCrawlerStrategy(
|
||||
browser_config=browser_config,
|
||||
browser_adapter=adapter
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(crawler_strategy=strategy, config=browser_config) as crawler:
|
||||
# Your crawling code
|
||||
```
|
||||
|
||||
**When to use**: Sites with sophisticated bot detection (Cloudflare, DataDome, etc.)
|
||||
|
||||
### 7.3 Combining Both
|
||||
|
||||
For maximum evasion, combine stealth mode with undetected browser:
|
||||
|
||||
```python
|
||||
browser_config = BrowserConfig(
|
||||
enable_stealth=True, # Enable stealth
|
||||
headless=False
|
||||
)
|
||||
|
||||
adapter = UndetectedAdapter() # Use undetected browser
|
||||
```
|
||||
|
||||
### Choosing the Right Approach
|
||||
|
||||
| Detection Level | Recommended Approach |
|
||||
|----------------|---------------------|
|
||||
| No protection | Regular browser |
|
||||
| Basic checks | Regular + Stealth mode |
|
||||
| Advanced protection | Undetected browser |
|
||||
| Maximum evasion | Undetected + Stealth mode |
|
||||
|
||||
**Best Practice**: Start with regular browser + stealth mode. Only use undetected browser if needed, as it may be slightly slower.
|
||||
|
||||
See [Undetected Browser Mode](undetected-browser.md) for detailed examples.
|
||||
|
||||
---
|
||||
|
||||
## Conclusion & Next Steps
|
||||
|
||||
You've now explored several **advanced** features:
|
||||
|
||||
- **Proxy Usage**
|
||||
- **PDF & Screenshot** capturing for large or critical pages
|
||||
- **SSL Certificate** retrieval & exporting
|
||||
- **Custom Headers** for language or specialized requests
|
||||
- **Session Persistence** via storage state
|
||||
- **Robots.txt Compliance**
|
||||
- **Anti-Bot Features** (Stealth Mode & Undetected Browser)
|
||||
|
||||
With these power tools, you can build robust scraping workflows that mimic real user behavior, handle secure sites, capture detailed snapshots, manage sessions across multiple runs, and bypass bot detection—streamlining your entire data collection pipeline.
|
||||
|
||||
**Note**: In future versions, we may enable stealth mode and undetected browser by default. For now, users should explicitly enable these features when needed.
|
||||
|
||||
**Last Updated**: 2025-01-17
|
||||
@@ -0,0 +1,263 @@
|
||||
# Anti-Bot Detection & Fallback
|
||||
|
||||
When crawling sites protected by anti-bot systems (Akamai, Cloudflare, PerimeterX, DataDome, Imperva, etc.), requests often get blocked with CAPTCHAs, 403 responses, or empty pages. Crawl4AI provides a layered retry and fallback system that automatically detects blocking and escalates through multiple strategies until content is retrieved.
|
||||
|
||||
## How Detection Works
|
||||
|
||||
After each crawl attempt, Crawl4AI inspects the HTTP status code and HTML content for known anti-bot signals:
|
||||
|
||||
- **HTTP 403/429** with short or empty response bodies
|
||||
- **Challenge pages** — Cloudflare "Just a moment", Akamai "Access Denied", PerimeterX block pages
|
||||
- **CAPTCHA injection** — reCAPTCHA, hCaptcha, or vendor-specific challenges on otherwise empty pages
|
||||
- **Firewall blocks** — Imperva/Incapsula resource iframes, Sucuri firewall pages, Cloudflare error codes
|
||||
|
||||
Detection uses structural HTML markers (specific element IDs, script sources, form actions) rather than generic keywords to minimize false positives. A normal page that happens to mention "CAPTCHA" or "Cloudflare" in its content will not be flagged.
|
||||
|
||||
When all attempts fail and blocking is still detected, the result is returned with `success=False` and `error_message` describing the block reason.
|
||||
|
||||
## Configuration Options
|
||||
|
||||
All anti-bot retry options live on `CrawlerRunConfig`:
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|---|---|---|---|
|
||||
| `proxy_config` | `ProxyConfig`, `list[ProxyConfig]`, or `None` | `None` | Single proxy or ordered list of proxies to try. Each retry round iterates through the full list. Use `"direct"` or `ProxyConfig.DIRECT` in a list to explicitly try without a proxy. |
|
||||
| `max_retries` | `int` | `0` | Number of retry rounds when blocking is detected. `0` = no retries. |
|
||||
| `fallback_fetch_function` | `async (str) -> str` | `None` | Async function called as last resort. Takes URL, returns raw HTML. |
|
||||
|
||||
## Escalation Chain
|
||||
|
||||
Each retry round tries every proxy in `proxy_config` in order. If all rounds are exhausted and the page is still blocked, the fallback fetch function is called as a last resort.
|
||||
|
||||
```
|
||||
For each round (1 + max_retries rounds):
|
||||
1. Try proxy_config[0] (or direct if proxy_config is None)
|
||||
2. If blocked → try proxy_config[1]
|
||||
3. If blocked → try proxy_config[2]
|
||||
4. ... continue through all proxies
|
||||
5. If any attempt succeeds → done
|
||||
|
||||
If all rounds exhausted and still blocked:
|
||||
6. Call fallback_fetch_function(url) → process returned HTML
|
||||
```
|
||||
|
||||
Worst-case attempts before the fetch function: `(1 + max_retries) x len(proxy_config)`
|
||||
|
||||
## Crawl Stats
|
||||
|
||||
Every crawl result includes a `crawl_stats` dict with detailed attempt tracking:
|
||||
|
||||
```python
|
||||
result.crawl_stats = {
|
||||
"attempts": 3, # total browser attempts made
|
||||
"retries": 1, # retry rounds used (0 = succeeded first round)
|
||||
"proxies_used": [ # ordered list of every attempt
|
||||
{"proxy": None, "status_code": 403, "blocked": True, "reason": "Akamai block (Reference #)"},
|
||||
{"proxy": "proxy.io:8080", "status_code": 403, "blocked": True, "reason": "Akamai block (Reference #)"},
|
||||
{"proxy": "premium.io:9090", "status_code": 200, "blocked": False, "reason": ""},
|
||||
],
|
||||
"fallback_fetch_used": False, # whether fallback_fetch_function was called
|
||||
"resolved_by": "proxy", # "direct" | "proxy" | "fallback_fetch" | null (all failed)
|
||||
}
|
||||
```
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Simple Retry (No Proxy)
|
||||
|
||||
Retry the crawl up to 3 times when blocking is detected. Useful when blocks are intermittent or IP-based.
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig
|
||||
|
||||
async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
config=CrawlerRunConfig(max_retries=3),
|
||||
)
|
||||
```
|
||||
|
||||
### Single Proxy
|
||||
|
||||
Pass a single `ProxyConfig` — it's used on every attempt. Same behavior as always.
|
||||
|
||||
```python
|
||||
from crawl4ai.async_configs import ProxyConfig
|
||||
|
||||
config = CrawlerRunConfig(
|
||||
max_retries=2,
|
||||
proxy_config=ProxyConfig(
|
||||
server="http://proxy.example.com:8080",
|
||||
username="user",
|
||||
password="pass",
|
||||
),
|
||||
)
|
||||
```
|
||||
|
||||
### Direct-First, Then Proxies
|
||||
|
||||
Try without a proxy first, then escalate to proxies if blocked. Use `ProxyConfig.DIRECT` (or the string `"direct"`) in the list to represent a no-proxy attempt.
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(
|
||||
max_retries=1,
|
||||
proxy_config=[
|
||||
ProxyConfig.DIRECT, # Try without proxy first
|
||||
ProxyConfig(
|
||||
server="http://datacenter-proxy.example.com:8080",
|
||||
username="user",
|
||||
password="pass",
|
||||
),
|
||||
ProxyConfig(
|
||||
server="http://residential-proxy.example.com:9090",
|
||||
username="user",
|
||||
password="pass",
|
||||
),
|
||||
],
|
||||
)
|
||||
```
|
||||
|
||||
With this setup, each round tries direct first, then datacenter, then residential. With `max_retries=1`, worst case is 2 rounds x 3 steps = 6 attempts.
|
||||
|
||||
### Proxy List (Escalation)
|
||||
|
||||
Pass a list of proxies. They're tried in order — first one that works wins. Within each retry round, the entire list is tried again.
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(
|
||||
max_retries=1,
|
||||
proxy_config=[
|
||||
ProxyConfig(
|
||||
server="http://datacenter-proxy.example.com:8080",
|
||||
username="user",
|
||||
password="pass",
|
||||
),
|
||||
ProxyConfig(
|
||||
server="http://residential-proxy.example.com:9090",
|
||||
username="user",
|
||||
password="pass",
|
||||
),
|
||||
],
|
||||
)
|
||||
```
|
||||
|
||||
With this setup, each round tries the datacenter proxy first, then the residential proxy. With `max_retries=1`, worst case is 2 rounds x 2 proxies = 4 attempts.
|
||||
|
||||
### Fallback Fetch Function
|
||||
|
||||
When all browser-based attempts fail, call a custom async function as a last resort. This function receives the URL and must return raw HTML as a string. The returned HTML is processed through the normal pipeline (markdown generation, extraction, etc.).
|
||||
|
||||
This is useful when you have access to a scraping API, a pre-fetched cache, or any other source of HTML.
|
||||
|
||||
```python
|
||||
import aiohttp
|
||||
|
||||
async def my_scraping_api(url: str) -> str:
|
||||
"""Fetch HTML via an external scraping API."""
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.get(
|
||||
"https://api.my-scraping-service.com/fetch",
|
||||
params={"url": url, "format": "html"},
|
||||
headers={"Authorization": "Bearer MY_TOKEN"},
|
||||
) as resp:
|
||||
if resp.status == 200:
|
||||
return await resp.text()
|
||||
raise RuntimeError(f"API error: {resp.status}")
|
||||
|
||||
config = CrawlerRunConfig(
|
||||
max_retries=1,
|
||||
fallback_fetch_function=my_scraping_api,
|
||||
)
|
||||
```
|
||||
|
||||
The function can do anything — call an API, read from a database, return cached HTML, or make a simple HTTP request with a different library. Crawl4AI does not care how the HTML is obtained.
|
||||
|
||||
### Full Escalation (All Features Combined)
|
||||
|
||||
This example combines every layer: stealth mode, a list of proxies tried in order, retries, and a final fetch function.
|
||||
|
||||
```python
|
||||
import aiohttp
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig, ProxyConfig
|
||||
|
||||
# Last-resort: fetch HTML via an external service
|
||||
async def external_fetch(url: str) -> str:
|
||||
async with aiohttp.ClientSession() as session:
|
||||
async with session.post(
|
||||
"https://api.my-service.com/scrape",
|
||||
json={"url": url, "render_js": True},
|
||||
headers={"Authorization": "Bearer MY_TOKEN"},
|
||||
) as resp:
|
||||
return await resp.text()
|
||||
|
||||
browser_config = BrowserConfig(
|
||||
headless=True,
|
||||
enable_stealth=True,
|
||||
)
|
||||
|
||||
crawl_config = CrawlerRunConfig(
|
||||
magic=True,
|
||||
wait_until="load",
|
||||
max_retries=2,
|
||||
|
||||
# Proxies tried in order — cheapest first
|
||||
proxy_config=[
|
||||
ProxyConfig(
|
||||
server="http://datacenter-proxy.example.com:8080",
|
||||
username="user",
|
||||
password="pass",
|
||||
),
|
||||
ProxyConfig(
|
||||
server="http://residential-proxy.example.com:9090",
|
||||
username="user",
|
||||
password="pass",
|
||||
),
|
||||
],
|
||||
|
||||
# Last resort — called after all retries and proxies are exhausted
|
||||
fallback_fetch_function=external_fetch,
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://protected-site.com/products",
|
||||
config=crawl_config,
|
||||
)
|
||||
|
||||
if result.success:
|
||||
print(f"Got {len(result.markdown.raw_markdown)} chars of markdown")
|
||||
print(f"Resolved by: {result.crawl_stats['resolved_by']}")
|
||||
print(f"Attempts: {result.crawl_stats['attempts']}")
|
||||
else:
|
||||
print(f"All attempts failed: {result.error_message}")
|
||||
```
|
||||
|
||||
**What happens step by step:**
|
||||
|
||||
| Round | Attempt | What runs |
|
||||
|---|---|---|
|
||||
| 1 | 1 | Datacenter proxy — blocked |
|
||||
| 1 | 2 | Residential proxy — blocked |
|
||||
| 2 | 1 | Datacenter proxy — blocked |
|
||||
| 2 | 2 | Residential proxy — blocked |
|
||||
| 3 | 1 | Datacenter proxy — blocked |
|
||||
| 3 | 2 | Residential proxy — blocked |
|
||||
| - | - | `external_fetch(url)` called — returns HTML |
|
||||
|
||||
That's up to 6 browser attempts + 1 function call before giving up.
|
||||
|
||||
## Tips
|
||||
|
||||
- **Start with `max_retries=0`** and a `fallback_fetch_function` if you just want a safety net without burning time on retries.
|
||||
- **Order proxies cheapest-first** — datacenter proxies before residential, residential before premium.
|
||||
- **Combine with stealth mode** — `BrowserConfig(enable_stealth=True)` and `CrawlerRunConfig(magic=True)` reduce the chance of being blocked in the first place.
|
||||
- **`wait_until="load"`** is important for anti-bot sites — the default `domcontentloaded` can return before the anti-bot sensor finishes.
|
||||
- **Check `crawl_stats`** to understand what happened — how many attempts, which proxy worked, whether the fallback function was needed.
|
||||
|
||||
## See Also
|
||||
|
||||
- [Proxy & Security](proxy-security.md) — Proxy setup, authentication, and rotation
|
||||
- [Undetected Browser](undetected-browser.md) — Stealth mode and browser fingerprint evasion
|
||||
- [Session Management](session-management.md) — Maintaining sessions across requests
|
||||
@@ -0,0 +1,12 @@
|
||||
# Crawl Dispatcher
|
||||
|
||||
We’re excited to announce a **Crawl Dispatcher** module that can handle **thousands** of crawling tasks simultaneously. By efficiently managing system resources (memory, CPU, network), this dispatcher ensures high-performance data extraction at scale. It also provides **real-time monitoring** of each crawler’s status, memory usage, and overall progress.
|
||||
|
||||
Stay tuned—this feature is **coming soon** in an upcoming release of Crawl4AI! For the latest news, keep an eye on our changelogs and follow [@unclecode](https://twitter.com/unclecode) on X.
|
||||
|
||||
Below is a **sample** of how the dispatcher’s performance monitor might look in action:
|
||||
|
||||
|
||||
|
||||
|
||||
We can’t wait to bring you this streamlined, **scalable** approach to multi-URL crawling—**watch this space** for updates!
|
||||
@@ -0,0 +1,118 @@
|
||||
# Download Handling in Crawl4AI
|
||||
|
||||
This guide explains how to use Crawl4AI to handle file downloads during crawling. You'll learn how to trigger downloads, specify download locations, and access downloaded files.
|
||||
|
||||
## Enabling Downloads
|
||||
|
||||
To enable downloads, set the `accept_downloads` parameter in the `BrowserConfig` object and pass it to the crawler.
|
||||
|
||||
```python
|
||||
from crawl4ai.async_configs import BrowserConfig, AsyncWebCrawler
|
||||
|
||||
async def main():
|
||||
config = BrowserConfig(accept_downloads=True) # Enable downloads globally
|
||||
async with AsyncWebCrawler(config=config) as crawler:
|
||||
# ... your crawling logic ...
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Specifying Download Location
|
||||
|
||||
Specify the download directory using the `downloads_path` attribute in the `BrowserConfig` object. If not provided, Crawl4AI defaults to creating a "downloads" directory inside the `.crawl4ai` folder in your home directory.
|
||||
|
||||
```python
|
||||
from crawl4ai.async_configs import BrowserConfig
|
||||
import os
|
||||
|
||||
downloads_path = os.path.join(os.getcwd(), "my_downloads") # Custom download path
|
||||
os.makedirs(downloads_path, exist_ok=True)
|
||||
|
||||
config = BrowserConfig(accept_downloads=True, downloads_path=downloads_path)
|
||||
|
||||
async def main():
|
||||
async with AsyncWebCrawler(config=config) as crawler:
|
||||
result = await crawler.arun(url="https://example.com")
|
||||
# ...
|
||||
```
|
||||
|
||||
## Triggering Downloads
|
||||
|
||||
Downloads are typically triggered by user interactions on a web page, such as clicking a download button. Use `js_code` in `CrawlerRunConfig` to simulate these actions and `wait_for` to allow sufficient time for downloads to start.
|
||||
|
||||
```python
|
||||
from crawl4ai.async_configs import CrawlerRunConfig
|
||||
|
||||
config = CrawlerRunConfig(
|
||||
js_code="""
|
||||
const downloadLink = document.querySelector('a[href$=".exe"]');
|
||||
if (downloadLink) {
|
||||
downloadLink.click();
|
||||
}
|
||||
""",
|
||||
wait_for=5 # Wait 5 seconds for the download to start
|
||||
)
|
||||
|
||||
result = await crawler.arun(url="https://www.python.org/downloads/", config=config)
|
||||
```
|
||||
|
||||
## Accessing Downloaded Files
|
||||
|
||||
The `downloaded_files` attribute of the `CrawlResult` object contains paths to downloaded files.
|
||||
|
||||
```python
|
||||
if result.downloaded_files:
|
||||
print("Downloaded files:")
|
||||
for file_path in result.downloaded_files:
|
||||
print(f"- {file_path}")
|
||||
file_size = os.path.getsize(file_path)
|
||||
print(f"- File size: {file_size} bytes")
|
||||
else:
|
||||
print("No files downloaded.")
|
||||
```
|
||||
|
||||
## Example: Downloading Multiple Files
|
||||
|
||||
```python
|
||||
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
async def download_multiple_files(url: str, download_path: str):
|
||||
config = BrowserConfig(accept_downloads=True, downloads_path=download_path)
|
||||
async with AsyncWebCrawler(config=config) as crawler:
|
||||
run_config = CrawlerRunConfig(
|
||||
js_code="""
|
||||
const downloadLinks = document.querySelectorAll('a[download]');
|
||||
for (const link of downloadLinks) {
|
||||
link.click();
|
||||
// Delay between clicks
|
||||
await new Promise(r => setTimeout(r, 2000));
|
||||
}
|
||||
""",
|
||||
wait_for=10 # Wait for all downloads to start
|
||||
)
|
||||
result = await crawler.arun(url=url, config=run_config)
|
||||
|
||||
if result.downloaded_files:
|
||||
print("Downloaded files:")
|
||||
for file in result.downloaded_files:
|
||||
print(f"- {file}")
|
||||
else:
|
||||
print("No files downloaded.")
|
||||
|
||||
# Usage
|
||||
download_path = os.path.join(Path.home(), ".crawl4ai", "downloads")
|
||||
os.makedirs(download_path, exist_ok=True)
|
||||
|
||||
asyncio.run(download_multiple_files("https://www.python.org/downloads/windows/", download_path))
|
||||
```
|
||||
|
||||
## Important Considerations
|
||||
|
||||
- **Browser Context:** Downloads are managed within the browser context. Ensure `js_code` correctly targets the download triggers on the webpage.
|
||||
- **Timing:** Use `wait_for` in `CrawlerRunConfig` to manage download timing.
|
||||
- **Error Handling:** Handle errors to manage failed downloads or incorrect paths gracefully.
|
||||
- **Security:** Scan downloaded files for potential security threats before use.
|
||||
|
||||
This revised guide ensures consistency with the `Crawl4AI` codebase by using `BrowserConfig` and `CrawlerRunConfig` for all download-related configurations. Let me know if further adjustments are needed!
|
||||
@@ -0,0 +1,254 @@
|
||||
# Hooks & Auth in AsyncWebCrawler
|
||||
|
||||
Crawl4AI’s **hooks** let you customize the crawler at specific points in the pipeline:
|
||||
|
||||
1. **`on_browser_created`** – After browser creation.
|
||||
2. **`on_page_context_created`** – After a new context & page are created.
|
||||
3. **`before_goto`** – Just before navigating to a page.
|
||||
4. **`after_goto`** – Right after navigation completes.
|
||||
5. **`on_user_agent_updated`** – Whenever the user agent changes.
|
||||
6. **`on_execution_started`** – Once custom JavaScript execution begins.
|
||||
7. **`before_retrieve_html`** – Just before the crawler retrieves final HTML.
|
||||
8. **`before_return_html`** – Right before returning the HTML content.
|
||||
|
||||
**Important**: Avoid heavy tasks in `on_browser_created` since you don’t yet have a page context. If you need to *log in*, do so in **`on_page_context_created`**.
|
||||
|
||||
> note "Important Hook Usage Warning"
|
||||
**Avoid Misusing Hooks**: Do not manipulate page objects in the wrong hook or at the wrong time, as it can crash the pipeline or produce incorrect results. A common mistake is attempting to handle authentication prematurely—such as creating or closing pages in `on_browser_created`.
|
||||
|
||||
> **Use the Right Hook for Auth**: If you need to log in or set tokens, use `on_page_context_created`. This ensures you have a valid page/context to work with, without disrupting the main crawling flow.
|
||||
|
||||
> **Identity-Based Crawling**: For robust auth, consider identity-based crawling (or passing a session ID) to preserve state. Run your initial login steps in a separate, well-defined process, then feed that session to your main crawl—rather than shoehorning complex authentication into early hooks. Check out [Identity-Based Crawling](../advanced/identity-based-crawling.md) for more details.
|
||||
|
||||
> **Be Cautious**: Overwriting or removing elements in the wrong hook can compromise the final crawl. Keep hooks focused on smaller tasks (like route filters, custom headers), and let your main logic (crawling, data extraction) proceed normally.
|
||||
|
||||
|
||||
Below is an example demonstration.
|
||||
|
||||
---
|
||||
|
||||
## Example: Using Hooks in AsyncWebCrawler
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import json
|
||||
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode
|
||||
from playwright.async_api import Page, BrowserContext
|
||||
|
||||
async def main():
|
||||
print("🔗 Hooks Example: Demonstrating recommended usage")
|
||||
|
||||
# 1) Configure the browser
|
||||
browser_config = BrowserConfig(
|
||||
headless=True,
|
||||
verbose=True
|
||||
)
|
||||
|
||||
# 2) Configure the crawler run
|
||||
crawler_run_config = CrawlerRunConfig(
|
||||
js_code="window.scrollTo(0, document.body.scrollHeight);",
|
||||
wait_for="body",
|
||||
cache_mode=CacheMode.BYPASS
|
||||
)
|
||||
|
||||
# 3) Create the crawler instance
|
||||
crawler = AsyncWebCrawler(config=browser_config)
|
||||
|
||||
#
|
||||
# Define Hook Functions
|
||||
#
|
||||
|
||||
async def on_browser_created(browser, **kwargs):
|
||||
# Called once the browser instance is created (but no pages or contexts yet)
|
||||
print("[HOOK] on_browser_created - Browser created successfully!")
|
||||
# Typically, do minimal setup here if needed
|
||||
return browser
|
||||
|
||||
async def on_page_context_created(page: Page, context: BrowserContext, **kwargs):
|
||||
# Called right after a new page + context are created (ideal for auth or route config).
|
||||
print("[HOOK] on_page_context_created - Setting up page & context.")
|
||||
|
||||
# Example 1: Route filtering (e.g., block images)
|
||||
async def route_filter(route):
|
||||
if route.request.resource_type == "image":
|
||||
print(f"[HOOK] Blocking image request: {route.request.url}")
|
||||
await route.abort()
|
||||
else:
|
||||
await route.continue_()
|
||||
|
||||
await context.route("**", route_filter)
|
||||
|
||||
# Example 2: (Optional) Simulate a login scenario
|
||||
# (We do NOT create or close pages here, just do quick steps if needed)
|
||||
# e.g., await page.goto("https://example.com/login")
|
||||
# e.g., await page.fill("input[name='username']", "testuser")
|
||||
# e.g., await page.fill("input[name='password']", "password123")
|
||||
# e.g., await page.click("button[type='submit']")
|
||||
# e.g., await page.wait_for_selector("#welcome")
|
||||
# e.g., await context.add_cookies([...])
|
||||
# Then continue
|
||||
|
||||
# Example 3: Adjust the viewport
|
||||
await page.set_viewport_size({"width": 1080, "height": 600})
|
||||
return page
|
||||
|
||||
async def before_goto(
|
||||
page: Page, context: BrowserContext, url: str, **kwargs
|
||||
):
|
||||
# Called before navigating to each URL.
|
||||
print(f"[HOOK] before_goto - About to navigate: {url}")
|
||||
# e.g., inject custom headers
|
||||
await page.set_extra_http_headers({
|
||||
"Custom-Header": "my-value"
|
||||
})
|
||||
return page
|
||||
|
||||
async def after_goto(
|
||||
page: Page, context: BrowserContext,
|
||||
url: str, response, **kwargs
|
||||
):
|
||||
# Called after navigation completes.
|
||||
print(f"[HOOK] after_goto - Successfully loaded: {url}")
|
||||
# e.g., wait for a certain element if we want to verify
|
||||
try:
|
||||
await page.wait_for_selector('.content', timeout=1000)
|
||||
print("[HOOK] Found .content element!")
|
||||
except:
|
||||
print("[HOOK] .content not found, continuing anyway.")
|
||||
return page
|
||||
|
||||
async def on_user_agent_updated(
|
||||
page: Page, context: BrowserContext,
|
||||
user_agent: str, **kwargs
|
||||
):
|
||||
# Called whenever the user agent updates.
|
||||
print(f"[HOOK] on_user_agent_updated - New user agent: {user_agent}")
|
||||
return page
|
||||
|
||||
async def on_execution_started(page: Page, context: BrowserContext, **kwargs):
|
||||
# Called after custom JavaScript execution begins.
|
||||
print("[HOOK] on_execution_started - JS code is running!")
|
||||
return page
|
||||
|
||||
async def before_retrieve_html(page: Page, context: BrowserContext, **kwargs):
|
||||
# Called before final HTML retrieval.
|
||||
print("[HOOK] before_retrieve_html - We can do final actions")
|
||||
# Example: Scroll again
|
||||
await page.evaluate("window.scrollTo(0, document.body.scrollHeight);")
|
||||
return page
|
||||
|
||||
async def before_return_html(
|
||||
page: Page, context: BrowserContext, html: str, **kwargs
|
||||
):
|
||||
# Called just before returning the HTML in the result.
|
||||
print(f"[HOOK] before_return_html - HTML length: {len(html)}")
|
||||
return page
|
||||
|
||||
#
|
||||
# Attach Hooks
|
||||
#
|
||||
|
||||
crawler.crawler_strategy.set_hook("on_browser_created", on_browser_created)
|
||||
crawler.crawler_strategy.set_hook(
|
||||
"on_page_context_created", on_page_context_created
|
||||
)
|
||||
crawler.crawler_strategy.set_hook("before_goto", before_goto)
|
||||
crawler.crawler_strategy.set_hook("after_goto", after_goto)
|
||||
crawler.crawler_strategy.set_hook(
|
||||
"on_user_agent_updated", on_user_agent_updated
|
||||
)
|
||||
crawler.crawler_strategy.set_hook(
|
||||
"on_execution_started", on_execution_started
|
||||
)
|
||||
crawler.crawler_strategy.set_hook(
|
||||
"before_retrieve_html", before_retrieve_html
|
||||
)
|
||||
crawler.crawler_strategy.set_hook(
|
||||
"before_return_html", before_return_html
|
||||
)
|
||||
|
||||
await crawler.start()
|
||||
|
||||
# 4) Run the crawler on an example page
|
||||
url = "https://example.com"
|
||||
result = await crawler.arun(url, config=crawler_run_config)
|
||||
|
||||
if result.success:
|
||||
print("\nCrawled URL:", result.url)
|
||||
print("HTML length:", len(result.html))
|
||||
else:
|
||||
print("Error:", result.error_message)
|
||||
|
||||
await crawler.close()
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Hook Lifecycle Summary
|
||||
|
||||
1. **`on_browser_created`**:
|
||||
- Browser is up, but **no** pages or contexts yet.
|
||||
- Light setup only—don’t try to open or close pages here (that belongs in `on_page_context_created`).
|
||||
|
||||
2. **`on_page_context_created`**:
|
||||
- Perfect for advanced **auth** or route blocking.
|
||||
- You have a **page** + **context** ready but haven’t navigated to the target URL yet.
|
||||
|
||||
3. **`before_goto`**:
|
||||
- Right before navigation. Typically used for setting **custom headers** or logging the target URL.
|
||||
|
||||
4. **`after_goto`**:
|
||||
- After page navigation is done. Good place for verifying content or waiting on essential elements.
|
||||
|
||||
5. **`on_user_agent_updated`**:
|
||||
- Whenever the user agent changes (for stealth or different UA modes).
|
||||
|
||||
6. **`on_execution_started`**:
|
||||
- If you set `js_code` or run custom scripts, this runs once your JS is about to start.
|
||||
|
||||
7. **`before_retrieve_html`**:
|
||||
- Just before the final HTML snapshot is taken. Often you do a final scroll or lazy-load triggers here.
|
||||
|
||||
8. **`before_return_html`**:
|
||||
- The last hook before returning HTML to the `CrawlResult`. Good for logging HTML length or minor modifications.
|
||||
|
||||
---
|
||||
|
||||
## When to Handle Authentication
|
||||
|
||||
**Recommended**: Use **`on_page_context_created`** if you need to:
|
||||
|
||||
- Navigate to a login page or fill forms
|
||||
- Set cookies or localStorage tokens
|
||||
- Block resource routes to avoid ads
|
||||
|
||||
This ensures the newly created context is under your control **before** `arun()` navigates to the main URL.
|
||||
|
||||
---
|
||||
|
||||
## Additional Considerations
|
||||
|
||||
- **Session Management**: If you want multiple `arun()` calls to reuse a single session, pass `session_id=` in your `CrawlerRunConfig`. Hooks remain the same.
|
||||
- **Performance**: Hooks can slow down crawling if they do heavy tasks. Keep them concise.
|
||||
- **Error Handling**: If a hook fails, the overall crawl might fail. Catch exceptions or handle them gracefully.
|
||||
- **Concurrency**: If you run `arun_many()`, each URL triggers these hooks in parallel. Ensure your hooks are thread/async-safe.
|
||||
|
||||
---
|
||||
|
||||
## Conclusion
|
||||
|
||||
Hooks provide **fine-grained** control over:
|
||||
|
||||
- **Browser** creation (light tasks only)
|
||||
- **Page** and **context** creation (auth, route blocking)
|
||||
- **Navigation** phases
|
||||
- **Final HTML** retrieval
|
||||
|
||||
Follow the recommended usage:
|
||||
- **Login** or advanced tasks in `on_page_context_created`
|
||||
- **Custom headers** or logs in `before_goto` / `after_goto`
|
||||
- **Scrolling** or final checks in `before_retrieve_html` / `before_return_html`
|
||||
|
||||
@@ -0,0 +1,413 @@
|
||||
# Preserve Your Identity with Crawl4AI
|
||||
|
||||
Crawl4AI empowers you to navigate and interact with the web using your **authentic digital identity**, ensuring you’re recognized as a human and not mistaken for a bot. This tutorial covers:
|
||||
|
||||
1. **Managed Browsers** – The recommended approach for persistent profiles and identity-based crawling.
|
||||
2. **Magic Mode** – A simplified fallback solution for quick automation without persistent identity.
|
||||
|
||||
---
|
||||
|
||||
## 1. Managed Browsers: Your Digital Identity Solution
|
||||
|
||||
**Managed Browsers** let developers create and use **persistent browser profiles**. These profiles store local storage, cookies, and other session data, letting you browse as your **real self**—complete with logins, preferences, and cookies.
|
||||
|
||||
### Key Benefits
|
||||
|
||||
- **Authentic Browsing Experience**: Retain session data and browser fingerprints as though you’re a normal user.
|
||||
- **Effortless Configuration**: Once you log in or solve CAPTCHAs in your chosen data directory, you can re-run crawls without repeating those steps.
|
||||
- **Empowered Data Access**: If you can see the data in your own browser, you can automate its retrieval with your genuine identity.
|
||||
|
||||
---
|
||||
|
||||
Below is a **partial update** to your **Managed Browsers** tutorial, specifically the section about **creating a user-data directory** using **Playwright’s Chromium** binary rather than a system-wide Chrome/Edge. We’ll show how to **locate** that binary and launch it with a `--user-data-dir` argument to set up your profile. You can then point `BrowserConfig.user_data_dir` to that folder for subsequent crawls.
|
||||
|
||||
---
|
||||
|
||||
### Creating a User Data Directory (Command-Line Approach via Playwright)
|
||||
|
||||
If you installed Crawl4AI (which installs Playwright under the hood), you already have a Playwright-managed Chromium on your system. Follow these steps to launch that **Chromium** from your command line, specifying a **custom** data directory:
|
||||
|
||||
1. **Find** the Playwright Chromium binary:
|
||||
- On most systems, installed browsers go under a `~/.cache/ms-playwright/` folder or similar path.
|
||||
- To see an overview of installed browsers, run:
|
||||
```bash
|
||||
python -m playwright install --dry-run
|
||||
```
|
||||
or
|
||||
```bash
|
||||
playwright install --dry-run
|
||||
```
|
||||
(depending on your environment). This shows where Playwright keeps Chromium.
|
||||
|
||||
- For instance, you might see a path like:
|
||||
```
|
||||
~/.cache/ms-playwright/chromium-1234/chrome-linux/chrome
|
||||
```
|
||||
on Linux, or a corresponding folder on macOS/Windows.
|
||||
|
||||
2. **Launch** the Playwright Chromium binary with a **custom** user-data directory:
|
||||
```bash
|
||||
# Linux example
|
||||
~/.cache/ms-playwright/chromium-1234/chrome-linux/chrome \
|
||||
--user-data-dir=/home/<you>/my_chrome_profile
|
||||
```
|
||||
```bash
|
||||
# macOS example (Playwright’s internal binary)
|
||||
~/Library/Caches/ms-playwright/chromium-1234/chrome-mac/Chromium.app/Contents/MacOS/Chromium \
|
||||
--user-data-dir=/Users/<you>/my_chrome_profile
|
||||
```
|
||||
```powershell
|
||||
# Windows example (PowerShell/cmd)
|
||||
"C:\Users\<you>\AppData\Local\ms-playwright\chromium-1234\chrome-win\chrome.exe" ^
|
||||
--user-data-dir="C:\Users\<you>\my_chrome_profile"
|
||||
```
|
||||
|
||||
**Replace** the path with the actual subfolder indicated in your `ms-playwright` cache structure.
|
||||
- This **opens** a fresh Chromium with your new or existing data folder.
|
||||
- **Log into** any sites or configure your browser the way you want.
|
||||
- **Close** when done—your profile data is saved in that folder.
|
||||
|
||||
3. **Use** that folder in **`BrowserConfig.user_data_dir`**:
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig
|
||||
|
||||
browser_config = BrowserConfig(
|
||||
headless=True,
|
||||
use_managed_browser=True,
|
||||
user_data_dir="/home/<you>/my_chrome_profile",
|
||||
browser_type="chromium"
|
||||
)
|
||||
```
|
||||
- Next time you run your code, it reuses that folder—**preserving** your session data, cookies, local storage, etc.
|
||||
|
||||
---
|
||||
|
||||
### Creating a Profile Using the Crawl4AI CLI (Easiest)
|
||||
|
||||
If you prefer a guided, interactive setup, use the built-in CLI to create and manage persistent browser profiles.
|
||||
|
||||
1.⠀Launch the profile manager:
|
||||
```bash
|
||||
crwl profiles
|
||||
```
|
||||
|
||||
2.⠀Choose "Create new profile" and enter a profile name. A Chromium window opens so you can log in to sites and configure settings. When finished, return to the terminal and press `q` to save the profile.
|
||||
|
||||
3.⠀Profiles are saved under `~/.crawl4ai/profiles/<profile_name>` (for example: `/home/<you>/.crawl4ai/profiles/test_profile_1`) along with a `storage_state.json` for cookies and session data.
|
||||
|
||||
4.⠀Optionally, choose "List profiles" in the CLI to view available profiles and their paths.
|
||||
|
||||
5.⠀Use the saved path with `BrowserConfig.user_data_dir`:
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, BrowserConfig
|
||||
|
||||
profile_path = "/home/<you>/.crawl4ai/profiles/test_profile_1"
|
||||
|
||||
browser_config = BrowserConfig(
|
||||
headless=True,
|
||||
use_managed_browser=True,
|
||||
user_data_dir=profile_path,
|
||||
browser_type="chromium",
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
result = await crawler.arun(url="https://example.com/private")
|
||||
```
|
||||
|
||||
The CLI also supports listing and deleting profiles, and even testing a crawl directly from the menu.
|
||||
|
||||
---
|
||||
|
||||
## 3. Using Managed Browsers in Crawl4AI
|
||||
|
||||
Once you have a data directory with your session data, pass it to **`BrowserConfig`**:
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig
|
||||
|
||||
async def main():
|
||||
# 1) Reference your persistent data directory
|
||||
browser_config = BrowserConfig(
|
||||
headless=True, # 'True' for automated runs
|
||||
verbose=True,
|
||||
use_managed_browser=True, # Enables persistent browser strategy
|
||||
browser_type="chromium",
|
||||
user_data_dir="/path/to/my-chrome-profile"
|
||||
)
|
||||
|
||||
# 2) Standard crawl config
|
||||
crawl_config = CrawlerRunConfig(
|
||||
wait_for="css:.logged-in-content"
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
result = await crawler.arun(url="https://example.com/private", config=crawl_config)
|
||||
if result.success:
|
||||
print("Successfully accessed private data with your identity!")
|
||||
else:
|
||||
print("Error:", result.error_message)
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Workflow
|
||||
|
||||
1. **Login** externally (via CLI or your normal Chrome with `--user-data-dir=...`).
|
||||
2. **Close** that browser.
|
||||
3. **Use** the same folder in `user_data_dir=` in Crawl4AI.
|
||||
4. **Crawl** – The site sees your identity as if you’re the same user who just logged in.
|
||||
|
||||
---
|
||||
|
||||
## 4. Magic Mode: Simplified Automation
|
||||
|
||||
If you **don’t** need a persistent profile or identity-based approach, **Magic Mode** offers a quick way to simulate human-like browsing without storing long-term data.
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
config=CrawlerRunConfig(
|
||||
magic=True, # Simplifies a lot of interaction
|
||||
remove_overlay_elements=True,
|
||||
page_timeout=60000
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
**Magic Mode**:
|
||||
|
||||
- Simulates a user-like experience
|
||||
- Randomizes user agent & navigator
|
||||
- Randomizes interactions & timings
|
||||
- Masks automation signals
|
||||
- Attempts pop-up handling
|
||||
|
||||
**But** it’s no substitute for **true** user-based sessions if you want a fully legitimate identity-based solution.
|
||||
|
||||
---
|
||||
|
||||
## 5. Comparing Managed Browsers vs. Magic Mode
|
||||
|
||||
| Feature | **Managed Browsers** | **Magic Mode** |
|
||||
|----------------------------|---------------------------------------------------------------|-----------------------------------------------------|
|
||||
| **Session Persistence** | Full localStorage/cookies retained in user_data_dir | No persistent data (fresh each run) |
|
||||
| **Genuine Identity** | Real user profile with full rights & preferences | Emulated user-like patterns, but no actual identity |
|
||||
| **Complex Sites** | Best for login-gated sites or heavy config | Simple tasks, minimal login or config needed |
|
||||
| **Setup** | External creation of user_data_dir, then use in Crawl4AI | Single-line approach (`magic=True`) |
|
||||
| **Reliability** | Extremely consistent (same data across runs) | Good for smaller tasks, can be less stable |
|
||||
|
||||
---
|
||||
|
||||
## 6. Using the BrowserProfiler Class
|
||||
|
||||
Crawl4AI provides a dedicated `BrowserProfiler` class for managing browser profiles, making it easy to create, list, and delete profiles for identity-based browsing.
|
||||
|
||||
### Creating and Managing Profiles with BrowserProfiler
|
||||
|
||||
The `BrowserProfiler` class offers a comprehensive API for browser profile management:
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import BrowserProfiler
|
||||
|
||||
async def manage_profiles():
|
||||
# Create a profiler instance
|
||||
profiler = BrowserProfiler()
|
||||
|
||||
# Create a profile interactively - opens a browser window
|
||||
profile_path = await profiler.create_profile(
|
||||
profile_name="my-login-profile" # Optional: name your profile
|
||||
)
|
||||
|
||||
print(f"Profile saved at: {profile_path}")
|
||||
|
||||
# List all available profiles
|
||||
profiles = profiler.list_profiles()
|
||||
|
||||
for profile in profiles:
|
||||
print(f"Profile: {profile['name']}")
|
||||
print(f" Path: {profile['path']}")
|
||||
print(f" Created: {profile['created']}")
|
||||
print(f" Browser type: {profile['type']}")
|
||||
|
||||
# Get a specific profile path by name
|
||||
specific_profile = profiler.get_profile_path("my-login-profile")
|
||||
|
||||
# Delete a profile when no longer needed
|
||||
success = profiler.delete_profile("old-profile-name")
|
||||
|
||||
asyncio.run(manage_profiles())
|
||||
```
|
||||
|
||||
**How profile creation works:**
|
||||
1. A browser window opens for you to interact with
|
||||
2. You log in to websites, set preferences, etc.
|
||||
3. When you're done, press 'q' in the terminal to close the browser
|
||||
4. The profile is saved in the Crawl4AI profiles directory
|
||||
5. You can use the returned path with `BrowserConfig.user_data_dir`
|
||||
|
||||
### Interactive Profile Management
|
||||
|
||||
The `BrowserProfiler` also offers an interactive management console that guides you through profile creation, listing, and deletion:
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import BrowserProfiler, AsyncWebCrawler, BrowserConfig
|
||||
|
||||
# Define a function to use a profile for crawling
|
||||
async def crawl_with_profile(profile_path, url):
|
||||
browser_config = BrowserConfig(
|
||||
headless=True,
|
||||
use_managed_browser=True,
|
||||
user_data_dir=profile_path
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
result = await crawler.arun(url)
|
||||
return result
|
||||
|
||||
async def main():
|
||||
# Create a profiler instance
|
||||
profiler = BrowserProfiler()
|
||||
|
||||
# Launch the interactive profile manager
|
||||
# Passing the crawl function as a callback adds a "crawl with profile" option
|
||||
await profiler.interactive_manager(crawl_callback=crawl_with_profile)
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Legacy Methods
|
||||
|
||||
For backward compatibility, the previous methods on `ManagedBrowser` are still available, but they delegate to the new `BrowserProfiler` class:
|
||||
|
||||
```python
|
||||
from crawl4ai.browser_manager import ManagedBrowser
|
||||
|
||||
# These methods still work but use BrowserProfiler internally
|
||||
profiles = ManagedBrowser.list_profiles()
|
||||
```
|
||||
|
||||
### Complete Example
|
||||
|
||||
See the full example in `docs/examples/identity_based_browsing.py` for a complete demonstration of creating and using profiles for authenticated browsing using the new `BrowserProfiler` class.
|
||||
|
||||
---
|
||||
|
||||
## 7. Locale, Timezone, and Geolocation Control
|
||||
|
||||
In addition to using persistent profiles, Crawl4AI supports customizing your browser's locale, timezone, and geolocation settings. These features enhance your identity-based browsing experience by allowing you to control how websites perceive your location and regional settings.
|
||||
|
||||
### Setting Locale and Timezone
|
||||
|
||||
You can set the browser's locale and timezone through `CrawlerRunConfig`:
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
config=CrawlerRunConfig(
|
||||
# Set browser locale (language and region formatting)
|
||||
locale="fr-FR", # French (France)
|
||||
|
||||
# Set browser timezone
|
||||
timezone_id="Europe/Paris",
|
||||
|
||||
# Other normal options...
|
||||
magic=True,
|
||||
page_timeout=60000
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
**How it works:**
|
||||
- `locale` affects language preferences, date formats, number formats, etc.
|
||||
- `timezone_id` affects JavaScript's Date object and time-related functionality
|
||||
- These settings are applied when creating the browser context and maintained throughout the session
|
||||
|
||||
### Configuring Geolocation
|
||||
|
||||
Control the GPS coordinates reported by the browser's geolocation API:
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, GeolocationConfig
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://maps.google.com", # Or any location-aware site
|
||||
config=CrawlerRunConfig(
|
||||
# Configure precise GPS coordinates
|
||||
geolocation=GeolocationConfig(
|
||||
latitude=48.8566, # Paris coordinates
|
||||
longitude=2.3522,
|
||||
accuracy=100 # Accuracy in meters (optional)
|
||||
),
|
||||
|
||||
# This site will see you as being in Paris
|
||||
page_timeout=60000
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
**Important notes:**
|
||||
- When `geolocation` is specified, the browser is automatically granted permission to access location
|
||||
- Websites using the Geolocation API will receive the exact coordinates you specify
|
||||
- This affects map services, store locators, delivery services, etc.
|
||||
- Combined with the appropriate `locale` and `timezone_id`, you can create a fully consistent location profile
|
||||
|
||||
### Combining with Managed Browsers
|
||||
|
||||
These settings work perfectly with managed browsers for a complete identity solution:
|
||||
|
||||
```python
|
||||
from crawl4ai import (
|
||||
AsyncWebCrawler, BrowserConfig, CrawlerRunConfig,
|
||||
GeolocationConfig
|
||||
)
|
||||
|
||||
browser_config = BrowserConfig(
|
||||
use_managed_browser=True,
|
||||
user_data_dir="/path/to/my-profile",
|
||||
browser_type="chromium"
|
||||
)
|
||||
|
||||
crawl_config = CrawlerRunConfig(
|
||||
# Location settings
|
||||
locale="es-MX", # Spanish (Mexico)
|
||||
timezone_id="America/Mexico_City",
|
||||
geolocation=GeolocationConfig(
|
||||
latitude=19.4326, # Mexico City
|
||||
longitude=-99.1332
|
||||
)
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
result = await crawler.arun(url="https://example.com", config=crawl_config)
|
||||
```
|
||||
|
||||
Combining persistent profiles with precise geolocation and region settings gives you complete control over your digital identity.
|
||||
|
||||
## 8. Summary
|
||||
|
||||
- **Create** your user-data directory either:
|
||||
- By launching Chrome/Chromium externally with `--user-data-dir=/some/path`
|
||||
- Or by using the built-in `BrowserProfiler.create_profile()` method
|
||||
- Or through the interactive interface with `profiler.interactive_manager()`
|
||||
- **Log in** or configure sites as needed, then close the browser
|
||||
- **Reference** that folder in `BrowserConfig(user_data_dir="...")` + `use_managed_browser=True`
|
||||
- **Customize** identity aspects with `locale`, `timezone_id`, and `geolocation`
|
||||
- **List and reuse** profiles with `BrowserProfiler.list_profiles()`
|
||||
- **Manage** your profiles with the dedicated `BrowserProfiler` class
|
||||
- Enjoy **persistent** sessions that reflect your real identity
|
||||
- If you only need quick, ephemeral automation, **Magic Mode** might suffice
|
||||
|
||||
**Recommended**: Always prefer a **Managed Browser** for robust, identity-based crawling and simpler interactions with complex sites. Use **Magic Mode** for quick tasks or prototypes where persistent data is unnecessary.
|
||||
|
||||
With these approaches, you preserve your **authentic** browsing environment, ensuring the site sees you exactly as a normal user—no repeated logins or wasted time.
|
||||
@@ -0,0 +1,104 @@
|
||||
## Handling Lazy-Loaded Images
|
||||
|
||||
Many websites now load images **lazily** as you scroll. If you need to ensure they appear in your final crawl (and in `result.media`), consider:
|
||||
|
||||
1. **`wait_for_images=True`** – Wait for images to fully load.
|
||||
2. **`scan_full_page`** – Force the crawler to scroll the entire page, triggering lazy loads.
|
||||
3. **`scroll_delay`** – Add small delays between scroll steps.
|
||||
|
||||
**Note**: If the site requires multiple “Load More” triggers or complex interactions, see the [Page Interaction docs](../core/page-interaction.md). For sites with virtual scrolling (Twitter/Instagram style), see the [Virtual Scroll docs](virtual-scroll.md).
|
||||
|
||||
### Example: Ensuring Lazy Images Appear
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, BrowserConfig
|
||||
from crawl4ai.async_configs import CacheMode
|
||||
|
||||
async def main():
|
||||
config = CrawlerRunConfig(
|
||||
# Force the crawler to wait until images are fully loaded
|
||||
wait_for_images=True,
|
||||
|
||||
# Option 1: If you want to automatically scroll the page to load images
|
||||
scan_full_page=True, # Tells the crawler to try scrolling the entire page
|
||||
scroll_delay=0.5, # Delay (seconds) between scroll steps
|
||||
|
||||
# Option 2: If the site uses a 'Load More' or JS triggers for images,
|
||||
# you can also specify js_code or wait_for logic here.
|
||||
|
||||
cache_mode=CacheMode.BYPASS,
|
||||
verbose=True
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(config=BrowserConfig(headless=True)) as crawler:
|
||||
result = await crawler.arun("https://www.example.com/gallery", config=config)
|
||||
|
||||
if result.success:
|
||||
images = result.media.get("images", [])
|
||||
print("Images found:", len(images))
|
||||
for i, img in enumerate(images[:5]):
|
||||
print(f"[Image {i}] URL: {img['src']}, Score: {img.get('score','N/A')}")
|
||||
else:
|
||||
print("Error:", result.error_message)
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
**Explanation**:
|
||||
|
||||
- **`wait_for_images=True`**
|
||||
The crawler tries to ensure images have finished loading before finalizing the HTML.
|
||||
- **`scan_full_page=True`**
|
||||
Tells the crawler to attempt scrolling from top to bottom. Each scroll step helps trigger lazy loading.
|
||||
- **`scroll_delay=0.5`**
|
||||
Pause half a second between each scroll step. Helps the site load images before continuing.
|
||||
|
||||
**When to Use**:
|
||||
|
||||
- **Lazy-Loading**: If images appear only when the user scrolls into view, `scan_full_page` + `scroll_delay` helps the crawler see them.
|
||||
- **Heavier Pages**: If a page is extremely long, be mindful that scanning the entire page can be slow. Adjust `scroll_delay` or the max scroll steps as needed.
|
||||
|
||||
---
|
||||
|
||||
## Combining with Other Link & Media Filters
|
||||
|
||||
You can still combine **lazy-load** logic with the usual **exclude_external_images**, **exclude_domains**, or link filtration:
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(
|
||||
wait_for_images=True,
|
||||
scan_full_page=True,
|
||||
scroll_delay=0.5,
|
||||
|
||||
# Filter out external images if you only want local ones
|
||||
exclude_external_images=True,
|
||||
|
||||
# Exclude certain domains for links
|
||||
exclude_domains=["spammycdn.com"],
|
||||
)
|
||||
```
|
||||
|
||||
This approach ensures you see **all** images from the main domain while ignoring external ones, and the crawler physically scrolls the entire page so that lazy-loading triggers.
|
||||
|
||||
---
|
||||
|
||||
## Tips & Troubleshooting
|
||||
|
||||
1. **Long Pages**
|
||||
- Setting `scan_full_page=True` on extremely long or infinite-scroll pages can be resource-intensive.
|
||||
- Consider using [hooks](../core/page-interaction.md) or specialized logic to load specific sections or “Load More” triggers repeatedly.
|
||||
|
||||
2. **Mixed Image Behavior**
|
||||
- Some sites load images in batches as you scroll. If you’re missing images, increase your `scroll_delay` or call multiple partial scrolls in a loop with JS code or hooks.
|
||||
|
||||
3. **Combining with Dynamic Wait**
|
||||
- If the site has a placeholder that only changes to a real image after a certain event, you might do `wait_for="css:img.loaded"` or a custom JS `wait_for`.
|
||||
|
||||
4. **Caching**
|
||||
- If `cache_mode` is enabled, repeated crawls might skip some network fetches. If you suspect caching is missing new images, set `cache_mode=CacheMode.BYPASS` for fresh fetches.
|
||||
|
||||
---
|
||||
|
||||
With **lazy-loading** support, **wait_for_images**, and **scan_full_page** settings, you can capture the entire gallery or feed of images you expect—even if the site only loads them as the user scrolls. Combine these with the standard media filtering and domain exclusion for a complete link & media handling strategy.
|
||||
@@ -0,0 +1,604 @@
|
||||
# Advanced Multi-URL Crawling with Dispatchers
|
||||
|
||||
> **Heads Up**: Crawl4AI supports advanced dispatchers for **parallel** or **throttled** crawling, providing dynamic rate limiting and memory usage checks. The built-in `arun_many()` function uses these dispatchers to handle concurrency efficiently.
|
||||
|
||||
## 1. Introduction
|
||||
|
||||
When crawling many URLs:
|
||||
|
||||
- **Basic**: Use `arun()` in a loop (simple but less efficient)
|
||||
- **Better**: Use `arun_many()`, which efficiently handles multiple URLs with proper concurrency control
|
||||
- **Best**: Customize dispatcher behavior for your specific needs (memory management, rate limits, etc.)
|
||||
|
||||
**Why Dispatchers?**
|
||||
|
||||
- **Adaptive**: Memory-based dispatchers can pause or slow down based on system resources
|
||||
- **Rate-limiting**: Built-in rate limiting with exponential backoff for 429/503 responses
|
||||
- **Real-time Monitoring**: Live dashboard of ongoing tasks, memory usage, and performance
|
||||
- **Flexibility**: Choose between memory-adaptive or semaphore-based concurrency
|
||||
|
||||
---
|
||||
|
||||
## 2. Core Components
|
||||
|
||||
### 2.1 Rate Limiter
|
||||
|
||||
```python
|
||||
class RateLimiter:
|
||||
def __init__(
|
||||
# Random delay range between requests
|
||||
base_delay: Tuple[float, float] = (1.0, 3.0),
|
||||
|
||||
# Maximum backoff delay
|
||||
max_delay: float = 60.0,
|
||||
|
||||
# Retries before giving up
|
||||
max_retries: int = 3,
|
||||
|
||||
# Status codes triggering backoff
|
||||
rate_limit_codes: List[int] = [429, 503]
|
||||
)
|
||||
```
|
||||
|
||||
Here’s the revised and simplified explanation of the **RateLimiter**, focusing on constructor parameters and adhering to your markdown style and mkDocs guidelines.
|
||||
|
||||
#### RateLimiter Constructor Parameters
|
||||
|
||||
The **RateLimiter** is a utility that helps manage the pace of requests to avoid overloading servers or getting blocked due to rate limits. It operates internally to delay requests and handle retries but can be configured using its constructor parameters.
|
||||
|
||||
**Parameters of the `RateLimiter` constructor:**
|
||||
|
||||
1. **`base_delay`** (`Tuple[float, float]`, default: `(1.0, 3.0)`)
|
||||
The range for a random delay (in seconds) between consecutive requests to the same domain.
|
||||
|
||||
- A random delay is chosen between `base_delay[0]` and `base_delay[1]` for each request.
|
||||
- This prevents sending requests at a predictable frequency, reducing the chances of triggering rate limits.
|
||||
|
||||
**Example:**
|
||||
If `base_delay = (2.0, 5.0)`, delays could be randomly chosen as `2.3s`, `4.1s`, etc.
|
||||
|
||||
---
|
||||
|
||||
2. **`max_delay`** (`float`, default: `60.0`)
|
||||
The maximum allowable delay when rate-limiting errors occur.
|
||||
|
||||
- When servers return rate-limit responses (e.g., 429 or 503), the delay increases exponentially with jitter.
|
||||
- The `max_delay` ensures the delay doesn’t grow unreasonably high, capping it at this value.
|
||||
|
||||
**Example:**
|
||||
For a `max_delay = 30.0`, even if backoff calculations suggest a delay of `45s`, it will cap at `30s`.
|
||||
|
||||
---
|
||||
|
||||
3. **`max_retries`** (`int`, default: `3`)
|
||||
The maximum number of retries for a request if rate-limiting errors occur.
|
||||
|
||||
- After encountering a rate-limit response, the `RateLimiter` retries the request up to this number of times.
|
||||
- If all retries fail, the request is marked as failed, and the process continues.
|
||||
|
||||
**Example:**
|
||||
If `max_retries = 3`, the system retries a failed request three times before giving up.
|
||||
|
||||
---
|
||||
|
||||
4. **`rate_limit_codes`** (`List[int]`, default: `[429, 503]`)
|
||||
A list of HTTP status codes that trigger the rate-limiting logic.
|
||||
|
||||
- These status codes indicate the server is overwhelmed or actively limiting requests.
|
||||
- You can customize this list to include other codes based on specific server behavior.
|
||||
|
||||
**Example:**
|
||||
If `rate_limit_codes = [429, 503, 504]`, the crawler will back off on these three error codes.
|
||||
|
||||
---
|
||||
|
||||
**How to Use the `RateLimiter`:**
|
||||
|
||||
Here’s an example of initializing and using a `RateLimiter` in your project:
|
||||
|
||||
```python
|
||||
from crawl4ai import RateLimiter
|
||||
|
||||
# Create a RateLimiter with custom settings
|
||||
rate_limiter = RateLimiter(
|
||||
base_delay=(2.0, 4.0), # Random delay between 2-4 seconds
|
||||
max_delay=30.0, # Cap delay at 30 seconds
|
||||
max_retries=5, # Retry up to 5 times on rate-limiting errors
|
||||
rate_limit_codes=[429, 503] # Handle these HTTP status codes
|
||||
)
|
||||
|
||||
# RateLimiter will handle delays and retries internally
|
||||
# No additional setup is required for its operation
|
||||
```
|
||||
|
||||
The `RateLimiter` integrates seamlessly with dispatchers like `MemoryAdaptiveDispatcher` and `SemaphoreDispatcher`, ensuring requests are paced correctly without user intervention. Its internal mechanisms manage delays and retries to avoid overwhelming servers while maximizing efficiency.
|
||||
|
||||
|
||||
### 2.2 Crawler Monitor
|
||||
|
||||
The CrawlerMonitor provides real-time visibility into crawling operations:
|
||||
|
||||
```python
|
||||
from crawl4ai import CrawlerMonitor, DisplayMode
|
||||
monitor = CrawlerMonitor(
|
||||
# Maximum rows in live display
|
||||
max_visible_rows=15,
|
||||
|
||||
# DETAILED or AGGREGATED view
|
||||
display_mode=DisplayMode.DETAILED
|
||||
)
|
||||
```
|
||||
|
||||
**Display Modes**:
|
||||
|
||||
1. **DETAILED**: Shows individual task status, memory usage, and timing
|
||||
2. **AGGREGATED**: Displays summary statistics and overall progress
|
||||
|
||||
---
|
||||
|
||||
## 3. Available Dispatchers
|
||||
|
||||
### 3.1 MemoryAdaptiveDispatcher (Default)
|
||||
|
||||
Automatically manages concurrency based on system memory usage:
|
||||
|
||||
```python
|
||||
from crawl4ai.async_dispatcher import MemoryAdaptiveDispatcher
|
||||
|
||||
dispatcher = MemoryAdaptiveDispatcher(
|
||||
memory_threshold_percent=90.0, # Pause if memory exceeds this
|
||||
check_interval=1.0, # How often to check memory
|
||||
max_session_permit=10, # Maximum concurrent tasks
|
||||
rate_limiter=RateLimiter( # Optional rate limiting
|
||||
base_delay=(1.0, 2.0),
|
||||
max_delay=30.0,
|
||||
max_retries=2
|
||||
),
|
||||
monitor=CrawlerMonitor( # Optional monitoring
|
||||
max_visible_rows=15,
|
||||
display_mode=DisplayMode.DETAILED
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
**Constructor Parameters:**
|
||||
|
||||
1. **`memory_threshold_percent`** (`float`, default: `90.0`)
|
||||
Specifies the memory usage threshold (as a percentage). If system memory usage exceeds this value, the dispatcher pauses crawling to prevent system overload.
|
||||
|
||||
2. **`check_interval`** (`float`, default: `1.0`)
|
||||
The interval (in seconds) at which the dispatcher checks system memory usage.
|
||||
|
||||
3. **`max_session_permit`** (`int`, default: `10`)
|
||||
The maximum number of concurrent crawling tasks allowed. This ensures resource limits are respected while maintaining concurrency.
|
||||
|
||||
4. **`memory_wait_timeout`** (`float`, default: `600.0`)
|
||||
Optional timeout (in seconds). If memory usage exceeds `memory_threshold_percent` for longer than this duration, a `MemoryError` is raised.
|
||||
|
||||
5. **`rate_limiter`** (`RateLimiter`, default: `None`)
|
||||
Optional rate-limiting logic to avoid server-side blocking (e.g., for handling 429 or 503 errors). See **RateLimiter** for details.
|
||||
|
||||
6. **`monitor`** (`CrawlerMonitor`, default: `None`)
|
||||
Optional monitoring for real-time task tracking and performance insights. See **CrawlerMonitor** for details.
|
||||
|
||||
---
|
||||
|
||||
### 3.2 SemaphoreDispatcher
|
||||
|
||||
Provides simple concurrency control with a fixed limit:
|
||||
|
||||
```python
|
||||
from crawl4ai.async_dispatcher import SemaphoreDispatcher
|
||||
|
||||
dispatcher = SemaphoreDispatcher(
|
||||
max_session_permit=20, # Maximum concurrent tasks
|
||||
rate_limiter=RateLimiter( # Optional rate limiting
|
||||
base_delay=(0.5, 1.0),
|
||||
max_delay=10.0
|
||||
),
|
||||
monitor=CrawlerMonitor( # Optional monitoring
|
||||
max_visible_rows=15,
|
||||
display_mode=DisplayMode.DETAILED
|
||||
)
|
||||
)
|
||||
```
|
||||
|
||||
**Constructor Parameters:**
|
||||
|
||||
1. **`max_session_permit`** (`int`, default: `20`)
|
||||
The maximum number of concurrent crawling tasks allowed, irrespective of semaphore slots.
|
||||
|
||||
2. **`rate_limiter`** (`RateLimiter`, default: `None`)
|
||||
Optional rate-limiting logic to avoid overwhelming servers. See **RateLimiter** for details.
|
||||
|
||||
3. **`monitor`** (`CrawlerMonitor`, default: `None`)
|
||||
Optional monitoring for tracking task progress and resource usage. See **CrawlerMonitor** for details.
|
||||
|
||||
---
|
||||
|
||||
## 4. Usage Examples
|
||||
|
||||
### 4.1 Batch Processing (Default)
|
||||
|
||||
```python
|
||||
async def crawl_batch():
|
||||
browser_config = BrowserConfig(headless=True, verbose=False)
|
||||
run_config = CrawlerRunConfig(
|
||||
cache_mode=CacheMode.BYPASS,
|
||||
stream=False # Default: get all results at once
|
||||
)
|
||||
|
||||
dispatcher = MemoryAdaptiveDispatcher(
|
||||
memory_threshold_percent=70.0,
|
||||
check_interval=1.0,
|
||||
max_session_permit=10,
|
||||
monitor=CrawlerMonitor(
|
||||
display_mode=DisplayMode.DETAILED
|
||||
)
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
# Get all results at once
|
||||
results = await crawler.arun_many(
|
||||
urls=urls,
|
||||
config=run_config,
|
||||
dispatcher=dispatcher
|
||||
)
|
||||
|
||||
# Process all results after completion
|
||||
for result in results:
|
||||
if result.success:
|
||||
await process_result(result)
|
||||
else:
|
||||
print(f"Failed to crawl {result.url}: {result.error_message}")
|
||||
```
|
||||
|
||||
**Review:**
|
||||
- **Purpose:** Executes a batch crawl with all URLs processed together after crawling is complete.
|
||||
- **Dispatcher:** Uses `MemoryAdaptiveDispatcher` to manage concurrency and system memory.
|
||||
- **Stream:** Disabled (`stream=False`), so all results are collected at once for post-processing.
|
||||
- **Best Use Case:** When you need to analyze results in bulk rather than individually during the crawl.
|
||||
|
||||
---
|
||||
|
||||
### 4.2 Streaming Mode
|
||||
|
||||
```python
|
||||
async def crawl_streaming():
|
||||
browser_config = BrowserConfig(headless=True, verbose=False)
|
||||
run_config = CrawlerRunConfig(
|
||||
cache_mode=CacheMode.BYPASS,
|
||||
stream=True # Enable streaming mode
|
||||
)
|
||||
|
||||
dispatcher = MemoryAdaptiveDispatcher(
|
||||
memory_threshold_percent=70.0,
|
||||
check_interval=1.0,
|
||||
max_session_permit=10,
|
||||
monitor=CrawlerMonitor(
|
||||
display_mode=DisplayMode.DETAILED
|
||||
)
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
# Process results as they become available
|
||||
async for result in await crawler.arun_many(
|
||||
urls=urls,
|
||||
config=run_config,
|
||||
dispatcher=dispatcher
|
||||
):
|
||||
if result.success:
|
||||
# Process each result immediately
|
||||
await process_result(result)
|
||||
else:
|
||||
print(f"Failed to crawl {result.url}: {result.error_message}")
|
||||
```
|
||||
|
||||
**Review:**
|
||||
- **Purpose:** Enables streaming to process results as soon as they’re available.
|
||||
- **Dispatcher:** Uses `MemoryAdaptiveDispatcher` for concurrency and memory management.
|
||||
- **Stream:** Enabled (`stream=True`), allowing real-time processing during crawling.
|
||||
- **Best Use Case:** When you need to act on results immediately, such as for real-time analytics or progressive data storage.
|
||||
|
||||
---
|
||||
|
||||
### 4.3 Semaphore-based Crawling
|
||||
|
||||
```python
|
||||
async def crawl_with_semaphore(urls):
|
||||
browser_config = BrowserConfig(headless=True, verbose=False)
|
||||
run_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
|
||||
|
||||
dispatcher = SemaphoreDispatcher(
|
||||
semaphore_count=5,
|
||||
rate_limiter=RateLimiter(
|
||||
base_delay=(0.5, 1.0),
|
||||
max_delay=10.0
|
||||
),
|
||||
monitor=CrawlerMonitor(
|
||||
max_visible_rows=15,
|
||||
display_mode=DisplayMode.DETAILED
|
||||
)
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
results = await crawler.arun_many(
|
||||
urls,
|
||||
config=run_config,
|
||||
dispatcher=dispatcher
|
||||
)
|
||||
return results
|
||||
```
|
||||
|
||||
**Review:**
|
||||
- **Purpose:** Uses `SemaphoreDispatcher` to limit concurrency with a fixed number of slots.
|
||||
- **Dispatcher:** Configured with a semaphore to control parallel crawling tasks.
|
||||
- **Rate Limiter:** Prevents servers from being overwhelmed by pacing requests.
|
||||
- **Best Use Case:** When you want precise control over the number of concurrent requests, independent of system memory.
|
||||
|
||||
---
|
||||
|
||||
### 4.4 Robots.txt Consideration
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
|
||||
|
||||
async def main():
|
||||
urls = [
|
||||
"https://example1.com",
|
||||
"https://example2.com",
|
||||
"https://example3.com"
|
||||
]
|
||||
|
||||
config = CrawlerRunConfig(
|
||||
cache_mode=CacheMode.ENABLED,
|
||||
check_robots_txt=True, # Will respect robots.txt for each URL
|
||||
semaphore_count=3 # Max concurrent requests
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
async for result in crawler.arun_many(urls, config=config):
|
||||
if result.success:
|
||||
print(f"Successfully crawled {result.url}")
|
||||
elif result.status_code == 403 and "robots.txt" in result.error_message:
|
||||
print(f"Skipped {result.url} - blocked by robots.txt")
|
||||
else:
|
||||
print(f"Failed to crawl {result.url}: {result.error_message}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
**Review:**
|
||||
- **Purpose:** Ensures compliance with `robots.txt` rules for ethical and legal web crawling.
|
||||
- **Configuration:** Set `check_robots_txt=True` to validate each URL against `robots.txt` before crawling.
|
||||
- **Dispatcher:** Handles requests with concurrency limits (`semaphore_count=3`).
|
||||
- **Best Use Case:** When crawling websites that strictly enforce robots.txt policies or for responsible crawling practices.
|
||||
|
||||
---
|
||||
|
||||
## 5. Dispatch Results
|
||||
|
||||
Each crawl result includes dispatch information:
|
||||
|
||||
```python
|
||||
@dataclass
|
||||
class DispatchResult:
|
||||
task_id: str
|
||||
memory_usage: float
|
||||
peak_memory: float
|
||||
start_time: datetime
|
||||
end_time: datetime
|
||||
error_message: str = ""
|
||||
```
|
||||
|
||||
Access via `result.dispatch_result`:
|
||||
|
||||
```python
|
||||
for result in results:
|
||||
if result.success:
|
||||
dr = result.dispatch_result
|
||||
print(f"URL: {result.url}")
|
||||
print(f"Memory: {dr.memory_usage:.1f}MB")
|
||||
print(f"Duration: {dr.end_time - dr.start_time}")
|
||||
```
|
||||
|
||||
## 6. URL-Specific Configurations
|
||||
|
||||
When crawling diverse content types, you often need different configurations for different URLs. For example:
|
||||
- PDFs need specialized extraction
|
||||
- Blog pages benefit from content filtering
|
||||
- Dynamic sites need JavaScript execution
|
||||
- API endpoints need JSON parsing
|
||||
|
||||
### 6.1 Basic URL Pattern Matching
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, MatchMode
|
||||
from crawl4ai.processors.pdf import PDFContentScrapingStrategy
|
||||
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
|
||||
from crawl4ai.content_filter_strategy import PruningContentFilter
|
||||
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
|
||||
|
||||
async def crawl_mixed_content():
|
||||
# Configure different strategies for different content
|
||||
configs = [
|
||||
# PDF files - specialized extraction
|
||||
CrawlerRunConfig(
|
||||
url_matcher="*.pdf",
|
||||
scraping_strategy=PDFContentScrapingStrategy()
|
||||
),
|
||||
|
||||
# Blog/article pages - content filtering
|
||||
CrawlerRunConfig(
|
||||
url_matcher=["*/blog/*", "*/article/*"],
|
||||
markdown_generator=DefaultMarkdownGenerator(
|
||||
content_filter=PruningContentFilter(threshold=0.48)
|
||||
)
|
||||
),
|
||||
|
||||
# Dynamic pages - JavaScript execution
|
||||
CrawlerRunConfig(
|
||||
url_matcher=lambda url: 'github.com' in url,
|
||||
js_code="window.scrollTo(0, 500);"
|
||||
),
|
||||
|
||||
# API endpoints - JSON extraction
|
||||
CrawlerRunConfig(
|
||||
url_matcher=lambda url: 'api' in url or url.endswith('.json'),
|
||||
# Custome settings for JSON extraction
|
||||
),
|
||||
|
||||
# Default config for everything else
|
||||
CrawlerRunConfig() # No url_matcher means it matches ALL URLs (fallback)
|
||||
]
|
||||
|
||||
# Mixed URLs
|
||||
urls = [
|
||||
"https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf",
|
||||
"https://blog.python.org/",
|
||||
"https://github.com/microsoft/playwright",
|
||||
"https://httpbin.org/json",
|
||||
"https://example.com/"
|
||||
]
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
results = await crawler.arun_many(
|
||||
urls=urls,
|
||||
config=configs # Pass list of configs
|
||||
)
|
||||
|
||||
for result in results:
|
||||
print(f"{result.url}: {len(result.markdown)} chars")
|
||||
```
|
||||
|
||||
### 6.2 Advanced Pattern Matching
|
||||
|
||||
**Important**: A `CrawlerRunConfig` without `url_matcher` (or with `url_matcher=None`) matches ALL URLs. This makes it perfect as a default/fallback configuration.
|
||||
|
||||
The `url_matcher` parameter supports three types of patterns:
|
||||
|
||||
#### Glob Patterns (Strings)
|
||||
```python
|
||||
# Simple patterns
|
||||
"*.pdf" # Any PDF file
|
||||
"*/api/*" # Any URL with /api/ in path
|
||||
"https://*.example.com/*" # Subdomain matching
|
||||
"*://example.com/blog/*" # Any protocol
|
||||
```
|
||||
|
||||
#### Custom Functions
|
||||
```python
|
||||
# Complex logic with lambdas
|
||||
lambda url: url.startswith('https://') and 'secure' in url
|
||||
lambda url: len(url) > 50 and url.count('/') > 5
|
||||
lambda url: any(domain in url for domain in ['api.', 'data.', 'feed.'])
|
||||
```
|
||||
|
||||
#### Mixed Lists with AND/OR Logic
|
||||
```python
|
||||
# Combine multiple conditions
|
||||
CrawlerRunConfig(
|
||||
url_matcher=[
|
||||
"https://*", # Must be HTTPS
|
||||
lambda url: 'internal' in url, # Must contain 'internal'
|
||||
lambda url: not url.endswith('.pdf') # Must not be PDF
|
||||
],
|
||||
match_mode=MatchMode.AND # ALL conditions must match
|
||||
)
|
||||
```
|
||||
|
||||
### 6.3 Practical Example: News Site Crawler
|
||||
|
||||
```python
|
||||
async def crawl_news_site():
|
||||
dispatcher = MemoryAdaptiveDispatcher(
|
||||
memory_threshold_percent=70.0,
|
||||
rate_limiter=RateLimiter(base_delay=(1.0, 2.0))
|
||||
)
|
||||
|
||||
configs = [
|
||||
# Homepage - light extraction
|
||||
CrawlerRunConfig(
|
||||
url_matcher=lambda url: url.rstrip('/') == 'https://news.ycombinator.com',
|
||||
css_selector="nav, .headline",
|
||||
extraction_strategy=None
|
||||
),
|
||||
|
||||
# Article pages - full extraction
|
||||
CrawlerRunConfig(
|
||||
url_matcher="*/article/*",
|
||||
extraction_strategy=CosineStrategy(
|
||||
semantic_filter="article content",
|
||||
word_count_threshold=100
|
||||
),
|
||||
screenshot=True,
|
||||
excluded_tags=["nav", "aside", "footer"]
|
||||
),
|
||||
|
||||
# Author pages - metadata focus
|
||||
CrawlerRunConfig(
|
||||
url_matcher="*/author/*",
|
||||
extraction_strategy=JsonCssExtractionStrategy({
|
||||
"name": "h1.author-name",
|
||||
"bio": ".author-bio",
|
||||
"articles": "article.post-card h2"
|
||||
})
|
||||
),
|
||||
|
||||
# Everything else
|
||||
CrawlerRunConfig()
|
||||
]
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
results = await crawler.arun_many(
|
||||
urls=news_urls,
|
||||
config=configs,
|
||||
dispatcher=dispatcher
|
||||
)
|
||||
```
|
||||
|
||||
### 6.4 Best Practices
|
||||
|
||||
1. **Order Matters**: Configs are evaluated in order - put specific patterns before general ones
|
||||
2. **Default Config Behavior**:
|
||||
- A config without `url_matcher` matches ALL URLs
|
||||
- Always include a default config as the last item if you want to handle all URLs
|
||||
- Without a default config, unmatched URLs will fail with "No matching configuration found"
|
||||
3. **Test Your Patterns**: Use the config's `is_match()` method to test patterns:
|
||||
```python
|
||||
config = CrawlerRunConfig(url_matcher="*.pdf")
|
||||
print(config.is_match("https://example.com/doc.pdf")) # True
|
||||
|
||||
default_config = CrawlerRunConfig() # No url_matcher
|
||||
print(default_config.is_match("https://any-url.com")) # True - matches everything!
|
||||
```
|
||||
4. **Optimize for Performance**:
|
||||
- Disable JS for static content
|
||||
- Skip screenshots for data APIs
|
||||
- Use appropriate extraction strategies
|
||||
|
||||
## 7. Summary
|
||||
|
||||
1. **Two Dispatcher Types**:
|
||||
|
||||
- MemoryAdaptiveDispatcher (default): Dynamic concurrency based on memory
|
||||
- SemaphoreDispatcher: Fixed concurrency limit
|
||||
|
||||
2. **Optional Components**:
|
||||
|
||||
- RateLimiter: Smart request pacing and backoff
|
||||
- CrawlerMonitor: Real-time progress visualization
|
||||
|
||||
3. **Key Benefits**:
|
||||
|
||||
- Automatic memory management
|
||||
- Built-in rate limiting
|
||||
- Live progress monitoring
|
||||
- Flexible concurrency control
|
||||
|
||||
Choose the dispatcher that best fits your needs:
|
||||
|
||||
- **MemoryAdaptiveDispatcher**: For large crawls or limited resources
|
||||
- **SemaphoreDispatcher**: For simple, fixed-concurrency scenarios
|
||||
@@ -0,0 +1,205 @@
|
||||
# Network Requests & Console Message Capturing
|
||||
|
||||
Crawl4AI can capture all network requests and browser console messages during a crawl, which is invaluable for debugging, security analysis, or understanding page behavior.
|
||||
|
||||
## Configuration
|
||||
|
||||
To enable network and console capturing, use these configuration options:
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
|
||||
|
||||
# Enable both network request capture and console message capture
|
||||
config = CrawlerRunConfig(
|
||||
capture_network_requests=True, # Capture all network requests and responses
|
||||
capture_console_messages=True # Capture all browser console output
|
||||
)
|
||||
```
|
||||
|
||||
## Example Usage
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import json
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
|
||||
|
||||
async def main():
|
||||
# Enable both network request capture and console message capture
|
||||
config = CrawlerRunConfig(
|
||||
capture_network_requests=True,
|
||||
capture_console_messages=True
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
config=config
|
||||
)
|
||||
|
||||
if result.success:
|
||||
# Analyze network requests
|
||||
if result.network_requests:
|
||||
print(f"Captured {len(result.network_requests)} network events")
|
||||
|
||||
# Count request types
|
||||
request_count = len([r for r in result.network_requests if r.get("event_type") == "request"])
|
||||
response_count = len([r for r in result.network_requests if r.get("event_type") == "response"])
|
||||
failed_count = len([r for r in result.network_requests if r.get("event_type") == "request_failed"])
|
||||
|
||||
print(f"Requests: {request_count}, Responses: {response_count}, Failed: {failed_count}")
|
||||
|
||||
# Find API calls
|
||||
api_calls = [r for r in result.network_requests
|
||||
if r.get("event_type") == "request" and "api" in r.get("url", "")]
|
||||
if api_calls:
|
||||
print(f"Detected {len(api_calls)} API calls:")
|
||||
for call in api_calls[:3]: # Show first 3
|
||||
print(f" - {call.get('method')} {call.get('url')}")
|
||||
|
||||
# Analyze console messages
|
||||
if result.console_messages:
|
||||
print(f"Captured {len(result.console_messages)} console messages")
|
||||
|
||||
# Group by type
|
||||
message_types = {}
|
||||
for msg in result.console_messages:
|
||||
msg_type = msg.get("type", "unknown")
|
||||
message_types[msg_type] = message_types.get(msg_type, 0) + 1
|
||||
|
||||
print("Message types:", message_types)
|
||||
|
||||
# Show errors (often the most important)
|
||||
errors = [msg for msg in result.console_messages if msg.get("type") == "error"]
|
||||
if errors:
|
||||
print(f"Found {len(errors)} console errors:")
|
||||
for err in errors[:2]: # Show first 2
|
||||
print(f" - {err.get('text', '')[:100]}")
|
||||
|
||||
# Export all captured data to a file for detailed analysis
|
||||
with open("network_capture.json", "w") as f:
|
||||
json.dump({
|
||||
"url": result.url,
|
||||
"network_requests": result.network_requests or [],
|
||||
"console_messages": result.console_messages or []
|
||||
}, f, indent=2)
|
||||
|
||||
print("Exported detailed capture data to network_capture.json")
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Captured Data Structure
|
||||
|
||||
### Network Requests
|
||||
|
||||
The `result.network_requests` contains a list of dictionaries, each representing a network event with these common fields:
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| `event_type` | Type of event: `"request"`, `"response"`, or `"request_failed"` |
|
||||
| `url` | The URL of the request |
|
||||
| `timestamp` | Unix timestamp when the event was captured |
|
||||
|
||||
#### Request Event Fields
|
||||
|
||||
```json
|
||||
{
|
||||
"event_type": "request",
|
||||
"url": "https://example.com/api/data.json",
|
||||
"method": "GET",
|
||||
"headers": {"User-Agent": "...", "Accept": "..."},
|
||||
"post_data": "key=value&otherkey=value",
|
||||
"resource_type": "fetch",
|
||||
"is_navigation_request": false,
|
||||
"timestamp": 1633456789.123
|
||||
}
|
||||
```
|
||||
|
||||
#### Response Event Fields
|
||||
|
||||
```json
|
||||
{
|
||||
"event_type": "response",
|
||||
"url": "https://example.com/api/data.json",
|
||||
"status": 200,
|
||||
"status_text": "OK",
|
||||
"headers": {"Content-Type": "application/json", "Cache-Control": "..."},
|
||||
"from_service_worker": false,
|
||||
"request_timing": {"requestTime": 1234.56, "receiveHeadersEnd": 1234.78},
|
||||
"timestamp": 1633456789.456
|
||||
}
|
||||
```
|
||||
|
||||
#### Failed Request Event Fields
|
||||
|
||||
```json
|
||||
{
|
||||
"event_type": "request_failed",
|
||||
"url": "https://example.com/missing.png",
|
||||
"method": "GET",
|
||||
"resource_type": "image",
|
||||
"failure_text": "net::ERR_ABORTED 404",
|
||||
"timestamp": 1633456789.789
|
||||
}
|
||||
```
|
||||
|
||||
### Console Messages
|
||||
|
||||
The `result.console_messages` contains a list of dictionaries, each representing a console message with these common fields:
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| `type` | Message type: `"log"`, `"error"`, `"warning"`, `"info"`, etc. |
|
||||
| `text` | The message text |
|
||||
| `timestamp` | Unix timestamp when the message was captured |
|
||||
|
||||
#### Console Message Example
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "error",
|
||||
"text": "Uncaught TypeError: Cannot read property 'length' of undefined",
|
||||
"location": "https://example.com/script.js:123:45",
|
||||
"timestamp": 1633456790.123
|
||||
}
|
||||
```
|
||||
|
||||
## Key Benefits
|
||||
|
||||
- **Full Request Visibility**: Capture all network activity including:
|
||||
- Requests (URLs, methods, headers, post data)
|
||||
- Responses (status codes, headers, timing)
|
||||
- Failed requests (with error messages)
|
||||
|
||||
- **Console Message Access**: View all JavaScript console output:
|
||||
- Log messages
|
||||
- Warnings
|
||||
- Errors with stack traces
|
||||
- Developer debugging information
|
||||
|
||||
- **Debugging Power**: Identify issues such as:
|
||||
- Failed API calls or resource loading
|
||||
- JavaScript errors affecting page functionality
|
||||
- CORS or other security issues
|
||||
- Hidden API endpoints and data flows
|
||||
|
||||
- **Security Analysis**: Detect:
|
||||
- Unexpected third-party requests
|
||||
- Data leakage in request payloads
|
||||
- Suspicious script behavior
|
||||
|
||||
- **Performance Insights**: Analyze:
|
||||
- Request timing data
|
||||
- Resource loading patterns
|
||||
- Potential bottlenecks
|
||||
|
||||
## Use Cases
|
||||
|
||||
1. **API Discovery**: Identify hidden endpoints and data flows in single-page applications
|
||||
2. **Debugging**: Track down JavaScript errors affecting page functionality
|
||||
3. **Security Auditing**: Detect unwanted third-party requests or data leakage
|
||||
4. **Performance Analysis**: Identify slow-loading resources
|
||||
5. **Ad/Tracker Analysis**: Detect and catalog advertising or tracking calls
|
||||
|
||||
This capability is especially valuable for complex sites with heavy JavaScript, single-page applications, or when you need to understand the exact communication happening between a browser and servers.
|
||||
@@ -0,0 +1,201 @@
|
||||
# PDF Processing Strategies
|
||||
|
||||
Crawl4AI provides specialized strategies for handling and extracting content from PDF files. These strategies allow you to seamlessly integrate PDF processing into your crawling workflows, whether the PDFs are hosted online or stored locally.
|
||||
|
||||
## `PDFCrawlerStrategy`
|
||||
|
||||
### Overview
|
||||
`PDFCrawlerStrategy` is an implementation of `AsyncCrawlerStrategy` designed specifically for PDF documents. Instead of interpreting the input URL as an HTML webpage, this strategy treats it as a pointer to a PDF file. It doesn't perform deep crawling or HTML parsing itself but rather prepares the PDF source for a dedicated PDF scraping strategy. Its primary role is to identify the PDF source (web URL or local file) and pass it along the processing pipeline in a way that `AsyncWebCrawler` can handle.
|
||||
|
||||
### When to Use
|
||||
Use `PDFCrawlerStrategy` when you need to:
|
||||
- Process PDF files using the `AsyncWebCrawler`.
|
||||
- Handle PDFs from both web URLs (e.g., `https://example.com/document.pdf`) and local file paths (e.g., `file:///path/to/your/document.pdf`).
|
||||
- Integrate PDF content extraction into a unified `CrawlResult` object, allowing consistent handling of PDF data alongside web page data.
|
||||
|
||||
### Key Methods and Their Behavior
|
||||
- **`__init__(self, logger: AsyncLogger = None)`**:
|
||||
- Initializes the strategy.
|
||||
- `logger`: An optional `AsyncLogger` instance (from `crawl4ai.async_logger`) for logging purposes.
|
||||
- **`async crawl(self, url: str, **kwargs) -> AsyncCrawlResponse`**:
|
||||
- This method is called by the `AsyncWebCrawler` during the `arun` process.
|
||||
- It takes the `url` (which should point to a PDF) and creates a minimal `AsyncCrawlResponse`.
|
||||
- The `html` attribute of this response is typically empty or a placeholder, as the actual PDF content processing is deferred to the `PDFContentScrapingStrategy` (or a similar PDF-aware scraping strategy).
|
||||
- It sets `response_headers` to indicate "application/pdf" and `status_code` to 200.
|
||||
- **`async close(self)`**:
|
||||
- A method for cleaning up any resources used by the strategy. For `PDFCrawlerStrategy`, this is usually minimal.
|
||||
- **`async __aenter__(self)` / `async __aexit__(self, exc_type, exc_val, exc_tb)`**:
|
||||
- Enables asynchronous context management for the strategy, allowing it to be used with `async with`.
|
||||
|
||||
### Example Usage
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
|
||||
from crawl4ai.processors.pdf import PDFCrawlerStrategy, PDFContentScrapingStrategy
|
||||
|
||||
async def main():
|
||||
# Initialize the PDF crawler strategy
|
||||
pdf_crawler_strategy = PDFCrawlerStrategy()
|
||||
|
||||
# PDFCrawlerStrategy is typically used in conjunction with PDFContentScrapingStrategy
|
||||
# The scraping strategy handles the actual PDF content extraction
|
||||
pdf_scraping_strategy = PDFContentScrapingStrategy()
|
||||
run_config = CrawlerRunConfig(scraping_strategy=pdf_scraping_strategy)
|
||||
|
||||
async with AsyncWebCrawler(crawler_strategy=pdf_crawler_strategy) as crawler:
|
||||
# Example with a remote PDF URL
|
||||
pdf_url = "https://arxiv.org/pdf/2310.06825.pdf" # A public PDF from arXiv
|
||||
|
||||
print(f"Attempting to process PDF: {pdf_url}")
|
||||
result = await crawler.arun(url=pdf_url, config=run_config)
|
||||
|
||||
if result.success:
|
||||
print(f"Successfully processed PDF: {result.url}")
|
||||
print(f"Metadata Title: {result.metadata.get('title', 'N/A')}")
|
||||
# Further processing of result.markdown, result.media, etc.
|
||||
# would be done here, based on what PDFContentScrapingStrategy extracts.
|
||||
if result.markdown and hasattr(result.markdown, 'raw_markdown'):
|
||||
print(f"Extracted text (first 200 chars): {result.markdown.raw_markdown[:200]}...")
|
||||
else:
|
||||
print("No markdown (text) content extracted.")
|
||||
else:
|
||||
print(f"Failed to process PDF: {result.error_message}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Pros and Cons
|
||||
**Pros:**
|
||||
- Enables `AsyncWebCrawler` to handle PDF sources directly using familiar `arun` calls.
|
||||
- Provides a consistent interface for specifying PDF sources (URLs or local paths).
|
||||
- Abstracts the source handling, allowing a separate scraping strategy to focus on PDF content parsing.
|
||||
|
||||
**Cons:**
|
||||
- Does not perform any PDF data extraction itself; it strictly relies on a compatible scraping strategy (like `PDFContentScrapingStrategy`) to process the PDF.
|
||||
- Has limited utility on its own; most of its value comes from being paired with a PDF-specific content scraping strategy.
|
||||
|
||||
---
|
||||
|
||||
## `PDFContentScrapingStrategy`
|
||||
|
||||
### Overview
|
||||
`PDFContentScrapingStrategy` is an implementation of `ContentScrapingStrategy` designed to extract text, metadata, and optionally images from PDF documents. It is intended to be used in conjunction with a crawler strategy that can provide it with a PDF source, such as `PDFCrawlerStrategy`. This strategy uses the `NaivePDFProcessorStrategy` internally to perform the low-level PDF parsing.
|
||||
|
||||
### When to Use
|
||||
Use `PDFContentScrapingStrategy` when your `AsyncWebCrawler` (often configured with `PDFCrawlerStrategy`) needs to:
|
||||
- Extract textual content page by page from a PDF document.
|
||||
- Retrieve standard metadata embedded within the PDF (e.g., title, author, subject, creation date, page count).
|
||||
- Optionally, extract images contained within the PDF pages. These images can be saved to a local directory or made available for further processing.
|
||||
- Produce a `ScrapingResult` that can be converted into a `CrawlResult`, making PDF content accessible in a manner similar to HTML web content (e.g., text in `result.markdown`, metadata in `result.metadata`).
|
||||
|
||||
### Key Configuration Attributes
|
||||
When initializing `PDFContentScrapingStrategy`, you can configure its behavior using the following attributes:
|
||||
- **`extract_images: bool = False`**: If `True`, the strategy will attempt to extract images from the PDF.
|
||||
- **`save_images_locally: bool = False`**: If `True` (and `extract_images` is also `True`), extracted images will be saved to disk in the `image_save_dir`. If `False`, image data might be available in another form (e.g., base64, depending on the underlying processor) but not saved as separate files by this strategy.
|
||||
- **`image_save_dir: str = None`**: Specifies the directory where extracted images should be saved if `save_images_locally` is `True`. If `None`, a default or temporary directory might be used.
|
||||
- **`batch_size: int = 4`**: Defines how many PDF pages are processed in a single batch. This can be useful for managing memory when dealing with very large PDF documents.
|
||||
- **`logger: AsyncLogger = None`**: An optional `AsyncLogger` instance for logging.
|
||||
|
||||
### Key Methods and Their Behavior
|
||||
- **`__init__(self, save_images_locally: bool = False, extract_images: bool = False, image_save_dir: str = None, batch_size: int = 4, logger: AsyncLogger = None)`**:
|
||||
- Initializes the strategy with configurations for image handling, batch processing, and logging. It sets up an internal `NaivePDFProcessorStrategy` instance which performs the actual PDF parsing.
|
||||
- **`scrap(self, url: str, html: str, **params) -> ScrapingResult`**:
|
||||
- This is the primary synchronous method called by the crawler (via `ascrap`) to process the PDF.
|
||||
- `url`: The path or URL to the PDF file (provided by `PDFCrawlerStrategy` or similar).
|
||||
- `html`: Typically an empty string when used with `PDFCrawlerStrategy`, as the content is a PDF, not HTML.
|
||||
- It first ensures the PDF is accessible locally (downloads it to a temporary file if `url` is remote).
|
||||
- It then uses its internal PDF processor to extract text, metadata, and images (if configured).
|
||||
- The extracted information is compiled into a `ScrapingResult` object:
|
||||
- `cleaned_html`: Contains an HTML-like representation of the PDF, where each page's content is often wrapped in a `<div>` with page number information.
|
||||
- `media`: A dictionary where `media["images"]` will contain information about extracted images if `extract_images` was `True`.
|
||||
- `links`: A dictionary where `links["urls"]` can contain URLs found within the PDF content.
|
||||
- `metadata`: A dictionary holding PDF metadata (e.g., title, author, num_pages).
|
||||
- **`async ascrap(self, url: str, html: str, **kwargs) -> ScrapingResult`**:
|
||||
- The asynchronous version of `scrap`. Under the hood, it typically runs the synchronous `scrap` method in a separate thread using `asyncio.to_thread` to avoid blocking the event loop.
|
||||
- **`_get_pdf_path(self, url: str) -> str`**:
|
||||
- A private helper method to manage PDF file access. If the `url` is remote (http/https), it downloads the PDF to a temporary local file and returns its path. If `url` indicates a local file (`file://` or a direct path), it resolves and returns the local path.
|
||||
|
||||
### Example Usage
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
|
||||
from crawl4ai.processors.pdf import PDFCrawlerStrategy, PDFContentScrapingStrategy
|
||||
import os # For creating image directory
|
||||
|
||||
async def main():
|
||||
# Define the directory for saving extracted images
|
||||
image_output_dir = "./my_pdf_images"
|
||||
os.makedirs(image_output_dir, exist_ok=True)
|
||||
|
||||
# Configure the PDF content scraping strategy
|
||||
# Enable image extraction and specify where to save them
|
||||
pdf_scraping_cfg = PDFContentScrapingStrategy(
|
||||
extract_images=True,
|
||||
save_images_locally=True,
|
||||
image_save_dir=image_output_dir,
|
||||
batch_size=2 # Process 2 pages at a time for demonstration
|
||||
)
|
||||
|
||||
# The PDFCrawlerStrategy is needed to tell AsyncWebCrawler how to "crawl" a PDF
|
||||
pdf_crawler_cfg = PDFCrawlerStrategy()
|
||||
|
||||
# Configure the overall crawl run
|
||||
run_cfg = CrawlerRunConfig(
|
||||
scraping_strategy=pdf_scraping_cfg # Use our PDF scraping strategy
|
||||
)
|
||||
|
||||
# Initialize the crawler with the PDF-specific crawler strategy
|
||||
async with AsyncWebCrawler(crawler_strategy=pdf_crawler_cfg) as crawler:
|
||||
pdf_url = "https://arxiv.org/pdf/2310.06825.pdf" # Example PDF
|
||||
|
||||
print(f"Starting PDF processing for: {pdf_url}")
|
||||
result = await crawler.arun(url=pdf_url, config=run_cfg)
|
||||
|
||||
if result.success:
|
||||
print("\n--- PDF Processing Successful ---")
|
||||
print(f"Processed URL: {result.url}")
|
||||
|
||||
print("\n--- Metadata ---")
|
||||
for key, value in result.metadata.items():
|
||||
print(f" {key.replace('_', ' ').title()}: {value}")
|
||||
|
||||
if result.markdown and hasattr(result.markdown, 'raw_markdown'):
|
||||
print(f"\n--- Extracted Text (Markdown Snippet) ---")
|
||||
print(result.markdown.raw_markdown[:500].strip() + "...")
|
||||
else:
|
||||
print("\nNo text (markdown) content extracted.")
|
||||
|
||||
if result.media and result.media.get("images"):
|
||||
print(f"\n--- Image Extraction ---")
|
||||
print(f"Extracted {len(result.media['images'])} image(s).")
|
||||
for i, img_info in enumerate(result.media["images"][:2]): # Show info for first 2 images
|
||||
print(f" Image {i+1}:")
|
||||
print(f" Page: {img_info.get('page')}")
|
||||
print(f" Format: {img_info.get('format', 'N/A')}")
|
||||
if img_info.get('path'):
|
||||
print(f" Saved at: {img_info.get('path')}")
|
||||
else:
|
||||
print("\nNo images were extracted (or extract_images was False).")
|
||||
else:
|
||||
print(f"\n--- PDF Processing Failed ---")
|
||||
print(f"Error: {result.error_message}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
### Pros and Cons
|
||||
|
||||
**Pros:**
|
||||
- Provides a comprehensive way to extract text, metadata, and (optionally) images from PDF documents.
|
||||
- Handles both remote PDFs (via URL) and local PDF files.
|
||||
- Configurable image extraction allows saving images to disk or accessing their data.
|
||||
- Integrates smoothly with the `CrawlResult` object structure, making PDF-derived data accessible in a way consistent with web-scraped data.
|
||||
- The `batch_size` parameter can help in managing memory consumption when processing large or numerous PDF pages.
|
||||
|
||||
**Cons:**
|
||||
- Extraction quality and performance can vary significantly depending on the PDF's complexity, encoding, and whether it's image-based (scanned) or text-based.
|
||||
- Image extraction can be resource-intensive (both CPU and disk space if `save_images_locally` is true).
|
||||
- Relies on `NaivePDFProcessorStrategy` internally, which might have limitations with very complex layouts, encrypted PDFs, or forms compared to more sophisticated PDF parsing libraries. Scanned PDFs will not yield text unless an OCR step is performed (which is not part of this strategy by default).
|
||||
- Link extraction from PDFs can be basic and depends on how hyperlinks are embedded in the document.
|
||||
@@ -0,0 +1,308 @@
|
||||
# Proxy & Security
|
||||
|
||||
This guide covers proxy configuration and security features in Crawl4AI, including SSL certificate analysis and proxy rotation strategies.
|
||||
|
||||
## Understanding Proxy Configuration
|
||||
|
||||
Crawl4AI recommends configuring proxies per request through `CrawlerRunConfig.proxy_config`. This gives you precise control, enables rotation strategies, and keeps examples simple enough to copy, paste, and run.
|
||||
|
||||
## Basic Proxy Setup
|
||||
|
||||
Configure proxies that apply to each crawl operation:
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, ProxyConfig
|
||||
|
||||
run_config = CrawlerRunConfig(proxy_config=ProxyConfig(server="http://proxy.example.com:8080"))
|
||||
# run_config = CrawlerRunConfig(proxy_config={"server": "http://proxy.example.com:8080"})
|
||||
# run_config = CrawlerRunConfig(proxy_config="http://proxy.example.com:8080")
|
||||
|
||||
|
||||
async def main():
|
||||
browser_config = BrowserConfig()
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
result = await crawler.arun(url="https://example.com", config=run_config)
|
||||
print(f"Success: {result.success} -> {result.url}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
!!! note "Why request-level?"
|
||||
`CrawlerRunConfig.proxy_config` keeps each request self-contained, so swapping proxies or rotation strategies is just a matter of building a new run configuration.
|
||||
|
||||
## Supported Proxy Formats
|
||||
|
||||
The `ProxyConfig.from_string()` method supports multiple formats:
|
||||
|
||||
```python
|
||||
from crawl4ai import ProxyConfig
|
||||
|
||||
# HTTP proxy with authentication
|
||||
proxy1 = ProxyConfig.from_string("http://user:pass@192.168.1.1:8080")
|
||||
|
||||
# HTTPS proxy
|
||||
proxy2 = ProxyConfig.from_string("https://proxy.example.com:8080")
|
||||
|
||||
# SOCKS5 proxy
|
||||
proxy3 = ProxyConfig.from_string("socks5://proxy.example.com:1080")
|
||||
|
||||
# Simple IP:port format
|
||||
proxy4 = ProxyConfig.from_string("192.168.1.1:8080")
|
||||
|
||||
# IP:port:user:pass format
|
||||
proxy5 = ProxyConfig.from_string("192.168.1.1:8080:user:pass")
|
||||
```
|
||||
|
||||
## Authenticated Proxies
|
||||
|
||||
For proxies requiring authentication:
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler,BrowserConfig, CrawlerRunConfig, ProxyConfig
|
||||
|
||||
run_config = CrawlerRunConfig(
|
||||
proxy_config=ProxyConfig(
|
||||
server="http://proxy.example.com:8080",
|
||||
username="your_username",
|
||||
password="your_password",
|
||||
)
|
||||
)
|
||||
# Or dictionary style:
|
||||
# run_config = CrawlerRunConfig(proxy_config={
|
||||
# "server": "http://proxy.example.com:8080",
|
||||
# "username": "your_username",
|
||||
# "password": "your_password",
|
||||
# })
|
||||
|
||||
|
||||
async def main():
|
||||
browser_config = BrowserConfig()
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
result = await crawler.arun(url="https://example.com", config=run_config)
|
||||
print(f"Success: {result.success} -> {result.url}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Environment Variable Configuration
|
||||
|
||||
Load proxies from environment variables for easy configuration:
|
||||
|
||||
```python
|
||||
import os
|
||||
from crawl4ai import ProxyConfig, CrawlerRunConfig
|
||||
|
||||
# Set environment variable
|
||||
os.environ["PROXIES"] = "ip1:port1:user1:pass1,ip2:port2:user2:pass2,ip3:port3"
|
||||
|
||||
# Load all proxies
|
||||
proxies = ProxyConfig.from_env()
|
||||
print(f"Loaded {len(proxies)} proxies")
|
||||
|
||||
# Use first proxy
|
||||
if proxies:
|
||||
run_config = CrawlerRunConfig(proxy_config=proxies[0])
|
||||
```
|
||||
|
||||
## Rotating Proxies
|
||||
|
||||
Crawl4AI supports automatic proxy rotation to distribute requests across multiple proxy servers. Rotation is applied per request using a rotation strategy on `CrawlerRunConfig`.
|
||||
|
||||
### Proxy Rotation (recommended)
|
||||
```python
|
||||
import asyncio
|
||||
import re
|
||||
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode, ProxyConfig
|
||||
from crawl4ai.proxy_strategy import RoundRobinProxyStrategy
|
||||
|
||||
async def main():
|
||||
# Load proxies from environment
|
||||
proxies = ProxyConfig.from_env()
|
||||
if not proxies:
|
||||
print("No proxies found! Set PROXIES environment variable.")
|
||||
return
|
||||
|
||||
# Create rotation strategy
|
||||
proxy_strategy = RoundRobinProxyStrategy(proxies)
|
||||
|
||||
# Configure per-request with proxy rotation
|
||||
browser_config = BrowserConfig(headless=True, verbose=False)
|
||||
run_config = CrawlerRunConfig(
|
||||
cache_mode=CacheMode.BYPASS,
|
||||
proxy_rotation_strategy=proxy_strategy,
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
urls = ["https://httpbin.org/ip"] * (len(proxies) * 2) # Test each proxy twice
|
||||
|
||||
print(f"🚀 Testing {len(proxies)} proxies with rotation...")
|
||||
results = await crawler.arun_many(urls=urls, config=run_config)
|
||||
|
||||
for i, result in enumerate(results):
|
||||
if result.success:
|
||||
# Extract IP from response
|
||||
ip_match = re.search(r'(?:[0-9]{1,3}\.){3}[0-9]{1,3}', result.html)
|
||||
if ip_match:
|
||||
detected_ip = ip_match.group(0)
|
||||
proxy_index = i % len(proxies)
|
||||
expected_ip = proxies[proxy_index].ip
|
||||
|
||||
print(f"✅ Request {i+1}: Proxy {proxy_index+1} -> IP {detected_ip}")
|
||||
if detected_ip == expected_ip:
|
||||
print(" 🎯 IP matches proxy configuration")
|
||||
else:
|
||||
print(f" ⚠️ IP mismatch (expected {expected_ip})")
|
||||
else:
|
||||
print(f"❌ Request {i+1}: Could not extract IP from response")
|
||||
else:
|
||||
print(f"❌ Request {i+1}: Failed - {result.error_message}")
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## SSL Certificate Analysis
|
||||
|
||||
Combine proxy usage with SSL certificate inspection for enhanced security analysis. SSL certificate fetching is configured per request via `CrawlerRunConfig`.
|
||||
|
||||
### Per-Request SSL Certificate Analysis
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig
|
||||
|
||||
run_config = CrawlerRunConfig(
|
||||
proxy_config={
|
||||
"server": "http://proxy.example.com:8080",
|
||||
"username": "user",
|
||||
"password": "pass",
|
||||
},
|
||||
fetch_ssl_certificate=True, # Enable SSL certificate analysis for this request
|
||||
)
|
||||
|
||||
|
||||
async def main():
|
||||
browser_config = BrowserConfig()
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
result = await crawler.arun(url="https://example.com", config=run_config)
|
||||
|
||||
if result.success:
|
||||
print(f"✅ Crawled via proxy: {result.url}")
|
||||
|
||||
# Analyze SSL certificate
|
||||
if result.ssl_certificate:
|
||||
cert = result.ssl_certificate
|
||||
print("🔒 SSL Certificate Info:")
|
||||
print(f" Issuer: {cert.issuer}")
|
||||
print(f" Subject: {cert.subject}")
|
||||
print(f" Valid until: {cert.valid_until}")
|
||||
print(f" Fingerprint: {cert.fingerprint}")
|
||||
|
||||
# Export certificate
|
||||
cert.to_json("certificate.json")
|
||||
print("💾 Certificate exported to certificate.json")
|
||||
else:
|
||||
print("⚠️ No SSL certificate information available")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Security Best Practices
|
||||
|
||||
### 1. Proxy Rotation for Anonymity
|
||||
```python
|
||||
from crawl4ai import CrawlerRunConfig, ProxyConfig
|
||||
from crawl4ai.proxy_strategy import RoundRobinProxyStrategy
|
||||
|
||||
# Use multiple proxies to avoid IP blocking
|
||||
proxies = ProxyConfig.from_env("PROXIES")
|
||||
strategy = RoundRobinProxyStrategy(proxies)
|
||||
|
||||
# Configure rotation per request (recommended)
|
||||
run_config = CrawlerRunConfig(proxy_rotation_strategy=strategy)
|
||||
|
||||
# For a fixed proxy across all requests, just reuse the same run_config instance
|
||||
static_run_config = run_config
|
||||
```
|
||||
|
||||
### 2. SSL Certificate Verification
|
||||
```python
|
||||
from crawl4ai import CrawlerRunConfig
|
||||
|
||||
# Always verify SSL certificates when possible
|
||||
# Per-request (affects specific requests)
|
||||
run_config = CrawlerRunConfig(fetch_ssl_certificate=True)
|
||||
```
|
||||
|
||||
### 3. Environment Variable Security
|
||||
```bash
|
||||
# Use environment variables for sensitive proxy credentials
|
||||
# Avoid hardcoding usernames/passwords in code
|
||||
export PROXIES="ip1:port1:user1:pass1,ip2:port2:user2:pass2"
|
||||
```
|
||||
|
||||
### 4. SOCKS5 for Enhanced Security
|
||||
```python
|
||||
from crawl4ai import CrawlerRunConfig
|
||||
|
||||
# Prefer SOCKS5 proxies for better protocol support
|
||||
run_config = CrawlerRunConfig(proxy_config="socks5://proxy.example.com:1080")
|
||||
```
|
||||
|
||||
## Migration from Deprecated `proxy` Parameter
|
||||
|
||||
- "Deprecation Notice"
|
||||
The legacy `proxy` argument on `BrowserConfig` is deprecated. Configure proxies through `CrawlerRunConfig.proxy_config` so each request fully describes its network settings.
|
||||
|
||||
```python
|
||||
# Old (deprecated) approach
|
||||
# from crawl4ai import BrowserConfig
|
||||
# browser_config = BrowserConfig(proxy_config="http://proxy.example.com:8080")
|
||||
|
||||
# New (preferred) approach
|
||||
from crawl4ai import CrawlerRunConfig
|
||||
run_config = CrawlerRunConfig(proxy_config="http://proxy.example.com:8080")
|
||||
```
|
||||
|
||||
### Safe Logging of Proxies
|
||||
```python
|
||||
from crawl4ai import ProxyConfig
|
||||
|
||||
def safe_proxy_repr(proxy: ProxyConfig):
|
||||
if getattr(proxy, "username", None):
|
||||
return f"{proxy.server} (auth: ****)"
|
||||
return proxy.server
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
- "Proxy connection failed"
|
||||
- Verify the proxy server is reachable from your network.
|
||||
- Double-check authentication credentials.
|
||||
- Ensure the protocol matches (`http`, `https`, or `socks5`).
|
||||
|
||||
- "SSL certificate errors"
|
||||
- Some proxies break SSL inspection; switch proxies if you see repeated failures.
|
||||
- Consider temporarily disabling certificate fetching to isolate the issue.
|
||||
|
||||
- "Environment variables not loading"
|
||||
- Confirm `PROXIES` (or your custom env var) is set before running the script.
|
||||
- Check formatting: `ip:port:user:pass,ip:port:user:pass`.
|
||||
|
||||
- "Proxy rotation not working"
|
||||
- Ensure `ProxyConfig.from_env()` actually loaded entries (`len(proxies) > 0`).
|
||||
- Attach `proxy_rotation_strategy` to `CrawlerRunConfig`.
|
||||
- Validate the proxy definitions you pass into the strategy.
|
||||
|
||||
## See Also
|
||||
|
||||
- [Anti-Bot Detection & Fallback](anti-bot-and-fallback.md) — Automatic retry with proxy escalation and fallback functions when anti-bot blocking is detected
|
||||
@@ -0,0 +1,267 @@
|
||||
# Session Management
|
||||
|
||||
Session management in Crawl4AI is a powerful feature that allows you to maintain state across multiple requests, making it particularly suitable for handling complex multi-step crawling tasks. It enables you to reuse the same browser tab (or page object) across sequential actions and crawls, which is beneficial for:
|
||||
|
||||
- **Performing JavaScript actions before and after crawling.**
|
||||
- **Executing multiple sequential crawls faster** without needing to reopen tabs or allocate memory repeatedly.
|
||||
|
||||
**Note:** This feature is designed for sequential workflows and is not suitable for parallel operations.
|
||||
|
||||
---
|
||||
|
||||
#### Basic Session Usage
|
||||
|
||||
Use `BrowserConfig` and `CrawlerRunConfig` to maintain state with a `session_id`:
|
||||
|
||||
```python
|
||||
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
session_id = "my_session"
|
||||
|
||||
# Define configurations
|
||||
config1 = CrawlerRunConfig(
|
||||
url="https://example.com/page1", session_id=session_id
|
||||
)
|
||||
config2 = CrawlerRunConfig(
|
||||
url="https://example.com/page2", session_id=session_id
|
||||
)
|
||||
|
||||
# First request
|
||||
result1 = await crawler.arun(config=config1)
|
||||
|
||||
# Subsequent request using the same session
|
||||
result2 = await crawler.arun(config=config2)
|
||||
|
||||
# Clean up when done
|
||||
await crawler.crawler_strategy.kill_session(session_id)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### Dynamic Content with Sessions
|
||||
|
||||
Here's an example of crawling GitHub commits across multiple pages while preserving session state:
|
||||
|
||||
```python
|
||||
from crawl4ai.async_configs import CrawlerRunConfig
|
||||
from crawl4ai import JsonCssExtractionStrategy
|
||||
from crawl4ai.cache_context import CacheMode
|
||||
|
||||
async def crawl_dynamic_content():
|
||||
url = "https://github.com/microsoft/TypeScript/commits/main"
|
||||
session_id = "wait_for_session"
|
||||
all_commits = []
|
||||
|
||||
js_next_page = """
|
||||
const commits = document.querySelectorAll('li[data-testid="commit-row-item"] h4');
|
||||
if (commits.length > 0) {
|
||||
window.lastCommit = commits[0].textContent.trim();
|
||||
}
|
||||
const button = document.querySelector('a[data-testid="pagination-next-button"]');
|
||||
if (button) {button.click(); console.log('button clicked') }
|
||||
"""
|
||||
|
||||
wait_for = """() => {
|
||||
const commits = document.querySelectorAll('li[data-testid="commit-row-item"] h4');
|
||||
if (commits.length === 0) return false;
|
||||
const firstCommit = commits[0].textContent.trim();
|
||||
return firstCommit !== window.lastCommit;
|
||||
}"""
|
||||
|
||||
schema = {
|
||||
"name": "Commit Extractor",
|
||||
"baseSelector": "li[data-testid='commit-row-item']",
|
||||
"fields": [
|
||||
{
|
||||
"name": "title",
|
||||
"selector": "h4 a",
|
||||
"type": "text",
|
||||
"transform": "strip",
|
||||
},
|
||||
],
|
||||
}
|
||||
extraction_strategy = JsonCssExtractionStrategy(schema, verbose=True)
|
||||
|
||||
|
||||
browser_config = BrowserConfig(
|
||||
verbose=True,
|
||||
headless=False,
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
for page in range(3):
|
||||
crawler_config = CrawlerRunConfig(
|
||||
session_id=session_id,
|
||||
css_selector="li[data-testid='commit-row-item']",
|
||||
extraction_strategy=extraction_strategy,
|
||||
js_code=js_next_page if page > 0 else None,
|
||||
wait_for=wait_for if page > 0 else None,
|
||||
js_only=page > 0,
|
||||
cache_mode=CacheMode.BYPASS,
|
||||
capture_console_messages=True,
|
||||
)
|
||||
|
||||
result = await crawler.arun(url=url, config=crawler_config)
|
||||
|
||||
if result.console_messages:
|
||||
print(f"Page {page + 1} console messages:", result.console_messages)
|
||||
|
||||
if result.extracted_content:
|
||||
# print(f"Page {page + 1} result:", result.extracted_content)
|
||||
commits = json.loads(result.extracted_content)
|
||||
all_commits.extend(commits)
|
||||
print(f"Page {page + 1}: Found {len(commits)} commits")
|
||||
else:
|
||||
print(f"Page {page + 1}: No content extracted")
|
||||
|
||||
print(f"Successfully crawled {len(all_commits)} commits across 3 pages")
|
||||
# Clean up session
|
||||
await crawler.crawler_strategy.kill_session(session_id)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Example 1: Basic Session-Based Crawling
|
||||
|
||||
A simple example using session-based crawling:
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai.async_configs import BrowserConfig, CrawlerRunConfig
|
||||
from crawl4ai.cache_context import CacheMode
|
||||
|
||||
async def basic_session_crawl():
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
session_id = "dynamic_content_session"
|
||||
url = "https://example.com/dynamic-content"
|
||||
|
||||
for page in range(3):
|
||||
config = CrawlerRunConfig(
|
||||
url=url,
|
||||
session_id=session_id,
|
||||
js_code="document.querySelector('.load-more-button').click();" if page > 0 else None,
|
||||
css_selector=".content-item",
|
||||
cache_mode=CacheMode.BYPASS
|
||||
)
|
||||
|
||||
result = await crawler.arun(config=config)
|
||||
print(f"Page {page + 1}: Found {result.extracted_content.count('.content-item')} items")
|
||||
|
||||
await crawler.crawler_strategy.kill_session(session_id)
|
||||
|
||||
asyncio.run(basic_session_crawl())
|
||||
```
|
||||
|
||||
This example shows:
|
||||
1. Reusing the same `session_id` across multiple requests.
|
||||
2. Executing JavaScript to load more content dynamically.
|
||||
3. Properly closing the session to free resources.
|
||||
|
||||
---
|
||||
|
||||
## Advanced Technique 1: Custom Execution Hooks
|
||||
|
||||
> Warning: You might feel confused by the end of the next few examples 😅, so make sure you are comfortable with the order of the parts before you start this.
|
||||
|
||||
Use custom hooks to handle complex scenarios, such as waiting for content to load dynamically:
|
||||
|
||||
```python
|
||||
async def advanced_session_crawl_with_hooks():
|
||||
first_commit = ""
|
||||
|
||||
async def on_execution_started(page):
|
||||
nonlocal first_commit
|
||||
try:
|
||||
while True:
|
||||
await page.wait_for_selector("li.commit-item h4")
|
||||
commit = await page.query_selector("li.commit-item h4")
|
||||
commit = await commit.evaluate("(element) => element.textContent").strip()
|
||||
if commit and commit != first_commit:
|
||||
first_commit = commit
|
||||
break
|
||||
await asyncio.sleep(0.5)
|
||||
except Exception as e:
|
||||
print(f"Warning: New content didn't appear: {e}")
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
session_id = "commit_session"
|
||||
url = "https://github.com/example/repo/commits/main"
|
||||
crawler.crawler_strategy.set_hook("on_execution_started", on_execution_started)
|
||||
|
||||
js_next_page = """document.querySelector('a.pagination-next').click();"""
|
||||
|
||||
for page in range(3):
|
||||
config = CrawlerRunConfig(
|
||||
url=url,
|
||||
session_id=session_id,
|
||||
js_code=js_next_page if page > 0 else None,
|
||||
css_selector="li.commit-item",
|
||||
js_only=page > 0,
|
||||
cache_mode=CacheMode.BYPASS
|
||||
)
|
||||
|
||||
result = await crawler.arun(config=config)
|
||||
print(f"Page {page + 1}: Found {len(result.extracted_content)} commits")
|
||||
|
||||
await crawler.crawler_strategy.kill_session(session_id)
|
||||
|
||||
asyncio.run(advanced_session_crawl_with_hooks())
|
||||
```
|
||||
|
||||
This technique ensures new content loads before the next action.
|
||||
|
||||
---
|
||||
|
||||
## Advanced Technique 2: Integrated JavaScript Execution and Waiting
|
||||
|
||||
Combine JavaScript execution and waiting logic for concise handling of dynamic content:
|
||||
|
||||
```python
|
||||
async def integrated_js_and_wait_crawl():
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
session_id = "integrated_session"
|
||||
url = "https://github.com/example/repo/commits/main"
|
||||
|
||||
js_next_page_and_wait = """
|
||||
(async () => {
|
||||
const getCurrentCommit = () => document.querySelector('li.commit-item h4').textContent.trim();
|
||||
const initialCommit = getCurrentCommit();
|
||||
document.querySelector('a.pagination-next').click();
|
||||
while (getCurrentCommit() === initialCommit) {
|
||||
await new Promise(resolve => setTimeout(resolve, 100));
|
||||
}
|
||||
})();
|
||||
"""
|
||||
|
||||
for page in range(3):
|
||||
config = CrawlerRunConfig(
|
||||
url=url,
|
||||
session_id=session_id,
|
||||
js_code=js_next_page_and_wait if page > 0 else None,
|
||||
css_selector="li.commit-item",
|
||||
js_only=page > 0,
|
||||
cache_mode=CacheMode.BYPASS
|
||||
)
|
||||
|
||||
result = await crawler.arun(config=config)
|
||||
print(f"Page {page + 1}: Found {len(result.extracted_content)} commits")
|
||||
|
||||
await crawler.crawler_strategy.kill_session(session_id)
|
||||
|
||||
asyncio.run(integrated_js_and_wait_crawl())
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### Common Use Cases for Sessions
|
||||
|
||||
1. **Authentication Flows**: Login and interact with secured pages.
|
||||
|
||||
2. **Pagination Handling**: Navigate through multiple pages.
|
||||
|
||||
3. **Form Submissions**: Fill forms, submit, and process results.
|
||||
|
||||
4. **Multi-step Processes**: Complete workflows that span multiple actions.
|
||||
|
||||
5. **Dynamic Content Navigation**: Handle JavaScript-rendered or event-triggered content.
|
||||
@@ -0,0 +1,179 @@
|
||||
# `SSLCertificate` Reference
|
||||
|
||||
The **`SSLCertificate`** class encapsulates an SSL certificate’s data and allows exporting it in various formats (PEM, DER, JSON, or text). It’s used within **Crawl4AI** whenever you set **`fetch_ssl_certificate=True`** in your **`CrawlerRunConfig`**.
|
||||
|
||||
## 1. Overview
|
||||
|
||||
**Location**: `crawl4ai/ssl_certificate.py`
|
||||
|
||||
```python
|
||||
class SSLCertificate:
|
||||
"""
|
||||
Represents an SSL certificate with methods to export in various formats.
|
||||
|
||||
Main Methods:
|
||||
- from_url(url, timeout=10)
|
||||
- from_file(file_path)
|
||||
- from_binary(binary_data)
|
||||
- to_json(filepath=None)
|
||||
- to_pem(filepath=None)
|
||||
- to_der(filepath=None)
|
||||
...
|
||||
|
||||
Common Properties:
|
||||
- issuer
|
||||
- subject
|
||||
- valid_from
|
||||
- valid_until
|
||||
- fingerprint
|
||||
"""
|
||||
```
|
||||
|
||||
### Typical Use Case
|
||||
1. You **enable** certificate fetching in your crawl by:
|
||||
```python
|
||||
CrawlerRunConfig(fetch_ssl_certificate=True, ...)
|
||||
```
|
||||
2. After `arun()`, if `result.ssl_certificate` is present, it’s an instance of **`SSLCertificate`**.
|
||||
3. You can **read** basic properties (issuer, subject, validity) or **export** them in multiple formats.
|
||||
|
||||
---
|
||||
|
||||
## 2. Construction & Fetching
|
||||
|
||||
### 2.1 **`from_url(url, timeout=10)`**
|
||||
Manually load an SSL certificate from a given URL (port 443). Typically used internally, but you can call it directly if you want:
|
||||
|
||||
```python
|
||||
cert = SSLCertificate.from_url("https://example.com")
|
||||
if cert:
|
||||
print("Fingerprint:", cert.fingerprint)
|
||||
```
|
||||
|
||||
### 2.2 **`from_file(file_path)`**
|
||||
Load from a file containing certificate data in ASN.1 or DER. Rarely needed unless you have local cert files:
|
||||
|
||||
```python
|
||||
cert = SSLCertificate.from_file("/path/to/cert.der")
|
||||
```
|
||||
|
||||
### 2.3 **`from_binary(binary_data)`**
|
||||
Initialize from raw binary. E.g., if you captured it from a socket or another source:
|
||||
|
||||
```python
|
||||
cert = SSLCertificate.from_binary(raw_bytes)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. Common Properties
|
||||
|
||||
After obtaining a **`SSLCertificate`** instance (e.g. `result.ssl_certificate` from a crawl), you can read:
|
||||
|
||||
1. **`issuer`** *(dict)*
|
||||
- E.g. `{"CN": "My Root CA", "O": "..."}`
|
||||
2. **`subject`** *(dict)*
|
||||
- E.g. `{"CN": "example.com", "O": "ExampleOrg"}`
|
||||
3. **`valid_from`** *(str)*
|
||||
- NotBefore date/time. Often in ASN.1/UTC format.
|
||||
4. **`valid_until`** *(str)*
|
||||
- NotAfter date/time.
|
||||
5. **`fingerprint`** *(str)*
|
||||
- The SHA-256 digest (lowercase hex).
|
||||
- E.g. `"d14d2e..."`
|
||||
|
||||
---
|
||||
|
||||
## 4. Export Methods
|
||||
|
||||
Once you have a **`SSLCertificate`** object, you can **export** or **inspect** it:
|
||||
|
||||
### 4.1 **`to_json(filepath=None)` → `Optional[str]`**
|
||||
- Returns a JSON string containing the parsed certificate fields.
|
||||
- If `filepath` is provided, saves it to disk instead, returning `None`.
|
||||
|
||||
**Usage**:
|
||||
```python
|
||||
json_data = cert.to_json() # returns JSON string
|
||||
cert.to_json("certificate.json") # writes file, returns None
|
||||
```
|
||||
|
||||
### 4.2 **`to_pem(filepath=None)` → `Optional[str]`**
|
||||
- Returns a PEM-encoded string (common for web servers).
|
||||
- If `filepath` is provided, saves it to disk instead.
|
||||
|
||||
```python
|
||||
pem_str = cert.to_pem() # in-memory PEM string
|
||||
cert.to_pem("/path/to/cert.pem") # saved to file
|
||||
```
|
||||
|
||||
### 4.3 **`to_der(filepath=None)` → `Optional[bytes]`**
|
||||
- Returns the original DER (binary ASN.1) bytes.
|
||||
- If `filepath` is specified, writes the bytes there instead.
|
||||
|
||||
```python
|
||||
der_bytes = cert.to_der()
|
||||
cert.to_der("certificate.der")
|
||||
```
|
||||
|
||||
### 4.4 (Optional) **`export_as_text()`**
|
||||
- If you see a method like `export_as_text()`, it typically returns an OpenSSL-style textual representation.
|
||||
- Not always needed, but can help for debugging or manual inspection.
|
||||
|
||||
---
|
||||
|
||||
## 5. Example Usage in Crawl4AI
|
||||
|
||||
Below is a minimal sample showing how the crawler obtains an SSL cert from a site, then reads or exports it. The code snippet:
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
import os
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, CacheMode
|
||||
|
||||
async def main():
|
||||
tmp_dir = "tmp"
|
||||
os.makedirs(tmp_dir, exist_ok=True)
|
||||
|
||||
config = CrawlerRunConfig(
|
||||
fetch_ssl_certificate=True,
|
||||
cache_mode=CacheMode.BYPASS
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun("https://example.com", config=config)
|
||||
if result.success and result.ssl_certificate:
|
||||
cert = result.ssl_certificate
|
||||
# 1. Basic Info
|
||||
print("Issuer CN:", cert.issuer.get("CN", ""))
|
||||
print("Valid until:", cert.valid_until)
|
||||
print("Fingerprint:", cert.fingerprint)
|
||||
|
||||
# 2. Export
|
||||
cert.to_json(os.path.join(tmp_dir, "certificate.json"))
|
||||
cert.to_pem(os.path.join(tmp_dir, "certificate.pem"))
|
||||
cert.to_der(os.path.join(tmp_dir, "certificate.der"))
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 6. Notes & Best Practices
|
||||
|
||||
1. **Timeout**: `SSLCertificate.from_url` internally uses a default **10s** socket connect and wraps SSL.
|
||||
2. **Binary Form**: The certificate is loaded in ASN.1 (DER) form, then re-parsed by `OpenSSL.crypto`.
|
||||
3. **Validation**: This does **not** validate the certificate chain or trust store. It only fetches and parses.
|
||||
4. **Integration**: Within Crawl4AI, you typically just set `fetch_ssl_certificate=True` in `CrawlerRunConfig`; the final result’s `ssl_certificate` is automatically built.
|
||||
5. **Export**: If you need to store or analyze a cert, the `to_json` and `to_pem` are quite universal.
|
||||
|
||||
---
|
||||
|
||||
### Summary
|
||||
|
||||
- **`SSLCertificate`** is a convenience class for capturing and exporting the **TLS certificate** from your crawled site(s).
|
||||
- Common usage is in the **`CrawlResult.ssl_certificate`** field, accessible after setting `fetch_ssl_certificate=True`.
|
||||
- Offers quick access to essential certificate details (`issuer`, `subject`, `fingerprint`) and is easy to export (PEM, DER, JSON) for further analysis or server usage.
|
||||
|
||||
Use it whenever you need **insight** into a site’s certificate or require some form of cryptographic or compliance check.
|
||||
@@ -0,0 +1,395 @@
|
||||
# Undetected Browser Mode
|
||||
|
||||
## Overview
|
||||
|
||||
Crawl4AI offers two powerful anti-bot features to help you access websites with bot detection:
|
||||
|
||||
1. **Stealth Mode** - Uses playwright-stealth to modify browser fingerprints and behaviors
|
||||
2. **Undetected Browser Mode** - Advanced browser adapter with deep-level patches for sophisticated bot detection
|
||||
|
||||
This guide covers both features and helps you choose the right approach for your needs.
|
||||
|
||||
## Anti-Bot Features Comparison
|
||||
|
||||
| Feature | Regular Browser | Stealth Mode | Undetected Browser |
|
||||
|---------|----------------|--------------|-------------------|
|
||||
| WebDriver Detection | ❌ | ✅ | ✅ |
|
||||
| Navigator Properties | ❌ | ✅ | ✅ |
|
||||
| Plugin Emulation | ❌ | ✅ | ✅ |
|
||||
| CDP Detection | ❌ | Partial | ✅ |
|
||||
| Deep Browser Patches | ❌ | ❌ | ✅ |
|
||||
| Performance Impact | None | Minimal | Moderate |
|
||||
| Setup Complexity | None | None | Minimal |
|
||||
|
||||
## When to Use Each Approach
|
||||
|
||||
### Use Regular Browser + Stealth Mode When:
|
||||
- Sites have basic bot detection (checking navigator.webdriver, plugins, etc.)
|
||||
- You need good performance with basic protection
|
||||
- Sites check for common automation indicators
|
||||
|
||||
### Use Undetected Browser When:
|
||||
- Sites employ sophisticated bot detection services (Cloudflare, DataDome, etc.)
|
||||
- Stealth mode alone isn't sufficient
|
||||
- You're willing to trade some performance for better evasion
|
||||
|
||||
### Best Practice: Progressive Enhancement
|
||||
1. **Start with**: Regular browser + Stealth mode
|
||||
2. **If blocked**: Switch to Undetected browser
|
||||
3. **If still blocked**: Combine Undetected browser + Stealth mode
|
||||
|
||||
## Stealth Mode
|
||||
|
||||
Stealth mode is the simpler anti-bot solution that works with both regular and undetected browsers:
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, BrowserConfig
|
||||
|
||||
# Enable stealth mode with regular browser
|
||||
browser_config = BrowserConfig(
|
||||
enable_stealth=True, # Simple flag to enable
|
||||
headless=False # Better for avoiding detection
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
result = await crawler.arun("https://example.com")
|
||||
```
|
||||
|
||||
### What Stealth Mode Does:
|
||||
- Removes `navigator.webdriver` flag
|
||||
- Modifies browser fingerprints
|
||||
- Emulates realistic plugin behavior
|
||||
- Adjusts navigator properties
|
||||
- Fixes common automation leaks
|
||||
|
||||
## Undetected Browser Mode
|
||||
|
||||
For sites with sophisticated bot detection that stealth mode can't bypass, use the undetected browser adapter:
|
||||
|
||||
### Key Features
|
||||
|
||||
- **Drop-in Replacement**: Uses the same API as regular browser mode
|
||||
- **Enhanced Stealth**: Built-in patches to evade common detection methods
|
||||
- **Browser Adapter Pattern**: Seamlessly switch between regular and undetected modes
|
||||
- **Automatic Installation**: `crawl4ai-setup` installs all necessary browser dependencies
|
||||
|
||||
### Quick Start
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import (
|
||||
AsyncWebCrawler,
|
||||
BrowserConfig,
|
||||
CrawlerRunConfig,
|
||||
UndetectedAdapter
|
||||
)
|
||||
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy
|
||||
|
||||
async def main():
|
||||
# Create the undetected adapter
|
||||
undetected_adapter = UndetectedAdapter()
|
||||
|
||||
# Create browser config
|
||||
browser_config = BrowserConfig(
|
||||
headless=False, # Headless mode can be detected easier
|
||||
verbose=True,
|
||||
)
|
||||
|
||||
# Create the crawler strategy with undetected adapter
|
||||
crawler_strategy = AsyncPlaywrightCrawlerStrategy(
|
||||
browser_config=browser_config,
|
||||
browser_adapter=undetected_adapter
|
||||
)
|
||||
|
||||
# Create the crawler with our custom strategy
|
||||
async with AsyncWebCrawler(
|
||||
crawler_strategy=crawler_strategy,
|
||||
config=browser_config
|
||||
) as crawler:
|
||||
# Your crawling code here
|
||||
result = await crawler.arun(
|
||||
url="https://example.com",
|
||||
config=CrawlerRunConfig()
|
||||
)
|
||||
print(result.markdown[:500])
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Combining Both Features
|
||||
|
||||
For maximum evasion, combine stealth mode with undetected browser:
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, BrowserConfig, UndetectedAdapter
|
||||
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy
|
||||
|
||||
# Create browser config with stealth enabled
|
||||
browser_config = BrowserConfig(
|
||||
enable_stealth=True, # Enable stealth mode
|
||||
headless=False
|
||||
)
|
||||
|
||||
# Create undetected adapter
|
||||
adapter = UndetectedAdapter()
|
||||
|
||||
# Create strategy with both features
|
||||
strategy = AsyncPlaywrightCrawlerStrategy(
|
||||
browser_config=browser_config,
|
||||
browser_adapter=adapter
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(
|
||||
crawler_strategy=strategy,
|
||||
config=browser_config
|
||||
) as crawler:
|
||||
result = await crawler.arun("https://protected-site.com")
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
### Example 1: Basic Stealth Mode
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig
|
||||
|
||||
async def test_stealth_mode():
|
||||
# Simple stealth mode configuration
|
||||
browser_config = BrowserConfig(
|
||||
enable_stealth=True,
|
||||
headless=False
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://bot.sannysoft.com",
|
||||
config=CrawlerRunConfig(screenshot=True)
|
||||
)
|
||||
|
||||
if result.success:
|
||||
print("✓ Successfully accessed bot detection test site")
|
||||
# Save screenshot to verify detection results
|
||||
if result.screenshot:
|
||||
import base64
|
||||
with open("stealth_test.png", "wb") as f:
|
||||
f.write(base64.b64decode(result.screenshot))
|
||||
print("✓ Screenshot saved - check for green (passed) tests")
|
||||
|
||||
asyncio.run(test_stealth_mode())
|
||||
```
|
||||
|
||||
### Example 2: Undetected Browser Mode
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import (
|
||||
AsyncWebCrawler,
|
||||
BrowserConfig,
|
||||
CrawlerRunConfig,
|
||||
UndetectedAdapter
|
||||
)
|
||||
from crawl4ai.async_crawler_strategy import AsyncPlaywrightCrawlerStrategy
|
||||
|
||||
|
||||
async def main():
|
||||
# Create browser config
|
||||
browser_config = BrowserConfig(
|
||||
headless=False,
|
||||
verbose=True,
|
||||
)
|
||||
|
||||
# Create the undetected adapter
|
||||
undetected_adapter = UndetectedAdapter()
|
||||
|
||||
# Create the crawler strategy with the undetected adapter
|
||||
crawler_strategy = AsyncPlaywrightCrawlerStrategy(
|
||||
browser_config=browser_config,
|
||||
browser_adapter=undetected_adapter
|
||||
)
|
||||
|
||||
# Create the crawler with our custom strategy
|
||||
async with AsyncWebCrawler(
|
||||
crawler_strategy=crawler_strategy,
|
||||
config=browser_config
|
||||
) as crawler:
|
||||
# Configure the crawl
|
||||
crawler_config = CrawlerRunConfig(
|
||||
markdown_generator=DefaultMarkdownGenerator(
|
||||
content_filter=PruningContentFilter()
|
||||
),
|
||||
capture_console_messages=True, # Test adapter console capture
|
||||
)
|
||||
|
||||
# Test on a site that typically detects bots
|
||||
print("Testing undetected adapter...")
|
||||
result: CrawlResult = await crawler.arun(
|
||||
url="https://www.helloworld.org",
|
||||
config=crawler_config
|
||||
)
|
||||
|
||||
print(f"Status: {result.status_code}")
|
||||
print(f"Success: {result.success}")
|
||||
print(f"Console messages captured: {len(result.console_messages or [])}")
|
||||
print(f"Markdown content (first 500 chars):\n{result.markdown.raw_markdown[:500]}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
## Browser Adapter Pattern
|
||||
|
||||
The undetected browser support is implemented using an adapter pattern, allowing seamless switching between different browser implementations:
|
||||
|
||||
```python
|
||||
# Regular browser adapter (default)
|
||||
from crawl4ai import PlaywrightAdapter
|
||||
regular_adapter = PlaywrightAdapter()
|
||||
|
||||
# Undetected browser adapter
|
||||
from crawl4ai import UndetectedAdapter
|
||||
undetected_adapter = UndetectedAdapter()
|
||||
```
|
||||
|
||||
The adapter handles:
|
||||
- JavaScript execution
|
||||
- Console message capture
|
||||
- Error handling
|
||||
- Browser-specific optimizations
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Avoid Headless Mode**: Detection is easier in headless mode
|
||||
```python
|
||||
browser_config = BrowserConfig(headless=False)
|
||||
```
|
||||
|
||||
2. **Use Reasonable Delays**: Don't rush through pages
|
||||
```python
|
||||
crawler_config = CrawlerRunConfig(
|
||||
wait_time=3.0, # Wait 3 seconds after page load
|
||||
delay_before_return_html=2.0 # Additional delay
|
||||
)
|
||||
```
|
||||
|
||||
3. **Rotate User Agents**: You can customize user agents
|
||||
```python
|
||||
browser_config = BrowserConfig(
|
||||
headers={"User-Agent": "your-user-agent"}
|
||||
)
|
||||
```
|
||||
|
||||
4. **Handle Failures Gracefully**: Some sites may still detect and block
|
||||
```python
|
||||
if not result.success:
|
||||
print(f"Crawl failed: {result.error_message}")
|
||||
```
|
||||
|
||||
## Advanced Usage Tips
|
||||
|
||||
### Progressive Detection Handling
|
||||
|
||||
```python
|
||||
async def crawl_with_progressive_evasion(url):
|
||||
# Step 1: Try regular browser with stealth
|
||||
browser_config = BrowserConfig(
|
||||
enable_stealth=True,
|
||||
headless=False
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
result = await crawler.arun(url)
|
||||
if result.success and "Access Denied" not in result.html:
|
||||
return result
|
||||
|
||||
# Step 2: If blocked, try undetected browser
|
||||
print("Regular + stealth blocked, trying undetected browser...")
|
||||
|
||||
adapter = UndetectedAdapter()
|
||||
strategy = AsyncPlaywrightCrawlerStrategy(
|
||||
browser_config=browser_config,
|
||||
browser_adapter=adapter
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(
|
||||
crawler_strategy=strategy,
|
||||
config=browser_config
|
||||
) as crawler:
|
||||
result = await crawler.arun(url)
|
||||
return result
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
The undetected browser dependencies are automatically installed when you run:
|
||||
|
||||
```bash
|
||||
crawl4ai-setup
|
||||
```
|
||||
|
||||
This command installs all necessary browser dependencies for both regular and undetected modes.
|
||||
|
||||
## Limitations
|
||||
|
||||
- **Performance**: Slightly slower than regular mode due to additional patches
|
||||
- **Headless Detection**: Some sites can still detect headless mode
|
||||
- **Resource Usage**: May use more resources than regular mode
|
||||
- **Not 100% Guaranteed**: Advanced anti-bot services are constantly evolving
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Browser Not Found
|
||||
|
||||
Run the setup command:
|
||||
```bash
|
||||
crawl4ai-setup
|
||||
```
|
||||
|
||||
### Detection Still Occurring
|
||||
|
||||
Try combining with other features:
|
||||
```python
|
||||
crawler_config = CrawlerRunConfig(
|
||||
simulate_user=True, # Add user simulation
|
||||
magic=True, # Enable magic mode
|
||||
wait_time=5.0, # Longer waits
|
||||
)
|
||||
```
|
||||
|
||||
### Performance Issues
|
||||
|
||||
If experiencing slow performance:
|
||||
```python
|
||||
# Use selective undetected mode only for protected sites
|
||||
if is_protected_site(url):
|
||||
adapter = UndetectedAdapter()
|
||||
else:
|
||||
adapter = PlaywrightAdapter() # Default adapter
|
||||
```
|
||||
|
||||
## Future Plans
|
||||
|
||||
**Note**: In future versions of Crawl4AI, we may enable stealth mode and undetected browser by default to provide better out-of-the-box success rates. For now, users should explicitly enable these features when needed.
|
||||
|
||||
## Conclusion
|
||||
|
||||
Crawl4AI provides flexible anti-bot solutions:
|
||||
|
||||
1. **Start Simple**: Use regular browser + stealth mode for most sites
|
||||
2. **Escalate if Needed**: Switch to undetected browser for sophisticated protection
|
||||
3. **Combine for Maximum Effect**: Use both features together when facing the toughest challenges
|
||||
|
||||
Remember:
|
||||
- Always respect robots.txt and website terms of service
|
||||
- Use appropriate delays to avoid overwhelming servers
|
||||
- Consider the performance trade-offs of each approach
|
||||
- Test progressively to find the minimum necessary evasion level
|
||||
|
||||
## See Also
|
||||
|
||||
- [Advanced Features](advanced-features.md) - Overview of all advanced features
|
||||
- [Proxy & Security](proxy-security.md) - Using proxies with anti-bot features
|
||||
- [Session Management](session-management.md) - Maintaining sessions across requests
|
||||
- [Identity Based Crawling](identity-based-crawling.md) - Additional anti-detection strategies
|
||||
- [Anti-Bot Detection & Fallback](anti-bot-and-fallback.md) - Automatic retry and proxy escalation when blocking is detected
|
||||
@@ -0,0 +1,309 @@
|
||||
# Virtual Scroll
|
||||
|
||||
Modern websites increasingly use **virtual scrolling** (also called windowed rendering or viewport rendering) to handle large datasets efficiently. This technique only renders visible items in the DOM, replacing content as users scroll. Popular examples include Twitter's timeline, Instagram's feed, and many data tables.
|
||||
|
||||
Crawl4AI's Virtual Scroll feature automatically detects and handles these scenarios, ensuring you capture **all content**, not just what's initially visible.
|
||||
|
||||
## Understanding Virtual Scroll
|
||||
|
||||
### The Problem
|
||||
|
||||
Traditional infinite scroll **appends** new content to existing content. Virtual scroll **replaces** content to maintain performance:
|
||||
|
||||
```
|
||||
Traditional Scroll: Virtual Scroll:
|
||||
┌─────────────┐ ┌─────────────┐
|
||||
│ Item 1 │ │ Item 11 │ <- Items 1-10 removed
|
||||
│ Item 2 │ │ Item 12 │ <- Only visible items
|
||||
│ ... │ │ Item 13 │ in DOM
|
||||
│ Item 10 │ │ Item 14 │
|
||||
│ Item 11 NEW │ │ Item 15 │
|
||||
│ Item 12 NEW │ └─────────────┘
|
||||
└─────────────┘
|
||||
DOM keeps growing DOM size stays constant
|
||||
```
|
||||
|
||||
Without proper handling, crawlers only capture the currently visible items, missing the rest of the content.
|
||||
|
||||
### Three Scrolling Scenarios
|
||||
|
||||
Crawl4AI's Virtual Scroll detects and handles three scenarios:
|
||||
|
||||
1. **No Change** - Content doesn't update on scroll (static page or end reached)
|
||||
2. **Content Appended** - New items added to existing ones (traditional infinite scroll)
|
||||
3. **Content Replaced** - Items replaced with new ones (true virtual scroll)
|
||||
|
||||
Only scenario 3 requires special handling, which Virtual Scroll automates.
|
||||
|
||||
## Basic Usage
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, VirtualScrollConfig
|
||||
|
||||
# Configure virtual scroll
|
||||
virtual_config = VirtualScrollConfig(
|
||||
container_selector="#feed", # CSS selector for scrollable container
|
||||
scroll_count=20, # Number of scrolls to perform
|
||||
scroll_by="container_height", # How much to scroll each time
|
||||
wait_after_scroll=0.5 # Wait time (seconds) after each scroll
|
||||
)
|
||||
|
||||
# Use in crawler configuration
|
||||
config = CrawlerRunConfig(
|
||||
virtual_scroll_config=virtual_config
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(url="https://example.com", config=config)
|
||||
# result.html contains ALL items from the virtual scroll
|
||||
```
|
||||
|
||||
## Configuration Parameters
|
||||
|
||||
### VirtualScrollConfig
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `container_selector` | `str` | Required | CSS selector for the scrollable container |
|
||||
| `scroll_count` | `int` | `10` | Maximum number of scrolls to perform |
|
||||
| `scroll_by` | `str` or `int` | `"container_height"` | Scroll amount per step |
|
||||
| `wait_after_scroll` | `float` | `0.5` | Seconds to wait after each scroll |
|
||||
|
||||
### Scroll By Options
|
||||
|
||||
- `"container_height"` - Scroll by the container's visible height
|
||||
- `"page_height"` - Scroll by the viewport height
|
||||
- `500` (integer) - Scroll by exact pixel amount
|
||||
|
||||
## Real-World Examples
|
||||
|
||||
### Twitter-like Timeline
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, VirtualScrollConfig, BrowserConfig
|
||||
|
||||
async def crawl_twitter_timeline():
|
||||
# Twitter replaces tweets as you scroll
|
||||
virtual_config = VirtualScrollConfig(
|
||||
container_selector="[data-testid='primaryColumn']",
|
||||
scroll_count=30,
|
||||
scroll_by="container_height",
|
||||
wait_after_scroll=1.0 # Twitter needs time to load
|
||||
)
|
||||
|
||||
browser_config = BrowserConfig(headless=True) # Set to False to watch it work
|
||||
config = CrawlerRunConfig(
|
||||
virtual_scroll_config=virtual_config
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://twitter.com/search?q=AI",
|
||||
config=config
|
||||
)
|
||||
|
||||
# Extract tweet count
|
||||
import re
|
||||
tweets = re.findall(r'data-testid="tweet"', result.html)
|
||||
print(f"Captured {len(tweets)} tweets")
|
||||
```
|
||||
|
||||
### Instagram Grid
|
||||
|
||||
```python
|
||||
async def crawl_instagram_grid():
|
||||
# Instagram uses virtualized grid for performance
|
||||
virtual_config = VirtualScrollConfig(
|
||||
container_selector="article", # Main feed container
|
||||
scroll_count=50, # More scrolls for grid layout
|
||||
scroll_by=800, # Fixed pixel scrolling
|
||||
wait_after_scroll=0.8
|
||||
)
|
||||
|
||||
config = CrawlerRunConfig(
|
||||
virtual_scroll_config=virtual_config,
|
||||
screenshot=True # Capture final state
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://www.instagram.com/explore/tags/photography/",
|
||||
config=config
|
||||
)
|
||||
|
||||
# Count posts
|
||||
posts = result.html.count('class="post"')
|
||||
print(f"Captured {posts} posts from virtualized grid")
|
||||
```
|
||||
|
||||
### Mixed Content (News Feed)
|
||||
|
||||
Some sites mix static and virtualized content:
|
||||
|
||||
```python
|
||||
async def crawl_mixed_feed():
|
||||
# Featured articles stay, regular articles virtualize
|
||||
virtual_config = VirtualScrollConfig(
|
||||
container_selector=".main-feed",
|
||||
scroll_count=25,
|
||||
scroll_by="container_height",
|
||||
wait_after_scroll=0.5
|
||||
)
|
||||
|
||||
config = CrawlerRunConfig(
|
||||
virtual_scroll_config=virtual_config
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
url="https://news.example.com",
|
||||
config=config
|
||||
)
|
||||
|
||||
# Featured articles remain throughout
|
||||
featured = result.html.count('class="featured-article"')
|
||||
regular = result.html.count('class="regular-article"')
|
||||
|
||||
print(f"Featured (static): {featured}")
|
||||
print(f"Regular (virtualized): {regular}")
|
||||
```
|
||||
|
||||
## Virtual Scroll vs scan_full_page
|
||||
|
||||
Both features handle dynamic content, but serve different purposes:
|
||||
|
||||
| Feature | Virtual Scroll | scan_full_page |
|
||||
|---------|---------------|----------------|
|
||||
| **Purpose** | Capture content that's replaced during scroll | Load content that's appended during scroll |
|
||||
| **Use Case** | Twitter, Instagram, virtual tables | Traditional infinite scroll, lazy-loaded images |
|
||||
| **DOM Behavior** | Replaces elements | Adds elements |
|
||||
| **Memory Usage** | Efficient (merges content) | Can grow large |
|
||||
| **Configuration** | Requires container selector | Works on full page |
|
||||
|
||||
### When to Use Which?
|
||||
|
||||
Use **Virtual Scroll** when:
|
||||
- Content disappears as you scroll (Twitter timeline)
|
||||
- DOM element count stays relatively constant
|
||||
- You need ALL items from a virtualized list
|
||||
- Container-based scrolling (not full page)
|
||||
|
||||
Use **scan_full_page** when:
|
||||
- Content accumulates as you scroll
|
||||
- Images load lazily
|
||||
- Simple "load more" behavior
|
||||
- Full page scrolling
|
||||
|
||||
## Combining with Extraction
|
||||
|
||||
Virtual Scroll works seamlessly with extraction strategies:
|
||||
|
||||
```python
|
||||
from crawl4ai import LLMExtractionStrategy, LLMConfig
|
||||
|
||||
# Define extraction schema
|
||||
schema = {
|
||||
"type": "array",
|
||||
"items": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"author": {"type": "string"},
|
||||
"content": {"type": "string"},
|
||||
"timestamp": {"type": "string"}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Configure both virtual scroll and extraction
|
||||
config = CrawlerRunConfig(
|
||||
virtual_scroll_config=VirtualScrollConfig(
|
||||
container_selector="#timeline",
|
||||
scroll_count=20
|
||||
),
|
||||
extraction_strategy=LLMExtractionStrategy(
|
||||
llm_config=LLMConfig(provider="openai/gpt-4o-mini"),
|
||||
schema=schema
|
||||
)
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(url="...", config=config)
|
||||
|
||||
# Extracted data from ALL scrolled content
|
||||
import json
|
||||
posts = json.loads(result.extracted_content)
|
||||
print(f"Extracted {len(posts)} posts from virtual scroll")
|
||||
```
|
||||
|
||||
## Performance Tips
|
||||
|
||||
1. **Container Selection**: Be specific with selectors. Using the correct container improves performance.
|
||||
|
||||
2. **Scroll Count**: Start conservative and increase as needed:
|
||||
```python
|
||||
# Start with fewer scrolls
|
||||
virtual_config = VirtualScrollConfig(
|
||||
container_selector="#feed",
|
||||
scroll_count=10 # Test with 10, increase if needed
|
||||
)
|
||||
```
|
||||
|
||||
3. **Wait Times**: Adjust based on site speed:
|
||||
```python
|
||||
# Fast sites
|
||||
wait_after_scroll=0.2
|
||||
|
||||
# Slower sites or heavy content
|
||||
wait_after_scroll=1.5
|
||||
```
|
||||
|
||||
4. **Debug Mode**: Set `headless=False` to watch scrolling:
|
||||
```python
|
||||
browser_config = BrowserConfig(headless=False)
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
# Watch the scrolling happen
|
||||
```
|
||||
|
||||
## How It Works Internally
|
||||
|
||||
1. **Detection Phase**: Scrolls and compares HTML to detect behavior
|
||||
2. **Capture Phase**: For replaced content, stores HTML chunks at each position
|
||||
3. **Merge Phase**: Combines all chunks, removing duplicates based on text content
|
||||
4. **Result**: Complete HTML with all unique items
|
||||
|
||||
The deduplication uses normalized text (lowercase, no spaces/symbols) to ensure accurate merging without false positives.
|
||||
|
||||
## Error Handling
|
||||
|
||||
Virtual Scroll handles errors gracefully:
|
||||
|
||||
```python
|
||||
# If container not found or scrolling fails
|
||||
result = await crawler.arun(url="...", config=config)
|
||||
|
||||
if result.success:
|
||||
# Virtual scroll worked or wasn't needed
|
||||
print(f"Captured {len(result.html)} characters")
|
||||
else:
|
||||
# Crawl failed entirely
|
||||
print(f"Error: {result.error_message}")
|
||||
```
|
||||
|
||||
If the container isn't found, crawling continues normally without virtual scroll.
|
||||
|
||||
## Complete Example
|
||||
|
||||
See our [comprehensive example](/docs/examples/virtual_scroll_example.py) that demonstrates:
|
||||
- Twitter-like feeds
|
||||
- Instagram grids
|
||||
- Traditional infinite scroll
|
||||
- Mixed content scenarios
|
||||
- Performance comparisons
|
||||
|
||||
```bash
|
||||
# Run the examples
|
||||
cd docs/examples
|
||||
python virtual_scroll_example.py
|
||||
```
|
||||
|
||||
The example includes a local test server with different scrolling behaviors for experimentation.
|
||||
Reference in New Issue
Block a user