chore: import upstream snapshot with attribution
This commit is contained in:
wehub-resource-sync
@@ -0,0 +1,343 @@
|
||||
# 🚀 Crawl4AI v0.7.0: The Adaptive Intelligence Update
|
||||
|
||||
*January 28, 2025 • 10 min read*
|
||||
|
||||
---
|
||||
|
||||
Today I'm releasing Crawl4AI v0.7.0—the Adaptive Intelligence Update. This release introduces fundamental improvements in how Crawl4AI handles modern web complexity through adaptive learning, intelligent content discovery, and advanced extraction capabilities.
|
||||
|
||||
## 🎯 What's New at a Glance
|
||||
|
||||
- **Adaptive Crawling**: Your crawler now learns and adapts to website patterns
|
||||
- **Virtual Scroll Support**: Complete content extraction from infinite scroll pages
|
||||
- **Link Preview with Intelligent Scoring**: Intelligent link analysis and prioritization
|
||||
- **Async URL Seeder**: Discover thousands of URLs in seconds with intelligent filtering
|
||||
- **Performance Optimizations**: Significant speed and memory improvements
|
||||
|
||||
## 🧠 Adaptive Crawling: Intelligence Through Pattern Learning
|
||||
|
||||
**The Problem:** Websites change. Class names shift. IDs disappear. Your carefully crafted selectors break at 3 AM, and you wake up to empty datasets and angry stakeholders.
|
||||
|
||||
**My Solution:** I implemented an adaptive learning system that observes patterns, builds confidence scores, and adjusts extraction strategies on the fly. It's like having a junior developer who gets better at their job with every page they scrape.
|
||||
|
||||
### Technical Deep-Dive
|
||||
|
||||
The Adaptive Crawler maintains a persistent state for each domain, tracking:
|
||||
- Pattern success rates
|
||||
- Selector stability over time
|
||||
- Content structure variations
|
||||
- Extraction confidence scores
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, AdaptiveCrawler, AdaptiveConfig
|
||||
import asyncio
|
||||
|
||||
async def main():
|
||||
|
||||
# Configure adaptive crawler
|
||||
config = AdaptiveConfig(
|
||||
strategy="statistical", # or "embedding" for semantic understanding
|
||||
max_pages=10,
|
||||
confidence_threshold=0.7, # Stop at 70% confidence
|
||||
top_k_links=3, # Follow top 3 links per page
|
||||
min_gain_threshold=0.05 # Need 5% information gain to continue
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(verbose=False) as crawler:
|
||||
adaptive = AdaptiveCrawler(crawler, config)
|
||||
|
||||
print("Starting adaptive crawl about Python decorators...")
|
||||
result = await adaptive.digest(
|
||||
start_url="https://docs.python.org/3/glossary.html",
|
||||
query="python decorators functions wrapping"
|
||||
)
|
||||
|
||||
print(f"\n✅ Crawling Complete!")
|
||||
print(f"• Confidence Level: {adaptive.confidence:.0%}")
|
||||
print(f"• Pages Crawled: {len(result.crawled_urls)}")
|
||||
print(f"• Knowledge Base: {len(adaptive.state.knowledge_base)} documents")
|
||||
|
||||
# Get most relevant content
|
||||
relevant = adaptive.get_relevant_content(top_k=3)
|
||||
print(f"\nMost Relevant Pages:")
|
||||
for i, page in enumerate(relevant, 1):
|
||||
print(f"{i}. {page['url']} (relevance: {page['score']:.2%})")
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **News Aggregation**: Maintain 95%+ extraction accuracy even as news sites update their templates
|
||||
- **E-commerce Monitoring**: Track product changes across hundreds of stores without constant maintenance
|
||||
- **Research Data Collection**: Build robust academic datasets that survive website redesigns
|
||||
- **Reduced Maintenance**: Cut selector update time by 80% for frequently-changing sites
|
||||
|
||||
## 🌊 Virtual Scroll: Complete Content Capture
|
||||
|
||||
**The Problem:** Modern web apps only render what's visible. Scroll down, new content appears, old content vanishes into the void. Traditional crawlers capture that first viewport and miss 90% of the content. It's like reading only the first page of every book.
|
||||
|
||||
**My Solution:** I built Virtual Scroll support that mimics human browsing behavior, capturing content as it loads and preserving it before the browser's garbage collector strikes.
|
||||
|
||||
### Implementation Details
|
||||
|
||||
```python
|
||||
from crawl4ai import VirtualScrollConfig
|
||||
|
||||
# For social media feeds (Twitter/X style)
|
||||
twitter_config = VirtualScrollConfig(
|
||||
container_selector="[data-testid='primaryColumn']",
|
||||
scroll_count=20, # Number of scrolls
|
||||
scroll_by="container_height", # Smart scrolling by container size
|
||||
wait_after_scroll=1.0 # Let content load
|
||||
)
|
||||
|
||||
# For e-commerce product grids (Instagram style)
|
||||
grid_config = VirtualScrollConfig(
|
||||
container_selector="main .product-grid",
|
||||
scroll_count=30,
|
||||
scroll_by=800, # Fixed pixel scrolling
|
||||
wait_after_scroll=1.5 # Images need time
|
||||
)
|
||||
|
||||
# For news feeds with lazy loading
|
||||
news_config = VirtualScrollConfig(
|
||||
container_selector=".article-feed",
|
||||
scroll_count=50,
|
||||
scroll_by="page_height", # Viewport-based scrolling
|
||||
wait_after_scroll=0.5 # Wait for content to load
|
||||
)
|
||||
|
||||
# Use it in your crawl
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
"https://twitter.com/trending",
|
||||
config=CrawlerRunConfig(
|
||||
virtual_scroll_config=twitter_config,
|
||||
# Combine with other features
|
||||
extraction_strategy=JsonCssExtractionStrategy({
|
||||
"tweets": {
|
||||
"selector": "[data-testid='tweet']",
|
||||
"fields": {
|
||||
"text": {"selector": "[data-testid='tweetText']", "type": "text"},
|
||||
"likes": {"selector": "[data-testid='like']", "type": "text"}
|
||||
}
|
||||
}
|
||||
})
|
||||
)
|
||||
)
|
||||
|
||||
print(f"Captured {len(result.extracted_content['tweets'])} tweets")
|
||||
```
|
||||
|
||||
**Key Capabilities:**
|
||||
- **DOM Recycling Awareness**: Detects and handles virtual DOM element recycling
|
||||
- **Smart Scroll Physics**: Three modes - container height, page height, or fixed pixels
|
||||
- **Content Preservation**: Captures content before it's destroyed
|
||||
- **Intelligent Stopping**: Stops when no new content appears
|
||||
- **Memory Efficient**: Streams content instead of holding everything in memory
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Social Media Analysis**: Capture entire Twitter threads with hundreds of replies, not just top 10
|
||||
- **E-commerce Scraping**: Extract 500+ products from infinite scroll catalogs vs. 20-50 with traditional methods
|
||||
- **News Aggregation**: Get all articles from modern news sites, not just above-the-fold content
|
||||
- **Research Applications**: Complete data extraction from academic databases using virtual pagination
|
||||
|
||||
## 🔗 Link Preview: Intelligent Link Analysis and Scoring
|
||||
|
||||
**The Problem:** You crawl a page and get 200 links. Which ones matter? Which lead to the content you actually want? Traditional crawlers force you to follow everything or build complex filters.
|
||||
|
||||
**My Solution:** I implemented a three-layer scoring system that analyzes links like a human would—considering their position, context, and relevance to your goals.
|
||||
|
||||
### Intelligent Link Analysis and Scoring
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import CrawlerRunConfig, CacheMode, AsyncWebCrawler
|
||||
from crawl4ai.adaptive_crawler import LinkPreviewConfig
|
||||
|
||||
async def main():
|
||||
# Configure intelligent link analysis
|
||||
link_config = LinkPreviewConfig(
|
||||
include_internal=True,
|
||||
include_external=False,
|
||||
max_links=10,
|
||||
concurrency=5,
|
||||
query="python tutorial", # For contextual scoring
|
||||
score_threshold=0.3,
|
||||
verbose=True
|
||||
)
|
||||
# Use in your crawl
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
"https://www.geeksforgeeks.org/",
|
||||
config=CrawlerRunConfig(
|
||||
link_preview_config=link_config,
|
||||
score_links=True, # Enable intrinsic scoring
|
||||
cache_mode=CacheMode.BYPASS
|
||||
)
|
||||
)
|
||||
|
||||
# Access scored and sorted links
|
||||
if result.success and result.links:
|
||||
for link in result.links.get("internal", []):
|
||||
text = link.get('text', 'No text')[:40]
|
||||
print(
|
||||
text,
|
||||
f"{link.get('intrinsic_score', 0):.1f}/10" if link.get('intrinsic_score') is not None else "0.0/10",
|
||||
f"{link.get('contextual_score', 0):.2f}/1" if link.get('contextual_score') is not None else "0.00/1",
|
||||
f"{link.get('total_score', 0):.3f}" if link.get('total_score') is not None else "0.000"
|
||||
)
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
**Scoring Components:**
|
||||
|
||||
1. **Intrinsic Score**: Based on link quality indicators
|
||||
- Position on page (navigation, content, footer)
|
||||
- Link attributes (rel, title, class names)
|
||||
- Anchor text quality and length
|
||||
- URL structure and depth
|
||||
|
||||
2. **Contextual Score**: Relevance to your query using BM25 algorithm
|
||||
- Keyword matching in link text and title
|
||||
- Meta description analysis
|
||||
- Content preview scoring
|
||||
|
||||
3. **Total Score**: Combined score for final ranking
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Research Efficiency**: Find relevant papers 10x faster by following only high-score links
|
||||
- **Competitive Analysis**: Automatically identify important pages on competitor sites
|
||||
- **Content Discovery**: Build topic-focused crawlers that stay on track
|
||||
- **SEO Audits**: Identify and prioritize high-value internal linking opportunities
|
||||
|
||||
## 🎣 Async URL Seeder: Automated URL Discovery at Scale
|
||||
|
||||
**The Problem:** You want to crawl an entire domain but only have the homepage. Or worse, you want specific content types across thousands of pages. Manual URL discovery? That's a job for machines, not humans.
|
||||
|
||||
**My Solution:** I built Async URL Seeder—a turbocharged URL discovery engine that combines multiple sources with intelligent filtering and relevance scoring.
|
||||
|
||||
### Technical Architecture
|
||||
|
||||
```python
|
||||
import asyncio
|
||||
from crawl4ai import AsyncUrlSeeder, SeedingConfig
|
||||
|
||||
async def main():
|
||||
async with AsyncUrlSeeder() as seeder:
|
||||
# Discover Python tutorial URLs
|
||||
config = SeedingConfig(
|
||||
source="sitemap", # Use sitemap
|
||||
pattern="*python*", # URL pattern filter
|
||||
extract_head=True, # Get metadata
|
||||
query="python tutorial", # For relevance scoring
|
||||
scoring_method="bm25",
|
||||
score_threshold=0.2,
|
||||
max_urls=10
|
||||
)
|
||||
|
||||
print("Discovering Python async tutorial URLs...")
|
||||
urls = await seeder.urls("https://www.geeksforgeeks.org/", config)
|
||||
|
||||
print(f"\n✅ Found {len(urls)} relevant URLs:")
|
||||
for i, url_info in enumerate(urls[:5], 1):
|
||||
print(f"\n{i}. {url_info['url']}")
|
||||
if url_info.get('relevance_score'):
|
||||
print(f" Relevance: {url_info['relevance_score']:.3f}")
|
||||
if url_info.get('head_data', {}).get('title'):
|
||||
print(f" Title: {url_info['head_data']['title'][:60]}...")
|
||||
|
||||
asyncio.run(main())
|
||||
```
|
||||
|
||||
**Discovery Methods:**
|
||||
- **Sitemap Mining**: Parses robots.txt and all linked sitemaps
|
||||
- **Common Crawl**: Queries the Common Crawl index for historical URLs
|
||||
- **Intelligent Crawling**: Follows links with smart depth control
|
||||
- **Pattern Analysis**: Learns URL structures and generates variations
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Migration Projects**: Discover 10,000+ URLs from legacy sites in under 60 seconds
|
||||
- **Market Research**: Map entire competitor ecosystems automatically
|
||||
- **Academic Research**: Build comprehensive datasets without manual URL collection
|
||||
- **SEO Audits**: Find every indexable page with content scoring
|
||||
- **Content Archival**: Ensure no content is left behind during site migrations
|
||||
|
||||
## ⚡ Performance Optimizations
|
||||
|
||||
This release includes significant performance improvements through optimized resource handling, better concurrency management, and reduced memory footprint.
|
||||
|
||||
### What We Optimized
|
||||
|
||||
```python
|
||||
# Optimized crawling with v0.7.0 improvements
|
||||
results = []
|
||||
for url in urls:
|
||||
result = await crawler.arun(
|
||||
url,
|
||||
config=CrawlerRunConfig(
|
||||
# Performance optimizations
|
||||
wait_until="domcontentloaded", # Faster than networkidle
|
||||
cache_mode=CacheMode.ENABLED # Enable caching
|
||||
)
|
||||
)
|
||||
results.append(result)
|
||||
```
|
||||
|
||||
**Performance Gains:**
|
||||
- **Startup Time**: 70% faster browser initialization
|
||||
- **Page Loading**: 40% reduction with smart resource blocking
|
||||
- **Extraction**: 3x faster with compiled CSS selectors
|
||||
- **Memory Usage**: 60% reduction with streaming processing
|
||||
- **Concurrent Crawls**: Handle 5x more parallel requests
|
||||
|
||||
|
||||
## 🔧 Important Changes
|
||||
|
||||
### Breaking Changes
|
||||
- `link_extractor` renamed to `link_preview` (better reflects functionality)
|
||||
- Minimum Python version now 3.9
|
||||
- `CrawlerConfig` split into `CrawlerRunConfig` and `BrowserConfig`
|
||||
|
||||
### Migration Guide
|
||||
```python
|
||||
# Old (v0.6.x)
|
||||
from crawl4ai import CrawlerConfig
|
||||
config = CrawlerConfig(timeout=30000)
|
||||
|
||||
# New (v0.7.0)
|
||||
from crawl4ai import CrawlerRunConfig, BrowserConfig
|
||||
browser_config = BrowserConfig(timeout=30000)
|
||||
run_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)
|
||||
```
|
||||
|
||||
## 🤖 Coming Soon: Intelligent Web Automation
|
||||
|
||||
I'm currently working on bringing advanced automation capabilities to Crawl4AI. This includes:
|
||||
|
||||
- **Crawl Agents**: Autonomous crawlers that understand your goals and adapt their strategies
|
||||
- **Auto JS Generation**: Automatic JavaScript code generation for complex interactions
|
||||
- **Smart Form Handling**: Intelligent form detection and filling
|
||||
- **Context-Aware Actions**: Crawlers that understand page context and make decisions
|
||||
|
||||
These features are under active development and will revolutionize how we approach web automation. Stay tuned!
|
||||
|
||||
## 🚀 Get Started
|
||||
|
||||
```bash
|
||||
pip install crawl4ai==0.7.0
|
||||
```
|
||||
|
||||
Check out the [updated documentation](https://docs.crawl4ai.com).
|
||||
|
||||
Questions? Issues? I'm always listening:
|
||||
- GitHub: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
|
||||
- Discord: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
|
||||
- Twitter: [@unclecode](https://x.com/unclecode)
|
||||
|
||||
Happy crawling! 🕷️
|
||||
|
||||
---
|
||||
|
||||
*P.S. If you're using Crawl4AI in production, I'd love to hear about it. Your use cases inspire the next features.*
|
||||
@@ -0,0 +1,43 @@
|
||||
# 🛠️ Crawl4AI v0.7.1: Minor Cleanup Update
|
||||
|
||||
*July 17, 2025 • 2 min read*
|
||||
|
||||
---
|
||||
|
||||
A small maintenance release that removes unused code and improves documentation.
|
||||
|
||||
## 🎯 What's Changed
|
||||
|
||||
- **Removed unused StealthConfig** from `crawl4ai/browser_manager.py`
|
||||
- **Updated documentation** with better examples and parameter explanations
|
||||
- **Fixed virtual scroll configuration** examples in docs
|
||||
|
||||
## 🧹 Code Cleanup
|
||||
|
||||
Removed unused `StealthConfig` import and configuration that wasn't being used anywhere in the codebase. The project uses its own custom stealth implementation through JavaScript injection instead.
|
||||
|
||||
```python
|
||||
# Removed unused code:
|
||||
from playwright_stealth import StealthConfig
|
||||
stealth_config = StealthConfig(...) # This was never used
|
||||
```
|
||||
|
||||
## 📖 Documentation Updates
|
||||
|
||||
- Fixed adaptive crawling parameter examples
|
||||
- Updated session management documentation
|
||||
- Corrected virtual scroll configuration examples
|
||||
|
||||
## 🚀 Installation
|
||||
|
||||
```bash
|
||||
pip install crawl4ai==0.7.1
|
||||
```
|
||||
|
||||
No breaking changes - upgrade directly from v0.7.0.
|
||||
|
||||
---
|
||||
|
||||
Questions? Issues?
|
||||
- GitHub: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
|
||||
- Discord: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
|
||||
@@ -0,0 +1,350 @@
|
||||
# 🚀 Crawl4AI v0.7.3: The Multi-Config Intelligence Update
|
||||
|
||||
*August 6, 2025 • 5 min read*
|
||||
|
||||
---
|
||||
|
||||
Today I'm releasing Crawl4AI v0.7.3—the Multi-Config Intelligence Update. This release brings smarter URL-specific configurations, flexible Docker deployments, important bug fixes, and documentation improvements that make Crawl4AI more robust and production-ready.
|
||||
|
||||
## 🎯 What's New at a Glance
|
||||
|
||||
- **🕵️ Undetected Browser Support**: Stealth mode for bypassing bot detection systems
|
||||
- **🎨 Multi-URL Configurations**: Different crawling strategies for different URL patterns in a single batch
|
||||
- **🐳 Flexible Docker LLM Providers**: Configure LLM providers via environment variables
|
||||
- **🧠 Memory Monitoring**: Enhanced memory usage tracking and optimization tools
|
||||
- **📊 Enhanced Table Extraction**: Improved table access and DataFrame conversion
|
||||
- **💰 GitHub Sponsors**: 4-tier sponsorship system with custom arrangements
|
||||
- **🔧 Bug Fixes**: Resolved several critical issues for better stability
|
||||
- **📚 Documentation Updates**: Clearer examples and improved API documentation
|
||||
|
||||
## 🎨 Multi-URL Configurations: One Size Doesn't Fit All
|
||||
|
||||
**The Problem:** You're crawling a mix of documentation sites, blogs, and API endpoints. Each needs different handling—caching for docs, fresh content for news, structured extraction for APIs. Previously, you'd run separate crawls or write complex conditional logic.
|
||||
|
||||
**My Solution:** I implemented URL-specific configurations that let you define different strategies for different URL patterns in a single crawl batch. First match wins, with optional fallback support.
|
||||
|
||||
### Technical Implementation
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, MatchMode
|
||||
|
||||
# Define specialized configs for different content types
|
||||
configs = [
|
||||
# Documentation sites - aggressive caching, include links
|
||||
CrawlerRunConfig(
|
||||
url_matcher=["*docs*", "*documentation*"],
|
||||
cache_mode="write",
|
||||
markdown_generator_options={"include_links": True}
|
||||
),
|
||||
|
||||
# News/blog sites - fresh content, scroll for lazy loading
|
||||
CrawlerRunConfig(
|
||||
url_matcher=lambda url: 'blog' in url or 'news' in url,
|
||||
cache_mode="bypass",
|
||||
js_code="window.scrollTo(0, document.body.scrollHeight/2);"
|
||||
),
|
||||
|
||||
# API endpoints - structured extraction
|
||||
CrawlerRunConfig(
|
||||
url_matcher=["*.json", "*api*"],
|
||||
extraction_strategy=LLMExtractionStrategy(
|
||||
provider="openai/gpt-4o-mini",
|
||||
extraction_type="structured"
|
||||
)
|
||||
),
|
||||
|
||||
# Default fallback for everything else
|
||||
CrawlerRunConfig() # No url_matcher = matches everything
|
||||
]
|
||||
|
||||
# Crawl multiple URLs with appropriate configs
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
results = await crawler.arun_many(
|
||||
urls=[
|
||||
"https://docs.python.org/3/", # → Uses documentation config
|
||||
"https://blog.python.org/", # → Uses blog config
|
||||
"https://api.github.com/users", # → Uses API config
|
||||
"https://example.com/" # → Uses default config
|
||||
],
|
||||
config=configs
|
||||
)
|
||||
```
|
||||
|
||||
**Matching Capabilities:**
|
||||
- **String Patterns**: Wildcards like `"*.pdf"`, `"*/blog/*"`
|
||||
- **Function Matchers**: Lambda functions for complex logic
|
||||
- **Mixed Matchers**: Combine strings and functions with AND/OR logic
|
||||
- **Fallback Support**: Default config when nothing matches
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Mixed Content Sites**: Handle blogs, docs, and downloads in one crawl
|
||||
- **Multi-Domain Crawling**: Different strategies per domain without separate runs
|
||||
- **Reduced Complexity**: No more if/else forests in your extraction code
|
||||
- **Better Performance**: Each URL gets exactly the processing it needs
|
||||
|
||||
## 🕵️ Undetected Browser Support: Stealth Mode Activated
|
||||
|
||||
**The Problem:** Modern websites employ sophisticated bot detection systems. Cloudflare, Akamai, and custom solutions block automated crawlers, limiting access to valuable content.
|
||||
|
||||
**My Solution:** I implemented undetected browser support with a flexible adapter pattern. Now Crawl4AI can bypass most bot detection systems using stealth techniques.
|
||||
|
||||
### Technical Implementation
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, BrowserConfig
|
||||
|
||||
# Enable undetected mode for stealth crawling
|
||||
browser_config = BrowserConfig(
|
||||
browser_type="undetected", # Use undetected Chrome
|
||||
headless=True, # Can run headless with stealth
|
||||
extra_args=[
|
||||
"--disable-blink-features=AutomationControlled",
|
||||
"--disable-web-security",
|
||||
"--disable-features=VizDisplayCompositor"
|
||||
]
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
# This will bypass most bot detection systems
|
||||
result = await crawler.arun("https://protected-site.com")
|
||||
|
||||
if result.success:
|
||||
print("✅ Successfully bypassed bot detection!")
|
||||
print(f"Content length: {len(result.markdown)}")
|
||||
```
|
||||
|
||||
**Advanced Anti-Bot Strategies:**
|
||||
|
||||
```python
|
||||
# Combine multiple stealth techniques
|
||||
from crawl4ai import CrawlerRunConfig
|
||||
|
||||
config = CrawlerRunConfig(
|
||||
# Random user agents and headers
|
||||
headers={
|
||||
"Accept-Language": "en-US,en;q=0.9",
|
||||
"Accept-Encoding": "gzip, deflate, br",
|
||||
"DNT": "1"
|
||||
},
|
||||
|
||||
# Human-like behavior simulation
|
||||
js_code="""
|
||||
// Random mouse movements
|
||||
const simulateHuman = () => {
|
||||
const event = new MouseEvent('mousemove', {
|
||||
clientX: Math.random() * window.innerWidth,
|
||||
clientY: Math.random() * window.innerHeight
|
||||
});
|
||||
document.dispatchEvent(event);
|
||||
};
|
||||
setInterval(simulateHuman, 100 + Math.random() * 200);
|
||||
|
||||
// Random scrolling
|
||||
const randomScroll = () => {
|
||||
const scrollY = Math.random() * (document.body.scrollHeight - window.innerHeight);
|
||||
window.scrollTo(0, scrollY);
|
||||
};
|
||||
setTimeout(randomScroll, 500 + Math.random() * 1000);
|
||||
""",
|
||||
|
||||
# Delay to appear more human
|
||||
delay_before_return_html=2.0
|
||||
)
|
||||
|
||||
result = await crawler.arun("https://bot-protected-site.com", config=config)
|
||||
```
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Enterprise Scraping**: Access previously blocked corporate sites and databases
|
||||
- **Market Research**: Gather data from competitor sites with protection
|
||||
- **Price Monitoring**: Track e-commerce sites that block automated access
|
||||
- **Content Aggregation**: Collect news and social media despite anti-bot measures
|
||||
- **Compliance Testing**: Verify your own site's bot protection effectiveness
|
||||
|
||||
## 🧠 Memory Monitoring & Optimization
|
||||
|
||||
**The Problem:** Long-running crawl sessions consuming excessive memory, especially when processing large batches or heavy JavaScript sites.
|
||||
|
||||
**My Solution:** Built comprehensive memory monitoring and optimization utilities that track usage patterns and provide actionable insights.
|
||||
|
||||
### Memory Tracking Implementation
|
||||
|
||||
```python
|
||||
from crawl4ai.memory_utils import MemoryMonitor, get_memory_info
|
||||
|
||||
# Monitor memory during crawling
|
||||
monitor = MemoryMonitor()
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
# Start monitoring
|
||||
monitor.start_monitoring()
|
||||
|
||||
# Perform memory-intensive operations
|
||||
results = await crawler.arun_many([
|
||||
"https://heavy-js-site.com",
|
||||
"https://large-images-site.com",
|
||||
"https://dynamic-content-site.com"
|
||||
])
|
||||
|
||||
# Get detailed memory report
|
||||
memory_report = monitor.get_report()
|
||||
print(f"Peak memory usage: {memory_report['peak_mb']:.1f} MB")
|
||||
print(f"Memory efficiency: {memory_report['efficiency']:.1f}%")
|
||||
|
||||
# Automatic cleanup suggestions
|
||||
if memory_report['peak_mb'] > 1000: # > 1GB
|
||||
print("💡 Consider batch size optimization")
|
||||
print("💡 Enable aggressive garbage collection")
|
||||
```
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Production Stability**: Prevent memory-related crashes in long-running services
|
||||
- **Cost Optimization**: Right-size server resources based on actual usage
|
||||
- **Performance Tuning**: Identify memory bottlenecks and optimization opportunities
|
||||
- **Scalability Planning**: Understand memory patterns for horizontal scaling
|
||||
|
||||
## 📊 Enhanced Table Extraction
|
||||
|
||||
**The Problem:** Table data was accessed through the generic `result.media` interface, making DataFrame conversion cumbersome and unclear.
|
||||
|
||||
**My Solution:** Dedicated `result.tables` interface with direct DataFrame conversion and improved detection algorithms.
|
||||
|
||||
### New Table Access Pattern
|
||||
|
||||
```python
|
||||
# Old way (deprecated)
|
||||
# tables_data = result.media.get('tables', [])
|
||||
|
||||
# New way (v0.7.3+)
|
||||
result = await crawler.arun("https://site-with-tables.com")
|
||||
|
||||
# Direct table access
|
||||
if result.tables:
|
||||
print(f"Found {len(result.tables)} tables")
|
||||
|
||||
# Convert to pandas DataFrame instantly
|
||||
import pandas as pd
|
||||
|
||||
for i, table in enumerate(result.tables):
|
||||
df = pd.DataFrame(table['data'])
|
||||
print(f"Table {i}: {df.shape[0]} rows × {df.shape[1]} columns")
|
||||
print(df.head())
|
||||
|
||||
# Table metadata
|
||||
print(f"Source: {table.get('source_xpath', 'Unknown')}")
|
||||
print(f"Headers: {table.get('headers', [])}")
|
||||
```
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Data Analysis**: Faster transition from web data to analysis-ready DataFrames
|
||||
- **ETL Pipelines**: Cleaner integration with data processing workflows
|
||||
- **Reporting**: Simplified table extraction for automated reporting systems
|
||||
|
||||
## 💰 Community Support: GitHub Sponsors
|
||||
|
||||
I've launched GitHub Sponsors to ensure Crawl4AI's continued development and support our growing community.
|
||||
|
||||
**Sponsorship Tiers:**
|
||||
- **🌱 Supporter ($5/month)**: Community support + early feature previews
|
||||
- **🚀 Professional ($25/month)**: Priority support + beta access
|
||||
- **🏢 Business ($100/month)**: Direct consultation + custom integrations
|
||||
- **🏛️ Enterprise ($500/month)**: Dedicated support + feature development
|
||||
|
||||
**Why Sponsor?**
|
||||
- Ensure continuous development and maintenance
|
||||
- Get priority support and feature requests
|
||||
- Access to premium documentation and examples
|
||||
- Direct line to the development team
|
||||
|
||||
[**Become a Sponsor →**](https://github.com/sponsors/unclecode)
|
||||
|
||||
## 🐳 Docker: Flexible LLM Provider Configuration
|
||||
|
||||
**The Problem:** Hardcoded LLM providers in Docker deployments. Want to switch from OpenAI to Groq? Rebuild and redeploy. Testing different models? Multiple Docker images.
|
||||
|
||||
**My Solution:** Configure LLM providers via environment variables. Switch providers without touching code or rebuilding images.
|
||||
|
||||
### Deployment Flexibility
|
||||
|
||||
```bash
|
||||
# Option 1: Direct environment variables
|
||||
docker run -d \
|
||||
-e LLM_PROVIDER="groq/llama-3.2-3b-preview" \
|
||||
-e GROQ_API_KEY="your-key" \
|
||||
-p 11235:11235 \
|
||||
unclecode/crawl4ai:latest
|
||||
|
||||
# Option 2: Using .llm.env file (recommended for production)
|
||||
# Create .llm.env file:
|
||||
# LLM_PROVIDER=openai/gpt-4o-mini
|
||||
# OPENAI_API_KEY=your-openai-key
|
||||
# GROQ_API_KEY=your-groq-key
|
||||
|
||||
docker run -d \
|
||||
--env-file .llm.env \
|
||||
-p 11235:11235 \
|
||||
unclecode/crawl4ai:latest
|
||||
```
|
||||
|
||||
Override per request when needed:
|
||||
```python
|
||||
# Use default provider from .llm.env
|
||||
response = requests.post("http://localhost:11235/crawl", json={
|
||||
"url": "https://example.com",
|
||||
"extraction_strategy": {"type": "llm"}
|
||||
})
|
||||
|
||||
# Override to use different provider for this specific request
|
||||
response = requests.post("http://localhost:11235/crawl", json={
|
||||
"url": "https://complex-page.com",
|
||||
"extraction_strategy": {
|
||||
"type": "llm",
|
||||
"provider": "openai/gpt-4" # Override default
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Cost Optimization**: Use cheaper models for simple tasks, premium for complex
|
||||
- **A/B Testing**: Compare provider performance without deployment changes
|
||||
- **Fallback Strategies**: Switch providers on-the-fly during outages
|
||||
- **Development Flexibility**: Test locally with one provider, deploy with another
|
||||
- **Secure Configuration**: Keep API keys in `.llm.env` file, not in commands
|
||||
|
||||
## 🔧 Bug Fixes & Improvements
|
||||
|
||||
This release includes several important bug fixes that improve stability and reliability:
|
||||
|
||||
- **URL Matcher Fallback**: Fixed edge cases in URL pattern matching logic
|
||||
- **Memory Management**: Resolved memory leaks in long-running crawl sessions
|
||||
- **Sitemap Processing**: Fixed redirect handling in sitemap fetching
|
||||
- **Table Extraction**: Improved table detection and extraction accuracy
|
||||
- **Error Handling**: Better error messages and recovery from network failures
|
||||
|
||||
## 📚 Documentation Enhancements
|
||||
|
||||
Based on community feedback, we've updated:
|
||||
- Clearer examples for multi-URL configuration
|
||||
- Improved CrawlResult documentation with all available fields
|
||||
- Fixed typos and inconsistencies across documentation
|
||||
- Added real-world URLs in examples for better understanding
|
||||
- New comprehensive demo showcasing all v0.7.3 features
|
||||
|
||||
## 🙏 Acknowledgments
|
||||
|
||||
Thanks to our contributors and the entire community for feedback and bug reports.
|
||||
|
||||
## 📚 Resources
|
||||
|
||||
- [Full Documentation](https://docs.crawl4ai.com)
|
||||
- [GitHub Repository](https://github.com/unclecode/crawl4ai)
|
||||
- [Discord Community](https://discord.gg/crawl4ai)
|
||||
- [Feature Demo](https://github.com/unclecode/crawl4ai/blob/main/docs/releases_review/demo_v0.7.3.py)
|
||||
|
||||
---
|
||||
|
||||
*Crawl4AI continues to evolve with your needs. This release makes it smarter, more flexible, and more stable. Try the new multi-config feature and flexible Docker deployment—they're game changers!*
|
||||
|
||||
**Happy Crawling! 🕷️**
|
||||
|
||||
*- The Crawl4AI Team*
|
||||
@@ -0,0 +1,270 @@
|
||||
# 🚀 Crawl4AI v0.7.4: The Intelligent Table Extraction & Performance Update
|
||||
|
||||
*August 17, 2025 • 6 min read*
|
||||
|
||||
---
|
||||
|
||||
Today I'm releasing Crawl4AI v0.7.4—the Intelligent Table Extraction & Performance Update. This release introduces revolutionary LLM-powered table extraction with intelligent chunking, significant performance improvements for concurrent crawling, enhanced browser management, and critical stability fixes that make Crawl4AI more robust for production workloads.
|
||||
|
||||
## 🎯 What's New at a Glance
|
||||
|
||||
- **🚀 LLMTableExtraction**: Revolutionary table extraction with intelligent chunking for massive tables
|
||||
- **⚡ Enhanced Concurrency**: True concurrency improvements for fast-completing tasks in batch operations
|
||||
- **🔧 Browser Manager Fixes**: Resolved race conditions in concurrent page creation
|
||||
- **⌨️ Cross-Platform Browser Profiler**: Improved keyboard handling and quit mechanisms
|
||||
- **🔗 Advanced URL Processing**: Better handling of raw URLs and base tag link resolution
|
||||
- **🛡️ Enhanced Proxy Support**: Flexible proxy configuration with dict and string formats
|
||||
- **🐳 Docker Improvements**: Better API handling and raw HTML support
|
||||
|
||||
## 🚀 LLMTableExtraction: Revolutionary Table Processing
|
||||
|
||||
**The Problem:** Complex tables with rowspan, colspan, nested structures, or massive datasets that traditional HTML parsing can't handle effectively. Large tables that exceed token limits crash extraction processes.
|
||||
|
||||
**My Solution:** I developed LLMTableExtraction—an intelligent table extraction strategy that uses Large Language Models with automatic chunking to handle tables of any size and complexity.
|
||||
|
||||
### Technical Implementation
|
||||
|
||||
```python
|
||||
from crawl4ai import (
|
||||
AsyncWebCrawler,
|
||||
CrawlerRunConfig,
|
||||
LLMConfig,
|
||||
LLMTableExtraction,
|
||||
CacheMode
|
||||
)
|
||||
|
||||
# Configure LLM for table extraction
|
||||
llm_config = LLMConfig(
|
||||
provider="openai/gpt-4.1-mini",
|
||||
api_token="env:OPENAI_API_KEY",
|
||||
temperature=0.1, # Low temperature for consistency
|
||||
max_tokens=32000
|
||||
)
|
||||
|
||||
# Create intelligent table extraction strategy
|
||||
table_strategy = LLMTableExtraction(
|
||||
llm_config=llm_config,
|
||||
verbose=True,
|
||||
max_tries=2,
|
||||
enable_chunking=True, # Handle massive tables
|
||||
chunk_token_threshold=5000, # Smart chunking threshold
|
||||
overlap_threshold=100, # Maintain context between chunks
|
||||
extraction_type="structured" # Get structured data output
|
||||
)
|
||||
|
||||
# Apply to crawler configuration
|
||||
config = CrawlerRunConfig(
|
||||
table_extraction_strategy=table_strategy,
|
||||
cache_mode=CacheMode.BYPASS
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
# Extract complex tables with intelligence
|
||||
result = await crawler.arun(
|
||||
"https://en.wikipedia.org/wiki/List_of_countries_by_GDP",
|
||||
config=config
|
||||
)
|
||||
|
||||
# Access extracted tables directly
|
||||
for i, table in enumerate(result.tables):
|
||||
print(f"Table {i}: {len(table['data'])} rows × {len(table['headers'])} columns")
|
||||
|
||||
# Convert to pandas DataFrame instantly
|
||||
import pandas as pd
|
||||
df = pd.DataFrame(table['data'], columns=table['headers'])
|
||||
print(df.head())
|
||||
```
|
||||
|
||||
**Intelligent Chunking for Massive Tables:**
|
||||
|
||||
```python
|
||||
# Handle tables that exceed token limits
|
||||
large_table_strategy = LLMTableExtraction(
|
||||
llm_config=llm_config,
|
||||
enable_chunking=True,
|
||||
chunk_token_threshold=3000, # Conservative threshold
|
||||
overlap_threshold=150, # Preserve context
|
||||
max_concurrent_chunks=3, # Parallel processing
|
||||
merge_strategy="intelligent" # Smart chunk merging
|
||||
)
|
||||
|
||||
# Process Wikipedia comparison tables, financial reports, etc.
|
||||
config = CrawlerRunConfig(
|
||||
table_extraction_strategy=large_table_strategy,
|
||||
# Target specific table containers
|
||||
css_selector="div.wikitable, table.sortable",
|
||||
delay_before_return_html=2.0
|
||||
)
|
||||
|
||||
result = await crawler.arun(
|
||||
"https://en.wikipedia.org/wiki/Comparison_of_operating_systems",
|
||||
config=config
|
||||
)
|
||||
|
||||
# Tables are automatically chunked, processed, and merged
|
||||
print(f"Extracted {len(result.tables)} complex tables")
|
||||
for table in result.tables:
|
||||
print(f"Merged table: {len(table['data'])} total rows")
|
||||
```
|
||||
|
||||
**Advanced Features:**
|
||||
|
||||
- **Intelligent Chunking**: Automatically splits massive tables while preserving structure
|
||||
- **Context Preservation**: Overlapping chunks maintain column relationships
|
||||
- **Parallel Processing**: Concurrent chunk processing for speed
|
||||
- **Smart Merging**: Reconstructs complete tables from processed chunks
|
||||
- **Complex Structure Support**: Handles rowspan, colspan, nested tables
|
||||
- **Metadata Extraction**: Captures table context, captions, and relationships
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Financial Analysis**: Extract complex earnings tables and financial statements
|
||||
- **Research & Academia**: Process large datasets from Wikipedia, research papers
|
||||
- **E-commerce**: Handle product comparison tables with complex layouts
|
||||
- **Government Data**: Extract census data, statistical tables from official sources
|
||||
- **Competitive Intelligence**: Process competitor pricing and feature tables
|
||||
|
||||
## ⚡ Enhanced Concurrency: True Performance Gains
|
||||
|
||||
**The Problem:** The `arun_many()` method wasn't achieving true concurrency for fast-completing tasks, leading to sequential processing bottlenecks in batch operations.
|
||||
|
||||
**My Solution:** I implemented true concurrency improvements in the dispatcher that enable genuine parallel processing for fast-completing tasks.
|
||||
|
||||
### Performance Optimization
|
||||
|
||||
```python
|
||||
# Before v0.7.4: Sequential-like behavior for fast tasks
|
||||
# After v0.7.4: True concurrency
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
# These will now run with true concurrency
|
||||
urls = [
|
||||
"https://httpbin.org/delay/1",
|
||||
"https://httpbin.org/delay/1",
|
||||
"https://httpbin.org/delay/1",
|
||||
"https://httpbin.org/delay/1"
|
||||
]
|
||||
|
||||
# Processes in truly parallel fashion
|
||||
results = await crawler.arun_many(urls)
|
||||
|
||||
# Performance improvement: ~4x faster for fast-completing tasks
|
||||
print(f"Processed {len(results)} URLs with true concurrency")
|
||||
```
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **API Crawling**: 3-4x faster processing of REST endpoints and API documentation
|
||||
- **Batch URL Processing**: Significant speedup for large URL lists
|
||||
- **Monitoring Systems**: Faster health checks and status page monitoring
|
||||
- **Data Aggregation**: Improved performance for real-time data collection
|
||||
|
||||
## 🔧 Critical Stability Fixes
|
||||
|
||||
### Browser Manager Race Condition Resolution
|
||||
|
||||
**The Problem:** Concurrent page creation in persistent browser contexts caused "Target page/context closed" errors during high-concurrency operations.
|
||||
|
||||
**My Solution:** Implemented thread-safe page creation with proper locking mechanisms.
|
||||
|
||||
```python
|
||||
# Fixed: Safe concurrent page creation
|
||||
browser_config = BrowserConfig(
|
||||
browser_type="chromium",
|
||||
use_persistent_context=True, # Now thread-safe
|
||||
max_concurrent_sessions=10 # Safely handle concurrent requests
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
# These concurrent operations are now stable
|
||||
tasks = [crawler.arun(url) for url in url_list]
|
||||
results = await asyncio.gather(*tasks) # No more race conditions
|
||||
```
|
||||
|
||||
### Enhanced Browser Profiler
|
||||
|
||||
**The Problem:** Inconsistent keyboard handling across platforms and unreliable quit mechanisms.
|
||||
|
||||
**My Solution:** Cross-platform keyboard listeners with improved quit handling.
|
||||
|
||||
### Advanced URL Processing
|
||||
|
||||
**The Problem:** Raw URL formats (`raw://` and `raw:`) weren't properly handled, and base tag link resolution was incomplete.
|
||||
|
||||
**My Solution:** Enhanced URL preprocessing and base tag support.
|
||||
|
||||
```python
|
||||
# Now properly handles all URL formats
|
||||
urls = [
|
||||
"https://example.com",
|
||||
"raw://static-html-content",
|
||||
"raw:file://local-file.html"
|
||||
]
|
||||
|
||||
# Base tag links are now correctly resolved
|
||||
config = CrawlerRunConfig(
|
||||
include_links=True, # Links properly resolved with base tags
|
||||
resolve_absolute_urls=True
|
||||
)
|
||||
```
|
||||
|
||||
## 🛡️ Enhanced Proxy Configuration
|
||||
|
||||
**The Problem:** Proxy configuration only accepted specific formats, limiting flexibility.
|
||||
|
||||
**My Solution:** Enhanced ProxyConfig to support both dictionary and string formats.
|
||||
|
||||
```python
|
||||
# Multiple proxy configuration formats now supported
|
||||
from crawl4ai import BrowserConfig, ProxyConfig
|
||||
|
||||
# String format
|
||||
proxy_config = ProxyConfig("http://proxy.example.com:8080")
|
||||
|
||||
# Dictionary format
|
||||
proxy_config = ProxyConfig({
|
||||
"server": "http://proxy.example.com:8080",
|
||||
"username": "user",
|
||||
"password": "pass"
|
||||
})
|
||||
|
||||
# Use with crawler
|
||||
browser_config = BrowserConfig(proxy_config=proxy_config)
|
||||
async with AsyncWebCrawler(config=browser_config) as crawler:
|
||||
result = await crawler.arun("https://httpbin.org/ip")
|
||||
```
|
||||
|
||||
## 🐳 Docker & Infrastructure Improvements
|
||||
|
||||
This release includes several Docker and infrastructure improvements:
|
||||
|
||||
- **Better API Token Handling**: Improved Docker example scripts with correct endpoints
|
||||
- **Raw HTML Support**: Enhanced Docker API to handle raw HTML content properly
|
||||
- **Documentation Updates**: Comprehensive Docker deployment examples
|
||||
- **Test Coverage**: Expanded test suite with better coverage
|
||||
|
||||
## 📚 Documentation & Examples
|
||||
|
||||
Enhanced documentation includes:
|
||||
|
||||
- **LLM Table Extraction Guide**: Comprehensive examples and best practices
|
||||
- **Migration Documentation**: Updated patterns for new table extraction methods
|
||||
- **Docker Deployment**: Complete deployment guide with examples
|
||||
- **Performance Optimization**: Guidelines for concurrent crawling
|
||||
|
||||
## 🙏 Acknowledgments
|
||||
|
||||
Thanks to our contributors and community for feedback, bug reports, and feature requests that made this release possible.
|
||||
|
||||
## 📚 Resources
|
||||
|
||||
- [Full Documentation](https://docs.crawl4ai.com)
|
||||
- [GitHub Repository](https://github.com/unclecode/crawl4ai)
|
||||
- [Discord Community](https://discord.gg/crawl4ai)
|
||||
- [LLM Table Extraction Examples](https://github.com/unclecode/crawl4ai/blob/main/docs/examples/llm_table_extraction_example.py)
|
||||
|
||||
---
|
||||
|
||||
*Crawl4AI v0.7.4 delivers intelligent table extraction and significant performance improvements. The new LLMTableExtraction strategy handles complex tables that were previously impossible to process, while concurrency improvements make batch operations 3-4x faster. Try the intelligent table extraction—it's a game changer for data extraction workflows!*
|
||||
|
||||
**Happy Crawling! 🕷️**
|
||||
|
||||
*- The Crawl4AI Team*
|
||||
@@ -0,0 +1,318 @@
|
||||
# 🚀 Crawl4AI v0.7.5: The Docker Hooks & Security Update
|
||||
|
||||
*September 29, 2025 • 8 min read*
|
||||
|
||||
---
|
||||
|
||||
Today I'm releasing Crawl4AI v0.7.5—focused on extensibility and security. This update introduces the Docker Hooks System for pipeline customization, enhanced LLM integration, and important security improvements.
|
||||
|
||||
## 🎯 What's New at a Glance
|
||||
|
||||
- **Docker Hooks System**: Custom Python functions at key pipeline points with function-based API
|
||||
- **Function-Based Hooks**: New `hooks_to_string()` utility with Docker client auto-conversion
|
||||
- **Enhanced LLM Integration**: Custom providers with temperature control
|
||||
- **HTTPS Preservation**: Secure internal link handling
|
||||
- **Bug Fixes**: Resolved multiple community-reported issues
|
||||
- **Improved Docker Error Handling**: Better debugging and reliability
|
||||
|
||||
## 🔧 Docker Hooks System: Pipeline Customization
|
||||
|
||||
Every scraping project needs custom logic—authentication, performance optimization, content processing. Traditional solutions require forking or complex workarounds. Docker Hooks let you inject custom Python functions at 8 key points in the crawling pipeline.
|
||||
|
||||
### Real Example: Authentication & Performance
|
||||
|
||||
```python
|
||||
import requests
|
||||
|
||||
# Real working hooks for httpbin.org
|
||||
hooks_config = {
|
||||
"on_page_context_created": """
|
||||
async def hook(page, context, **kwargs):
|
||||
print("Hook: Setting up page context")
|
||||
# Block images to speed up crawling
|
||||
await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
|
||||
print("Hook: Images blocked")
|
||||
return page
|
||||
""",
|
||||
|
||||
"before_retrieve_html": """
|
||||
async def hook(page, context, **kwargs):
|
||||
print("Hook: Before retrieving HTML")
|
||||
# Scroll to bottom to load lazy content
|
||||
await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
|
||||
await page.wait_for_timeout(1000)
|
||||
print("Hook: Scrolled to bottom")
|
||||
return page
|
||||
""",
|
||||
|
||||
"before_goto": """
|
||||
async def hook(page, context, url, **kwargs):
|
||||
print(f"Hook: About to navigate to {url}")
|
||||
# Add custom headers
|
||||
await page.set_extra_http_headers({
|
||||
'X-Test-Header': 'crawl4ai-hooks-test'
|
||||
})
|
||||
return page
|
||||
"""
|
||||
}
|
||||
|
||||
# Test with Docker API
|
||||
payload = {
|
||||
"urls": ["https://httpbin.org/html"],
|
||||
"hooks": {
|
||||
"code": hooks_config,
|
||||
"timeout": 30
|
||||
}
|
||||
}
|
||||
|
||||
response = requests.post("http://localhost:11235/crawl", json=payload)
|
||||
result = response.json()
|
||||
|
||||
if result.get('success'):
|
||||
print("✅ Hooks executed successfully!")
|
||||
print(f"Content length: {len(result.get('markdown', ''))} characters")
|
||||
```
|
||||
|
||||
**Available Hook Points:**
|
||||
- `on_browser_created`: Browser setup
|
||||
- `on_page_context_created`: Page context configuration
|
||||
- `before_goto`: Pre-navigation setup
|
||||
- `after_goto`: Post-navigation processing
|
||||
- `on_user_agent_updated`: User agent changes
|
||||
- `on_execution_started`: Crawl initialization
|
||||
- `before_retrieve_html`: Pre-extraction processing
|
||||
- `before_return_html`: Final HTML processing
|
||||
|
||||
### Function-Based Hooks API
|
||||
|
||||
Writing hooks as strings works, but lacks IDE support and type checking. v0.7.5 introduces a function-based approach with automatic conversion!
|
||||
|
||||
**Option 1: Using the `hooks_to_string()` Utility**
|
||||
|
||||
```python
|
||||
from crawl4ai import hooks_to_string
|
||||
import requests
|
||||
|
||||
# Define hooks as regular Python functions (with full IDE support!)
|
||||
async def on_page_context_created(page, context, **kwargs):
|
||||
"""Block images to speed up crawling"""
|
||||
await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
|
||||
await page.set_viewport_size({"width": 1920, "height": 1080})
|
||||
return page
|
||||
|
||||
async def before_goto(page, context, url, **kwargs):
|
||||
"""Add custom headers"""
|
||||
await page.set_extra_http_headers({
|
||||
'X-Crawl4AI': 'v0.7.5',
|
||||
'X-Custom-Header': 'my-value'
|
||||
})
|
||||
return page
|
||||
|
||||
# Convert functions to strings
|
||||
hooks_code = hooks_to_string({
|
||||
"on_page_context_created": on_page_context_created,
|
||||
"before_goto": before_goto
|
||||
})
|
||||
|
||||
# Use with REST API
|
||||
payload = {
|
||||
"urls": ["https://httpbin.org/html"],
|
||||
"hooks": {"code": hooks_code, "timeout": 30}
|
||||
}
|
||||
response = requests.post("http://localhost:11235/crawl", json=payload)
|
||||
```
|
||||
|
||||
**Option 2: Docker Client with Automatic Conversion (Recommended!)**
|
||||
|
||||
```python
|
||||
from crawl4ai.docker_client import Crawl4aiDockerClient
|
||||
|
||||
# Define hooks as functions (same as above)
|
||||
async def on_page_context_created(page, context, **kwargs):
|
||||
await context.route("**/*.{png,jpg,jpeg,gif,webp}", lambda route: route.abort())
|
||||
return page
|
||||
|
||||
async def before_retrieve_html(page, context, **kwargs):
|
||||
# Scroll to load lazy content
|
||||
await page.evaluate("window.scrollTo(0, document.body.scrollHeight)")
|
||||
await page.wait_for_timeout(1000)
|
||||
return page
|
||||
|
||||
# Use Docker client - conversion happens automatically!
|
||||
client = Crawl4aiDockerClient(base_url="http://localhost:11235")
|
||||
|
||||
results = await client.crawl(
|
||||
urls=["https://httpbin.org/html"],
|
||||
hooks={
|
||||
"on_page_context_created": on_page_context_created,
|
||||
"before_retrieve_html": before_retrieve_html
|
||||
},
|
||||
hooks_timeout=30
|
||||
)
|
||||
|
||||
if results and results.success:
|
||||
print(f"✅ Hooks executed! HTML length: {len(results.html)}")
|
||||
```
|
||||
|
||||
**Benefits of Function-Based Hooks:**
|
||||
- ✅ Full IDE support (autocomplete, syntax highlighting)
|
||||
- ✅ Type checking and linting
|
||||
- ✅ Easier to test and debug
|
||||
- ✅ Reusable across projects
|
||||
- ✅ Automatic conversion in Docker client
|
||||
- ✅ No breaking changes - string hooks still work!
|
||||
|
||||
## 🤖 Enhanced LLM Integration
|
||||
|
||||
Enhanced LLM integration with custom providers, temperature control, and base URL configuration.
|
||||
|
||||
### Multi-Provider Support
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
|
||||
from crawl4ai.extraction_strategy import LLMExtractionStrategy
|
||||
|
||||
# Test with different providers
|
||||
async def test_llm_providers():
|
||||
# OpenAI with custom temperature
|
||||
openai_strategy = LLMExtractionStrategy(
|
||||
provider="gemini/gemini-2.5-flash-lite",
|
||||
api_token="your-api-token",
|
||||
temperature=0.7, # New in v0.7.5
|
||||
instruction="Summarize this page in one sentence"
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
"https://example.com",
|
||||
config=CrawlerRunConfig(extraction_strategy=openai_strategy)
|
||||
)
|
||||
|
||||
if result.success:
|
||||
print("✅ LLM extraction completed")
|
||||
print(result.extracted_content)
|
||||
|
||||
# Docker API with enhanced LLM config
|
||||
llm_payload = {
|
||||
"url": "https://example.com",
|
||||
"f": "llm",
|
||||
"q": "Summarize this page in one sentence.",
|
||||
"provider": "gemini/gemini-2.5-flash-lite",
|
||||
"temperature": 0.7
|
||||
}
|
||||
|
||||
response = requests.post("http://localhost:11235/md", json=llm_payload)
|
||||
```
|
||||
|
||||
**New Features:**
|
||||
- Custom `temperature` parameter for creativity control
|
||||
- `base_url` for custom API endpoints
|
||||
- Multi-provider environment variable support
|
||||
- Docker API integration
|
||||
|
||||
## 🔒 HTTPS Preservation
|
||||
|
||||
**The Problem:** Modern web apps require HTTPS everywhere. When crawlers downgrade internal links from HTTPS to HTTP, authentication breaks and security warnings appear.
|
||||
|
||||
**Solution:** HTTPS preservation maintains secure protocols throughout crawling.
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig, FilterChain, URLPatternFilter, BFSDeepCrawlStrategy
|
||||
|
||||
async def test_https_preservation():
|
||||
# Enable HTTPS preservation
|
||||
url_filter = URLPatternFilter(
|
||||
patterns=["^(https:\/\/)?quotes\.toscrape\.com(\/.*)?$"]
|
||||
)
|
||||
|
||||
config = CrawlerRunConfig(
|
||||
exclude_external_links=True,
|
||||
preserve_https_for_internal_links=True, # New in v0.7.5
|
||||
deep_crawl_strategy=BFSDeepCrawlStrategy(
|
||||
max_depth=2,
|
||||
max_pages=5,
|
||||
filter_chain=FilterChain([url_filter])
|
||||
)
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
async for result in await crawler.arun(
|
||||
url="https://quotes.toscrape.com",
|
||||
config=config
|
||||
):
|
||||
# All internal links maintain HTTPS
|
||||
internal_links = [link['href'] for link in result.links['internal']]
|
||||
https_links = [link for link in internal_links if link.startswith('https://')]
|
||||
|
||||
print(f"HTTPS links preserved: {len(https_links)}/{len(internal_links)}")
|
||||
for link in https_links[:3]:
|
||||
print(f" → {link}")
|
||||
```
|
||||
|
||||
## 🛠️ Bug Fixes and Improvements
|
||||
|
||||
### Major Fixes
|
||||
- **URL Processing**: Fixed '+' sign preservation in query parameters (#1332)
|
||||
- **Proxy Configuration**: Enhanced proxy string parsing (old `proxy` parameter deprecated)
|
||||
- **Docker Error Handling**: Comprehensive error messages with status codes
|
||||
- **Memory Management**: Fixed leaks in long-running sessions
|
||||
- **JWT Authentication**: Fixed Docker JWT validation issues (#1442)
|
||||
- **Playwright Stealth**: Fixed stealth features for Playwright integration (#1481)
|
||||
- **API Configuration**: Fixed config handling to prevent overriding user-provided settings (#1505)
|
||||
- **Docker Filter Serialization**: Resolved JSON encoding errors in deep crawl strategy (#1419)
|
||||
- **LLM Provider Support**: Fixed custom LLM provider integration for adaptive crawler (#1291)
|
||||
- **Performance Issues**: Resolved backoff strategy failures and timeout handling (#989)
|
||||
|
||||
### Community-Reported Issues Fixed
|
||||
This release addresses multiple issues reported by the community through GitHub issues and Discord discussions:
|
||||
- Fixed browser configuration reference errors
|
||||
- Resolved dependency conflicts with cssselect
|
||||
- Improved error messaging for failed authentications
|
||||
- Enhanced compatibility with various proxy configurations
|
||||
- Fixed edge cases in URL normalization
|
||||
|
||||
### Configuration Updates
|
||||
```python
|
||||
# Old proxy config (deprecated)
|
||||
# browser_config = BrowserConfig(proxy="http://proxy:8080")
|
||||
|
||||
# New enhanced proxy config
|
||||
browser_config = BrowserConfig(
|
||||
proxy_config={
|
||||
"server": "http://proxy:8080",
|
||||
"username": "optional-user",
|
||||
"password": "optional-pass"
|
||||
}
|
||||
)
|
||||
```
|
||||
|
||||
## 🔄 Breaking Changes
|
||||
|
||||
1. **Python 3.10+ Required**: Upgrade from Python 3.9
|
||||
2. **Proxy Parameter Deprecated**: Use new `proxy_config` structure
|
||||
3. **New Dependency**: Added `cssselect` for better CSS handling
|
||||
|
||||
## 🚀 Get Started
|
||||
|
||||
```bash
|
||||
# Install latest version
|
||||
pip install crawl4ai==0.7.5
|
||||
|
||||
# Docker deployment
|
||||
docker pull unclecode/crawl4ai:latest
|
||||
docker run -p 11235:11235 unclecode/crawl4ai:latest
|
||||
```
|
||||
|
||||
**Try the Demo:**
|
||||
```bash
|
||||
# Run working examples
|
||||
python docs/releases_review/demo_v0.7.5.py
|
||||
```
|
||||
|
||||
**Resources:**
|
||||
- 📖 Documentation: [docs.crawl4ai.com](https://docs.crawl4ai.com)
|
||||
- 🐙 GitHub: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
|
||||
- 💬 Discord: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
|
||||
- 🐦 Twitter: [@unclecode](https://x.com/unclecode)
|
||||
|
||||
Happy crawling! 🕷️
|
||||
@@ -0,0 +1,314 @@
|
||||
# Crawl4AI v0.7.6 Release Notes
|
||||
|
||||
*Release Date: October 22, 2025*
|
||||
|
||||
I'm excited to announce Crawl4AI v0.7.6, featuring a complete webhook infrastructure for the Docker job queue API! This release eliminates polling and brings real-time notifications to both crawling and LLM extraction workflows.
|
||||
|
||||
## 🎯 What's New
|
||||
|
||||
### Webhook Support for Docker Job Queue API
|
||||
|
||||
The headline feature of v0.7.6 is comprehensive webhook support for asynchronous job processing. No more constant polling to check if your jobs are done - get instant notifications when they complete!
|
||||
|
||||
**Key Capabilities:**
|
||||
|
||||
- ✅ **Universal Webhook Support**: Both `/crawl/job` and `/llm/job` endpoints now support webhooks
|
||||
- ✅ **Flexible Delivery Modes**: Choose notification-only or include full data in the webhook payload
|
||||
- ✅ **Reliable Delivery**: Exponential backoff retry mechanism (5 attempts: 1s → 2s → 4s → 8s → 16s)
|
||||
- ✅ **Custom Authentication**: Add custom headers for webhook authentication
|
||||
- ✅ **Global Configuration**: Set default webhook URL in `config.yml` for all jobs
|
||||
- ✅ **Task Type Identification**: Distinguish between `crawl` and `llm_extraction` tasks
|
||||
|
||||
### How It Works
|
||||
|
||||
Instead of constantly checking job status:
|
||||
|
||||
**OLD WAY (Polling):**
|
||||
```python
|
||||
# Submit job
|
||||
response = requests.post("http://localhost:11235/crawl/job", json=payload)
|
||||
task_id = response.json()['task_id']
|
||||
|
||||
# Poll until complete
|
||||
while True:
|
||||
status = requests.get(f"http://localhost:11235/crawl/job/{task_id}")
|
||||
if status.json()['status'] == 'completed':
|
||||
break
|
||||
time.sleep(5) # Wait and try again
|
||||
```
|
||||
|
||||
**NEW WAY (Webhooks):**
|
||||
```python
|
||||
# Submit job with webhook
|
||||
payload = {
|
||||
"urls": ["https://example.com"],
|
||||
"webhook_config": {
|
||||
"webhook_url": "https://myapp.com/webhook",
|
||||
"webhook_data_in_payload": True
|
||||
}
|
||||
}
|
||||
response = requests.post("http://localhost:11235/crawl/job", json=payload)
|
||||
|
||||
# Done! Webhook will notify you when complete
|
||||
# Your webhook handler receives the results automatically
|
||||
```
|
||||
|
||||
### Crawl Job Webhooks
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:11235/crawl/job \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"urls": ["https://example.com"],
|
||||
"browser_config": {"headless": true},
|
||||
"crawler_config": {"cache_mode": "bypass"},
|
||||
"webhook_config": {
|
||||
"webhook_url": "https://myapp.com/webhooks/crawl-complete",
|
||||
"webhook_data_in_payload": false,
|
||||
"webhook_headers": {
|
||||
"X-Webhook-Secret": "your-secret-token"
|
||||
}
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
### LLM Extraction Job Webhooks (NEW!)
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:11235/llm/job \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"url": "https://example.com/article",
|
||||
"q": "Extract the article title, author, and publication date",
|
||||
"schema": "{\"type\":\"object\",\"properties\":{\"title\":{\"type\":\"string\"}}}",
|
||||
"provider": "openai/gpt-4o-mini",
|
||||
"webhook_config": {
|
||||
"webhook_url": "https://myapp.com/webhooks/llm-complete",
|
||||
"webhook_data_in_payload": true
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
### Webhook Payload Structure
|
||||
|
||||
**Success (with data):**
|
||||
```json
|
||||
{
|
||||
"task_id": "llm_1698765432",
|
||||
"task_type": "llm_extraction",
|
||||
"status": "completed",
|
||||
"timestamp": "2025-10-22T10:30:00.000000+00:00",
|
||||
"urls": ["https://example.com/article"],
|
||||
"data": {
|
||||
"extracted_content": {
|
||||
"title": "Understanding Web Scraping",
|
||||
"author": "John Doe",
|
||||
"date": "2025-10-22"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Failure:**
|
||||
```json
|
||||
{
|
||||
"task_id": "crawl_abc123",
|
||||
"task_type": "crawl",
|
||||
"status": "failed",
|
||||
"timestamp": "2025-10-22T10:30:00.000000+00:00",
|
||||
"urls": ["https://example.com"],
|
||||
"error": "Connection timeout after 30s"
|
||||
}
|
||||
```
|
||||
|
||||
### Simple Webhook Handler Example
|
||||
|
||||
```python
|
||||
from flask import Flask, request, jsonify
|
||||
|
||||
app = Flask(__name__)
|
||||
|
||||
@app.route('/webhook', methods=['POST'])
|
||||
def handle_webhook():
|
||||
payload = request.json
|
||||
|
||||
task_id = payload['task_id']
|
||||
task_type = payload['task_type']
|
||||
status = payload['status']
|
||||
|
||||
if status == 'completed':
|
||||
if 'data' in payload:
|
||||
# Process data directly
|
||||
data = payload['data']
|
||||
else:
|
||||
# Fetch from API
|
||||
endpoint = 'crawl' if task_type == 'crawl' else 'llm'
|
||||
response = requests.get(f'http://localhost:11235/{endpoint}/job/{task_id}')
|
||||
data = response.json()
|
||||
|
||||
# Your business logic here
|
||||
print(f"Job {task_id} completed!")
|
||||
|
||||
elif status == 'failed':
|
||||
error = payload.get('error', 'Unknown error')
|
||||
print(f"Job {task_id} failed: {error}")
|
||||
|
||||
return jsonify({"status": "received"}), 200
|
||||
|
||||
app.run(port=8080)
|
||||
```
|
||||
|
||||
## 📊 Performance Improvements
|
||||
|
||||
- **Reduced Server Load**: Eliminates constant polling requests
|
||||
- **Lower Latency**: Instant notification vs. polling interval delay
|
||||
- **Better Resource Usage**: Frees up client connections while jobs run in background
|
||||
- **Scalable Architecture**: Handles high-volume crawling workflows efficiently
|
||||
|
||||
## 🐛 Bug Fixes
|
||||
|
||||
- Fixed webhook configuration serialization for Pydantic HttpUrl fields
|
||||
- Improved error handling in webhook delivery service
|
||||
- Enhanced Redis task storage for webhook config persistence
|
||||
|
||||
## 🌍 Expected Real-World Impact
|
||||
|
||||
### For Web Scraping Workflows
|
||||
- **Reduced Costs**: Less API calls = lower bandwidth and server costs
|
||||
- **Better UX**: Instant notifications improve user experience
|
||||
- **Scalability**: Handle 100s of concurrent jobs without polling overhead
|
||||
|
||||
### For LLM Extraction Pipelines
|
||||
- **Async Processing**: Submit LLM extraction jobs and move on
|
||||
- **Batch Processing**: Queue multiple extractions, get notified as they complete
|
||||
- **Integration**: Easy integration with workflow automation tools (Zapier, n8n, etc.)
|
||||
|
||||
### For Microservices
|
||||
- **Event-Driven**: Perfect for event-driven microservice architectures
|
||||
- **Decoupling**: Decouple job submission from result processing
|
||||
- **Reliability**: Automatic retries ensure webhooks are delivered
|
||||
|
||||
## 🔄 Breaking Changes
|
||||
|
||||
**None!** This release is fully backward compatible.
|
||||
|
||||
- Webhook configuration is optional
|
||||
- Existing code continues to work without modification
|
||||
- Polling is still supported for jobs without webhook config
|
||||
|
||||
## 📚 Documentation
|
||||
|
||||
### New Documentation
|
||||
- **[WEBHOOK_EXAMPLES.md](../deploy/docker/WEBHOOK_EXAMPLES.md)** - Comprehensive webhook usage guide
|
||||
- **[docker_webhook_example.py](../docs/examples/docker_webhook_example.py)** - Working code examples
|
||||
|
||||
### Updated Documentation
|
||||
- **[Docker README](../deploy/docker/README.md)** - Added webhook sections
|
||||
- API documentation with webhook examples
|
||||
|
||||
## 🛠️ Migration Guide
|
||||
|
||||
No migration needed! Webhooks are opt-in:
|
||||
|
||||
1. **To use webhooks**: Add `webhook_config` to your job payload
|
||||
2. **To keep polling**: Continue using your existing code
|
||||
|
||||
### Quick Start
|
||||
|
||||
```python
|
||||
# Just add webhook_config to your existing payload
|
||||
payload = {
|
||||
# Your existing configuration
|
||||
"urls": ["https://example.com"],
|
||||
"browser_config": {...},
|
||||
"crawler_config": {...},
|
||||
|
||||
# NEW: Add webhook configuration
|
||||
"webhook_config": {
|
||||
"webhook_url": "https://myapp.com/webhook",
|
||||
"webhook_data_in_payload": True
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 🔧 Configuration
|
||||
|
||||
### Global Webhook Configuration (config.yml)
|
||||
|
||||
```yaml
|
||||
webhooks:
|
||||
enabled: true
|
||||
default_url: "https://myapp.com/webhooks/default" # Optional
|
||||
data_in_payload: false
|
||||
retry:
|
||||
max_attempts: 5
|
||||
initial_delay_ms: 1000
|
||||
max_delay_ms: 32000
|
||||
timeout_ms: 30000
|
||||
headers:
|
||||
User-Agent: "Crawl4AI-Webhook/1.0"
|
||||
```
|
||||
|
||||
## 🚀 Upgrade Instructions
|
||||
|
||||
### Docker
|
||||
|
||||
```bash
|
||||
# Pull the latest image
|
||||
docker pull unclecode/crawl4ai:0.7.6
|
||||
|
||||
# Or use latest tag
|
||||
docker pull unclecode/crawl4ai:latest
|
||||
|
||||
# Run with webhook support
|
||||
docker run -d \
|
||||
-p 11235:11235 \
|
||||
--env-file .llm.env \
|
||||
--name crawl4ai \
|
||||
unclecode/crawl4ai:0.7.6
|
||||
```
|
||||
|
||||
### Python Package
|
||||
|
||||
```bash
|
||||
pip install --upgrade crawl4ai
|
||||
```
|
||||
|
||||
## 💡 Pro Tips
|
||||
|
||||
1. **Use notification-only mode** for large results - fetch data separately to avoid large webhook payloads
|
||||
2. **Set custom headers** for webhook authentication and request tracking
|
||||
3. **Configure global default webhook** for consistent handling across all jobs
|
||||
4. **Implement idempotent webhook handlers** - same webhook may be delivered multiple times on retry
|
||||
5. **Use structured schemas** with LLM extraction for predictable webhook data
|
||||
|
||||
## 🎬 Demo
|
||||
|
||||
Try the release demo:
|
||||
|
||||
```bash
|
||||
python docs/releases_review/demo_v0.7.6.py
|
||||
```
|
||||
|
||||
This comprehensive demo showcases:
|
||||
- Crawl job webhooks (notification-only and with data)
|
||||
- LLM extraction webhooks (with JSON schema support)
|
||||
- Custom headers for authentication
|
||||
- Webhook retry mechanism
|
||||
- Real-time webhook receiver
|
||||
|
||||
## 🙏 Acknowledgments
|
||||
|
||||
Thank you to the community for the feedback that shaped this feature! Special thanks to everyone who requested webhook support for asynchronous job processing.
|
||||
|
||||
## 📞 Support
|
||||
|
||||
- **Documentation**: https://docs.crawl4ai.com
|
||||
- **GitHub Issues**: https://github.com/unclecode/crawl4ai/issues
|
||||
- **Discord**: https://discord.gg/crawl4ai
|
||||
|
||||
---
|
||||
|
||||
**Happy crawling with webhooks!** 🕷️🪝
|
||||
|
||||
*- unclecode*
|
||||
@@ -0,0 +1,626 @@
|
||||
# 🚀 Crawl4AI v0.7.7: The Self-Hosting & Monitoring Update
|
||||
|
||||
*November 14, 2025 • 10 min read*
|
||||
|
||||
---
|
||||
|
||||
Today I'm releasing Crawl4AI v0.7.7—the Self-Hosting & Monitoring Update. This release transforms Crawl4AI Docker from a simple containerized crawler into a complete self-hosting platform with enterprise-grade real-time monitoring, full operational transparency, and production-ready observability.
|
||||
|
||||
## 🎯 What's New at a Glance
|
||||
|
||||
- **📊 Real-time Monitoring Dashboard**: Interactive web UI with live system metrics and browser pool status
|
||||
- **🔌 Comprehensive Monitor API**: Complete REST API for programmatic access to all monitoring data
|
||||
- **⚡ WebSocket Streaming**: Real-time updates every 2 seconds for custom dashboards
|
||||
- **🎮 Control Actions**: Manual browser management (kill, restart, cleanup)
|
||||
- **🔥 Smart Browser Pool**: 3-tier architecture (permanent/hot/cold) with automatic promotion
|
||||
- **🧹 Janitor Cleanup System**: Automatic resource management with event logging
|
||||
- **📈 Production Metrics**: 6 critical metrics for operational excellence
|
||||
- **🏭 Integration Ready**: Prometheus, alerting, and log aggregation examples
|
||||
- **🐛 Critical Bug Fixes**: Async LLM extraction, DFS crawling, viewport config, and more
|
||||
|
||||
## 📊 Real-time Monitoring Dashboard: Complete Visibility
|
||||
|
||||
**The Problem:** Running Crawl4AI in Docker was like flying blind. Users had no visibility into what was happening inside the container—memory usage, active requests, browser pools, or errors. Troubleshooting required checking logs, and there was no way to monitor performance or manually intervene when issues occurred.
|
||||
|
||||
**My Solution:** I built a complete real-time monitoring system with an interactive dashboard, comprehensive REST API, WebSocket streaming, and manual control actions. Now you have full transparency and control over your crawling infrastructure.
|
||||
|
||||
### The Self-Hosting Value Proposition
|
||||
|
||||
Before v0.7.7, Docker was just a containerized crawler. After v0.7.7, it's a complete self-hosting platform that gives you:
|
||||
|
||||
- **🔒 Data Privacy**: Your data never leaves your infrastructure
|
||||
- **💰 Cost Control**: No per-request pricing or rate limits
|
||||
- **🎯 Full Customization**: Complete control over configurations and strategies
|
||||
- **📊 Complete Transparency**: Real-time visibility into every aspect
|
||||
- **⚡ Performance**: Direct access without network overhead
|
||||
- **🛡️ Enterprise Security**: Keep workflows behind your firewall
|
||||
|
||||
### Interactive Monitoring Dashboard
|
||||
|
||||
Access the dashboard at `http://localhost:11235/dashboard` to see:
|
||||
|
||||
- **System Health Overview**: CPU, memory, network, and uptime in real-time
|
||||
- **Live Request Tracking**: Active and completed requests with full details
|
||||
- **Browser Pool Management**: Interactive table with permanent/hot/cold browsers
|
||||
- **Janitor Events Log**: Automatic cleanup activities
|
||||
- **Error Monitoring**: Full context error logs
|
||||
|
||||
The dashboard updates every 2 seconds via WebSocket, giving you live visibility into your crawling operations.
|
||||
|
||||
## 🔌 Monitor API: Programmatic Access
|
||||
|
||||
**The Problem:** Monitoring dashboards are great for humans, but automation and integration require programmatic access.
|
||||
|
||||
**My Solution:** A comprehensive REST API that exposes all monitoring data for integration with your existing infrastructure.
|
||||
|
||||
### System Health Endpoint
|
||||
|
||||
```python
|
||||
import httpx
|
||||
import asyncio
|
||||
|
||||
async def monitor_system_health():
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.get("http://localhost:11235/monitor/health")
|
||||
health = response.json()
|
||||
|
||||
print(f"Container Metrics:")
|
||||
print(f" CPU: {health['container']['cpu_percent']:.1f}%")
|
||||
print(f" Memory: {health['container']['memory_percent']:.1f}%")
|
||||
print(f" Uptime: {health['container']['uptime_seconds']}s")
|
||||
|
||||
print(f"\nBrowser Pool:")
|
||||
print(f" Permanent: {health['pool']['permanent']['active']} active")
|
||||
print(f" Hot Pool: {health['pool']['hot']['count']} browsers")
|
||||
print(f" Cold Pool: {health['pool']['cold']['count']} browsers")
|
||||
|
||||
print(f"\nStatistics:")
|
||||
print(f" Total Requests: {health['stats']['total_requests']}")
|
||||
print(f" Success Rate: {health['stats']['success_rate_percent']:.1f}%")
|
||||
print(f" Avg Latency: {health['stats']['avg_latency_ms']:.0f}ms")
|
||||
|
||||
asyncio.run(monitor_system_health())
|
||||
```
|
||||
|
||||
### Request Tracking
|
||||
|
||||
```python
|
||||
async def track_requests():
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.get("http://localhost:11235/monitor/requests")
|
||||
requests_data = response.json()
|
||||
|
||||
print(f"Active Requests: {len(requests_data['active'])}")
|
||||
print(f"Completed Requests: {len(requests_data['completed'])}")
|
||||
|
||||
# See details of recent requests
|
||||
for req in requests_data['completed'][:5]:
|
||||
status_icon = "✅" if req['success'] else "❌"
|
||||
print(f"{status_icon} {req['endpoint']} - {req['latency_ms']:.0f}ms")
|
||||
```
|
||||
|
||||
### Browser Pool Management
|
||||
|
||||
```python
|
||||
async def monitor_browser_pool():
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.get("http://localhost:11235/monitor/browsers")
|
||||
browsers = response.json()
|
||||
|
||||
print(f"Pool Summary:")
|
||||
print(f" Total Browsers: {browsers['summary']['total_count']}")
|
||||
print(f" Total Memory: {browsers['summary']['total_memory_mb']} MB")
|
||||
print(f" Reuse Rate: {browsers['summary']['reuse_rate_percent']:.1f}%")
|
||||
|
||||
# List all browsers
|
||||
for browser in browsers['permanent']:
|
||||
print(f"🔥 Permanent: {browser['browser_id'][:8]}... | "
|
||||
f"Requests: {browser['request_count']} | "
|
||||
f"Memory: {browser['memory_mb']:.0f} MB")
|
||||
```
|
||||
|
||||
### Endpoint Performance Statistics
|
||||
|
||||
```python
|
||||
async def get_endpoint_stats():
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.get("http://localhost:11235/monitor/endpoints/stats")
|
||||
stats = response.json()
|
||||
|
||||
print("Endpoint Analytics:")
|
||||
for endpoint, data in stats.items():
|
||||
print(f" {endpoint}:")
|
||||
print(f" Requests: {data['count']}")
|
||||
print(f" Avg Latency: {data['avg_latency_ms']:.0f}ms")
|
||||
print(f" Success Rate: {data['success_rate_percent']:.1f}%")
|
||||
```
|
||||
|
||||
### Complete API Reference
|
||||
|
||||
The Monitor API includes these endpoints:
|
||||
|
||||
- `GET /monitor/health` - System health with pool statistics
|
||||
- `GET /monitor/requests` - Active and completed request tracking
|
||||
- `GET /monitor/browsers` - Browser pool details and efficiency
|
||||
- `GET /monitor/endpoints/stats` - Per-endpoint performance analytics
|
||||
- `GET /monitor/timeline?minutes=5` - Time-series data for charts
|
||||
- `GET /monitor/logs/janitor?limit=10` - Cleanup activity logs
|
||||
- `GET /monitor/logs/errors?limit=10` - Error logs with context
|
||||
- `POST /monitor/actions/cleanup` - Force immediate cleanup
|
||||
- `POST /monitor/actions/kill_browser` - Kill specific browser
|
||||
- `POST /monitor/actions/restart_browser` - Restart browser
|
||||
- `POST /monitor/stats/reset` - Reset accumulated statistics
|
||||
|
||||
## ⚡ WebSocket Streaming: Real-time Updates
|
||||
|
||||
**The Problem:** Polling the API every few seconds wastes resources and adds latency. Real-time dashboards need instant updates.
|
||||
|
||||
**My Solution:** WebSocket streaming with 2-second update intervals for building custom real-time dashboards.
|
||||
|
||||
### WebSocket Integration Example
|
||||
|
||||
```python
|
||||
import websockets
|
||||
import json
|
||||
import asyncio
|
||||
|
||||
async def monitor_realtime():
|
||||
uri = "ws://localhost:11235/monitor/ws"
|
||||
|
||||
async with websockets.connect(uri) as websocket:
|
||||
print("Connected to real-time monitoring stream")
|
||||
|
||||
while True:
|
||||
# Receive update every 2 seconds
|
||||
data = await websocket.recv()
|
||||
update = json.loads(data)
|
||||
|
||||
# Access all monitoring data
|
||||
print(f"\n--- Update at {update['timestamp']} ---")
|
||||
print(f"Memory: {update['health']['container']['memory_percent']:.1f}%")
|
||||
print(f"Active Requests: {len(update['requests']['active'])}")
|
||||
print(f"Total Browsers: {update['browsers']['summary']['total_count']}")
|
||||
|
||||
if update['errors']:
|
||||
print(f"⚠️ Recent Errors: {len(update['errors'])}")
|
||||
|
||||
asyncio.run(monitor_realtime())
|
||||
```
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Custom Dashboards**: Build tailored monitoring UIs for your team
|
||||
- **Real-time Alerting**: Trigger alerts instantly when metrics exceed thresholds
|
||||
- **Integration**: Feed live data into monitoring tools like Grafana
|
||||
- **Automation**: React to events in real-time without polling
|
||||
|
||||
## 🔥 Smart Browser Pool: 3-Tier Architecture
|
||||
|
||||
**The Problem:** Creating a new browser for every request is slow and memory-intensive. Traditional browser pools are static and inefficient.
|
||||
|
||||
**My Solution:** A smart 3-tier browser pool that automatically adapts to usage patterns.
|
||||
|
||||
### How It Works
|
||||
|
||||
```python
|
||||
import httpx
|
||||
|
||||
async def demonstrate_browser_pool():
|
||||
async with httpx.AsyncClient() as client:
|
||||
# Request 1-3: Default config → Uses permanent browser
|
||||
print("Phase 1: Using permanent browser")
|
||||
for i in range(3):
|
||||
await client.post(
|
||||
"http://localhost:11235/crawl",
|
||||
json={"urls": [f"https://httpbin.org/html?req={i}"]}
|
||||
)
|
||||
print(f" Request {i+1}: Reused permanent browser")
|
||||
|
||||
# Request 4-6: Custom viewport → Cold pool (first use)
|
||||
print("\nPhase 2: Custom config creates cold pool browser")
|
||||
viewport_config = {"viewport": {"width": 1280, "height": 720}}
|
||||
for i in range(4):
|
||||
await client.post(
|
||||
"http://localhost:11235/crawl",
|
||||
json={
|
||||
"urls": [f"https://httpbin.org/json?v={i}"],
|
||||
"browser_config": viewport_config
|
||||
}
|
||||
)
|
||||
if i < 2:
|
||||
print(f" Request {i+1}: Cold pool browser")
|
||||
else:
|
||||
print(f" Request {i+1}: Promoted to hot pool! (after 3 uses)")
|
||||
|
||||
# Check pool status
|
||||
response = await client.get("http://localhost:11235/monitor/browsers")
|
||||
browsers = response.json()
|
||||
|
||||
print(f"\nPool Status:")
|
||||
print(f" Permanent: {len(browsers['permanent'])} (always active)")
|
||||
print(f" Hot: {len(browsers['hot'])} (frequently used configs)")
|
||||
print(f" Cold: {len(browsers['cold'])} (on-demand)")
|
||||
print(f" Reuse Rate: {browsers['summary']['reuse_rate_percent']:.1f}%")
|
||||
|
||||
asyncio.run(demonstrate_browser_pool())
|
||||
```
|
||||
|
||||
**Pool Tiers:**
|
||||
|
||||
- **🔥 Permanent Browser**: Always-on, default configuration, instant response
|
||||
- **♨️ Hot Pool**: Browsers promoted after 3+ uses, kept warm for quick access
|
||||
- **❄️ Cold Pool**: On-demand browsers for variant configs, cleaned up when idle
|
||||
|
||||
**Expected Real-World Impact:**
|
||||
- **Memory Efficiency**: 10x reduction in memory usage vs creating browsers per request
|
||||
- **Performance**: Instant access to frequently-used configurations
|
||||
- **Automatic Optimization**: Pool adapts to your usage patterns
|
||||
- **Resource Management**: Janitor automatically cleans up idle browsers
|
||||
|
||||
## 🧹 Janitor System: Automatic Cleanup
|
||||
|
||||
**The Problem:** Long-running crawlers accumulate idle browsers and consume memory over time.
|
||||
|
||||
**My Solution:** An automatic janitor system that monitors and cleans up idle resources.
|
||||
|
||||
```python
|
||||
async def monitor_janitor_activity():
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.get("http://localhost:11235/monitor/logs/janitor?limit=5")
|
||||
logs = response.json()
|
||||
|
||||
print("Recent Cleanup Activities:")
|
||||
for log in logs:
|
||||
print(f" {log['timestamp']}: {log['message']}")
|
||||
|
||||
# Example output:
|
||||
# 2025-11-14 10:30:00: Cleaned up 2 cold pool browsers (idle > 5min)
|
||||
# 2025-11-14 10:25:00: Browser reuse rate: 85.3%
|
||||
# 2025-11-14 10:20:00: Hot pool browser promoted (10 requests)
|
||||
```
|
||||
|
||||
## 🎮 Control Actions: Manual Management
|
||||
|
||||
**The Problem:** Sometimes you need to manually intervene—kill a stuck browser, force cleanup, or restart resources.
|
||||
|
||||
**My Solution:** Manual control actions via the API for operational troubleshooting.
|
||||
|
||||
### Force Cleanup
|
||||
|
||||
```python
|
||||
async def force_cleanup():
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.post("http://localhost:11235/monitor/actions/cleanup")
|
||||
result = response.json()
|
||||
|
||||
print(f"Cleanup completed:")
|
||||
print(f" Browsers cleaned: {result.get('cleaned_count', 0)}")
|
||||
print(f" Memory freed: {result.get('memory_freed_mb', 0):.1f} MB")
|
||||
```
|
||||
|
||||
### Kill Specific Browser
|
||||
|
||||
```python
|
||||
async def kill_stuck_browser(browser_id: str):
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.post(
|
||||
"http://localhost:11235/monitor/actions/kill_browser",
|
||||
json={"browser_id": browser_id}
|
||||
)
|
||||
|
||||
if response.status_code == 200:
|
||||
print(f"✅ Browser {browser_id} killed successfully")
|
||||
```
|
||||
|
||||
### Reset Statistics
|
||||
|
||||
```python
|
||||
async def reset_stats():
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.post("http://localhost:11235/monitor/stats/reset")
|
||||
print("📊 Statistics reset for fresh monitoring")
|
||||
```
|
||||
|
||||
## 📈 Production Integration Patterns
|
||||
|
||||
### Prometheus Integration
|
||||
|
||||
```python
|
||||
# Export metrics for Prometheus scraping
|
||||
async def export_prometheus_metrics():
|
||||
async with httpx.AsyncClient() as client:
|
||||
health = await client.get("http://localhost:11235/monitor/health")
|
||||
data = health.json()
|
||||
|
||||
# Export in Prometheus format
|
||||
metrics = f"""
|
||||
# HELP crawl4ai_memory_usage_percent Memory usage percentage
|
||||
# TYPE crawl4ai_memory_usage_percent gauge
|
||||
crawl4ai_memory_usage_percent {data['container']['memory_percent']}
|
||||
|
||||
# HELP crawl4ai_request_success_rate Request success rate
|
||||
# TYPE crawl4ai_request_success_rate gauge
|
||||
crawl4ai_request_success_rate {data['stats']['success_rate_percent']}
|
||||
|
||||
# HELP crawl4ai_browser_pool_count Total browsers in pool
|
||||
# TYPE crawl4ai_browser_pool_count gauge
|
||||
crawl4ai_browser_pool_count {data['pool']['permanent']['active'] + data['pool']['hot']['count'] + data['pool']['cold']['count']}
|
||||
"""
|
||||
return metrics
|
||||
```
|
||||
|
||||
### Alerting Example
|
||||
|
||||
```python
|
||||
async def check_alerts():
|
||||
async with httpx.AsyncClient() as client:
|
||||
health = await client.get("http://localhost:11235/monitor/health")
|
||||
data = health.json()
|
||||
|
||||
# Memory alert
|
||||
if data['container']['memory_percent'] > 80:
|
||||
print("🚨 ALERT: Memory usage above 80%")
|
||||
# Trigger cleanup
|
||||
await client.post("http://localhost:11235/monitor/actions/cleanup")
|
||||
|
||||
# Success rate alert
|
||||
if data['stats']['success_rate_percent'] < 90:
|
||||
print("🚨 ALERT: Success rate below 90%")
|
||||
# Check error logs
|
||||
errors = await client.get("http://localhost:11235/monitor/logs/errors")
|
||||
print(f"Recent errors: {len(errors.json())}")
|
||||
|
||||
# Latency alert
|
||||
if data['stats']['avg_latency_ms'] > 5000:
|
||||
print("🚨 ALERT: Average latency above 5s")
|
||||
```
|
||||
|
||||
### Key Metrics to Track
|
||||
|
||||
```python
|
||||
CRITICAL_METRICS = {
|
||||
"memory_usage": {
|
||||
"current": "container.memory_percent",
|
||||
"target": "<80%",
|
||||
"alert_threshold": ">80%",
|
||||
"action": "Force cleanup or scale"
|
||||
},
|
||||
"success_rate": {
|
||||
"current": "stats.success_rate_percent",
|
||||
"target": ">95%",
|
||||
"alert_threshold": "<90%",
|
||||
"action": "Check error logs"
|
||||
},
|
||||
"avg_latency": {
|
||||
"current": "stats.avg_latency_ms",
|
||||
"target": "<2000ms",
|
||||
"alert_threshold": ">5000ms",
|
||||
"action": "Investigate slow requests"
|
||||
},
|
||||
"browser_reuse_rate": {
|
||||
"current": "browsers.summary.reuse_rate_percent",
|
||||
"target": ">80%",
|
||||
"alert_threshold": "<60%",
|
||||
"action": "Check pool configuration"
|
||||
},
|
||||
"total_browsers": {
|
||||
"current": "browsers.summary.total_count",
|
||||
"target": "<15",
|
||||
"alert_threshold": ">20",
|
||||
"action": "Check for browser leaks"
|
||||
},
|
||||
"error_frequency": {
|
||||
"current": "len(errors)",
|
||||
"target": "<5/hour",
|
||||
"alert_threshold": ">10/hour",
|
||||
"action": "Review error patterns"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## 🐛 Critical Bug Fixes
|
||||
|
||||
This release includes significant bug fixes that improve stability and performance:
|
||||
|
||||
### Async LLM Extraction (#1590)
|
||||
|
||||
**The Problem:** LLM extraction was blocking async execution, causing URLs to be processed sequentially instead of in parallel (issue #1055).
|
||||
|
||||
**The Fix:** Resolved the blocking issue to enable true parallel processing for LLM extraction.
|
||||
|
||||
```python
|
||||
# Before v0.7.7: Sequential processing
|
||||
# After v0.7.7: True parallel processing
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
urls = ["url1", "url2", "url3", "url4"]
|
||||
|
||||
# Now processes truly in parallel with LLM extraction
|
||||
results = await crawler.arun_many(
|
||||
urls,
|
||||
config=CrawlerRunConfig(
|
||||
extraction_strategy=LLMExtractionStrategy(...)
|
||||
)
|
||||
)
|
||||
# 4x faster for parallel LLM extraction!
|
||||
```
|
||||
|
||||
**Expected Impact:** Major performance improvement for batch LLM extraction workflows.
|
||||
|
||||
### DFS Deep Crawling (#1607)
|
||||
|
||||
**The Problem:** DFS (Depth-First Search) deep crawl strategy had implementation issues.
|
||||
|
||||
**The Fix:** Enhanced DFSDeepCrawlStrategy with proper seen URL tracking and improved documentation.
|
||||
|
||||
### Browser & Crawler Config Documentation (#1609)
|
||||
|
||||
**The Problem:** Documentation didn't match the actual `async_configs.py` implementation.
|
||||
|
||||
**The Fix:** Updated all configuration documentation to accurately reflect the current implementation.
|
||||
|
||||
### Sitemap Seeder (#1598)
|
||||
|
||||
**The Problem:** Sitemap parsing and URL normalization issues in AsyncUrlSeeder (issue #1559).
|
||||
|
||||
**The Fix:** Added comprehensive tests and fixes for sitemap namespace parsing and URL normalization.
|
||||
|
||||
### Remove Overlay Elements (#1529)
|
||||
|
||||
**The Problem:** The `remove_overlay_elements` functionality wasn't working (issue #1396).
|
||||
|
||||
**The Fix:** Fixed by properly calling the injected JavaScript function.
|
||||
|
||||
### Viewport Configuration (#1495)
|
||||
|
||||
**The Problem:** Viewport configuration wasn't working in managed browsers (issue #1490).
|
||||
|
||||
**The Fix:** Added proper viewport size configuration support for browser launch.
|
||||
|
||||
### Managed Browser CDP Timing (#1528)
|
||||
|
||||
**The Problem:** CDP (Chrome DevTools Protocol) endpoint verification had timing issues causing connection failures (issue #1445).
|
||||
|
||||
**The Fix:** Added exponential backoff for CDP endpoint verification to handle timing variations.
|
||||
|
||||
### Security Updates
|
||||
|
||||
- **pyOpenSSL**: Updated from >=24.3.0 to >=25.3.0 to address security vulnerability
|
||||
- Added verification tests for the security update
|
||||
|
||||
### Docker Fixes
|
||||
|
||||
- **Port Standardization**: Fixed inconsistent port usage (11234 vs 11235) - now standardized to 11235
|
||||
- **LLM Environment**: Fixed LLM API key handling for multi-provider support (PR #1537)
|
||||
- **Error Handling**: Improved Docker API error messages with comprehensive status codes
|
||||
- **Serialization**: Fixed `fit_html` property serialization in `/crawl` and `/crawl/stream` endpoints
|
||||
|
||||
### Other Important Fixes
|
||||
|
||||
- **arun_many Returns**: Fixed function to always return a list, even on exception (PR #1530)
|
||||
- **Webhook Serialization**: Properly serialize Pydantic HttpUrl in webhook config
|
||||
- **LLMConfig Documentation**: Fixed casing and variable name consistency (issue #1551)
|
||||
- **Python Version**: Dropped Python 3.9 support, now requires Python >=3.10
|
||||
|
||||
## 📊 Expected Real-World Impact
|
||||
|
||||
### For DevOps & Infrastructure Teams
|
||||
- **Full Visibility**: Know exactly what's happening inside your crawling infrastructure
|
||||
- **Proactive Monitoring**: Catch issues before they become problems
|
||||
- **Resource Optimization**: Identify memory leaks and performance bottlenecks
|
||||
- **Operational Control**: Manual intervention when automated systems need help
|
||||
|
||||
### For Production Deployments
|
||||
- **Enterprise Observability**: Prometheus, Grafana, and alerting integration
|
||||
- **Debugging**: Real-time logs and error tracking
|
||||
- **Capacity Planning**: Historical metrics for scaling decisions
|
||||
- **SLA Monitoring**: Track success rates and latency against targets
|
||||
|
||||
### For Development Teams
|
||||
- **Local Monitoring**: Understand crawler behavior during development
|
||||
- **Performance Testing**: Measure impact of configuration changes
|
||||
- **Troubleshooting**: Quickly identify and fix issues
|
||||
- **Learning**: See exactly how the browser pool works
|
||||
|
||||
## 🔄 Breaking Changes
|
||||
|
||||
**None!** This release is fully backward compatible.
|
||||
|
||||
- All existing Docker configurations continue to work
|
||||
- No API changes to existing endpoints
|
||||
- Monitoring is additive functionality
|
||||
- No migration required
|
||||
|
||||
## 🚀 Upgrade Instructions
|
||||
|
||||
### Docker
|
||||
|
||||
```bash
|
||||
# Pull the latest version
|
||||
docker pull unclecode/crawl4ai:0.7.7
|
||||
|
||||
# Or use the latest tag
|
||||
docker pull unclecode/crawl4ai:latest
|
||||
|
||||
# Run with monitoring enabled (default)
|
||||
docker run -d \
|
||||
-p 11235:11235 \
|
||||
--shm-size=1g \
|
||||
--name crawl4ai \
|
||||
unclecode/crawl4ai:0.7.7
|
||||
|
||||
# Access the monitoring dashboard
|
||||
open http://localhost:11235/dashboard
|
||||
```
|
||||
|
||||
### Python Package
|
||||
|
||||
```bash
|
||||
# Upgrade to latest version
|
||||
pip install --upgrade crawl4ai
|
||||
|
||||
# Or install specific version
|
||||
pip install crawl4ai==0.7.7
|
||||
```
|
||||
|
||||
## 🎬 Try the Demo
|
||||
|
||||
Run the comprehensive demo that showcases all monitoring features:
|
||||
|
||||
```bash
|
||||
python docs/releases_review/demo_v0.7.7.py
|
||||
```
|
||||
|
||||
**The demo includes:**
|
||||
1. System health overview with live metrics
|
||||
2. Request tracking with active/completed monitoring
|
||||
3. Browser pool management (permanent/hot/cold)
|
||||
4. Complete Monitor API endpoint examples
|
||||
5. WebSocket streaming demonstration
|
||||
6. Control actions (cleanup, kill, restart)
|
||||
7. Production metrics and alerting patterns
|
||||
8. Self-hosting value proposition
|
||||
|
||||
## 📚 Documentation
|
||||
|
||||
### New Documentation
|
||||
- **[Self-Hosting Guide](https://docs.crawl4ai.com/core/self-hosting/)** - Complete self-hosting documentation with monitoring
|
||||
- **Demo Script**: `docs/releases_review/demo_v0.7.7.py` - Working examples
|
||||
|
||||
### Updated Documentation
|
||||
- **Docker Deployment** → **Self-Hosting** (renamed for better positioning)
|
||||
- Added comprehensive monitoring sections
|
||||
- Production integration patterns
|
||||
- WebSocket streaming examples
|
||||
|
||||
## 💡 Pro Tips
|
||||
|
||||
1. **Start with the dashboard** - Visit `/dashboard` to get familiar with the monitoring system
|
||||
2. **Track the 6 key metrics** - Memory, success rate, latency, reuse rate, browser count, errors
|
||||
3. **Set up alerting early** - Use the Monitor API to build alerts before issues occur
|
||||
4. **Monitor browser pool efficiency** - Aim for >80% reuse rate for optimal performance
|
||||
5. **Use WebSocket for custom dashboards** - Build tailored monitoring UIs for your team
|
||||
6. **Leverage Prometheus integration** - Export metrics for long-term storage and analysis
|
||||
7. **Check janitor logs** - Understand automatic cleanup patterns
|
||||
8. **Use control actions judiciously** - Manual interventions are for exceptional cases
|
||||
|
||||
## 🙏 Acknowledgments
|
||||
|
||||
Thank you to our community for the feedback, bug reports, and feature requests that shaped this release. Special thanks to everyone who contributed to the issues that were fixed in this version.
|
||||
|
||||
The monitoring system was built based on real user needs for production deployments, and your input made it comprehensive and practical.
|
||||
|
||||
## 📞 Support & Resources
|
||||
|
||||
- **📖 Documentation**: [docs.crawl4ai.com](https://docs.crawl4ai.com)
|
||||
- **🐙 GitHub**: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
|
||||
- **💬 Discord**: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
|
||||
- **🐦 Twitter**: [@unclecode](https://x.com/unclecode)
|
||||
- **📊 Dashboard**: `http://localhost:11235/dashboard` (when running)
|
||||
|
||||
---
|
||||
|
||||
**Crawl4AI v0.7.7 delivers complete self-hosting with enterprise-grade monitoring. You now have full visibility and control over your web crawling infrastructure. The monitoring dashboard, comprehensive API, and WebSocket streaming give you everything needed for production deployments. Try the self-hosting platform—it's a game changer for operational excellence!**
|
||||
|
||||
**Happy crawling with full visibility!** 🕷️📊
|
||||
|
||||
*- unclecode*
|
||||
@@ -0,0 +1,327 @@
|
||||
# Crawl4AI v0.7.8: Stability & Bug Fix Release
|
||||
|
||||
*December 2025*
|
||||
|
||||
---
|
||||
|
||||
I'm releasing Crawl4AI v0.7.8—a focused stability release that addresses 11 bugs reported by the community. While there are no new features in this release, these fixes resolve important issues affecting Docker deployments, LLM extraction, URL handling, and dependency compatibility.
|
||||
|
||||
## What's Fixed at a Glance
|
||||
|
||||
- **Docker API**: Fixed ContentRelevanceFilter deserialization, ProxyConfig serialization, and cache folder permissions
|
||||
- **LLM Extraction**: Configurable rate limiter backoff, HTML input format support, and proper URL handling for raw HTML
|
||||
- **URL Handling**: Correct relative URL resolution after JavaScript redirects
|
||||
- **Dependencies**: Replaced deprecated PyPDF2 with pypdf, Pydantic v2 ConfigDict compatibility
|
||||
- **AdaptiveCrawler**: Fixed query expansion to actually use LLM instead of hardcoded mock data
|
||||
|
||||
## Bug Fixes
|
||||
|
||||
### Docker & API Fixes
|
||||
|
||||
#### ContentRelevanceFilter Deserialization (#1642)
|
||||
|
||||
**The Problem:** When sending deep crawl requests to the Docker API with `ContentRelevanceFilter`, the server failed to deserialize the filter, causing requests to fail.
|
||||
|
||||
**The Fix:** I added `ContentRelevanceFilter` to the public exports and enhanced the deserialization logic with dynamic imports.
|
||||
|
||||
```python
|
||||
# This now works correctly in Docker API
|
||||
import httpx
|
||||
|
||||
request = {
|
||||
"urls": ["https://docs.example.com"],
|
||||
"crawler_config": {
|
||||
"deep_crawl_strategy": {
|
||||
"type": "BFSDeepCrawlStrategy",
|
||||
"max_depth": 2,
|
||||
"filter_chain": [
|
||||
{
|
||||
"type": "ContentRelevanceFilter",
|
||||
"query": "API documentation",
|
||||
"threshold": 0.3
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
async with httpx.AsyncClient() as client:
|
||||
response = await client.post("http://localhost:11235/crawl", json=request)
|
||||
# Previously failed, now works!
|
||||
```
|
||||
|
||||
#### ProxyConfig JSON Serialization (#1629)
|
||||
|
||||
**The Problem:** `BrowserConfig.to_dict()` failed when `proxy_config` was set because `ProxyConfig` wasn't being serialized to a dictionary.
|
||||
|
||||
**The Fix:** `ProxyConfig.to_dict()` is now called during serialization.
|
||||
|
||||
```python
|
||||
from crawl4ai import BrowserConfig
|
||||
from crawl4ai.async_configs import ProxyConfig
|
||||
|
||||
proxy = ProxyConfig(
|
||||
server="http://proxy.example.com:8080",
|
||||
username="user",
|
||||
password="pass"
|
||||
)
|
||||
|
||||
config = BrowserConfig(headless=True, proxy_config=proxy)
|
||||
|
||||
# Previously raised TypeError, now works
|
||||
config_dict = config.to_dict()
|
||||
json.dumps(config_dict) # Valid JSON
|
||||
```
|
||||
|
||||
#### Docker Cache Folder Permissions (#1638)
|
||||
|
||||
**The Problem:** The `.cache` folder in the Docker image had incorrect permissions, causing crawling to fail when caching was enabled.
|
||||
|
||||
**The Fix:** Corrected ownership and permissions during image build.
|
||||
|
||||
```bash
|
||||
# Cache now works correctly in Docker
|
||||
docker run -d -p 11235:11235 \
|
||||
--shm-size=1g \
|
||||
-v ./my-cache:/app/.cache \
|
||||
unclecode/crawl4ai:0.7.8
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### LLM & Extraction Fixes
|
||||
|
||||
#### Configurable Rate Limiter Backoff (#1269)
|
||||
|
||||
**The Problem:** The LLM rate limiting backoff parameters were hardcoded, making it impossible to adjust retry behavior for different API rate limits.
|
||||
|
||||
**The Fix:** `LLMConfig` now accepts three new parameters for complete control over retry behavior.
|
||||
|
||||
```python
|
||||
from crawl4ai import LLMConfig
|
||||
|
||||
# Default behavior (unchanged)
|
||||
default_config = LLMConfig(provider="openai/gpt-4o-mini")
|
||||
# backoff_base_delay=2, backoff_max_attempts=3, backoff_exponential_factor=2
|
||||
|
||||
# Custom configuration for APIs with strict rate limits
|
||||
custom_config = LLMConfig(
|
||||
provider="openai/gpt-4o-mini",
|
||||
backoff_base_delay=5, # Wait 5 seconds on first retry
|
||||
backoff_max_attempts=5, # Try up to 5 times
|
||||
backoff_exponential_factor=3 # Multiply delay by 3 each attempt
|
||||
)
|
||||
|
||||
# Retry sequence: 5s -> 15s -> 45s -> 135s -> 405s
|
||||
```
|
||||
|
||||
#### LLM Strategy HTML Input Support (#1178)
|
||||
|
||||
**The Problem:** `LLMExtractionStrategy` always sent markdown to the LLM, but some extraction tasks work better with HTML structure preserved.
|
||||
|
||||
**The Fix:** Added `input_format` parameter supporting `"markdown"`, `"html"`, `"fit_markdown"`, `"cleaned_html"`, and `"fit_html"`.
|
||||
|
||||
```python
|
||||
from crawl4ai import LLMExtractionStrategy, LLMConfig
|
||||
|
||||
# Default: markdown input (unchanged)
|
||||
markdown_strategy = LLMExtractionStrategy(
|
||||
llm_config=LLMConfig(provider="openai/gpt-4o-mini"),
|
||||
instruction="Extract product information"
|
||||
)
|
||||
|
||||
# NEW: HTML input - preserves table/list structure
|
||||
html_strategy = LLMExtractionStrategy(
|
||||
llm_config=LLMConfig(provider="openai/gpt-4o-mini"),
|
||||
instruction="Extract the data table preserving structure",
|
||||
input_format="html"
|
||||
)
|
||||
|
||||
# NEW: Filtered markdown - only relevant content
|
||||
fit_strategy = LLMExtractionStrategy(
|
||||
llm_config=LLMConfig(provider="openai/gpt-4o-mini"),
|
||||
instruction="Summarize the main content",
|
||||
input_format="fit_markdown"
|
||||
)
|
||||
```
|
||||
|
||||
#### Raw HTML URL Variable (#1116)
|
||||
|
||||
**The Problem:** When using `url="raw:<html>..."`, the entire HTML content was being passed to extraction strategies as the URL parameter, polluting LLM prompts.
|
||||
|
||||
**The Fix:** The URL is now correctly set to `"Raw HTML"` for raw HTML inputs.
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
|
||||
|
||||
html = "<html><body><h1>Test</h1></body></html>"
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(
|
||||
url=f"raw:{html}",
|
||||
config=CrawlerRunConfig(extraction_strategy=my_strategy)
|
||||
)
|
||||
# extraction_strategy receives url="Raw HTML" instead of the HTML blob
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### URL Handling Fix
|
||||
|
||||
#### Relative URLs After Redirects (#1268)
|
||||
|
||||
**The Problem:** When JavaScript caused a page redirect, relative links were resolved against the original URL instead of the final URL.
|
||||
|
||||
**The Fix:** `redirected_url` now captures the actual page URL after all JavaScript execution completes.
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
# Page at /old-page redirects via JS to /new-page
|
||||
result = await crawler.arun(url="https://example.com/old-page")
|
||||
|
||||
# BEFORE: redirected_url = "https://example.com/old-page"
|
||||
# AFTER: redirected_url = "https://example.com/new-page"
|
||||
|
||||
# Links are now correctly resolved against the final URL
|
||||
for link in result.links['internal']:
|
||||
print(link['href']) # Relative links resolved correctly
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Dependency & Compatibility Fixes
|
||||
|
||||
#### PyPDF2 Replaced with pypdf (#1412)
|
||||
|
||||
**The Problem:** PyPDF2 was deprecated in 2022 and is no longer maintained.
|
||||
|
||||
**The Fix:** Replaced with the actively maintained `pypdf` library.
|
||||
|
||||
```python
|
||||
# Installation (unchanged)
|
||||
pip install crawl4ai[pdf]
|
||||
|
||||
# The PDF processor now uses pypdf internally
|
||||
# No code changes required - API remains the same
|
||||
```
|
||||
|
||||
#### Pydantic v2 ConfigDict Compatibility (#678)
|
||||
|
||||
**The Problem:** Using the deprecated `class Config` syntax caused deprecation warnings with Pydantic v2.
|
||||
|
||||
**The Fix:** Migrated to `model_config = ConfigDict(...)` syntax.
|
||||
|
||||
```python
|
||||
# No more deprecation warnings when importing crawl4ai models
|
||||
from crawl4ai.models import CrawlResult
|
||||
from crawl4ai import CrawlerRunConfig, BrowserConfig
|
||||
|
||||
# All models are now Pydantic v2 compatible
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### AdaptiveCrawler Fix
|
||||
|
||||
#### Query Expansion Using LLM (#1621)
|
||||
|
||||
**The Problem:** The `EmbeddingStrategy` in AdaptiveCrawler had commented-out LLM code and was using hardcoded mock query variations instead.
|
||||
|
||||
**The Fix:** Uncommented and activated the LLM call for actual query expansion.
|
||||
|
||||
```python
|
||||
# AdaptiveCrawler query expansion now actually uses the LLM
|
||||
# Instead of hardcoded variations like:
|
||||
# variations = {'queries': ['what are the best vegetables...']}
|
||||
|
||||
# The LLM generates relevant query variations based on your actual query
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Code Formatting Fix
|
||||
|
||||
#### Import Statement Formatting (#1181)
|
||||
|
||||
**The Problem:** When extracting code from web pages, import statements were sometimes concatenated without proper line separation.
|
||||
|
||||
**The Fix:** Import statements now maintain proper newline separation.
|
||||
|
||||
```python
|
||||
# BEFORE: "import osimport sysfrom pathlib import Path"
|
||||
# AFTER:
|
||||
# import os
|
||||
# import sys
|
||||
# from pathlib import Path
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Breaking Changes
|
||||
|
||||
**None!** This release is fully backward compatible.
|
||||
|
||||
- All existing code continues to work without modification
|
||||
- New parameters have sensible defaults matching previous behavior
|
||||
- No API changes to existing functionality
|
||||
|
||||
---
|
||||
|
||||
## Upgrade Instructions
|
||||
|
||||
### Python Package
|
||||
|
||||
```bash
|
||||
pip install --upgrade crawl4ai
|
||||
# or
|
||||
pip install crawl4ai==0.7.8
|
||||
```
|
||||
|
||||
### Docker
|
||||
|
||||
```bash
|
||||
# Pull the latest version
|
||||
docker pull unclecode/crawl4ai:0.7.8
|
||||
|
||||
# Run
|
||||
docker run -d -p 11235:11235 --shm-size=1g unclecode/crawl4ai:0.7.8
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
Run the verification tests to confirm all fixes are working:
|
||||
|
||||
```bash
|
||||
python docs/releases_review/demo_v0.7.8.py
|
||||
```
|
||||
|
||||
This runs actual tests that verify each bug fix is properly implemented.
|
||||
|
||||
---
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
Thank you to everyone who reported these issues and provided detailed reproduction steps. Your bug reports make Crawl4AI better for everyone.
|
||||
|
||||
Issues fixed: #1642, #1638, #1629, #1621, #1412, #1269, #1268, #1181, #1178, #1116, #678
|
||||
|
||||
---
|
||||
|
||||
## Support & Resources
|
||||
|
||||
- **Documentation**: [docs.crawl4ai.com](https://docs.crawl4ai.com)
|
||||
- **GitHub**: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
|
||||
- **Discord**: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
|
||||
- **Twitter**: [@unclecode](https://x.com/unclecode)
|
||||
|
||||
---
|
||||
|
||||
**This stability release ensures Crawl4AI works reliably across Docker deployments, LLM extraction workflows, and various edge cases. Thank you for your continued support and feedback!**
|
||||
|
||||
**Happy crawling!**
|
||||
|
||||
*- unclecode*
|
||||
@@ -0,0 +1,243 @@
|
||||
# Crawl4AI v0.8.0 Release Notes
|
||||
|
||||
**Release Date**: January 2026
|
||||
**Previous Version**: v0.7.6
|
||||
**Status**: Release Candidate
|
||||
|
||||
---
|
||||
|
||||
## Highlights
|
||||
|
||||
- **Critical Security Fixes** for Docker API deployment
|
||||
- **11 New Features** including crash recovery, prefetch mode, and proxy improvements
|
||||
- **Breaking Changes** - see migration guide below
|
||||
|
||||
---
|
||||
|
||||
## Breaking Changes
|
||||
|
||||
### 1. Docker API: Hooks Disabled by Default
|
||||
|
||||
**What changed**: Hooks are now disabled by default on the Docker API.
|
||||
|
||||
**Why**: Security fix for Remote Code Execution (RCE) vulnerability.
|
||||
|
||||
**Who is affected**: Users of the Docker API who use the `hooks` parameter in `/crawl` requests.
|
||||
|
||||
**Migration**:
|
||||
```bash
|
||||
# To re-enable hooks (only if you trust all API users):
|
||||
export CRAWL4AI_HOOKS_ENABLED=true
|
||||
```
|
||||
|
||||
### 2. Docker API: file:// URLs Blocked
|
||||
|
||||
**What changed**: The endpoints `/execute_js`, `/screenshot`, `/pdf`, and `/html` now reject `file://` URLs.
|
||||
|
||||
**Why**: Security fix for Local File Inclusion (LFI) vulnerability.
|
||||
|
||||
**Who is affected**: Users who were reading local files via the Docker API.
|
||||
|
||||
**Migration**: Use the Python library directly for local file processing:
|
||||
```python
|
||||
# Instead of API call with file:// URL, use library:
|
||||
from crawl4ai import AsyncWebCrawler
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun(url="file:///path/to/file.html")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Security Fixes
|
||||
|
||||
### Critical: Remote Code Execution via Hooks (CVE Pending)
|
||||
|
||||
**Severity**: CRITICAL (CVSS 10.0)
|
||||
**Affected**: Docker API deployment (all versions before v0.8.0)
|
||||
**Vector**: `POST /crawl` with malicious `hooks` parameter
|
||||
|
||||
**Details**: The `__import__` builtin was available in hook code, allowing attackers to import `os`, `subprocess`, etc. and execute arbitrary commands.
|
||||
|
||||
**Fix**:
|
||||
1. Removed `__import__` from allowed builtins
|
||||
2. Hooks disabled by default (`CRAWL4AI_HOOKS_ENABLED=false`)
|
||||
|
||||
### High: Local File Inclusion via file:// URLs (CVE Pending)
|
||||
|
||||
**Severity**: HIGH (CVSS 8.6)
|
||||
**Affected**: Docker API deployment (all versions before v0.8.0)
|
||||
**Vector**: `POST /execute_js` (and other endpoints) with `file:///etc/passwd`
|
||||
|
||||
**Details**: API endpoints accepted `file://` URLs, allowing attackers to read arbitrary files from the server.
|
||||
|
||||
**Fix**: URL scheme validation now only allows `http://`, `https://`, and `raw:` URLs.
|
||||
|
||||
### Credits
|
||||
|
||||
Discovered by **Neo by ProjectDiscovery** ([projectdiscovery.io](https://projectdiscovery.io)) - December 2025
|
||||
|
||||
---
|
||||
|
||||
## New Features
|
||||
|
||||
### 1. init_scripts Support for BrowserConfig
|
||||
|
||||
Pre-page-load JavaScript injection for stealth evasions.
|
||||
|
||||
```python
|
||||
config = BrowserConfig(
|
||||
init_scripts=[
|
||||
"Object.defineProperty(navigator, 'webdriver', {get: () => false})"
|
||||
]
|
||||
)
|
||||
```
|
||||
|
||||
### 2. CDP Connection Improvements
|
||||
|
||||
- WebSocket URL support (`ws://`, `wss://`)
|
||||
- Proper cleanup with `cdp_cleanup_on_close=True`
|
||||
- Browser reuse across multiple connections
|
||||
|
||||
### 3. Crash Recovery for Deep Crawl Strategies
|
||||
|
||||
All deep crawl strategies (BFS, DFS, Best-First) now support crash recovery:
|
||||
|
||||
```python
|
||||
from crawl4ai.deep_crawling import BFSDeepCrawlStrategy
|
||||
|
||||
strategy = BFSDeepCrawlStrategy(
|
||||
max_depth=3,
|
||||
resume_state=saved_state, # Resume from checkpoint
|
||||
on_state_change=save_callback # Persist state in real-time
|
||||
)
|
||||
```
|
||||
|
||||
### 4. PDF and MHTML for raw:/file:// URLs
|
||||
|
||||
Generate PDFs and MHTML from cached HTML content.
|
||||
|
||||
### 5. Screenshots for raw:/file:// URLs
|
||||
|
||||
Render cached HTML and capture screenshots.
|
||||
|
||||
### 6. base_url Parameter for CrawlerRunConfig
|
||||
|
||||
Proper URL resolution for raw: HTML processing:
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(base_url='https://example.com')
|
||||
result = await crawler.arun(url='raw:{html}', config=config)
|
||||
```
|
||||
|
||||
### 7. Prefetch Mode for Two-Phase Deep Crawling
|
||||
|
||||
Fast link extraction without full page processing:
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(prefetch=True)
|
||||
```
|
||||
|
||||
### 8. Proxy Rotation and Configuration
|
||||
|
||||
Enhanced proxy rotation with sticky sessions support.
|
||||
|
||||
### 9. Proxy Support for HTTP Strategy
|
||||
|
||||
Non-browser crawler now supports proxies.
|
||||
|
||||
### 10. Browser Pipeline for raw:/file:// URLs
|
||||
|
||||
New `process_in_browser` parameter for browser operations on local content:
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(
|
||||
process_in_browser=True, # Force browser processing
|
||||
screenshot=True
|
||||
)
|
||||
result = await crawler.arun(url='raw:<html>...</html>', config=config)
|
||||
```
|
||||
|
||||
### 11. Smart TTL Cache for Sitemap URL Seeder
|
||||
|
||||
Intelligent cache invalidation for sitemaps:
|
||||
|
||||
```python
|
||||
config = SeedingConfig(
|
||||
cache_ttl_hours=24,
|
||||
validate_sitemap_lastmod=True
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Bug Fixes
|
||||
|
||||
### raw: URL Parsing Truncates at # Character
|
||||
|
||||
**Problem**: CSS color codes like `#eee` were being truncated.
|
||||
|
||||
**Before**: `raw:body{background:#eee}` → `body{background:`
|
||||
**After**: `raw:body{background:#eee}` → `body{background:#eee}`
|
||||
|
||||
### Caching System Improvements
|
||||
|
||||
Various fixes to cache validation and persistence.
|
||||
|
||||
---
|
||||
|
||||
## Documentation Updates
|
||||
|
||||
- Multi-sample schema generation documentation
|
||||
- URL seeder smart TTL cache parameters
|
||||
- Security documentation (SECURITY.md)
|
||||
|
||||
---
|
||||
|
||||
## Upgrade Guide
|
||||
|
||||
### From v0.7.x to v0.8.0
|
||||
|
||||
1. **Update the package**:
|
||||
```bash
|
||||
pip install --upgrade crawl4ai
|
||||
```
|
||||
|
||||
2. **Docker API users**:
|
||||
- Hooks are now disabled by default
|
||||
- If you need hooks: `export CRAWL4AI_HOOKS_ENABLED=true`
|
||||
- `file://` URLs no longer work on API (use library directly)
|
||||
|
||||
3. **Review security settings**:
|
||||
```yaml
|
||||
# config.yml - recommended for production
|
||||
security:
|
||||
enabled: true
|
||||
jwt_enabled: true
|
||||
```
|
||||
|
||||
4. **Test your integration** before deploying to production
|
||||
|
||||
### Breaking Change Checklist
|
||||
|
||||
- [ ] Check if you use `hooks` parameter in API calls
|
||||
- [ ] Check if you use `file://` URLs via the API
|
||||
- [ ] Update environment variables if needed
|
||||
- [ ] Review security configuration
|
||||
|
||||
---
|
||||
|
||||
## Full Changelog
|
||||
|
||||
See [CHANGELOG.md](../CHANGELOG.md) for complete version history.
|
||||
|
||||
---
|
||||
|
||||
## Contributors
|
||||
|
||||
Thanks to all contributors who made this release possible.
|
||||
|
||||
Special thanks to **Neo by ProjectDiscovery** for responsible security disclosure.
|
||||
|
||||
---
|
||||
|
||||
*For questions or issues, please open a [GitHub Issue](https://github.com/unclecode/crawl4ai/issues).*
|
||||
@@ -0,0 +1,384 @@
|
||||
# Crawl4AI v0.8.5: Anti-Bot, Shadow DOM & 60+ Bug Fixes
|
||||
|
||||
*March 2026 • 10 min read*
|
||||
|
||||
---
|
||||
|
||||
I'm releasing Crawl4AI v0.8.5—our biggest release since v0.8.0. This update brings automatic anti-bot detection with proxy escalation, Shadow DOM flattening, deep crawl cancellation, and over 60 bug fixes from both our team and the community. If you're running crawls at scale or dealing with protected sites, this one's for you.
|
||||
|
||||
## What's New at a Glance
|
||||
|
||||
- **Anti-Bot Detection & Proxy Escalation**: 3-tier detection with automatic retry, proxy chain, and fallback
|
||||
- **Shadow DOM Flattening**: Extract content hidden inside shadow DOM components
|
||||
- **Deep Crawl Cancellation**: Stop long crawls gracefully with `cancel()` or `should_cancel` callback
|
||||
- **Config Defaults API**: Set once, apply everywhere with `set_defaults()` / `get_defaults()` / `reset_defaults()`
|
||||
- **Source/Sibling Selector**: Extract data spanning sibling elements in JSON extraction schemas
|
||||
- **Consent Popup Removal**: Auto-dismiss cookie banners from 40+ CMP platforms
|
||||
- **Resource Filtering**: Block ads and CSS at the network level with `avoid_ads` / `avoid_css`
|
||||
- **Browser Recycling**: Memory-saving mode and automatic browser restart for long sessions
|
||||
- **GFM Table Compliance**: Proper `| col1 | col2 |` pipe delimiters in markdown output
|
||||
- **60+ Bug Fixes**: Security patches, browser stability, extraction accuracy, and more
|
||||
|
||||
---
|
||||
|
||||
## New Features
|
||||
|
||||
### 1. Anti-Bot Detection, Retry & Fallback
|
||||
|
||||
This is the headline feature. Crawl4AI now automatically detects when a page is blocked by anti-bot protection and takes action—retrying with different proxies or falling back to an alternative fetch method.
|
||||
|
||||
The detection uses three tiers:
|
||||
- **Tier 1**: Known vendor patterns (Cloudflare, Akamai, DataDome, PerimeterX, etc.)
|
||||
- **Tier 2**: Generic block indicators on small pages
|
||||
- **Tier 3**: Structural integrity checks (empty shells, script-heavy pages with no content)
|
||||
|
||||
```python
|
||||
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig
|
||||
from crawl4ai.async_configs import ProxyConfig
|
||||
|
||||
config = CrawlerRunConfig(
|
||||
# Try direct first, then proxy on bot detection
|
||||
proxy_config=[
|
||||
ProxyConfig.DIRECT,
|
||||
ProxyConfig(server="http://my-proxy:8080"),
|
||||
],
|
||||
max_retries=2,
|
||||
# Optional: fallback when all proxies fail
|
||||
fallback_fetch_function=my_web_unlocker_function,
|
||||
)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun("https://protected-site.com", config=config)
|
||||
|
||||
# Check what happened
|
||||
stats = result.crawl_stats
|
||||
print(f"Resolved by: {stats['resolved_by']}") # "direct", "proxy", or "fallback_fetch"
|
||||
print(f"Proxies tried: {len(stats['proxies_used'])}")
|
||||
```
|
||||
|
||||
The system errs on the side of caution—false positives are cheap (the fallback rescues them), but false negatives mean garbage results. After 5 iterations of real-world testing, it handles everything from Cloudflare challenges to Reddit's 180KB SPA block pages.
|
||||
|
||||
### 2. Shadow DOM Flattening
|
||||
|
||||
Web components with shadow DOM hide their content from regular DOM traversal. The new `flatten_shadow_dom` option serializes shadow DOM content into the light DOM before extraction.
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(flatten_shadow_dom=True)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
result = await crawler.arun("https://some-web-component-site.com", config=config)
|
||||
# Shadow DOM content is now visible in result.html, cleaned_html, and markdown
|
||||
```
|
||||
|
||||
The implementation patches `attachShadow` to force-open closed shadow roots, recursively resolves `<slot>` projections, and strips only shadow-scoped `<style>` tags. It also reorders the JS execution pipeline—`js_code` now runs after `wait_for` + `delay_before_return_html` so your scripts operate on the fully-hydrated page. If you need JS to run before waiting, use the new `js_code_before_wait` parameter.
|
||||
|
||||
### 3. Deep Crawl Cancellation
|
||||
|
||||
All deep crawl strategies (BFS, DFS, BestFirst) now support graceful cancellation:
|
||||
|
||||
```python
|
||||
from crawl4ai.deep_crawling import DFSDeepCrawlStrategy
|
||||
|
||||
pages_found = 0
|
||||
|
||||
def should_stop():
|
||||
return pages_found >= 50 # Stop after finding enough pages
|
||||
|
||||
async def on_state(state):
|
||||
nonlocal pages_found
|
||||
pages_found = state["pages_crawled"]
|
||||
|
||||
strategy = DFSDeepCrawlStrategy(
|
||||
max_depth=3,
|
||||
max_pages=1000,
|
||||
should_cancel=should_stop, # Sync or async callback
|
||||
on_state_change=on_state,
|
||||
)
|
||||
|
||||
config = CrawlerRunConfig(deep_crawl_strategy=strategy)
|
||||
|
||||
async with AsyncWebCrawler() as crawler:
|
||||
results = await crawler.arun("https://example.com", config=config)
|
||||
print(f"Cancelled: {strategy.cancelled}")
|
||||
```
|
||||
|
||||
You can also call `strategy.cancel()` directly from another thread or coroutine.
|
||||
|
||||
### 4. Config Defaults API
|
||||
|
||||
Tired of repeating the same parameters? Set defaults once and they apply to every new instance:
|
||||
|
||||
```python
|
||||
from crawl4ai import BrowserConfig, CrawlerRunConfig
|
||||
|
||||
# Set organization-wide defaults
|
||||
BrowserConfig.set_defaults(headless=True, text_mode=True)
|
||||
CrawlerRunConfig.set_defaults(verbose=False, remove_consent_popups=True)
|
||||
|
||||
# All new instances inherit defaults
|
||||
bc = BrowserConfig() # headless=True, text_mode=True
|
||||
rc = CrawlerRunConfig() # verbose=False, remove_consent_popups=True
|
||||
|
||||
# Explicit params always override
|
||||
bc2 = BrowserConfig(text_mode=False) # text_mode=False, headless still True
|
||||
|
||||
# Inspect and reset
|
||||
print(BrowserConfig.get_defaults()) # {"headless": True, "text_mode": True}
|
||||
BrowserConfig.reset_defaults() # Back to normal
|
||||
```
|
||||
|
||||
### 5. Source/Sibling Selector in JSON Extraction
|
||||
|
||||
Many sites split a single item's data across sibling elements (think Hacker News, where title and score are in separate `<tr>` rows). The new `"source"` field navigates to a sibling before extracting:
|
||||
|
||||
```python
|
||||
from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
|
||||
|
||||
schema = {
|
||||
"name": "HackerNewsItems",
|
||||
"baseSelector": "tr.athing",
|
||||
"fields": [
|
||||
{"name": "title", "selector": ".titleline > a", "type": "text"},
|
||||
{"name": "link", "selector": ".titleline > a", "type": "attribute", "attribute": "href"},
|
||||
# Navigate to the NEXT sibling <tr> to get the score
|
||||
{"name": "score", "selector": ".score", "type": "text", "source": "+ tr"},
|
||||
{"name": "author", "selector": ".hnuser", "type": "text", "source": "+ tr"},
|
||||
]
|
||||
}
|
||||
|
||||
strategy = JsonCssExtractionStrategy(schema=schema)
|
||||
```
|
||||
|
||||
Works in both `JsonCssExtractionStrategy` and `JsonXPathExtractionStrategy`. Falls back gracefully when siblings don't exist.
|
||||
|
||||
### 6. Consent Popup Removal
|
||||
|
||||
A single flag auto-dismisses cookie consent banners from 40+ CMP platforms:
|
||||
|
||||
```python
|
||||
config = CrawlerRunConfig(remove_consent_popups=True)
|
||||
```
|
||||
|
||||
Covers OneTrust, Cookiebot, Didomi, Quantcast, Sourcepoint, Google FundingChoices, TrustArc, ConsentManager, Osano, Iubenda, Complianz, LiveRamp, CookieYes, Klaro, Termly, and many more.
|
||||
|
||||
### 7. Resource Filtering: avoid_ads / avoid_css
|
||||
|
||||
Block ad trackers and CSS resources at the network level for faster, leaner crawls:
|
||||
|
||||
```python
|
||||
config = BrowserConfig(
|
||||
avoid_ads=True, # Blocks doubleclick, google-analytics, etc.
|
||||
avoid_css=True, # Blocks .css, .less, .scss resources
|
||||
)
|
||||
```
|
||||
|
||||
### 8. Browser Recycling & Memory-Saving Mode
|
||||
|
||||
For long-running crawl sessions:
|
||||
|
||||
```python
|
||||
config = BrowserConfig(
|
||||
memory_saving_mode=True, # Aggressive cache/V8 heap flags
|
||||
max_pages_before_recycle=100, # Auto-restart browser after N pages
|
||||
)
|
||||
```
|
||||
|
||||
This prevents memory leaks during sustained crawling. The recycling uses a version-based approach that's safe under concurrent load—we fixed three separate deadlock bugs to get this right.
|
||||
|
||||
### 9. GFM Table Compliance
|
||||
|
||||
Tables in markdown output now have proper GitHub-Flavored Markdown pipe delimiters:
|
||||
|
||||
**Before (v0.8.0)**:
|
||||
```
|
||||
Name | Age | City
|
||||
---|---|---
|
||||
Alice | 30 | NYC
|
||||
```
|
||||
|
||||
**After (v0.8.5)**:
|
||||
```
|
||||
| Name | Age | City |
|
||||
| --- | --- | --- |
|
||||
| Alice | 30 | NYC |
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Minor Features
|
||||
|
||||
- **`query_llm_config`**: Separate LLM config for adaptive crawler query expansion (#1682)
|
||||
- **`force_viewport_screenshot`**: Screenshot only the viewport, not the full page
|
||||
- **`device_scale_factor`**: Configurable screenshot DPI via BrowserConfig (#1463)
|
||||
- **`redirected_status_code`**: Now available on CrawlResult (#1435)
|
||||
- **`wait_for_images`**: Wait for images to load before taking screenshots (#1792)
|
||||
- **`score_threshold`**: Filter low-quality URLs in BestFirstCrawlingStrategy (#1804)
|
||||
- **`link_preview_timeout`**: Configurable timeout in AdaptiveConfig (#1793)
|
||||
- **`--json-ensure-ascii`**: CLI flag for Unicode preservation in JSON output (#1668)
|
||||
- **`type-list` pipeline**: Chained extraction like `["attribute", "regex"]` in JsonCssExtractionStrategy (#1290)
|
||||
|
||||
---
|
||||
|
||||
## Security Fixes
|
||||
|
||||
### Critical: RCE via Deserialization in Docker /crawl Endpoint
|
||||
|
||||
**Severity**: CRITICAL
|
||||
**Affected**: Docker API deployment (v0.8.0 and earlier)
|
||||
|
||||
The `/crawl` endpoint's deserialization logic used `eval()` for certain object types. I removed this entirely and added an allowlist (`ALLOWED_DESERIALIZE_TYPES`) so only known config classes can be instantiated.
|
||||
|
||||
### Critical: Redis CVE-2025-49844 (CVSS 10.0)
|
||||
|
||||
**Affected**: Docker deployments using Redis
|
||||
|
||||
Upgraded Redis to 7.2.7 which patches the Lua use-after-free vulnerability.
|
||||
|
||||
### Additional Security
|
||||
|
||||
- **XSS prevention**: Use DOMParser instead of innerHTML in iframe processing (#1796)
|
||||
- **API token enforcement**: `/token` endpoint now requires `api_token` when configured (#1795)
|
||||
- **Stealth improvements**: `sec-ch-ua` synced with User-Agent, WebGL kept alive in stealth mode
|
||||
|
||||
---
|
||||
|
||||
## Bug Fixes
|
||||
|
||||
### Browser & Page Management
|
||||
|
||||
- Fix page reuse race condition when `create_isolated_context=False`
|
||||
- Fix browser context memory leak — signature shrink + LRU eviction (#943)
|
||||
- Fix cascading context crash from duplicate `add_init_script` (#1768)
|
||||
- Fix `simulate_user` destroying page content via ArrowDown keypress
|
||||
- Fix browser recycling deadlock under sustained concurrent load (#1640)
|
||||
- Fix Docker monitor LOCK contention causing pod deadlock (#1754)
|
||||
|
||||
### Proxy & Network
|
||||
|
||||
- Fix proxy auth `ERR_INVALID_AUTH_CREDENTIALS` (#1281)
|
||||
- Fix proxy auth for persistent browser contexts
|
||||
- Fix proxy escalation not re-raising on first exception when chain has alternatives
|
||||
- Fix fallback fetch: run when all proxies crash, skip re-check, never return None
|
||||
|
||||
### Deep Crawling
|
||||
|
||||
- Fix `can_process_url()` to receive normalized URL
|
||||
- Fix `total_score` not calculated for links that fail head extraction
|
||||
- Fix `FilterChain.add_filter` AttributeError on tuple immutability
|
||||
- Fix URL Seeder forcing Common Crawl index for sitemaps (#1746)
|
||||
- Fix `is_external_url` port comparison (#1783)
|
||||
- Prevent AdaptiveCrawler from crawling external domains (#1805)
|
||||
|
||||
### Extraction & Content
|
||||
|
||||
- Fix `<base>` tag ignored in html2text relative link resolution (#1721)
|
||||
- Fix script tag removal losing adjacent text in `cleaned_html` (#1364)
|
||||
- Preserve `class` and `id` attributes in `cleaned_html` (#1782)
|
||||
- Fix nested brackets/parentheses in LINK_PATTERN regex (#1790)
|
||||
- Strip markdown fences in `force_json_response` path for LLM extraction
|
||||
- Guard against None LLM content, propagate `finish_reason` (#1788)
|
||||
- Fix `agenerate_schema()` JSON parsing for Anthropic models
|
||||
- Fix `from_serializable_dict` ignoring plain data dicts with "type" key
|
||||
- Fix MediaItem crash on non-numeric width values like "100%" (#1635)
|
||||
- Fix BM25ContentFilter returning duplicate chunks (#1213)
|
||||
- Fix `css_selector` ignored in LXML scraping for `raw://` URLs (#1484)
|
||||
|
||||
### CLI & Docker
|
||||
|
||||
- Fix deep-crawl CLI outputting only the first page (#1667)
|
||||
- Fix VersionManager ignoring `CRAWL4_AI_BASE_DIRECTORY` env var (#1296)
|
||||
- Fix Docker health endpoint to use dynamic version (#1686)
|
||||
- Add explicit UTF-8 encoding to CLI file output (#1789)
|
||||
- Handle `UnicodeEncodeError` in URL seeder, strip zero-width chars (#1784)
|
||||
- Add TTL expiry for Redis task data to prevent memory growth (#1730)
|
||||
- Add Windows support for crawler monitor keyboard input (#1794)
|
||||
- Fix `scroll_delay` ignored in full-page screenshot scroller
|
||||
- Fix MCP SSE endpoint crash — mount via raw ASGI Route (#1594)
|
||||
- Fix `/llm` per-request provider override, Redis config from host/port/password (#1611, #1817)
|
||||
- Fix screenshot respects `scan_full_page=False` (#1750)
|
||||
- Fix screenshot distortion on Elementor sites (#1370)
|
||||
- Fix deep crawl timeout and `arun_many` dispatcher bypass (#1818, #1509)
|
||||
|
||||
### Other
|
||||
|
||||
- Replace `tf-playwright-stealth` with `playwright-stealth` (#1553)
|
||||
- Allow local embeddings by removing OpenAI fallback (#1658)
|
||||
- Include GoogleSearchCrawler `script.js` in package distribution (#1711)
|
||||
- Fix bs4 deprecation warning (`text` → `string`) (#1077)
|
||||
- Run blocking `chardet.detect` in thread executor (#1751)
|
||||
- Wire `mean_delay`/`max_range` from CrawlerRunConfig into dispatcher rate limiter (#1786)
|
||||
|
||||
---
|
||||
|
||||
## Tests
|
||||
|
||||
Added a comprehensive **291-test regression suite** covering all major subsystems: core crawl, content processing, extraction strategies, deep crawling, browser management, config serialization, utilities, and edge cases.
|
||||
|
||||
---
|
||||
|
||||
## Breaking Changes
|
||||
|
||||
### `cleaned_html` Now Preserves `class` and `id` Attributes
|
||||
|
||||
If you have downstream code that parses `cleaned_html` and assumes no class/id attributes are present, this may need updating. This change enables users to do CSS-based analysis on cleaned HTML.
|
||||
|
||||
### Docker: Redis Upgraded to 7.2.7
|
||||
|
||||
If you pin Redis versions in your deployment, update to 7.2.7 or later.
|
||||
|
||||
---
|
||||
|
||||
## Upgrade Instructions
|
||||
|
||||
### Python Package
|
||||
|
||||
```bash
|
||||
pip install --upgrade crawl4ai
|
||||
# or
|
||||
pip install crawl4ai==0.8.5
|
||||
```
|
||||
|
||||
### Docker
|
||||
|
||||
```bash
|
||||
docker pull unclecode/crawl4ai:0.8.5
|
||||
|
||||
docker run -d -p 11235:11235 --shm-size=1g unclecode/crawl4ai:0.8.5
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Verification
|
||||
|
||||
Run the verification tests to confirm all features are working:
|
||||
|
||||
```bash
|
||||
python docs/releases_review/demo_v0.8.5.py
|
||||
```
|
||||
|
||||
This runs 13 actual tests that crawl real URLs and verify each feature end-to-end.
|
||||
|
||||
---
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
This release includes contributions from a large number of community members. Thank you to everyone who submitted PRs, reported issues, and provided reproduction steps. Special thanks to all contributors listed in [CONTRIBUTORS.md](../CONTRIBUTORS.md).
|
||||
|
||||
Issues fixed: #462, #880, #943, #1031, #1077, #1183, #1213, #1251, #1281, #1290, #1296, #1308, #1354, #1364, #1370, #1374, #1424, #1435, #1463, #1484, #1487, #1489, #1494, #1503, #1509, #1512, #1520, #1553, #1594, #1601, #1606, #1611, #1622, #1635, #1640, #1658, #1666, #1667, #1668, #1671, #1682, #1686, #1711, #1715, #1716, #1721, #1730, #1731, #1746, #1750, #1751, #1754, #1758, #1762, #1768, #1770, #1776, #1782, #1783, #1784, #1786, #1788, #1789, #1790, #1792, #1793, #1794, #1795, #1796, #1797, #1801, #1803, #1804, #1805, #1815, #1817, #1818, #1824
|
||||
|
||||
---
|
||||
|
||||
## Support & Resources
|
||||
|
||||
- **Documentation**: [docs.crawl4ai.com](https://docs.crawl4ai.com)
|
||||
- **GitHub**: [github.com/unclecode/crawl4ai](https://github.com/unclecode/crawl4ai)
|
||||
- **Discord**: [discord.gg/crawl4ai](https://discord.gg/jP8KfhDhyN)
|
||||
- **Twitter**: [@unclecode](https://x.com/unclecode)
|
||||
|
||||
---
|
||||
|
||||
**This is a massive release—10 new features, critical security patches, and 60+ bug fixes. Whether you're dealing with anti-bot protection, shadow DOM sites, or just want more reliable crawls at scale, v0.8.5 has you covered. Thank you for your continued support!**
|
||||
|
||||
**Happy crawling!**
|
||||
|
||||
*- unclecode*
|
||||
@@ -0,0 +1,113 @@
|
||||
# Crawl4AI v0.8.7: Security Hardening, DomainMapper & Community Fixes
|
||||
|
||||
*June 2026 - 7 min read*
|
||||
|
||||
---
|
||||
|
||||
I'm releasing Crawl4AI v0.8.7, a security-hardening release. It bundles every responsibly-disclosed vulnerability we patched since v0.8.6, adds the new DomainMapper feature, and ships a batch of scraping, deep-crawl, and LLM fixes from our team and the community.
|
||||
|
||||
If you self-host the Docker API server, please upgrade right away. This release closes several critical issues, and two GitHub Security Advisories accompany it.
|
||||
|
||||
## What's New at a Glance
|
||||
|
||||
- **Security hardening**: critical RCE, SSRF, auth-bypass, file-write, XSS, and hardcoded-secret fixes across the Docker API
|
||||
- **DomainMapper**: comprehensive domain URL discovery with subdomain and per-source timeout controls
|
||||
- **arun_many config-list in the Docker API**: send per-URL configs in one request
|
||||
- **Markdown fidelity fixes**: mermaid SVG text, table rowspan/colspan, trailing text, sentence order
|
||||
- **Deep crawl and dispatcher fixes**: streaming ContextVar bug, semaphore wiring
|
||||
- **LLM and provider fixes**: Bedrock auth, schema-first extraction, table-extraction allowlist
|
||||
- **Logging and MCP fixes**: stderr by default, non-TTY width, CJK preservation
|
||||
|
||||
---
|
||||
|
||||
## Security
|
||||
|
||||
v0.8.7 is, first and foremost, a security release. Every issue below was reported responsibly by the community, and every reporter is credited in `SECURITY-CREDITS.md` and the published advisories.
|
||||
|
||||
### Remote Code Execution
|
||||
|
||||
- **AST sandbox escape (CVSS 9.8)**: a `gi_frame.f_back` frame-chain walk escaped the computed-field expression sandbox to reach the real `__import__`. We removed `eval()` from computed fields entirely. SDK users can still pass Python callables via the `function` key.
|
||||
- **Hook sandbox escape (CVSS 9.8)**: injected module objects (`asyncio`, `json`, `re`) carried a full `__builtins__`, providing an alternate path to `__import__`. We stripped those builtins and tightened the allowlist.
|
||||
|
||||
### Authentication and Secrets
|
||||
|
||||
- **Hardcoded JWT secret (CVSS 9.8)**: the signing key defaulted to `"mysecret"`. We removed the default, reject weak or short secrets at startup, and auto-generate an ephemeral key when JWT is enabled with no key set.
|
||||
- **Monitor endpoint auth bypass (CVSS 6.5)**: the `/monitor/*` routes, including destructive actions, ran without auth. They now require a token, and the WebSocket endpoint checks the token explicitly.
|
||||
|
||||
### SSRF
|
||||
|
||||
- **Webhook SSRF (CVSS 8.6)**: webhook URLs on `/crawl/job` and `/llm/job` could hit internal and cloud-metadata addresses. We added a blocklist and disabled redirect following.
|
||||
- **Direct crawl-endpoint SSRF (CVSS 8.6)**: `/crawl`, `/md`, and `/llm` fetched arbitrary URLs, and IPv6-mapped IPv4 addresses such as `[::ffff:169.254.169.254]` slipped past naive checks. We added destination validation on all entry points and normalize IPv6-mapped IPv4 before the blocklist check.
|
||||
|
||||
### File Write, XSS, and JS Execution
|
||||
|
||||
- **Arbitrary file write (CVSS 9.1)**: `/screenshot` and `/pdf` honored any `output_path`. Writes are now restricted to `CRAWL4AI_OUTPUT_DIR`, and `..` traversal is rejected.
|
||||
- **Stored XSS in the monitor dashboard (CVSS 6.1)**: crawled URLs were rendered via `innerHTML` without escaping. We escape on both the server and the client now.
|
||||
- **Arbitrary JS execution via `/execute_js` (CVSS 8.1)**: the endpoint is disabled by default behind `CRAWL4AI_EXECUTE_JS_ENABLED`, we removed `--disable-web-security` from default browser args, and added an SSRF blocklist on the destination.
|
||||
|
||||
We also replaced the `eval()` in `/config/dump` with Pydantic-validated JSON input, and added type validation for `markdown_generator` in `CrawlerRunConfig`.
|
||||
|
||||
---
|
||||
|
||||
## New Features
|
||||
|
||||
### DomainMapper
|
||||
|
||||
DomainMapper discovers URLs across an entire domain in one pass, combining multiple discovery sources. It supports an `include_subdomains` flag to widen or narrow the crawl boundary, and a per-source timeout so a single slow source cannot stall the whole map. See the [Domain Mapping guide](../core/domain-mapping.md).
|
||||
|
||||
### arun_many config-list in the Docker API
|
||||
|
||||
The Docker API now accepts a list of configs aligned with the list of URLs, so you can apply a different `CrawlerRunConfig` to each URL in a single `arun_many` request (#1837).
|
||||
|
||||
---
|
||||
|
||||
## Fixes
|
||||
|
||||
**Markdown and scraping**
|
||||
|
||||
- Preserve mermaid diagram text rendered as SVG, and prevent nested code fences (#1043)
|
||||
- Preserve table `rowspan` and `colspan` in cleaned HTML (#1920)
|
||||
- Preserve `.tail` text when removing empty elements (#1938)
|
||||
- Keep sentence order in `NlpSentenceChunking` (#1909)
|
||||
|
||||
**Deep crawl and dispatcher**
|
||||
|
||||
- Fix the deep-crawl streaming ContextVar bug by using `set(False)` instead of `reset(token)` (#1917)
|
||||
- Wire `semaphore_count` into the auto-created `MemoryAdaptiveDispatcher` and default it to 10 (#1927)
|
||||
|
||||
**LLM and providers**
|
||||
|
||||
- Add Bedrock to the provider prefixes so AWS credential auth works
|
||||
- Default `LLMExtractionStrategy.extraction_type` to schema
|
||||
- Add `LLMTableExtraction` to the Docker deserialization allowlist
|
||||
|
||||
**Crawler and downloads**
|
||||
|
||||
- Return `success=True` for binary downloads, and skip the block check when `downloaded_files` is set
|
||||
- Honor `<base href>` in prefetch `quick_extract_links` (#752)
|
||||
|
||||
**Logging and MCP**
|
||||
|
||||
- Route `AsyncLogger` output to stderr by default (#1968) and use `Console(width=200)` for non-TTY contexts
|
||||
- Use `ensure_ascii=False` in the MCP bridge to preserve CJK characters (#1967)
|
||||
|
||||
**Browser and misc**
|
||||
|
||||
- `browser_adapter` now uses the `Stealth` import, fixing a stealth import mismatch (#1960)
|
||||
- Correct the `arun()` return type to `CrawlResultContainer` (#1898)
|
||||
- Log the real failure reason before COMPLETE, fixing a misleading success line (#1949)
|
||||
- Assistant toolbar scroll fix and issue-1973 fix
|
||||
|
||||
---
|
||||
|
||||
## Upgrade
|
||||
|
||||
```bash
|
||||
pip install -U crawl4ai
|
||||
```
|
||||
|
||||
Docker users should pull the latest image once the Docker release workflow finishes.
|
||||
|
||||
## Security Credits
|
||||
|
||||
Thank you to the researchers who disclosed these issues responsibly: Song Binglin (q1uf3ng), by111 (August829), Jeongbean Jeon, wulonchia, secsys_codex, Velayutham Selvaraj, and IcySun. Full details are in `SECURITY-CREDITS.md`.
|
||||
@@ -0,0 +1,43 @@
|
||||
# Crawl4AI v0.8.8: Docker Server Security Patch
|
||||
|
||||
*June 2026 - 3 min read*
|
||||
|
||||
---
|
||||
|
||||
I'm releasing Crawl4AI v0.8.8, a focused security patch for the self-hosted Docker API server. It is backward compatible: upgrade in place, no configuration changes required.
|
||||
|
||||
If you run the Docker server, please upgrade. If it is exposed to a network, also set `CRAWL4AI_API_TOKEN`. Security advisories accompany this release.
|
||||
|
||||
## What it fixes
|
||||
|
||||
- **SSRF filter gaps**: the SSRF protection now rejects any resolved address that is not globally routable, and it covers IPv6 transition forms that previously slipped past the blocklist (NAT64, 6to4, IPv4-mapped, and the unspecified `::`). These could otherwise reach internal services and cloud-metadata endpoints. Error messages no longer echo the resolved address.
|
||||
- **Arbitrary file write via `output_path`**: `/screenshot` and `/pdf` now resolve symlinks and re-check containment before writing, and write with `O_NOFOLLOW`, closing a symlink/TOCTOU bypass of the output-directory restriction. Normal use is unchanged.
|
||||
- **LLM credential exfiltration**: the LLM endpoints no longer honor a request-supplied `base_url`, so the configured provider key cannot be redirected to an attacker endpoint, and `LLMConfig` will not resolve protected environment variables via `env:`.
|
||||
- **Hardening**: CRLF-safe logging and webhook request-header validation.
|
||||
|
||||
All changes are backward compatible. Details and credits are in the security advisories.
|
||||
|
||||
## Coming next: a secure-by-default Docker server (~1-2 weeks)
|
||||
|
||||
The next release is a larger, secure-by-default update for the Docker API server, and it has intentional breaking changes. I want to give everyone time to prepare, so here is the heads-up.
|
||||
|
||||
If you run the Docker server, plan for these and test in staging before upgrading:
|
||||
|
||||
- **Authentication on by default.** The server binds loopback unless you configure a credential (`CRAWL4AI_API_TOKEN`). Put a TLS-terminating reverse proxy in front to expose it.
|
||||
- **Stricter request validation and safer defaults.** TLS verification on, tighter outbound egress controls, and declarative hook actions instead of inline code.
|
||||
- **A few request options move server-side.** `/screenshot` and `/pdf` return an artifact id instead of a file path, and the LLM endpoint is selected by provider name.
|
||||
- **Hardened container defaults.** Least-privilege compose, Redis authentication, loopback bind.
|
||||
|
||||
A full migration guide will go out with the pre-announcement on Discord and X. Watch those channels.
|
||||
|
||||
## Upgrade
|
||||
|
||||
```bash
|
||||
pip install -U crawl4ai
|
||||
# Docker
|
||||
docker pull unclecode/crawl4ai:0.8.8
|
||||
```
|
||||
|
||||
Thanks to everyone who reports issues responsibly. Star and use Crawl4AI: https://github.com/unclecode/crawl4ai
|
||||
|
||||
Live long and import crawl4ai
|
||||
@@ -0,0 +1,37 @@
|
||||
# Crawl4AI v0.8.9: Proxy SSRF Patch
|
||||
|
||||
*June 2026 - 2 min read*
|
||||
|
||||
---
|
||||
|
||||
I'm releasing Crawl4AI v0.8.9, a follow-up security patch for the self-hosted Docker API server. It closes a server-side request forgery path that v0.8.8 did not cover. It is backward compatible: upgrade in place, no configuration changes required.
|
||||
|
||||
If you run the Docker server, please upgrade. If it is exposed to a network, also set `CRAWL4AI_API_TOKEN`. A security advisory accompanies this release.
|
||||
|
||||
## What it fixes
|
||||
|
||||
The SSRF destination check validated the crawl target URL, but not the proxy address. An unauthenticated `/crawl`, `/crawl/stream`, or `/crawl/job` request could point a proxy at an internal IP and route the browser through it, reaching internal services and cloud-metadata endpoints, even with a perfectly valid crawl URL.
|
||||
|
||||
v0.8.9 validates every proxy destination with the same global-routability check before the browser is built:
|
||||
|
||||
- `browser_config.proxy_config.server`
|
||||
- `browser_config.proxy` (deprecated field)
|
||||
- `crawler_config.proxy_config.server`
|
||||
- proxy / DNS-redirecting flags in `extra_args` (`--proxy-server`, `--host-resolver-rules`, `--proxy-bypass-list`, `--proxy-pac-url`) are stripped
|
||||
|
||||
A legitimate public proxy still works. The only behavior change: set proxies through `proxy_config` (which is validated) rather than raw `extra_args` flags.
|
||||
|
||||
## Upgrade
|
||||
|
||||
```bash
|
||||
pip install -U crawl4ai
|
||||
docker pull unclecode/crawl4ai:0.8.9
|
||||
```
|
||||
|
||||
## Still coming: a secure-by-default Docker server (~1-2 weeks)
|
||||
|
||||
The next release remains a larger, secure-by-default update for the Docker API server, with intentional breaking changes (authentication on by default, stricter request validation, safer deployment defaults). A full migration guide will accompany the pre-announcement on Discord and X. This proxy fix is already part of that release; 0.8.9 simply brings it forward because it is an unauthenticated SSRF.
|
||||
|
||||
Thanks to Geo for the responsible disclosure.
|
||||
|
||||
Live long and import crawl4ai
|
||||
@@ -0,0 +1,78 @@
|
||||
# Crawl4AI v0.9.0: Secure-by-Default Docker Server
|
||||
|
||||
*June 2026 - 6 min read*
|
||||
|
||||
---
|
||||
|
||||
I'm releasing Crawl4AI v0.9.0, a major, secure-by-default release of the Crawl4AI Docker API server. This is the biggest change to the self-hosted HTTP server since we shipped it. It moves the out-of-the-box deployment from an open, trust-the-caller posture to a closed, hardened one with defense in depth.
|
||||
|
||||
This is a breaking release for the Docker server only. The core pip library (the SDK and in-process use) is unchanged. If you only `pip install crawl4ai` and drive it from Python, nothing here affects you and you can upgrade freely.
|
||||
|
||||
If you self-host the Docker API server, please read the [migration guide](https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/MIGRATION.md) before you upgrade, and roll out behind a staging environment first.
|
||||
|
||||
## Why this release
|
||||
|
||||
Over the last few releases we patched a series of issues in the Docker server one at a time. 0.9.0 finishes the job by changing the architecture instead of patching behavior. The principle is simple: the server should be safe the moment you start it, and the network request body should be treated as untrusted input rather than a trusted control channel.
|
||||
|
||||
That means the permissive defaults are gone. Authentication is on by default. The server binds loopback unless you give it a token. The request body carries declarative options only. Everything that used to let a caller reach into browser internals or supply code now lives server-side, where the operator controls it.
|
||||
|
||||
## What changed at a glance
|
||||
|
||||
- **Auth on by default, loopback bind**: no unauthenticated API on `0.0.0.0`.
|
||||
- **Request trust boundary**: crawl request bodies are declarative and scalar; power fields are rejected at the network edge.
|
||||
- **Declarative hooks**: a fixed action set replaces request-supplied hook code.
|
||||
- **Artifact store**: `output_path` is gone; screenshots and PDFs return an artifact id you fetch with auth.
|
||||
- **Provider by name**: LLM endpoints select a provider by name, configured server-side.
|
||||
- **Hardened transport and infra**: TLS verification on, deny-by-default CORS, strict security headers, password-protected loopback-only Redis, a bounded job queue, and generic error responses with correlation ids.
|
||||
|
||||
## Hardening details
|
||||
|
||||
### Authentication and binding
|
||||
|
||||
The server no longer serves an unauthenticated API on `0.0.0.0`. With no token configured it binds `127.0.0.1` only and prints a one-off token at startup for local use. To expose it, set a token and put a TLS-terminating reverse proxy in front:
|
||||
|
||||
```bash
|
||||
export CRAWL4AI_API_TOKEN="$(openssl rand -hex 32)"
|
||||
```
|
||||
|
||||
Every request except `GET /health` then needs `Authorization: Bearer <token>`. WebSocket clients that cannot set headers may pass `?token=...`. The JWT implementation changed, so tokens from older versions are no longer valid; re-mint via `POST /token`.
|
||||
|
||||
### The request trust boundary
|
||||
|
||||
A crawl request body now carries declarative, scalar options only. Fields that previously let a caller drive browser internals or arbitrary code are rejected with HTTP 400 at the network boundary, including `js_code`, `c4a_script`, `proxy_config`, `extra_args`, `user_data_dir`, `cdp_url`, `cookies`, `headers`, `init_scripts`, `base_url`, `deep_crawl_strategy`, `simulate_user`, `magic`, and `process_in_browser`. Configure these server-side, or use the in-process SDK where you keep full control. Unknown fields are dropped, and timeouts, viewport, and scroll counts are clamped to safe maximums.
|
||||
|
||||
Request-supplied browser launch arguments (`browser_config.extra_args`) are part of this boundary and are now rejected, closing a Chromium launch-argument injection class.
|
||||
|
||||
### Declarative hooks
|
||||
|
||||
`hooks.code` (Python strings) is replaced by a fixed set of declarative actions: `block_resources`, `add_cookies`, `set_headers`, `scroll_to_bottom`, and `wait_for_timeout`. Call `GET /hooks/info` for the parameter schemas. Arbitrary hook code remains available in a self-hosted in-process build.
|
||||
|
||||
### Downloads, screenshots, and PDFs
|
||||
|
||||
Download sinks now confine writes with basename plus realpath plus `O_NOFOLLOW`, removing a path-traversal-to-file-write class. `output_path` is removed from `/screenshot` and `/pdf`; the server stores the result and returns an `artifact_id` plus a URL, which you fetch with authenticated `GET /artifacts/{artifact_id}` (artifacts have a TTL and a storage quota).
|
||||
|
||||
### SSRF on the streaming path
|
||||
|
||||
Destination validation now covers the streaming crawl handler. `/crawl/stream` and `/crawl` with `stream=true` validate the target and return HTTP 400 for disallowed destinations, matching the non-streaming handlers.
|
||||
|
||||
### Transport and infrastructure
|
||||
|
||||
TLS verification is on; self-signed or internal targets fail by default, with explicit escape hatches (`CRAWL4AI_ALLOW_INSECURE_TLS`, `CRAWL4AI_ALLOW_INTERNAL_URLS`) for trusted internal testing. CORS is deny-by-default; allowlist your frontend origin under `security.cors_allow_origins`. Redis runs in-container, loopback-only, password-protected, with its port no longer published. Background jobs run on a bounded queue, and request size, wall-clock, and per-principal concurrency are capped (all configurable, `0` = unbounded). 5xx responses return a generic body with a correlation id you can match in the logs.
|
||||
|
||||
## Migrating
|
||||
|
||||
How much you have to do scales with how much you drove through the API. A plain "crawl these URLs with a normal config" user only needs to set a token and re-issue tokens. Everything else applies only if you used that specific feature.
|
||||
|
||||
Read the [migration guide](https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/MIGRATION.md) first, then follow `deploy/docker/SECURITY-VERIFY.md` for the deployment checklist.
|
||||
|
||||
## Upgrade
|
||||
|
||||
```bash
|
||||
pip install -U crawl4ai
|
||||
```
|
||||
|
||||
Docker users should pull the latest image once the Docker release workflow finishes.
|
||||
|
||||
## Security Credits
|
||||
|
||||
Thank you to the researchers who disclosed these issues responsibly: Y4tacker, KOH Jun Sheng, and UDU_RisePho ([hoanggxyuuki](https://github.com/hoanggxyuuki)). Full details are in `SECURITY-CREDITS.md`.
|
||||
@@ -0,0 +1,90 @@
|
||||
# Crawl4AI v0.9.1: Bug Fixes & PruningContentFilter Whitelist
|
||||
|
||||
*July 2026 - 3 min read*
|
||||
|
||||
---
|
||||
|
||||
I'm releasing Crawl4AI v0.9.1, a patch release that ships 12 bug fixes across Docker, browser, core, and extraction, plus one new feature for `PruningContentFilter`.
|
||||
|
||||
No breaking changes. If you're on v0.9.0, upgrade freely.
|
||||
|
||||
## What's new at a glance
|
||||
|
||||
- **PruningContentFilter whitelist**: New `preserve_classes` / `preserve_tags` parameters to protect specific elements from density-based pruning
|
||||
- **Windows browser fix**: Default `channel='chromium'` no longer crashes Playwright on Windows
|
||||
- **Docker hardening**: 6 fixes for auth gate, supervisord, redis, tmpfs, and FastAPI compatibility
|
||||
- **HTTP timeout fix**: `page_timeout` was passed in milliseconds to aiohttp (which expects seconds), effectively disabling timeouts in HTTP mode
|
||||
- **Dependency**: lxml ceiling widened to allow 6.x
|
||||
|
||||
## PruningContentFilter whitelist
|
||||
|
||||
PruningContentFilter's density-based scoring is great at stripping boilerplate, but it sometimes takes short metadata elements — author names, timestamps, attribution lines — along with it. The new `preserve_classes` and `preserve_tags` parameters let you whitelist specific CSS classes or HTML tags that should never be pruned, regardless of their density score.
|
||||
|
||||
```python
|
||||
from crawl4ai.content_filter_strategy import PruningContentFilter
|
||||
from crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator
|
||||
|
||||
filter = PruningContentFilter(
|
||||
threshold=0.48,
|
||||
preserve_classes=["author", "byline", "dateline"],
|
||||
preserve_tags=["time", "address"],
|
||||
)
|
||||
generator = DefaultMarkdownGenerator(content_filter=filter)
|
||||
|
||||
config = CrawlerRunConfig(markdown_generator=generator)
|
||||
```
|
||||
|
||||
Whitelisted nodes skip scoring entirely. Default is empty sets — no behavior change for existing users. (#1900, thanks @hafezparast)
|
||||
|
||||
## Bug fixes
|
||||
|
||||
### Docker (6 fixes)
|
||||
|
||||
- **Auth gate UI**: Dashboard and playground now load when JWT auth is active. Token input bar added to both UIs; all fetch calls use `authFetch()` with Bearer token. API routes remain fail-closed. (#2037)
|
||||
- **Supervisord/Redis dirs**: Pidfile and RDB snapshot dirs moved to writable paths for read-only rootfs deployments. (#2047, thanks @TobiasWallura-xitaso)
|
||||
- **tmpfs writable**: Read-only tmpfs mounts made writable. (#2027, thanks @nightcityblade)
|
||||
- **FastAPI cap**: Pinned FastAPI below 0.137 to avoid compatibility breakage. (#2025, thanks @nightcityblade)
|
||||
- **Redis auth**: Rate-limit Redis storage now authenticates with the configured password. (#2040, thanks @harshmathurx)
|
||||
- **Posture tests updated**: Dashboard/playground moved to public UI assertions to match auth gate fix.
|
||||
|
||||
### Browser (2 fixes)
|
||||
|
||||
- **Windows channel crash**: `channel='chromium'` (the default) caused Playwright to look for a system Chrome install instead of the bundled binary, crashing on Windows with `TargetClosedError`. The default channel is no longer passed to Playwright. (#2051, thanks @fstark96)
|
||||
- **Context snapshot leak**: Browser contexts from snapshot are now properly closed. (#1999, thanks @nightcityblade)
|
||||
|
||||
### Core (2 fixes)
|
||||
|
||||
- **HTTP timeout**: `page_timeout` (60000ms) was passed directly to `aiohttp.ClientTimeout` which expects seconds, making the effective timeout 16.7 hours. Now correctly divided by 1000. (#1894, thanks @hafezparast)
|
||||
- **Best-first ordering**: Batch ordering in `BestFirstCrawlingStrategy` stabilized for deterministic crawl order. (#1998, thanks @nightcityblade)
|
||||
|
||||
### Extraction (1 fix)
|
||||
|
||||
- **Table attributes**: `html2text` now preserves all attributes on table tags when `bypass_tables` is enabled. (#2007)
|
||||
|
||||
### Dependencies (1 fix)
|
||||
|
||||
- **lxml 6.x**: Widened lxml ceiling from `<6` to `<7` so crawl4ai can co-install with packages requiring lxml 6.x (e.g. scrapling). (#2019)
|
||||
|
||||
### Housekeeping
|
||||
|
||||
- Removed dead `normalize_url` duplicates and accidental `adaptive_crawler` copy. (thanks @RajanChavada)
|
||||
- Sponsor logos hosted locally to fix broken GitHub rendering.
|
||||
|
||||
## Upgrade
|
||||
|
||||
```bash
|
||||
pip install -U crawl4ai
|
||||
crawl4ai-doctor # verify installation
|
||||
```
|
||||
|
||||
Docker users: pull the latest image once the Docker release workflow finishes.
|
||||
|
||||
## Acknowledgments
|
||||
|
||||
Thanks to the community contributors who made this release possible: @hafezparast (#1894, #1900), @nightcityblade (#1998, #1999, #2025, #2027), @fstark96 (#2051), @TobiasWallura-xitaso (#2047), @harshmathurx (#2040), @RajanChavada (#2042).
|
||||
|
||||
## Support & Resources
|
||||
|
||||
- [Documentation](https://docs.crawl4ai.com)
|
||||
- [GitHub Issues](https://github.com/unclecode/crawl4ai/issues)
|
||||
- [Discord Community](https://discord.gg/crawl4ai)
|
||||
Reference in New Issue
Block a user